Add AppRepository to enable tools inside apps

[nicolas-grekas]

This is a tentative PR towards a better separation of tools' dependencies vs app's dependencies. This is related to #9636 and the discussion that is happening there.

Inside a dummy app, I created a tools/test/composer.json with the following content:

{
    "require": {
        "app/app": "*",
        "phpunit/phpunit": "^9.5"
    },
    "repositories": [
        {
            "type": "app",
            "url": "../../"
        }
    ],
    "config": {
        "bin-dir": "../../vendor/bin",
        "vendor-dir": "../../vendor/composer/tools/test"
    }
}

Running composer install inside this tools/test/ directory installs PHPUnit and its dependencies in vendor/composer/tools/test, with symlinks to the root app for shared dependencies.

Since many apps don't have a name in their composer.json file, the app/app that you see in the require section is a convention I added to allow tools to reference the app itself as a dependency when needed (for PHPUnit typically). If a tool doesn't need access to the app's source code, this entry should be just skipped.

By default, packages found in the app's vendor/ directory are locked: a tool that shares a dependency with an app will always resolve that dependency to the same exact version as the app. This can be changed by making the above app repository "non-canonical", as explained in Composer's doc.

I feel like this PR is a small but useful step towards splitting tools' deps from app's ones. The DX could be improved for sure, but this can be done in future PRs I think.

Before adding tests/doc: is this worth pursuing? I think yes :) What about you?

TL;DR: with this PR, running phpunit is done by typing $> ./vendor/bin/phpunit and phpunit's deps don't mess up with your app's ones.

[glensc]

TL;DR: running phpunit is done by typing $> ./tools/test/phpunit

shouldn't that be ./vendor/bin/phpunit?

[gmsantos]

It would be nice if composer scripts can find ./tools/test/phpunit somehow https://getcomposer.org/doc/articles/scripts.md#writing-custom-commands

[naderman]

I think adding a separate composer.json file for this which may exist in different directories based on the project is really a step backward from standardized composer.json at the root of the project. If this is to be implemented I think it should be as part of the project's composer.json as discussed in previous issues and the one you linked.

[nicolas-grekas]

@naderman I totally agree: as I suggested in the description, this PR is a step towards this target. I feel safer to take this path than to dream about an issue without anyone actually contributing some code :)

[mabar]

If I for example installed phpstan and phpunit as a tools, would I be able to analyse phpunit tests with phpstan? From what I understand, phpstan would know nothing about phpunit.

I think phars are better solution for tools. Phar can be published as a separate package, with no dependencies, so it can be still installed via Composer. For example phpstan do that - phpstan/phpstan-src is published via phpstan/phpstan phar

[nicolas-grekas]

@mabar yes, phpstan would read tools/test/vendor/ to learn everything it needs to know about phpunit.
About phars, this has been discussed in the linked issue #9636: they don't work well with extensions/plugins, and they require careful care by authors, which means we cannot expect all tools to have a corresponding phar anyway. Both reasons rules them out for a generic solution.

[nicolas-grekas]

@gmsantos this works already, what else would you need?

    "scripts": {
        "test": [
            "./tools/test/phpunit"
        ]
    },

[gmsantos]

@nicolas-grekas it would be related with this note from the docs:

Note: Before executing scripts, Composer's bin-dir is temporarily pushed on top of the PATH environment variable so that binaries of dependencies are directly accessible. In this example no matter if the phpunit binary is actually in vendor/bin/phpunit or bin/phpunit it will be found and executed.

If you just use phpunit, ./tools/test/phpunit would not be found, i.e.

"scripts": {
    "test": [
        "phpunit"
    ]
},

[nicolas-grekas]

@glensc @gmsantos now achieved by changing the bin-dir in the composer.json file, see updated description.

[Jean85]

Nice experiment!

How would the autoloaders interact in such a case?

I can see multiple situations in which I would need to load the app and multiple tools at the same time... For example, PHPUnit, phpspec/prophecy-phpunit (and its transient dependency Prophecy) and the app's code. Plus, I would want to run static analysis (and its plugins!) on top of that.

[nicolas-grekas]

@Jean85 there is one autoloader per "tools" directory, which can load the app or not depending on the app/app in the require above. When multiple tools should share the same autoloader, they should be required in the same tools/foo/ directory. I suppose there won't be many of them actually, as grouping many tools in one set of shared deps could be fine, as long as individual tools allow it.

[Jean85]

@nicolas-grekas and what about transient dependencies? Let's say, both the app and the tool require symfony/console (which is very likely), the first to be loaded wins? Wouldn't that create clashing?

[nicolas-grekas]

I suppose you mean transitive dependencies, right? What I wrote in the description: the vendor/ dir of the app will be used as the package repository for shared deps. This means that the tool's console directory will be a symlink to the app's one, by default at least (see PR description, about canonical repo for more).

[Jean85]

So, let me rephrase is to see if I get this right: transitive (yep, sorry I meant that) dependencies that are present in both the app and the tool wouldn't get duplicated, but instead the tool will use the same from the app, with a symlink if needed. You approach will reduce the "clash" because transitive dependencies that do NOT overlap with the app will be resolved in a different tree, reducing the "pollution" that the app would get from the normal require-dev procedure.

Did I get this right? If this is the case, we still have the (big) issue of common transitive deps, and that it's a tough problem to solve.

[nicolas-grekas]

I think you've got this right. Which issue are you talking about? To me they are all solved:

• if a tool needs to be able to autoload the classes of your app, any shared deps must be compatible with the app's deps. This PR provides that.
• if a tool doesn't need to autoload the classes of your app, this approach can provide opportunistic sharing of shared deps, but allows any other version of said deps to be installed for the need of a specific tool.

[Jean85]

I'm just pondering the full impact of your solution... surely you solved a big chunk of possible conflicts, but the tools still needs to be compatible with your own set of dependencies. Anyway, as said above, this shouldn't be a big issue for applications (where lock is present and target is known) and for libraries like Symfony too (where you should aim to reduce your dependencies). Good! 👍

[mxr576]

I see that this looks like a valid workaround for the problem but is it the right way to get it solved? Aren't we trying to abuse Composer?

Can we collect and compare how others are dealing with this problem in the industry? (Like how it happened with Enum PSR recently.)

I am more like with those who asked would not be better using PHAR for development dependencies. Would not it be better figuring out best practices around that instead of trying to add tricks to a dependency manager?

PHAR is similar to Snap/Flatpak in my vocabulary and they were born for a reason.

Related: https://tomasvotruba.com/blog/2019/12/02/how-to-box-symfony-app-to-phar-without-killing-yourself/

• https://github.com/humbug/php-scoper
• https://github.com/box-project/box

[Jean85]

The limitation of the PHAR approach is that every single maintainer has to apply it to its project; if we can bake in a similar solution inside Composer, maybe with a scoper too, everyone will benefit from that for free.

[WinterSilence]

Is it still needed? Composer\InstalledVersions contains similar functionality.