Move on from Arborist?

[kemitchell]

#101 has me seeing again that pnpm and other alternative package managers are apparently creating node_modules trees that Node.js import and require find acceptable, but that our "find all the packages in node_modules" package, Arborist, a component of npm, handles differently.

I don't know exactly what those differences are. I believe all the current frontrunners—npm, pnpm, yarn, bun—now extensively use symlinks to cached copies of packages. But I haven't seen or done a comparative study of how the trees they create differ.

I wonder how difficult it would be to re-implement just the recursive descent of node_modules that we need, independent of Arborist. At the same time, I've learned enough to suspect that's likely far more nuanced than it appears:

• We would still need to differentiate production and development dependencies.
• Especially with symlinks galore, we'd need to detect cycles.
• npm and Arborist support Windows, Mac, and Linux. We'd need to be attuned to what node:fs does and doesn't abstract there.
• This is fundamentally a filesystem task, and filesystems mean quirky edge cases galore.

[ljharb]

"what npm does" is the standard, and arborist is what npm uses, so I think not using it would be a very poor choice.

If arborist isn't working on a non-npm-installed tree, then i think that whatever installed it either needs to fix their bug; arborist needs to be extended to handle it; or, whatever installed it needs to provide an arborist replacement so that the rest of the ecosystem doesn't have to implement its own tree explorer.

[kemitchell]

@ljharb all the new package managers clearly owe a debt to npm. But I'm seeing signs of broader take-up of new alternatives. npm has also massively changed what it does to disk over time, in large part responding to the perf competition.

What I don't see is good motivation to actually keep Arborist reliably reading all the different downloaders' trees. They have some test rigging for other package managers, but we've seen it falling behind in real time in issues here.

Of course, each of the package managers gets to implement its own, much simpler routine for detecting packages installed its own way. [Here's pnpm's, for example.]https://github.com/pnpm/pnpm/blob/main/fs/read-modules-dir/src/index.ts) But none of those has to be up-to-date and functional for all the other package managers to pass tests and get released in the package manager it comes from.

Meanwhile, Node.js require/import is accepting all their node_modules trees fine—the new package managers wouldn't get adopted otherwise. I think that's also why folks open issues on Licensee surprised to see errors. If it runs, it runs. No notion of "standard" required.

For Licensee, we've been free riding for a while: Some shared subroutine for listing installed dependencies "should exist" and should work for all the package managers, who happen to be in very direct competition, for benefit of toolmakers like us. If I were going to take on the burden of implement and maintaining that subroutine, though, it's be much less burden on me as its own, small, separate package, not bound up with any particular package manager. Not buried in some irrelevant monorepo.

[ljharb]

signs of broader take-up of new alternatives.

i don't see any signs that the lion's share are anywhere but using npm, but either way it's not just "owe a debt", npm is the gold standard.

Any tooling that looks at installed deps as a whole has to use arborist or another solution to look at the tree, and will break in similar ways. This isn't a problem unique to licensee, and i continue to believe that if a CLI installs deps such they don't work with all tools that work with npm i, then they are broken, and the ecosystem shouldn't cave and work around those bugs.

[kemitchell]

@ljharb understood. I certainly wasn't expecting you to lift this for me.

[ljharb]

I'm not trying to shirk any work, to be clear :-) if an npm alternative that doesn't work with licensee has an arborist-like package, I'm happy to make a PR to integrate it.

[kemitchell]

I looked, but didn't see any generic alternatives.