Parallelize cache key computation

[jerenkrantz]

Welcome

• Yes, I've searched similar issues on GitHub and didn't find any.
• I agree to follow this project's Code of Conduct

How did you install golangci-lint?

go install

Your feature request related to a problem? Please describe

Cache key computation (loadIssuesFromCache) can take 600ms+ even for a single-package run because it recursively hashes every file in every transitive dependency, sequentially, on one goroutine. The global mutex in FileHash further serializes concurrent callers.

Describe the solution you'd like

• Replace hashFileCache global mutex with sync.Map + singleflight.Group for lock-free cache hits and deduplicated concurrent misses
• Hash all files within a package concurrently in computePkgHash
• Add PrecomputePackageHashes that walks the full dependency graph in topological order, computing hashes leaf-to-root across all available cores

Describe alternatives you've considered

We confirmed caching of go build and lint caching is present - a significant amount of time during lint is the serial processing of the hashes to compute whether the caches are valid.

Additional context

Draft PR that speeds up hash computation via parallelization in #6468.

Supporter

• I am a sponsor/backer of this project.

[ldez]

This has zero impact.
And the issue provide no information about the effective performances.

It feels like AI-slop.

[jerenkrantz]

It's not slop. It's a real issue. As above, a single go module doing linting takes ~650ms to walk the entire dependency tree serially. Parallelizing it makes it significantly faster - on the order of 617us.

[ldez]

~650ms on what? 2 seconds, 2 minutes, 2 hours?

[jerenkrantz]

In our setup, it takes ~4 seconds total on a no-op lint when all of the go and lint caches are hit; so, the extra 600ms savings is noticeable.

[ldez]

on a no-op lint

on a no-op lint? so not in the reality of a run of golangci-lint.

I tested your PR: I saw no changes on the duration (with and without cache).

[jerenkrantz]

How big is the Go dependency graph in that test?

Yes, a pure no-op lint is a best case scenario. When we are doing interactive edits, we want to run lints before commit/push. We already do a lot of optimization to try to reduce the scope of what we ask golangci-lint to evaluate due to the overall lint performance. We only lint the files that actually changed - yet, we're spending a lot of time walking the entire Go dependency tree for files that haven't changed to compute the cache keys.

[ldez]

How big is the Go dependency graph in that test?

Big enough for the analysis to take ~17s without cache and ~1s with cache.

[jerenkrantz]

To put a specific number on it, in the immediate case we were testing against, our dep graph is 1517 packages:

$ go list -json -deps pkg | jq -s 'length - 1'
1517

[ldez]

$ go list -json -deps ./... | jq -s 'length - 1'
1486

[mitar]

Is maybe difference in number of parallel threads you two are running with? If @jerenkrantz has more cores or higher GOMAXPROCS, maybe his PR has more impact?