Go builder caching policies might need another look

[dr2chase]

What version of Go are you using (go version)?

go version devel +9f7f82ee86 Thu May 30 17:29:58 2019 -0400 linux/amd64

Does this issue reproduce with the latest release?

Yes

What did you do?

Ran thousands of benchmarks over several days using compilebench, each with a slightly different compiler.

What did you expect to see?

Not 200 gigabytes of Go cache contents accumulated over time, completely filling the disk.

What did you see instead?

200 gigabytes of Go cache and a full disk.
go clean -cache solved the problem, but it took a few minutes, and I almost hurt myself laughing.

I realize this is not a typical use case, and I've added a preventive go clean -cache to my benchmarking script, but if there's an easy check for this we ought to do it.

[ALTree]

I also hit this while fuzzing the compiler. For example, gosmith's driver uses go build and that means you'll have a hundreds-of-GBs cache after a few hours of fuzzing. And when I noticed, deleting the cache took hours on windows.

@josharian had the same issue in #31931. Moreover, you can't even configure your driver to periodically clean the cache while still building, because concurrent build and cache clean is unsafe (#31948).

Unfortunately #31931 was closed so it seems that for this kind of use-cases we are expected to clean the cache manually.

[ALTree]

Also relevant (and still open): #29561 (cmd/go: clean GOCACHE based on disk usage).

[josharian]

cc @bcmills @jayconrod

I imagine y’all will close this as a duplicate, but cc’ing you to make that call. Also notable that there have now been a fair few of us that have been bitten by this.

[jayconrod]

Yep, let's keep #29561 as the canonical issue. Sorry we haven't gotten to it yet. This is high on my list of things to do in 1.14 though.