How can I cache binaries to a new location without rebuilding?

[jdx-john]

Thanks to help here, I have now successfully enabled binary-caching (#19380) and it looks great.

However, I had already built most of my ports without it. When I re-ran vcpkg install for every port/triplet we use and passed --binarysource=files,c:\vcpkg_cache,readwrite only the ones which got built appear there. All those which were already built did not. We have ~250 of these in total and it takes about 4 hours to do a clean build, so I'm hoping there is some way to avoid that?

I do have all the default caching in %LOCALAPPDATA%\vcpkg\archives is there maybe a command to migrate them, or could I even copy these archives over?

[autoantwort]

Simply copying should work

[JackBoosY]

According to our docs, you have 2 ways:

1. upload the caches to your CI system, see here.
2. copy them to the other machine, add --binarysource=<files,ABS_PATH_TO_CACHES> to the install command or set the environment VCPKG_BINARY_SOURCES, see here.

[jdx-john]

Thanks @JackBoosY I hadn't realised the files were stable to simply copy around the way hashing was performed. That seems easy though a small shame you cannot AFAIK 'regenerate' your cache in a new location. e.g. I've been using the default file location, then changed it after half the ports were built so I have some things in each location.

I will simply rationalise them to a single location. Maybe when everything is tested I'll wipe vcpkg and rebuild totally from scratch overnight :)

[jdx-john]

Well that didn't work.

I copied everything over to my local machine, but when I passed this path to vcpkg install I got this for every single project:

Could not locate cached archive: C:\Users\john\AppData\Local\vcpkg\archives\e0\e067149b284....zip
Could not locate cached archive: c:\vcpkg_cache\e0\e067149b284....zip
Starting package 1/760:

My thought was the hashing must be different for some reason. In the docs (https://vcpkg.readthedocs.io/en/stable/users/binarycaching/#implementation-notes-internal-details-subject-to-change-without-notice) it says how hashes are generated.

I let the first project build (abseil) and compared the file "vcpkg_abi_info.txt" between the two locations. They are identical except for:

triplet_abi 4469e2d62e620868b79d43cf03f77b457f92f26a9a98a023681335b3d118e902-b569c9954a47274946415ff01b1a344c8549f3fc19cb931d50bc09d1e5630c1d-78c2c298802bb1bd9f53cf1ccebfe314ca591206
triplet_abi 4469e2d62e620868b79d43cf03f77b457f92f26a9a98a023681335b3d118e902-b569c9954a47274946415ff01b1a344c8549f3fc19cb931d50bc09d1e5630c1d-674cc9191bda876ee7d19ae851648ca0819aa44f

edit: see next comment

[jdx-john]

I tried to search for "triplet_abi" and couldn't find it in code, but did find a similar discussion in #16615 about a different part of this field.
Notably, in #16615 (comment) @vicroms said:

The triplet_abi entry is composed of {triplet file hash}-{toolchain file hash}-*{compiler info hash}*.

I have VS2019 installed on both machines so how can I check what compiler info it is hashing?

[JackBoosY]

@jdx-john Maybe the two vcpkg are different.

[jdx-john]

I checked that. I updated both machines to have the newest VS and it was resolved but it doesn't answer the question where and how the check is done.
It's going to be really fragile if anytime someone updates their VS to a new minor version, they cannot use binary-cache... if we get a new developer join our team in a month the chances are VS is a slightly newer version. We are constrained to keep our build machine stable.

So the question is, does it consider the full VS version or does it allow any variance? This discussion was had in #16615 too about minor differences in CMake/PWSH version but I don't know the resolution.

Is this worth splitting off as a separate issue since my main query is now resolved @JackBoosY ?

[JackBoosY]

I prefer @ras0219-msft to answer this part.