update_checkout: add boringssl to the checkout set

[compnerd]

This library dependency is already present in the Android SDK builds. Replicate the dependency in the checkouts to allow building for the Android SDK on non-Linux platforms.

[compnerd]

@swift-ci please smoke test

[compnerd]

@swift-ci please smoke test

[finagolfin]

@marcprux, won't this break Android CI, since we've been checking out this repo separately, as discussed in the linked issue?

[marcprux]

It seems like it ought to break the Android and Static Linux SDK builds, and yet Static Linux also checks out its own libxml2 and doesn't break despite that dependency being listed in update-checkout-config.json. WDYT, @al45tair?

[finagolfin]

Oh, I didn't read that script source carefully enough, 🫠 looks like the [[ -d boringssl ]] || makes sure there isn't a source directory already, so this should be fine.

I guess the only question then is whether we should use this fips branch checked out here, or stick with the tag override we're doing there. Heh, never mind, we already use the fips branch on Android too. 😸

[finagolfin]

The static linux and Android SDKs were already downloading this source separately, so good to pull this repo in in one place. Can you add it for the 6.3 branch also, as we already use it there?

[marcprux]

Heh, never mind, we already use the fips branch on Android too.

Note that Static Linux just bumped its BoringSSL dependency to a hash aligned with Swift NIO: See swiftlang/swift-docker#488 (comment) and apple/swift-nio-ssl#506.

I had been wondering if we should do the same for Android, but for the time being we are still on fips-20220613, so I guess once this PR gets merged we can just remove the BoringSSL checkout from swift-ci/sdks/android/scripts/fetch-source.sh…

[compnerd]

Okay, I think that aligning on the same version of boringssl is a good idea. I wonder if we can also somehow manage to not build boringssl twice (or rather thrice). It does take a while to build.

[compnerd]

@swift-ci please smoke test

[marcprux]

Once this is merged, will this allow us to resurrect the libcurl bump to 8.17.0 in #86321 that was reverted in #86762?

[compnerd]

No, this will need to be integrated into build.ps1 before we can restore that change.

[compnerd]

Hmm, it appears that 817ab07ebb53da35afea409ab9328f578492832d is not on the GH mirror. I don't know how to add the googlesource repo to the update-checkout.

Perhaps @charles-zablit knows how to setup update-checkout to use https://boringssl.googlesource.com/boringssl for the checkout.

[charles-zablit]

AFAIK github.com is hardcoded here:

swift/utils/update_checkout/update-checkout-config.json

Lines 2 to 3 in 001e421

	"ssh-clone-pattern": "git@github.com:%s.git",
	"https-clone-pattern": "https://github.com/%s.git",

[charles-zablit]

swift/utils/update_checkout/update_checkout/update_checkout.py

Lines 631 to 639 in 001e421

	remote_repo_info = repo_info["remote"]
	if "url" in remote_repo_info:
	remote = remote_repo_info["url"]
	else:
	remote_repo_id = remote_repo_info["id"]
	if args.clone_with_ssh is True or "https-clone-pattern" not in config:
	remote = config["ssh-clone-pattern"] % remote_repo_id
	else:
	remote = config["https-clone-pattern"] % remote_repo_id

looks like it can be overriden with url in remote:

{
  "boringssl": {
    "remote": {
      "id": "google/boringssl",
      "url": "https://boringssl.googlesource.com/boringssl"
    }
  }
}

[compnerd]

@swift-ci please smoke test

[compnerd]

@swift-ci please smoke test Windows platform

[finagolfin]

The issue is not the repo used, but that the Windows version of update-checkout seems to work differently, ie both Mac and linux runs passed with the earlier GitHub version of this pull, but the Windows run always failed both versions of this pull with this same error:

[swift-PR-windows] 'git clone --config core.symlinks=true --config core.autocrlf=false --recursive --depth 1 --branch 817ab07ebb53da35afea409ab9328f578492832d https://boringssl.googlesource.com/boringssl boringssl' returned (128) with the following Cloning into 'boringssl'...
warning: Could not find remote branch 817ab07ebb53da35afea409ab9328f578492832d to clone.
fatal: Remote branch 817ab07ebb53da35afea409ab9328f578492832d not found in upstream origin
.
update-checkout failed, fix errors and try again

This implies that the source checkout is being done differently with git on Windows, ie there is something different or wrong about the Windows execution of this checkout script.

[charles-zablit]

The issue is not the repo used, but that the Windows version of update-checkout seems to work differently, ie both Mac and linux runs passed with the earlier GitHub version of this pull, but the Windows run always failed both versions of this pull with this same error:

[swift-PR-windows] 'git clone --config core.symlinks=true --config core.autocrlf=false --recursive --depth 1 --branch 817ab07ebb53da35afea409ab9328f578492832d https://boringssl.googlesource.com/boringssl boringssl' returned (128) with the following Cloning into 'boringssl'...
warning: Could not find remote branch 817ab07ebb53da35afea409ab9328f578492832d to clone.
fatal: Remote branch 817ab07ebb53da35afea409ab9328f578492832d not found in upstream origin
.
update-checkout failed, fix errors and try again

This implies that the source checkout is being done differently with git on Windows, ie there is something different or wrong about the Windows execution of this checkout script.

On Linux and macOS, there is a warning:

Updating [50/54] (llvm-project, cmake, curl, boringssl)                         Cannot determine if 817ab07ebb53da35afea409ab9328f578492832d is a branch or a tag

Linux and macOS are using Git 2.32, Windows is using Git 2.47. Maybe this warning became an error in more recent versions of Git?

The commit hash should be changed to a branch or tag: https://github.com/swiftlang/swift/pull/86823/changes#r2758582368

[charles-zablit]

This should be a branch or a tag, not a commit hash.

[charles-zablit]

This should be a branch or a tag, not a commit hash.

Same here.

[compnerd]

@charles-zablit we cannot use a branch or a tag, the NIO folks discuss the state of boringssl with upstream and choose specific commits, so this must remain a commit hash. I know that I can do this with repo, how do we do this with update-checkout? Alternatively, I would be fine with floating this to a branch if we can float NIO as well.

[charles-zablit]

Looking into this closer: update-checkout works with commit hashes, however the --skip-repository option does not. I have fixed this here: #86977

Once the patch above is merged, this patch will work.

[compnerd]

@swift-ci please smoke test macOS platform

[compnerd]

@swift-ci please test Windows platform

[compnerd]

@swift-ci please smoke test macOS platform

[finagolfin]

@compnerd, can we cherry-pick these two pulls to 6.3 also?