-
Notifications
You must be signed in to change notification settings - Fork 7
Getting spm working on the App Store build
A bit of background on what the swift compiler is/does and what swiftpm is/does (bear with me, you'll see why).
The swift compiler has as its main job, reading text source files and making object files. This has been true since compilers were invented many decades ago. Generally, compilers make object files for the platform they're running on. When I wrote FORTRAN on VAX/VMS, the fortran compiler would read FORTRAN source code files and produce corresponding object files for a VAX running VMS. I'd then link all the object files in my program and standard libraries with a separate linker program to make an executable, which I'd then run and test.
The standard swift compiler can do this too (be a normal "host" compiler), and if you build a swift compiler out of the box from source code by default this is exactly what it does. If you build the compiler on your mac, by default the compiler you build produces macOS object files for the architecture of your mac (apple silicon these days). But, more interestingly, you can add an ability for the compiler to target other architectures. In our case we want it to produce AVR object files. Because you're making object files for a different platform from the one the compiler is actually running on, this is called "cross compilation". You can optionally build the swift compiler to do this, to be able to produce "host" object files on macOS/apple silicon OR to produce AVR object files, you pass the flag "-target" to the swift command line to tell it which one to produce. You can make a swift compiler that can target a lot of possible different back ends, macOS/arm, macOS/intel, iOS/arm, linux/intel, avr/bare, etc. In fact, your daily use of Xcode you're mostly doing cross compilation for iOS.
Unfortunately we can't yet build working and valid AVR with a standard compiler, and our patches are against an older tree, so we can't build an out of the box cross compiler that works on all these targets, although we are getting close.
Nowadays people mostly don’t want to remember to link programs as it’s an annoying implementation detail. So modern compilers like gcc, clang, rustc, and swift have another "personality". You can ask them to drive the whole compilation process, including linking. To do this they know that they're being invoked in "driver" mode or "compiler" mode and in "driver" mode they actually just run a set of programs to do a complete build from source code including running the linker. You can see this part of the source code of the swift compiler in lib/driver.
However in the last year or two, the community took this in a new direction. They wrote a brand new, standalone program written in pure swift that acts as the driver. When you invoke swift to build an executable, you're actually just running a symbolic link to the swift-driver program. That then runs the compilation jobs and linking. It also, if requested, invokes swiftpm.
Swiftpm itself is just a command line tool, or rather a set of related command line tools that share a lot of code. When you build a swift compiler toolchain from source code and ask it to also build swiftpm, you'll see a number of files in the /bin folder it produces, including swift-driver, swift-frontend (the actual compiler), swift and swiftc (both are just symlinks to swift-driver), swift-package, swift-build, swift-run and swift-test. The latter 3 are command line executables that enact the functionality of the swift package manager.
When you run the command "swift run" in a project that is a swift package, you are of course running swift-driver (it's a symlink) and passing "run" as the first parameter. If swift-driver sees a command it doesn't recognise, such as "swift byegones", all it does is search for an executable program (or executable script) called "swift-byegones" in the usual places (locally in the same directory or on the search path), then run it and pass the rest of the parameters to it. So "swift run" will actually just run the command line program swift-run. And "swift package" will run the command line program "swift-package".
The fundamental purpose of package managers is to allow your project to specify a list of dependencies somehow then download (and cache) the appropriate version of them next time you run your package manager, it may check for newer versions if needed or may rely on the downloaded/cached versions it already has. npm and pip work exactly like this. swiftpm is analogous, but it must also compile the dependencies because imported libraries/modules in the swift world are binaries (typically a compiled .swiftmodule and optionally a compiled library of object files). This complicates it a bit but it's essentially the same process as npm or pip, but with a build step added for each package after the resolve and source code downloading steps.
Another wrinkle in swiftpm is the package file is written in swift itself. To make this work, swift-package actually compiles Package.swift into a simple command line program (a sort of command line mini tool), by linking it with the PackageDependency module and others. It then runs this mini tool, which outputs a custom json document describing the package and its dependencies and the package manager code reads the special json file and acts according to that spec. This means that swift-package actually invokes the full swift compiler to make macOS object files and link them... an important detail!
In our build toolchain, to avoid a lot of complexity, we just run swift-package, then we use a trick to build the downloaded/cached packages using our own toolchain and import them. This all works nicely.
As I said above, our swift cross compiler isn't really a normal one. It's going to take a lot of work to get the normal mainline compiler working properly for AVR. We are on the road, but there's a lot to bring the community with us, it might take many months of focused effort, and relies on them continuing to engage. We are one of a lot of competing priorities and voices for the community.
In the meantime, our IDE still uses a custom built swift cross compiler that doesn't work as anything else. It cannot compile macOS object files… we don’t even have a standard library that works for Mac in our build… and it doesn't have a working swiftpm set of binaries (swift-package, swift-run, swift-test)… until we get the mainstream compiler working for both normal use and our platform, we are unlikely to have these crucial pieces of swiftpm in our IDE toolchain.
To use swiftpm binaries we either need to build a whole “regular” toolchain separately or use an already installed version from somewhere. In our current developer ID IDE version 6.1 this is provided by just invoking out to the command line tools installed with Xcode. Because developer ID apps are less restricted, our IDE and its XPC buildengine service can invoke outside programs installed on the same Mac without serious restrictions. So our package manager relies on Xcode. Not perfect but fine for most of us.
The app store restrictions include sandboxing and code signing strict “laws”. One of which is all executable programs must exist in only one directory in the app, Contents/MacOS. They must be correctly signed and they cannot invoke much outside of their own programs except the most basic parts of unix/posix (ls or sed and so on).
One small exception is our build engine is an XPC service, which is a bundle in Contents/XPCServices/BuildEngine.xpc. This has the same restriction again, so its own tools/binaries must exist only in its own Contents/MacOS.
Now comes the "fun" part. our BuildEngine/Contents/MacOS contains our AVR compiler binaries, our swift-frontend, swift-driver, swift and other tools such as avrdude for uploading the hex.
Sandbox restrictions won't let us run the swift compiler or swiftpm tools from Xcode inside a sandboxed app. So we'd have to install a full swift and swiftpm toolchain somewhere in our IDE bundle to get around that.
To make swiftpm work without relying on Xcode we would have to deploy a full toolchain. But where would we put it? As stated above, the only place you can put binaries would be in BuildEngine/Contents/MacOS or you'll fail app store signing/validation (it might run in debug on your developer machine but would be blocked from the app store). So we would have to put all the swiftpm binaries and all the swift compiler parts they rely on to function (that build MacOS object files and the simple command line mini tool) in BuildEngine/Contents/MacOS too!
BUT.. we already have a swift compiler in there, the one we use for the actual AVR platform itself. We cannot have two compiler binaries named exactly the same name in the same directory! We can’t have two swift-driver(s) and two swift-fronted(s) one for AVR and one for Mac both in the same /MacOS folder.
So we are really stuck here.
The solution is to work with Apple to get the mainstream compiler properly working for AVR. It's a lot of work but will bring a lot of benefits and is "the right thing to do" in the end.
One hack I have thought of, that I don't fancy doing, is to make yet another XPC service, just for containing swiftpm! Then that XPC will have its own separate Contents/MacOS folder, separate from the one in the BuildEngine XPC. I'm not sure this would work as it would have to share some runtime sandbox folders with the IDE and the XPC build service so they can work on the same build files/folders, which I think is possible with shared group containers. But it's going to be tricky even if it does work.. the build engine will have to pause and hand over steps in the build process to this XPC service, which will be a bit nightmarish. I'm not even sure if an XPC service can invoke another XPC service? Or would it be some horror like the build service sending a message back to the IDE which then triggered the packaging build service... ugh...
Anyway, the above is all pretty complicated but hopefully it explains why I'm stuck with swiftpm on the app store.
Carl