Allow compilation with mainline compiler?

[jas4711]

Hi. Thanks for this project. To allow easier rebuilding of the firmware blob, it would be nice if it was possible to compile it using some mainline compiler. My goal is to offer a Debian package that rebuilds the blob (without including a patched llvm in the bcm5719 firmware source package). Is this likely to ever be possible?

[meklort]

A few comments here:

• The MIPS (RX) cpu is non-standard and requires a custom compiler to ensure it does not emit instructions like multiply.
• I only validate the version of the compiler used. Using an alternative compiler, even for the ARM (APE) cpu has a high chance of breaking if a different compiler is used.
• I do periodically upgrade the compiler version, however this usually causes some breakage that requires fixing.
  For those reason, using anything else than the official build instructions is not supported.

All that said, can you let me know what the goal is for packaging the firmware into Debian?

• Firmware is not loaded (automatically) by Linux into the BCM5719. Instead, it's written into flash, and then loaded at the next power off / power on event (a reboot is not sufficient in most cases, but this may be improved by adding a soft reset in fw for fwupd no longer resets the APE after fw update. APE resets may have failed in the past. #271, but still wont fix switching from proprietary fw to oss fw).
• If the goal is to use this with a card not embedded in the Talos II / Blackbird, the fw (currently) doesn't support items like WoL, PXE ROMs, NCSI over SMBus, or Jumbo Frames. These are all items that should exist for Add in Card, or there will be reduced functionality - in fact, I'd argue it'd be better to erase the firmware entirely and run without firmware for the add in cards for most users.
• The firmware only targets the Talos II / Blackbird, and release versions are included in LVFS, so if a user would like to switch, they can do so using the fwupd commands.
• If the goal is to have the firmware built from source, this seems reasonable, but please check the file hashes after builds complete. The repository supports reproducible builds. So long as you use the same compiler, you should get the same result. If the hash does not match the release version, the build should be marked as a fail.

Please let me know how you'd like to proceed, and maybe we can come up with an option that works for you and isn't difficult to support / maintain.

[meklort]

Just to clarify the custom compiler comment, the upstream RE effort has a wrapper around clang that does enable the upstream compiler by disabling specific passes and has more details. Something like this could be done, but... I'd still want to validate the specific compiler version.

https://github.com/hlandau/ortega/blob/master/cc_mips

[meklort]

(note that it also is could be possible to emulate the missing instructions, which might be another good alternative)

[jas4711]

Thanks for followup! A wrapper script like that seems like a good approach. My only goal is to package this in Debian so that it gets built using the Debian build infrastructure and can take advantage of all QA related efforts that go into Debian. If it doesn't result in the exact same hash as you get, there is good reason to debug that -- and I agree that the build should compare hash value and fail on mismatches. I have some Talos II machines to test things on.

As you may guess, this is not a serious problem -- but just cleaning up the build process and get continuous testing of the build via Debian. I worry that in 10 years reproducing the same binary we get today will no longer be that easy.