First draft of Reproducible builds RFC

Author: cmaglie

RFCs/0003-build-profiles.md

@@ -0,0 +1,216 @@
+# Sketch build profiles and reproducible builds
+
+## Overview
+
+The Arduino build system and Sketch specifications lack a way to share and archive sketches and guarantee reproducibility of the built firmware at any time in the future as it was last successfully built.
+
+These incompatibilities might arise due to newer versions of a Platform, Tool, or Library.
+
+## Problem
+
+An Arduino project is known as Sketch. It comprises its main .ino file plus optional local source code (in the same folder) and an optional extra src folder containing additional sources if required.
+
+The sketch compiles against a set of globally installed libraries and platforms. When said library and/or part of the hardware platform is updated (sometimes introducing some breaking changes) that sketch might not compile anymore or behave differently.
+
+Currently, Arduino guarantees no portability and reproducibility of a build at a later time unless exact instructions are provided by the developer to the final user such as the required platform (including version and possibly a 3rd party platform URL to download and install the platform) or all the required libraries including the exact version of each one.
+
+## Goals
+
+This RFC aims to propose new augmentations to the current library management and a build system that can be entirely transparent to the user and retains backward compatibility yet allowing more flexibility for the future.
+
+The following should be true:
+
+- Implement ways for any Arduino Sketch to behave as a reproducible build with an option for versioning
+
+- Allow sharing of such sketch and “environment settings” from
+
+  - IDE 1.8 and 2.0 using Archive Sketch
+  - Arduino Cloud using Share/Download Sketch
+
+## Constraints
+
+- Must be compatible across build systems
+
+  - Arduino CLI
+  - Arduino IDE 2.0
+  - Arduino IDE 1.8
+  - Arduino Cloud
+
+Must not break pre-existing setups or be completely transparent to / ignored by previous versions of the tools
+
+## Proposed Solution
+
+The proposal is to add a project file called `sketch.yaml`. This file will be in YAML format and it will contain a description of all the resources required to compile a sketch, in particular:
+
+- the board FQBN
+- the target core platform version (and the 3rd party platform index URL if needed)
+- a possible core platform that is a dependency of the target core platform (again with version and index URL)
+- the libraries used (including their version)
+
+We will call the set of the information above a **profile**. Each profile will be saved in the file `sketch.yaml` and this file may have one or more profiles.
+
+The format of the file is the following:
+
+```
+profiles:
+  <PROFILE_NAME>:
+    notes: <USER_NOTES>
+    fqbn: <FQBN>
+    platforms:
+      - platform: <PLATFORM> (<PLATFORM_VERSION>)
+        platform_index_url: <3RD_PARTY_PLATFORM_URL>
+      - platform: <PLATFORM_DEPENDENCY> (<PLATFORM_DEPENDENCY_VERSION>)
+        platform_index_url: <3RD_PARTY_PLATFORM_DEPENDENCY_URL>
+    libraries:
+      - <LIB_NAME> (<LIB_VERSION>)
+      - <LIB_NAME> (<LIB_VERSION>)
+      - <LIB_NAME> (<LIB_VERSION>)
+
+...more profiles here...
+
+default_profile: <DEFAULT_PROFILE_NAME>
+```
+
+We have a `profiles:` section containing all the profiles. Each field is self-explanatory, in particular:
+
+- `<PROFILE*NAME>` is the profile identifier, it’s a user-defined field, and the allowed characters are alphanumerics, underscore `_`, dot `.`, and dash `-`
+- `<PLATFORM>` is the target core platform identifier, for example, `arduino:avr` or `adafruit:samd`
+- `<PLATFORM_VERSION>` is the target core platform version required
+- `<3RD_PARTY_PLATFORM_URL>` is the index URL to download the target core platform (also known as “Additional Boards Manager URLs” in the Arduino IDE). This field can be omitted for the official `arduino:*` platforms.
+- `<PLATFORM_DEPENDENCY>`, `<PLATFORM_DEPENDENCY_VERSION>`, and `<3RD_PARTY_PLATFORM_DEPENDENCY_URL>` contains the same information as `<PLATFORM>`, `<PLATFORM_VERSION>`, and `<3RD_PARTY_PLATFORM_URL>` but for the core platform dependency of the main core platform. These fields are optional.
+- `<LIB_VERSION>` is the version required for the library, for example, `1.0.0`
+- `<USER_NOTES>` is a free text string available to the developer to add comments
+- `<DEFAULT_PROFILE_NAME>` is the profile used by default (more on that later)
+
+A complete example of a `sketch.yaml` may be the following:
+
+```
+profiles:
+  nanorp:
+  fqbn: arduino:mbed_nano:nanorp2040connect
+  platforms:
+    - platform: arduino:mbed_nano (2.1.0)
+  libraries:
+    - ArduinoIoTCloud (1.0.2)
+    - Arduino_ConnectionHandler (0.6.4)
+    - TinyDHT sensor library (1.1.0)
+
+  another_profile_name:
+    notes: testing the limit of the AVR platform, may be unstable
+    fqbn: arduino:avr:uno
+    platforms:
+      - platform: arduino:avr (1.8.4)
+    libraries:
+      - VitconMQTT (1.0.1)
+      - Arduino_ConnectionHandler (0.6.4)
+      - TinyDHT sensor library (1.1.0)
+
+  tiny:
+    notes: testing the very limit of the AVR platform, it will be very unstable
+    fqbn: attiny:avr:ATtinyX5:cpu=attiny85,clock=internal16
+    platforms:
+      - platform: attiny:[email protected]
+        platform_index_url: https://raw.githubusercontent.com/damellis/attiny/ide-1.6.x-boards-manager/package_damellis_attiny_index.json
+     - platform: arduino:[email protected]
+    libraries:
+      - ArduinoIoTCloud (1.0.2)
+      - Arduino_ConnectionHandler (0.6.4)
+      - TinyDHT sensor library (1.1.0)
+
+  feather:
+    fqbn: adafruit:samd:adafruit_feather_m0
+    platforms:
+      - platform: adafruit:samd (1.6.0)
+        platform_index_url: https://adafruit.github.io/arduino-board-index/package_adafruit_index.json
+    libraries:
+      - ArduinoIoTCloud (1.0.2)
+      - Arduino_ConnectionHandler (0.6.4)
+      - TinyDHT sensor library (1.1.0)
+
+default_profile: nanorp
+```
+
+## Building a sketch
+
+When a `sketch.yaml` file exists in the sketch, it can be leveraged to compile the sketch with the `--profile` flag in the compile command:
+
+```
+arduino-cli compile [sketch] --profile nanorp
+```
+
+In this case, the sketch will be compiled using the core platform and libraries specified in the `nanorp` profile. If a core platform or a library is missing it will be automatically downloaded and installed on the fly in a dedicated directory inside the data folder. The dedicated storage is not accessible to the user and is meant as a “cache” of the resources used to build the sketch.
+
+When using the “profile” based build, the globally installed platforms and libraries are excluded from the compile and **may not be used in any way**, in other words, the build is isolated from the system and will rely only on the resources specified in the profile: this will ensure that the build is portable and reproducible independently from the platforms and libraries installed in the system.
+
+## Using a default profile
+
+If a `default_profile` is specified in the `sketch.yaml` then the “classic” compile command:
+
+```
+arduino-cli compile [sketch]
+```
+
+will trigger a profile-based build using the default profile indicated in the `sketch.yaml`.
+
+## Vendoring libraries with custom modifications
+
+We will add the possibility to add custom libraries directly in the sketch using the `libraries` subdirectory in the sketch root folder.
+
+A typical usage scenario is when the sketch needs a library that is not part of the library manager, or if the sketch needs a library with some customizations that are not available in the upstream release.
+
+To accomplish this the idea is to put the custom libraries' source code directly in the sketch `libraries` subdirectory (exactly as we do for the globally installed libraries in the `libraries` subdirectory of the sketchbook root folder). For example:
+
+```
+~/Arduino/HumiditySensor$ tree
+HumiditySensor
+├── HumiditySensor.ino
+├── sketch.yaml
+└── libraries
+    ├── DHTSensorCustom
+    │   ├── library.properties
+    │   ├── README
+    │   └── src
+    │       ├── DHT.cpp
+    │       ├── DHT.h
+    │       └── DHTSensorCustom.h
+    └── NTPClientEth
+        ├── NTPClient.h
+        └── NTPClient.cpp
+```
+
+If we run a “classic” build, without profiles, then the libraries `DHTSensorCustom` and `NTPClientEth` will be included at compile time as they were installed globally.
+
+If we want to run a reproducible build with profiles, we need to explicitly specify the bundled libraries we intend to use in the `sketch.yaml` in this way:
+
+```
+profiles:
+  nanorp:
+    fqbn: arduino:mbed_nano:nanorp2040connect
+    platforms:
+      - platform: arduino:mbed_nano (2.1.0)
+    libraries:
+      - ArduinoIoTCloud (1.0.2)
+      - Arduino_ConnectionHandler (0.6.4)
+      - TinyDHT sensor library (1.1.0)
+    vendored_libraries:
+      - NTPClientEth
+
+  uno:
+    notes: testing the limit of the AVR platform, may be unstable
+    fqbn: arduino:avr:uno
+    platforms:
+      - platform: arduino:avr (1.8.4)
+    libraries:
+      - VitconMQTT (1.0.1)
+      - Arduino_ConnectionHandler (0.6.4)
+    vendored_libraries:
+      - DHTSensorCustom
+      - NTPClientEth
+```
+
+In this case:
+
+- running a build using the profile `nanorp` will use the bundled `NTPClientEth` library, but not `DHTSensorCustom`
+- running a build with the profile `uno` will use both `NTPClientEth` and `DHTSensorCustom`
+
+This behavior is required to allow different profiles to use a different set of vendored libraries.