feat: Add S3 backend support

[delsner]

Motivation

Part of a series of PRs for implementing conda/rattler#960.

Changes

Add S3 backend support for pixi.toml and global pixi config.

Joint work with @moritzwilksch and @pavelzw.

[ruben-arts]

Thanks guys, I'm asking @baszalmstra for a review of this work.

[pavelzw]

this first version should be ready for review as well, only a couple smaller things missing

[wolfv]

fyi we temporarily removed the secrets to see if that fixes our release build.

[ruben-arts]

This is a great addition, please finalize the config CLI and tell me how the aws example should work and then this should be ready!

[ruben-arts]

I'll continue to look what is going on, but not doing anything this example gives me this error, with this version of pixi.

pixi/examples/s3-package-server on  s3 via 🐍 v3.13.1 
❯ pixi i
    ⠁ validate cache       [00:00:00] [━━━━━━━━━━╾─────────] markupsafe (+20)
    ⠁ download & extract   [00:00:00] [────────────────────]      0 B @ 0 B/s    email_validator (+16)
    ⠁ installing packages  [00:00:00] [━───────────────────]     2/61 python_abi (+18)                                                                         Error: 
  × failed to fetch my-webserver-0.1.0-pyh4616a5c_0.conda
      validate cache       [00:00:00] [━━━━━━━━━━━━━━━━━━━━] markupsafe (+16)
      download & extract   [00:00:00] [━━━━━━━━━━━━━━━━━━━━] 3.42 MiB @ 227.13 MiB/s email_validator (+16)
    ⠁ installing packages  [00:00:00] [━───────────────────]     2/61 python_abi (+18)                                                                         
pixi/examples/s3-package-server on  s3 via 🐍 v3.13.1 
❯ pixi i -vvv
...
 INFO pixi::lock_file::update: the lock-file is up-to-date
 INFO pixi::lock_file::update: Updating prefix
DEBUG pixi_utils::reqwest: Using s3_config: {"rattler-s3-testing": Custom { endpoint_url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("e1a7cde76f1780ec06bac859036dbaf7.eu.r2.cloudflarestorage.com")), port: None, path: "/", query: None, fragment: None }, region: "auto", force_path_style: true }}
 INFO rattler_networking::s3_middleware: Creating S3 middleware using {"rattler-s3-testing": Custom { endpoint_url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("e1a7cde76f1780ec06bac859036dbaf7.eu.r2.cloudflarestorage.com")), port: None, path: "/", query: None, fragment: None }, region: "auto", force_path_style: true }}
DEBUG pixi_utils::reqwest: s3_middleware: S3Middleware { s3: S3 { auth_storage: AuthenticationStorage { backends: [KeyringAuthenticationStorage { store_key: "rattler" }, FileStorage { path: "/Users/rubenarts/.rattler/credentials.json", cache: RwLock { data: FileStorageCache { content: {} }, poisoned: false, .. } }, NetRcStorage { machines: {} }], cache: Mutex { data: {}, poisoned: false, .. } }, config: {"rattler-s3-testing": Custom { endpoint_url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("e1a7cde76f1780ec06bac859036dbaf7.eu.r2.cloudflarestorage.com")), port: None, path: "/", query: None, fragment: None }, region: "auto", force_path_style: true }}, expiration: 300s } }
DEBUG pixi::environment::conda_prefix: updating prefix for 'default'
Error: 
  × EOF while parsing a value at line 1 column 0

[ruben-arts]

Follow the story that started by this error here: #3125

[ruben-arts]

Should I run something locally to test this examples?

[pavelzw]

pixi auth login --s3-access-key-id <access-key-id> --s3-secret-access-key <secret-access-key> s3://rattler-s3-testing. does it work afterwards?

does pixi update --no-install work? this should only fetch repodata.json

[pavelzw]

sort-of related to conda/rattler#1057

[ruben-arts]

Let's remove the example 👍

[ruben-arts]

Amazing work! Going to use this as an example for contributors.

For future visitors, this contribution is specifically good because:

• Including good tests
• Including the required CLI updates: pixi config ...
• Updated the model.py for the JSON schema.
• Updated documentation, in a separate PR wouldn't be needed, but it's there!
• You went to Discord for direct help to speed up the process.

It's a big feature, so it's not something easy to accept!