`SettingsService#Merge` and `SettingsService#Write` work incorrectly between `daemon` sessions #2275

kittaakos · 2023-08-23T09:45:32Z

Describe the problem

Originally from arduino/arduino-ide#2185 (review).

When IDE2 sets the network#proxy to manual proxy settings, then unsets it in another daemon session, it does not work. It works perfectly when setting and then unsetting network#proxy within the same daemon session.

To reproduce

You need two shells.

In shell #1:

./arduino-cli version
arduino-cli  Version: 0.34.0 Commit: 304d48cd Date: 2023-08-23T08:01:50Z

cat ./my-cli-config.yaml

board_manager:
  additional_urls:
  - https://www.pjrc.com/teensy/package_issue1588_index.json
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  builtin:
    libraries: /Users/a.kitta/Library/Arduino15/libraries
  data: /Users/a.kitta/Library/Arduino15
  downloads: /Users/a.kitta/Library/Arduino15/staging
  user: /Users/a.kitta/Documents/Arduino
library:
  enable_unsafe_install: false
locale: en
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

./arduino-cli daemon --config-file ./my-cli-config.yaml
Daemon is now listening on 127.0.0.1:50051
{"IP":"127.0.0.1","Port":"50051"}

In shell 2:

Get all config values

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.GetAll | jq -r .jsonData | jq .

{
  "board_manager": {
    "additional_urls": [
      "https://www.pjrc.com/teensy/package_issue1588_index.json"
    ]
  },
  "build_cache": {
    "compilations_before_purge": 10,
    "ttl": "720h0m0s"
  },
  "daemon": {
    "port": "50051"
  },
  "directories": {
    "builtin": {
      "libraries": "/Users/a.kitta/Library/Arduino15/libraries"
    },
    "data": "/Users/a.kitta/Library/Arduino15",
    "downloads": "/Users/a.kitta/Library/Arduino15/staging",
    "user": "/Users/a.kitta/Documents/Arduino"
  },
  "library": {
    "enable_unsafe_install": false
  },
  "locale": "en",
  "logging": {
    "file": "",
    "format": "text",
    "level": "info"
  },
  "metrics": {
    "addr": ":9090",
    "enabled": true
  },
  "network": {
    "user_agent_ext": "daemon"
  },
  "output": {
    "no_color": false
  },
  "sketch": {
    "always_export_binaries": false
  },
  "updater": {
    "enable_notification": true
  }
}

Compare with content on disk:

cat ./my-cli-config.yaml

board_manager:
  additional_urls:
   - https://www.pjrc.com/teensy/package_issue1588_index.json
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  builtin:
    libraries: /Users/a.kitta/Library/Arduino15/libraries
  data: /Users/a.kitta/Library/Arduino15
  downloads: /Users/a.kitta/Library/Arduino15/staging
  user: /Users/a.kitta/Documents/Arduino
library:
  enable_unsafe_install: false
locale: en
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

Update network#proxy config value with manual proxy settings

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  -d '{"jsonData": "{\n  \"board_manager\": {\n    \"additional_urls\": [\n      \"https://www.pjrc.com/teensy/package_issue1588_index.json\"\n    ]\n  },\n  \"build_cache\": {\n    \"compilations_before_purge\": 10,\n    \"ttl\": \"720h0m0s\"\n  },\n  \"daemon\": {\n    \"port\": \"50051\"\n  },\n  \"directories\": {\n    \"data\": \"/Users/a.kitta/Library/Arduino15\",\n    \"user\": \"/Users/a.kitta/Documents/Arduino\"\n  },\n  \"library\": {\n    \"enable_unsafe_install\": false\n  },\n  \"locale\": \"en\",\n  \"logging\": {\n    \"file\": \"\",\n    \"format\": \"text\",\n    \"level\": \"info\"\n  },\n  \"metrics\": {\n    \"addr\": \":9090\",\n    \"enabled\": true\n  },\n  \"output\": {\n    \"no_color\": false\n  },\n  \"sketch\": {\n    \"always_export_binaries\": false\n  },\n  \"updater\": {\n    \"enable_notification\": true\n  },\n  \"network\": {\n    \"proxy\": \"http://username:secret@hostename:1234/\"\n  }\n}"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.Merge

{
  
}

Write config state to disk

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  -d '{"filePath": "/Users/a.kitta/dev/git/arduino-cli/my-cli-config.yaml"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.Write

{
  
}

Verify network#proxy, manual proxy settings are configured

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.GetAll | jq -r .jsonData | jq .

{
  "board_manager": {
    "additional_urls": [
      "https://www.pjrc.com/teensy/package_issue1588_index.json"
    ]
  },
  "build_cache": {
    "compilations_before_purge": 10,
    "ttl": "720h0m0s"
  },
  "daemon": {
    "port": "50051"
  },
  "directories": {
    "builtin": {
      "libraries": "/Users/a.kitta/Library/Arduino15/libraries"
    },
    "data": "/Users/a.kitta/Library/Arduino15",
    "downloads": "/Users/a.kitta/Library/Arduino15/staging",
    "user": "/Users/a.kitta/Documents/Arduino"
  },
  "library": {
    "enable_unsafe_install": false
  },
  "locale": "en",
  "logging": {
    "file": "",
    "format": "text",
    "level": "info"
  },
  "metrics": {
    "addr": ":9090",
    "enabled": true
  },
  "network": {
    "proxy": "http://username:secret@hostename:1234/",
    "user_agent_ext": "daemon"
  },
  "output": {
    "no_color": false
  },
  "sketch": {
    "always_export_binaries": false
  },
  "updater": {
    "enable_notification": true
  }
}

cat ./my-cli-config.yaml

board_manager:
  additional_urls:
  - https://www.pjrc.com/teensy/package_issue1588_index.json
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  builtin:
    libraries: /Users/a.kitta/Library/Arduino15/libraries
  data: /Users/a.kitta/Library/Arduino15
  downloads: /Users/a.kitta/Library/Arduino15/staging
  user: /Users/a.kitta/Documents/Arduino
library:
  enable_unsafe_install: false
locale: en
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
network:
  proxy: http://username:secret@hostename:1234/
  user_agent_ext: daemon
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

Terminate the CLI daemon process in shell 1 and restart it with ./arduino-cli daemon --config-file ./my-cli-config.yaml.
❗ The bug does not happen if you do not restart the daemon process but continue with the grpcurl commands in shell 2.

In shell 2:

Unset network#proxy manual proxy configuration (using an empty object ({}))

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  -d '{"jsonData": "{\n  \"board_manager\": {\n    \"additional_urls\": [\n      \"https://www.pjrc.com/teensy/package_issue1588_index.json\"\n    ]\n  },\n  \"build_cache\": {\n    \"compilations_before_purge\": 10,\n    \"ttl\": \"720h0m0s\"\n  },\n  \"daemon\": {\n    \"port\": \"50051\"\n  },\n  \"directories\": {\n    \"data\": \"/Users/a.kitta/Library/Arduino15\",\n    \"user\": \"/Users/a.kitta/Documents/Arduino\"\n  },\n  \"library\": {\n    \"enable_unsafe_install\": false\n  },\n  \"locale\": \"en\",\n  \"logging\": {\n    \"file\": \"\",\n    \"format\": \"text\",\n    \"level\": \"info\"\n  },\n  \"metrics\": {\n    \"addr\": \":9090\",\n    \"enabled\": true\n  },\n  \"output\": {\n    \"no_color\": false\n  },\n  \"sketch\": {\n    \"always_export_binaries\": false\n  },\n  \"updater\": {\n    \"enable_notification\": true\n  },\n  \"network\": {}\n}"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.Merge

{
  
}

Write config state to disk

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  -d '{"filePath": "/Users/a.kitta/dev/git/arduino-cli/my-cli-config.yaml"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.Write

{
  
}

Verify network#proxy, manual proxy is not set (it is 🐛)

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.GetAll | jq -r .jsonData | jq .

{
  "board_manager": {
    "additional_urls": [
      "https://www.pjrc.com/teensy/package_issue1588_index.json"
    ]
  },
  "build_cache": {
    "compilations_before_purge": 10,
    "ttl": "720h0m0s"
  },
  "daemon": {
    "port": "50051"
  },
  "directories": {
    "builtin": {
      "libraries": "/Users/a.kitta/Library/Arduino15/libraries"
    },
    "data": "/Users/a.kitta/Library/Arduino15",
    "downloads": "/Users/a.kitta/Library/Arduino15/staging",
    "user": "/Users/a.kitta/Documents/Arduino"
  },
  "library": {
    "enable_unsafe_install": false
  },
  "locale": "en",
  "logging": {
    "file": "",
    "format": "text",
    "level": "info"
  },
  "metrics": {
    "addr": ":9090",
    "enabled": true
  },
  "network": {
    "proxy": "http://username:secret@hostename:1234/",
    "user_agent_ext": "daemon"
  },
  "output": {
    "no_color": false
  },
  "sketch": {
    "always_export_binaries": false
  },
  "updater": {
    "enable_notification": true
  }
}

Verify network#proxy by reading the CLI config from the disk, manual proxy is not set (it is 🐛)

cat ./my-cli-config.yaml

board_manager:
  additional_urls:
  - https://www.pjrc.com/teensy/package_issue1588_index.json
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  builtin:
    libraries: /Users/a.kitta/Library/Arduino15/libraries
  data: /Users/a.kitta/Library/Arduino15
  downloads: /Users/a.kitta/Library/Arduino15/staging
  user: /Users/a.kitta/Documents/Arduino
library:
  enable_unsafe_install: false
locale: en
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
network:
  proxy: http://username:secret@hostename:1234/
  user_agent_ext: daemon
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

Expected behavior

SettingsService#Merge and SettingsService#Write behaves deterministically independently how many daemon session I start.

Arduino CLI version

0.34.0

Operating system

macOS

Operating system version

13.5

Additional context

Downstream ref: arduino/arduino-ide#2184

Issue checklist

I searched for previous reports in the issue tracker
I verified the problem still occurs when using the nightly build
My report contains all necessary details

The text was updated successfully, but these errors were encountered:

MatteoPologruto · 2023-09-19T09:18:06Z

Hello @kittaakos, thanks for reporting this!

We actually fixed this issue in PR #2212, introducing a new command SettingsService#Delete. It was necessary to do it because Viper does not allow to unset a default configuration (which in this case is everything that is loaded from my-cli-config.yaml) by replacing it with an empty object.

To delete the network configuration, I would suggest to follow these steps:

Preliminary steps:

arduino-cli version
arduino-cli  Version: 0.34.2 Commit: 963c1a76 Date: 2023-09-11T10:05:42Z

cat ./my-cli-config.yaml

board_manager:
  additional_urls:
  - https://www.pjrc.com/teensy/package_issue1588_index.json
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  builtin:
    libraries: /home/matteo/.arduino15/libraries
  data: /home/matteo/.arduino15
  downloads: /home/matteo/.arduino15/staging
  user: /home/matteo/Arduino
library:
  enable_unsafe_install: false
locale: en
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
network:
  proxy: http://username:secret@hostename:1234/
  user_agent_ext: daemon
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

Start daemon in shell 1 and verify that network#proxy configuration is set in shell 2

arduino-cli daemon --config-file ./my-cli-config.yaml 
Daemon is now listening on 127.0.0.1:50051
{"IP":"127.0.0.1","Port":"50051"}

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.GetAll | jq -r .jsonData | jq .

{
  "board_manager": {
    "additional_urls": [
      "https://www.pjrc.com/teensy/package_issue1588_index.json"
    ]
  },
  "build_cache": {
    "compilations_before_purge": 10,
    "ttl": "720h0m0s"
  },
  "daemon": {
    "port": "50051"
  },
  "directories": {
    "builtin": {
      "libraries": "/home/matteo/.arduino15/libraries"
    },
    "data": "/home/matteo/.arduino15",
    "downloads": "/home/matteo/.arduino15/staging",
    "user": "/home/matteo/Arduino"
  },
  "library": {
    "enable_unsafe_install": false
  },
  "locale": "en",
  "logging": {
    "file": "",
    "format": "text",
    "level": "info"
  },
  "metrics": {
    "addr": ":9090",
    "enabled": true
  },
  "network": {
    "proxy": "http://username:secret@hostename:1234/",
    "user_agent_ext": "daemon"
  },
  "output": {
    "no_color": false
  },
  "sketch": {
    "always_export_binaries": false
  },
  "updater": {
    "enable_notification": true
  }
}

In shell 2:

Delete network using the DeleteRequest {"key": "network"}

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  -d '{"key": "network"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.Delete

Write config state to disk

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  -d '{"filePath": "/Users/a.kitta/dev/git/arduino-cli/my-cli-config.yaml"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.Write

Verify network#proxy, manual proxy is not set

grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/settings/v1/settings.proto \
  127.0.0.1:50051 \
  cc.arduino.cli.settings.v1.SettingsService.GetAll | jq -r .jsonData | jq .

{
  "board_manager": {
    "additional_urls": [
      "https://www.pjrc.com/teensy/package_issue1588_index.json"
    ]
  },
  "build_cache": {
    "compilations_before_purge": 10,
    "ttl": "720h0m0s"
  },
  "daemon": {
    "port": "50051"
  },
  "directories": {
    "builtin": {
      "libraries": "/home/matteo/.arduino15/libraries"
    },
    "data": "/home/matteo/.arduino15",
    "downloads": "/home/matteo/.arduino15/staging",
    "user": "/home/matteo/Arduino"
  },
  "library": {
    "enable_unsafe_install": false
  },
  "locale": "en",
  "logging": {
    "file": "",
    "format": "text",
    "level": "info"
  },
  "metrics": {
    "addr": ":9090",
    "enabled": true
  },
  "output": {
    "no_color": false
  },
  "sketch": {
    "always_export_binaries": false
  },
  "updater": {
    "enable_notification": true
  }
}

Verify network#proxy by reading the CLI config from the disk, manual proxy is not set

cat ./my-cli-config.yaml

board_manager:
  additional_urls:
  - https://www.pjrc.com/teensy/package_issue1588_index.json
build_cache:
  compilations_before_purge: 10
  ttl: 720h0m0s
daemon:
  port: "50051"
directories:
  builtin:
    libraries: /home/matteo/.arduino15/libraries
  data: /home/matteo/.arduino15
  downloads: /home/matteo/.arduino15/staging
  user: /home/matteo/Arduino
library:
  enable_unsafe_install: false
locale: en
logging:
  file: ""
  format: text
  level: info
metrics:
  addr: :9090
  enabled: true
output:
  no_color: false
sketch:
  always_export_binaries: false
updater:
  enable_notification: true

Please let me know if this process works for you or if you have any doubts!

kittaakos · 2023-09-19T09:34:38Z

introducing a new command SettingsService#Delete.

IDE2 does not know what has changed, that's why the Merge method has been created. If each CLI client has to keep track of what the config diff is, the merge API is useless.

kittaakos added type: imperfection Perceived defect in any part of project topic: CLI Related to the command line interface topic: gRPC Related to the gRPC interface labels Aug 23, 2023

This was referenced Aug 23, 2023

fix: can unset network#proxy in the CLI config arduino/arduino-ide#2185

Closed

[grpc] lib list and board details throw incorrect error when directories.data folder does not exist at daemon startup #1262

Closed

kittaakos changed the title ~~SettingsService#Merge and SettingsService#Write works incorrectly between daemon sessions~~ SettingsService#Merge and SettingsService#Write work incorrectly between daemon sessions Aug 24, 2023

umbynos added this to the Arduino CLI 0.35.0 milestone Sep 1, 2023

umbynos assigned MatteoPologruto Sep 18, 2023

umbynos modified the milestones: Arduino CLI v0.35.0, Arduino CLI v0.36.0 Oct 16, 2023

MatteoPologruto linked a pull request Jan 8, 2024 that will close this issue

Allow empty configuration merging #2432

Merged

6 tasks

umbynos closed this as completed in #2432 Jan 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`SettingsService#Merge` and `SettingsService#Write` work incorrectly between `daemon` sessions #2275

`SettingsService#Merge` and `SettingsService#Write` work incorrectly between `daemon` sessions #2275

kittaakos commented Aug 23, 2023 •

edited

Loading

MatteoPologruto commented Sep 19, 2023

kittaakos commented Sep 19, 2023

SettingsService#Merge and SettingsService#Write work incorrectly between daemon sessions #2275

SettingsService#Merge and SettingsService#Write work incorrectly between daemon sessions #2275

Comments

kittaakos commented Aug 23, 2023 • edited Loading

Describe the problem

To reproduce

Expected behavior

Arduino CLI version

Operating system

Operating system version

Additional context

Issue checklist

MatteoPologruto commented Sep 19, 2023

kittaakos commented Sep 19, 2023

`SettingsService#Merge` and `SettingsService#Write` work incorrectly between `daemon` sessions #2275

`SettingsService#Merge` and `SettingsService#Write` work incorrectly between `daemon` sessions #2275

kittaakos commented Aug 23, 2023 •

edited

Loading