Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transmit config to new worker with a protobuf type + protobuf channels #1000

Merged
merged 12 commits into from
Feb 9, 2024

Conversation

Keksoj
Copy link
Contributor

@Keksoj Keksoj commented Oct 11, 2023

When a new worker is created, the whole Config object is passed to it:

sozu/bin/src/worker.rs

Lines 340 to 342 in bc49d7f

main_to_worker_channel
.write_message(config)
.with_context(|| "Could not send config to the new worker using the channel")?;

This Config object contains necessary infos, like timeouts, but also proxy business infos like clusters, frontends etc. that are not used directly and are re-sent to the new worker later on in a different form.

Since we strive to have a protobuf-only communication both without and within Sōzu,
and since the Config object can be translated to protobuf only with difficulty, and no real purpose (as most of it is unused),
this PR changes the passing of the Config object to a ServerConfig. This struct was already derived from Config, was present in the Sōzu lib, but now:

  • it is completed with necessary fields (buffer sizes, etc)
  • it is written in protobuf
  • it is passed to a new worker when created

This PR paves the way to changing the Sōzu channels in order to stop serializing structs to JSON, but to make use of the protobuf binary serialization.

One thing remains in a JSON form, though: requests written to a state file and passed to sōzu using a LoadState message.

@Keksoj Keksoj added this to the v0.16.0 milestone Oct 11, 2023
@Keksoj Keksoj self-assigned this Oct 11, 2023
@Keksoj Keksoj force-pushed the transmit-protobuf-only branch from 542a679 to 64920c5 Compare October 11, 2023 13:27
Copy link
Member

@Wonshtrum Wonshtrum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Great update!

@Keksoj Keksoj changed the title Transmit config to new worker with a protobuf type Transmit config to new worker with a protobuf type + protobuf channels Oct 13, 2023
@Keksoj
Copy link
Contributor Author

Keksoj commented Oct 13, 2023

We merged #1001, a unique commit, into this branch, here is the abstract of this PR:

This is the last step in the implementation of protobuf in Sōzu: use protobuf for all communications of Sōzu, within and without. See #1000

Coded with @Wonshtrum , this PR:

  • makes channel transmit protobuf messages
  • in a binary serialization
  • with a fixed-size delimiter that tells how long the message is

Here is a simple benchmark of performance improvements:

We will serialize, write in a buffer, read from the buffer and deserialize, a 1000000 times, this object:
RequestHttpFrontend { cluster_id: Some("cluster_1"), address: "127.0.0.1:8080", hostname: "lolcatho.st", path: PathRule { kind: Prefix, value: "/" }, method: Some("GET"), position: Tree, tags: {"app_id": "a782f2e4-ee42-4384-9757-2b13d4198105", "owner": "Clever Cloud"} }
serde_json serialized it in 216 bytes
Serde JSON needed 23.838388363s
prost serialized it in 125 bytes
Prost needed 12.118231414s

@Keksoj Keksoj force-pushed the transmit-protobuf-only branch from 2e8a15e to 33d8150 Compare October 23, 2023 18:03
@Keksoj Keksoj force-pushed the transmit-protobuf-only branch from 29f6af0 to a17a585 Compare November 23, 2023 15:44
@Keksoj Keksoj force-pushed the transmit-protobuf-only branch from a17a585 to 323171d Compare November 24, 2023 13:14
@Keksoj Keksoj force-pushed the transmit-protobuf-only branch 4 times, most recently from ee289e6 to b87ada8 Compare November 30, 2023 17:21
@Keksoj Keksoj force-pushed the transmit-protobuf-only branch 6 times, most recently from 190f5b4 to 58314d0 Compare December 8, 2023 17:02
@Keksoj Keksoj force-pushed the transmit-protobuf-only branch from 49764d2 to b063ac4 Compare January 2, 2024 10:02
@Keksoj Keksoj mentioned this pull request Jan 22, 2024
4 tasks
@Keksoj Keksoj force-pushed the transmit-protobuf-only branch 2 times, most recently from c6005ec to 81dc275 Compare January 24, 2024 14:02
@Keksoj Keksoj force-pushed the transmit-protobuf-only branch from 81dc275 to 47ebb9f Compare January 30, 2024 21:14
@Keksoj Keksoj force-pushed the transmit-protobuf-only branch 9 times, most recently from 9bd9a79 to 17f3ba8 Compare February 1, 2024 12:41
@Keksoj Keksoj force-pushed the transmit-protobuf-only branch 2 times, most recently from 6a72c6f to bb876ee Compare February 2, 2024 14:00
@Keksoj Keksoj force-pushed the transmit-protobuf-only branch from bb876ee to cc0a221 Compare February 2, 2024 15:51
@FlorentinDUBOIS FlorentinDUBOIS merged commit e451177 into main Feb 9, 2024
13 checks passed
@FlorentinDUBOIS FlorentinDUBOIS deleted the transmit-protobuf-only branch February 9, 2024 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants