Add build and containerize pipeline #61

ValMobBIllich · 2025-01-13T11:57:27Z

Adds a new action (and a small composite action) that builds the streamer binary, creates an image with it and pushes it to a container registry. Users can then pull the image from the registry and run the streamer in a container without ever looking at the source code.

The pipeline build statically linked binaries of the zenoh-mqtt streamer and pushes them via a manifest so that users can pull whichever image is better for their platform.

There is also a new docker directory which showcases how to use the images in the container registry and with a complete example config setup. Maybe there is a way to structure this thats a bit easier to understand? For now I ll just add some documentation in the implementations Readme...

…ner, reference docker compose with config mounting

…iles

a composite action

push steps of the build pipeline

PLeVasseur · 2025-01-27T20:29:40Z

I'm guessing I should hold on review till the CI passes & you give me a heads up. My assumption though! Let me know.

ValMobBIllich · 2025-01-28T10:36:57Z

I'm guessing I should hold on review till the CI passes & you give me a heads up. My assumption though! Let me know.

Yeah I think we need to merge the other PR first so that the vsomeip stuff works again? The build and push to registry pipeline fails right now because Im using the organization name for the image URL and that has to be lowercase (our is ValtechMobility so its not...) but eclipse-uprotocol should be fine. Anyways the build pipeline wont run on PR but on push to main so it shouldnt block the PR once I change that.

build push action

reference docker-compose

PLeVasseur · 2025-02-13T19:01:37Z

Hey @ValMobBIllich -- looked like CI fell over here. Perhaps need to rebase? I'd like to get CI running and then take a look :)

…o use it.

…0.3.0

…+ stdlib

…ld_and_containerize_pipeline

PLeVasseur · 2025-03-05T14:21:44Z

Hey @ValMobBIllich! I'm going to start reviewing this now. Apologies if I pick over things you did a while back, I'm looking at it with fresh eyes. 🙏

PLeVasseur

Hey @ValMobBIllich! Thanks for the contribution to containerize. Could you take a look at some of the suggestions I left?

PLeVasseur · 2025-03-05T14:27:39Z

.github/actions/setup-rust-cross/action.yaml

+#  SPDX-License-Identifier: Apache-2.0
+# *******************************************************************************/
+
+name: "Setup Rust and Cross"


Do we need to use cross-rs? I have the stable Rust compiler installed and a

rustc --print target-list

already shows aarch64-unknown-linux-musl.

It's also listed on the targets page here as supported.

Hmm cross is just a low-setup way to cross compile. If we dont use cross then we have to provide the build toolchain manually for each target like its done here: https://github.com/eclipse-sdv-blueprints/fleet-management/blob/main/components/Dockerfile.fms-consumer

PLeVasseur · 2025-03-05T14:30:19Z

.github/workflows/containerize-and-push.yaml

+  push:
+    branches:
+      - main
+      - add_build_and_containerize_pipeline


This should be removed prior to merge, right? Just for testing?

yes! its actually a bit of a problem though. We cant run the pipeline before merging it, so Im not really sure what the process of testing it before merge would be... By having it trigger on PR I could at least test that the github action itself runs through (although only on our forked repo), but not sure how I can test if the access to the main repos ghcr for example works as intended without merging to main.

mayyybe since you have more access rights here, you can manually trigger the workflow from the actions section? I did allow the "workflow_dispatch" trigger after all, but there seems to be some strange behavior in github where it might still not like being run manually...

PLeVasseur · 2025-03-05T14:32:09Z

.github/workflows/containerize-and-push.yaml

+
+name: Containerize uStreamer and Push to Container Registry
+
+on:


Should this be done on each PR? Or only when we have a release? Would like your feedback.

Ideally I guess only for proper releases? Until we have a real release strategy though maybe I can just set it to "workflow_dispatch" provided that actually works. Then we can manually release whenever we feel like it.

PLeVasseur · 2025-03-05T14:33:21Z

.github/workflows/containerize-and-push.yaml

@@ -0,0 +1,164 @@
+# ********************************************************************************


Most of this I'm taking on faith it's being done correctly, since I'm not familiar enough with Docker and containerizing. 😅

A basic question though -- you've tested the resulting Docker image on your end to make sure it works?

Anyone else that is more familiar with Docker I'd be super appreciative if you could take a look! cc @AnotherDaniel @sophokles73

Yeah I tried around with the image and container by itself and found no issues yet. Im also using it in our work around the s2s blueprint and so far it behaves exactly like a streamer in a container should.

PLeVasseur · 2025-03-05T14:35:22Z

.github/actions/setup-rust-cross/action.yaml

@@ -0,0 +1,28 @@
+# ********************************************************************************


I can't leave a comment on the .DS_Store file, so I'll leave it here.

Please add this and other "macOS-isms" as new entries in the .gitignore :)

untracked the macos nonsense. thanks for noticing ^^

PLeVasseur · 2025-03-05T14:37:01Z

Cross.toml

+#  SPDX-License-Identifier: Apache-2.0
+# *******************************************************************************/
+
+[build.env]


Is this related to why you chose to use cross-rs? If we don't use cross-rs is this needed?

Yes this is the config for cross build. Instead of having to provide the whole toolchain directly we can use the default one for each target and just append it via the config here.

PLeVasseur · 2025-03-05T14:37:35Z

Dockerfile.zenoh-mqtt-amd

@@ -0,0 +1,23 @@
+# ********************************************************************************
+#  Copyright (c) 2024 Contributors to the Eclipse Foundation


Please reflect 2025 as the copyright year for any newly added files throughout as you've got a number of new files.

done! hope I didnt miss any

PLeVasseur · 2025-03-05T14:42:54Z

docker/docker-compose.yaml

@@ -0,0 +1,10 @@
+services:
+  ustreamer:
+    image: benediktillich/streamertest:latest


Sorry again for my Docker-ignorance. Claude is telling me that:

benediktillich is the username or organization on Docker Hub

Should this be changed to something specific to Eclipse uProtocol? Do we need to make an account?

Ahh yeah this was from testing. I removed this folder altogether. The example docker directory is now within the example-streamer-implementations folder and it should have the correct image names in the compose.yaml.

PLeVasseur · 2025-03-05T14:45:11Z

example-streamer-uses/src/bin/zenoh_client.rs

@@ -67,7 +67,7 @@ async fn main() -> Result<(), UStatus> {
    if !args.endpoint.is_empty() {
        // Specify the address to listen on using IPv4
        let ipv4_endpoint =
-            EndPoint::from_str(args.endpoint.as_str()).expect("Unable to set endpoint");
+            EndPoint::from_str("tcp/127.0.0.1:7447").expect("Unable to set endpoint");


We should respect if the user is choosing a different endpoint here.

Suggested change

EndPoint::from_str("tcp/127.0.0.1:7447").expect("Unable to set endpoint");

EndPoint::from_str(args.endpoint.as_str()).expect("Unable to set endpoint");

I see you've already set it as default, so if needed you can update the README.md to ensure that users are aware they should not override it if using the Docker-ized setup.

Ahh yeah that must have also been from testing. Thanks for catching all these...

the implementations folder

ValMobBIllich added 10 commits January 7, 2025 10:28

added the github action for building, docker file for streamer contai…

914bde2

…ner, reference docker compose with config mounting

removed unnecessary comments

ecab4db

building for amd instead of arm

d0bd754

added login, build and push steps

5f74dbf

added environment to action yaml

6665252

removed a slash from the bin directory

ab8c797

added debug step

14ac580

changed the debug step, explicitly added context to build and push step

9cc9ead

fixed the repo destination

5e5165e

changed the container registry to github and added metadata step

16743f7

ValMobBIllich had a problem deploying to production January 13, 2025 11:57 — with GitHub Actions Failure

ValMobBIllich added 3 commits January 13, 2025 13:00

removed debug step and environment

4c8211f

trying out build for musl

3ea223f

reverted back to -gnu

8da0461

PLeVasseur self-requested a review January 14, 2025 14:43

ValMobBIllich added 3 commits January 27, 2025 12:01

added pipeline for arm, changed to musl, fixed openssl issue

6f1ecc2

added on PR to action

1b0aa7b

et environment to production

529c4ab

ValMobBIllich had a problem deploying to production January 27, 2025 12:11 — with GitHub Actions Failure

changed back to ghcr

e521b1f

ValMobBIllich had a problem deploying to production January 27, 2025 12:21 — with GitHub Actions Failure

ValMobBIllich added 6 commits January 27, 2025 13:47

condensed builds for both archs into one pipeline, removed obsolete f…

a4ba005

…iles

parallelized builds for amd and arm, turned rust and cross install into

ca46b41

a composite action

moved composite action to the right spot

0df01ef

added comments to cross.toml, fixed getting the tags and labels in the

43c1ab6

push steps of the build pipeline

added create-manifest step to build pipeline for both arches

55c1aa8

reverted some debugging changes

4f20d65

trying lowercase registry

4250b28

ValMobBIllich added 7 commits January 29, 2025 14:51

fixed lowercase issue, added step to enfore lowercase

cdb0001

took the lowercase enforce out of the composite action

7da57b4

changed the create manifest job: now using custom bash script instead of

95a92c7

build push action

switched to using buildx for the manifest

bd4d6ab

added documentation for the streamer container and comments to the

9ebb3c3

reference docker-compose

changed build action to only run on push to main

cace560

added action on feature branch push for now

6896280

PLeVasseur added 7 commits March 4, 2025 03:20

Add shell script to set environment variables, updated CI workflows t…

ee7e944

…o use it.

Update up-transport-vsomeip-rust and up-streamer-rust to use up-rust …

534f76b

…0.3.0

Update up-transport-mqtt5 to 0.2.0

3f4b1fd

Add necessary piping through of environment variables for paths to C+…

4e1e6bc

…+ stdlib

Update to published up-transport-vsomeip 0.4.1

3c6cd5d

Add copyright notice to envsetup.sh

b2ef2c3

Update README.md on how to build using either config.toml or envsetup.sh

298ac22

PLeVasseur mentioned this pull request Mar 3, 2025

Update with up-transport-vsomeip-rust build changes #60

Merged

Merge branch 'feature/update-to-new-vsomeip-build-style' into add_bui…

a5b4316

…ld_and_containerize_pipeline

PLeVasseur requested changes Mar 5, 2025

View reviewed changes

ValMobBIllich added 6 commits March 5, 2025 15:55

removed macos relic from git

5b27a89

fixed the year in the licenses of new files

b9f1fb0

removed the example docker folder and fixed the example docker folder in

f948480

the implementations folder

made the zenoh endpoint variable again

c3ac216

Merge branch 'main' into add_build_and_containerize_pipeline

49219c0

added the unstable features to zenoh dependency

ce40614

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add build and containerize pipeline #61

Add build and containerize pipeline #61

ValMobBIllich commented Jan 13, 2025 •

edited

Loading

PLeVasseur commented Jan 27, 2025

ValMobBIllich commented Jan 28, 2025

PLeVasseur commented Feb 13, 2025

PLeVasseur commented Mar 5, 2025

PLeVasseur left a comment

PLeVasseur Mar 5, 2025

ValMobBIllich Mar 5, 2025

PLeVasseur Mar 5, 2025

ValMobBIllich Mar 5, 2025

PLeVasseur Mar 5, 2025

ValMobBIllich Mar 5, 2025

PLeVasseur Mar 5, 2025

ValMobBIllich Mar 5, 2025

PLeVasseur Mar 5, 2025

ValMobBIllich Mar 5, 2025

PLeVasseur Mar 5, 2025

ValMobBIllich Mar 5, 2025

PLeVasseur Mar 5, 2025

ValMobBIllich Mar 5, 2025

PLeVasseur Mar 5, 2025

ValMobBIllich Mar 5, 2025

PLeVasseur Mar 5, 2025

ValMobBIllich Mar 5, 2025


		name: Containerize uStreamer and Push to Container Registry

		on:

		@@ -0,0 +1,164 @@
		# ********************************************************************************

		@@ -0,0 +1,28 @@
		# ********************************************************************************

		@@ -0,0 +1,23 @@
		# ********************************************************************************
		# Copyright (c) 2024 Contributors to the Eclipse Foundation

	EndPoint::from_str("tcp/127.0.0.1:7447").expect("Unable to set endpoint");
	EndPoint::from_str(args.endpoint.as_str()).expect("Unable to set endpoint");

Add build and containerize pipeline #61

Are you sure you want to change the base?

Add build and containerize pipeline #61

Conversation

ValMobBIllich commented Jan 13, 2025 • edited Loading

PLeVasseur commented Jan 27, 2025

ValMobBIllich commented Jan 28, 2025

PLeVasseur commented Feb 13, 2025

PLeVasseur commented Mar 5, 2025

PLeVasseur left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ValMobBIllich commented Jan 13, 2025 •

edited

Loading