Skip to content

Commit 992bd1e

Browse files
adadoucheAbdelhalim Dadouche
adadouche
and
Abdelhalim Dadouche
authored
adjusting to aws4embeddedlinux-ci latest version (#18)
* moving the code to cdk folder * - switching from npm to yarn - updating libraries to new cdk version and requirements * adjusting the example to the new code from adadouche/aws4embeddedlinux-ci#dev-adadouche * doc adjustments * updated documentation * adjusting to aws4embeddedlinux-ci latest version * adjusting to aws4embeddedlinux-ci latest version --------- Co-authored-by: Abdelhalim Dadouche <[email protected]>
1 parent e525657 commit 992bd1e

29 files changed

+8247
-5929
lines changed

Diff for: ‎.editorconfig

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# EditorConfig is awesome: http://EditorConfig.org
2+
3+
# top-most EditorConfig file
4+
root = true
5+
6+
# Unix-style newlines with a newline ending every file
7+
[*]
8+
end_of_line = lf
9+
insert_final_newline = true
10+
11+
[*.{js,json,yml}]
12+
charset = utf-8
13+
indent_style = space
14+
indent_size = 2

Diff for: ‎.gitattributes

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
* text=auto eol=lf
2+
/.yarn/** linguist-vendored
3+
/.yarn/releases/* binary
4+
/.yarn/plugins/**/* binary
5+
/.pnp.* binary linguist-generated

Diff for: ‎.gitignore

+9-1
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,17 @@
11
node_modules
22
dist
33

4+
# yarn asset
5+
yarn-install.log
6+
.yarn
7+
!.yarn/releases/
8+
!.yarn/plugins/
9+
!.yarn/versions/
10+
!.yarn/sdks/
11+
.pnp.*
12+
413
# CDK asset staging directory
514
.cdk.staging
615
cdk.out
716

817
cdk.context.json
9-
/aws4embeddedlinux-ci-examples.iml

Diff for: ‎README.md

+127-114
Original file line numberDiff line numberDiff line change
@@ -1,169 +1,182 @@
11
# aws4embeddedlinux-ci-examples
22

3-
## Getting Started
43
This repository shows ways to use the [aws4embeddedlinux-ci](https://github.com/aws4embeddedlinux/aws4embeddedlinux-ci.git) library.
54

65
In order to use these examples, you must set up the [CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html), including
76
installing the CDK tool and bootstrapping the account you wish to deploy to. Additionally, you must have [Node](https://nodejs.org/en/) installed.
87

98
> [!NOTE]
10-
> This library is tested against Node Version 22. If these version are not available for your system, we recommend
11-
> using [NVM](https://github.com/nvm-sh/nvm) to install a compatible version.
9+
> This library is tested against Node Versions 22. If these version is not available for your system, we recommend using [NVM](https://github.com/nvm-sh/nvm) to install a compatible version.
1210
13-
### Clone and Setup NPM Project
14-
```bash
15-
git clone https://github.com/aws4embeddedlinux/aws4embeddedlinux-ci-examples.git
16-
cd aws4embeddedlinux-ci-examples
17-
npm install .
18-
npm run build
19-
```
20-
21-
Note that while the CDK projects often do not require that you invoke the build command separately, doing so will ensure various assets
22-
in the library are packaged correctly.
11+
---
2312

24-
### Deploying
13+
## Setup
2514

26-
To deploy _all_ the pipeline examples, you can use the CDK deploy command:
15+
### Setting environment variables
2716

2817
```bash
29-
cdk deploy --all
30-
```
18+
export AWS_PROFILE="default"
19+
export AWS_DEFAULT_REGION=$(aws configure get region --profile ${AWS_PROFILE})
20+
export AWS_DEFAULT_ACCOUNT=$(aws sts get-caller-identity --query Account --output text --profile ${AWS_PROFILE})
3121

32-
Alternatively, to deploy just a specific pipeline example, you can use the CDK deploy command:
22+
echo "PROFILE : $AWS_PROFILE"
23+
echo "ACCOUNT : $AWS_DEFAULT_ACCOUNT"
24+
echo "REGION : $AWS_DEFAULT_REGION"
25+
```
3326

34-
\<EXAMPLE\> can be one or more of: PokyPipeline, QemuEmbeddedLinuxPipeline, PokyAmiPipeline, KasPipeline, RenesasPipeline, NxpImxPipeline
27+
### Clone the project
3528

3629
```bash
37-
cdk deploy <EXAMPLE>
30+
git clone https://github.com/aws4embeddedlinux/aws4embeddedlinux-ci-examples.git
31+
cd aws4embeddedlinux-ci-examples
3832
```
3933

40-
The pipelines can be found in the `Developer Tools > Code Pipeline > Pipelines` Console page. The newly created
41-
pipeline `ubuntu_22_04BuildImagePipeline` should start automatically. If not, it will need to be run before other
42-
pipelines will work correctly. Once it is complete, the EmbeddedLinuxPipeline in the CodePipeline console page is ready to run.
34+
### Bootstrap CDK
4335

44-
### Removing Pipelines
45-
The `cdk destroy` command can be used to remove individual pipelines and their related resources. This can also be done in the CloudFormation Console Page.
46-
**Do not delete stacks while a CodePipeline is running, this can lead to unexpected failures!**
36+
> [!NOTE]
37+
> Only required once unless you upgrade your cdk version
4738
48-
To remove all the resources associated with this application:
4939
```bash
50-
cdk destroy --all
40+
cdk bootstrap aws://$AWS_DEFAULT_ACCOUNT/$AWS_DEFAULT_REGION
5141
```
5242

53-
## Examples
54-
Several example pipelines are provided. Each one demonstrates a different aspect of how to build a Yocto image with AWS.
55-
56-
### A Simple Poky Based Pipeline
57-
This example will build the `core-image-minimal` image from Poky using the repo tool to manage layers. CVE checking is also enabled in the buildspec file.
58-
59-
The recommended place to view this is from the `Developer Tools > Code Pipeline > Pipelines` page. The pipeline will start with `PokyPipeline-`
60-
followed by some unique identifier. From the pipeline page, you can find the CodeCommit source repository, the CodeBuild Project (with build logs),
61-
and the S3 bucket that the image is uploaded to, at the end.
62-
63-
Example stack name: PokyPipeline
64-
65-
Expected build time: 32min / rebuild (without any change, just use sstate cache): 8min
66-
67-
#### Using Kas
68-
The Kas example shows how to use a [Kas Config](https://github.com/aws4embeddedlinux/aws4embeddedlinux-ci/blob/main/source-repo/kas/kas.yml) to manage
69-
layers. This tool can help programatically manage layers and config with tighter Yocto integration than Git Submodules or the Repo tool.
70-
71-
See the AWS CodeBuild pipeline: KasPipeline-EmbeddedLinuxPipeline*
72-
73-
Example stack name: KasPipeline
74-
75-
Expected build time: 36min / rebuild (without any change, just use sstate cache): 11min
76-
77-
#### A slightly modified version building a qemu pipeline:
78-
This example builds a Qemu based image using [meta-aws-demos](https://github.com/aws4embeddedlinux/meta-aws-demos). The Qemu image can be run in
79-
the CodeBuild environment. Using SLIRP networking, [OEQA testing](https://docs.yoctoproject.org/singleindex.html#performing-automated-runtime-testing)
80-
such as ptest can be run in the pipeline.
81-
82-
See the AWS CodeBuild pipeline: QemuEmbeddedLinuxPipeline-EmbeddedLinuxPipeline*
43+
### Install packages and build the stack
8344

84-
Expected build time: 45min / rebuild (without any change, just use sstate cache): 14min
85-
86-
### A Poky Based EC2 AMI Pipeline
87-
Yocto can be used to create an EC2 AMI. This example builds an AMI based on Poky and meta-aws and exports it to your AMI registry using
88-
the [VM Import/Export Service](https://docs.aws.amazon.com/vm-import/latest/userguide/what-is-vmimport.html).
89-
90-
The pipeline name starts with `PokyAmiPipeline-` in the CodePipeline page.
45+
First move to the `cdk` folder:
46+
47+
```bash
48+
cd cdk
49+
```
9150

92-
Example stack name: PokyAmiPipeline
51+
Then you will need to install the CDK library including the `aws4embeddedlinux-ci` library either using `npm`:
9352

94-
Expected build time: 52min / rebuild (without any change, just use sstate cache): 17min
53+
```bash
54+
npm install
55+
npm run build
56+
```
9557

96-
### A NXP / IMX Pipeline
97-
This example will build an image for
98-
the [i.MX 6ULL EVK](https://www.nxp.com/design/development-boards/i-mx-evaluation-and-development-boards/evaluation-kit-for-the-i-mx-6ull-and-6ulz-applications-processor:MCIMX6ULL-EVK) board.
58+
of `yarn':
9959

100-
NXP requires users to accept and comply with a EULA in order to build and, for this reason, the buildspec will require modification before the build succeeds. See the [IMX Yocto Users Guide](https://www.nxp.com/docs/en/user-guide/IMX_YOCTO_PROJECT_USERS_GUIDE.pdf) for more detail.
60+
```bash
61+
yarn install
62+
yarn build
63+
```
10164

102-
The pipeline name starts with `NxpImxPipeline-` in the CodePipeline page.
65+
> If you are not familliar with Yarn, please refer to the [documentation](https://yarnpkg.com/getting-started).
10366
104-
Example stack name: NxpImxPipeline
67+
> [!NOTE]
68+
>
69+
> While the CDK projects often do not require that you invoke the build command separately, doing so will ensure various assets in the library are packaged correctly.
10570
106-
### Using pre-built, proprietary artifacts in a Pipeline
71+
### Deploying the base image pipeline stack
10772

108-
This example is based on this [work](https://elinux.org/R-Car/Boards/Yocto-Gen3/v5.9.0) to build an image for Renesas R-Car-H3 Starter Kit
109-
Premier board (unofficial name - H3ULCB) including the proprietary graphics and multimedia drivers from Renesas.
73+
First, you will need to deploy the *base ubuntu* image pipeline (`aws4el-ci-pipeline-base-image`).
11074

111-
Download the Multimedia and Graphics library and related Linux drivers from the following link (registration necessary):
112-
https://www.renesas.com/us/en/application/automotive/r-car-h3-m3-h2-m2-e2-documents-software
75+
```bash
76+
cdk deploy aws4el-ci-pipeline-base-image --require-approval never --progress bar
77+
```
11378

114-
#### Download two files:
79+
The created pipeline can be found in the AWS console under `Developer Tools > Pipeline - CodePipeline > Pipelines`.
11580

116-
- R-Car_Gen3_Series_Evaluation_Software_Package_for_Linux-20220121.zip
117-
- R-Car_Gen3_Series_Evaluation_Software_Package_of_Linux_Drivers-20220121.zip
81+
The newly created pipeline `aws4el-ci-pipeline-base-image` should start automatically. If not, you can start it manually.
11882

119-
Graphic drivers are required for Wayland. Multimedia drivers are optional.
83+
> _NOTE_:
84+
> The `aws4el-ci-pipeline-base-image` will need to be successfully completed before other pipelines can work correctly.
12085
121-
#### Steps to build the image
86+
**_Expected build times: 5 minutes_**
12287

123-
1. Create a folder named `proprietary` in the root of the source repo, and put those two downloaded files into this folder.
124-
1. Deploy the build pipeline and uncomment the `#TODO` in the build.sh file.
125-
1. A build should automatically start. Once it succeeds you will get an image containing the proprietary graphics and multimedia drivers.
88+
You can check that the pipeline completed sucessfully when the following command returns an ***imageIds** entry :
12689

127-
See the AWS CodeBuild pipeline: RenesasPipeline-EmbeddedLinuxPipeline*
90+
```bash
91+
aws ecr list-images \
92+
--repository-name "aws4el-ci-$AWS_DEFAULT_ACCOUNT-$AWS_DEFAULT_REGION-repo" \
93+
--query "imageIds[?imageTag=='aws4el-ci-pipeline-base-image']"
94+
```
12895

129-
Example stack name: RenesasPipeline
96+
Once the pipeline completes and the image is available in the ECR repository, the other `EmbeddedLinuxPipeline` stacks can be created and executed.
13097

131-
Expected build time: 27min / rebuild (without any change, just use sstate cache): 9min
98+
### Deploying the project pipeline stack
13299

133-
### A AWS CodeBuild Project
134-
This will create an Embedded Linux ready AWS CodeBuild project that can be used to connect to a source, e.g. [GitHub Actions](https://docs.aws.amazon.com/codebuild/latest/userguide/action-runner.html). This is not using any CodePipeline.
100+
To deploy a specific pipeline type, you can use the following CDK deploy command:
135101

136-
And use the EFS to share downloads and sstate cache between the runners.
102+
```bash
103+
cdk deploy <pipeline-id> --require-approval
104+
```
137105

138-
The connection to the CodeBuild source must be performed manually.
106+
where **\<pipeline-id\>** can be one or more of the following:
107+
108+
| Name | Pipeline stack id |
109+
|---------------------|-------------------------------|
110+
| Poky | `aws4el-ci-pipeline-poky` |
111+
| Poky Ami | `aws4el-ci-pipeline-poky-ami` |
112+
| Qemu Embedded Linux | `aws4el-ci-pipeline-qemu` |
113+
| Kas | `aws4el-ci-pipeline-kas` |
114+
| Renesas | `aws4el-ci-pipeline-renesas` |
115+
| NXP-IMX | `aws4el-ci-pipeline-nxp-imx` |
116+
| Custom | `aws4el-ci-pipeline-custom` |
117+
118+
Again, the created pipeline can be found in the AWS console under `Developer Tools > Pipeline - CodePipeline > Pipelines`.
119+
120+
> [!**NXP-IMX**]
121+
>
122+
> The deployed pipeline for **NXP-IMX** will not complete as you should first accept the EULA and update the `build.buildspec.yml` file accordingly. See the [IMX Yocto Users Guide](https://www.nxp.com/docs/en/user-guide/IMX_YOCTO_PROJECT_USERS_GUIDE.pdf) for more detail.
123+
>
124+
> The source files are available in a S3 bucket that you can get with the following command:
125+
>
126+
> ```sh
127+
> aws cloudformation describe-stacks --stack-name aws4el-ci-pipeline-nxp-imx --output text --query "Stacks[0].Outputs[?OutputKey=='SourceURI'].OutputValue"
128+
> ```
129+
>
130+
> Once you have adjusted the content, you can update and upload the zip back to Amazon S3, and the pipeline will restart.
131+
>
132+
133+
> [!**Renesas**]
134+
>
135+
> The deployed pipeline for **Renesas** will complete. However, it won't include the Multimedia and Graphics library and related Linux drivers. See the [Renesas](https://github.com/adadouche/aws4embeddedlinux-ci/blob/dev-adadouche/README.md#renesas) section for more detail.
136+
> The source files are available in a S3 bucket that you can get with the following command:
137+
>
138+
> ```sh
139+
> aws cloudformation describe-stacks --stack-name aws4el-ci-pipeline-renesas --output text --query "Stacks[0].Outputs[?OutputKey=='SourceURI'].OutputValue"
140+
> ```
141+
>
142+
> Once you have adjusted the content, you can update and upload the zip back to Amazon S3, and the pipeline will restart.
143+
>
144+
145+
> [!**Custom Pipeline**]
146+
>
147+
> When using the **Custom** pipeline, you will need to provide your own `build.buildspec.yml` file.
148+
>
149+
> To do so, you will provide a path in one of the `EmbeddedLinuxCodePipelineStack` props (`sourceCustomPath` property).
150+
> This path repsent the folder where the `build.buildspec.yml` file is located.
151+
> Make sure to use a full path instead of a relative path to avoid any issues.
152+
>
153+
> If the `build.buildspec.yml` file is not present in the provided folder path, the stack will fail to deploy.
154+
>
155+
> You can also provide any additional files need to execute your build in the same folder.
156+
>
157+
158+
To deploy _all_ the example pipelines, you can use the CDK deploy command:
139159
140-
Also you can clone the CodeBuild project and share the efs between the CodeBuild projects.
160+
```bash
161+
cdk deploy aws4el-ci-pipelines --require-approval never --concurrency 3
162+
```
141163
142-
See the AWS CodeBuild pipeline: EmbeddedLinuxCodebuildProje-*
164+
> [!*NOTE*]
165+
>
166+
> `aws4el-ci-pipelines`is an empty stack that depends on the other stacks, so that if you deploy it, it will deploy the others.
167+
>
143168
144-
To make a source connection to GitHub you need to:
145-
- Select a "Source provider"->"GitHub"
146-
- Select "Primary source webhook events" -> "Webhook - optional" -> "Rebuild every time a code change is pushed to this repository"
147-
- Add "Filter group 1" -> "WORKFLOW_JOB_QUEUED"
148-
- Modify the GitHub action `runs-on: ${{ vars.CODEBUILD_RUNNER_NAME }}-${{ github.run_id }}-${{ github.run_attempt }}`
149-
CODEBUILD_RUNNER_NAME should be `codebuild-EmbeddedLinuxCodebuildProjeNAME` with prefix `codebuild-`. See example [here](https://github.com/aws4embeddedlinux/meta-aws-demos/blob/master/.github/workflows/build-gg.yml).
169+
### Cleanup
150170
151-
Example stack name: EmbeddedLinuxCodeBuildProject
171+
The `cdk destroy` command can be used to remove individual pipelines and their related resources. This can also be done in the CloudFormation Console Page.
152172
153-
## Useful NPM and CDK commands
173+
> **Do not delete stacks while a CodePipeline is running, this can lead to unexpected failures!**
154174
155-
- `npm run build` compile typescript to js
156-
- `npm run watch` watch for changes and compile
157-
- `npm run test` perform the jest unit tests
158-
- `cdk deploy` deploy this stack to your default AWS account/region
159-
- `cdk diff` compare deployed stack with current state
160-
- `cdk synth` emits the synthesized CloudFormation template
175+
To remove all the resources associated with this application:
161176
162-
Project Specific:
163-
- `npm run format` runs prettier and eslint on the repository
164-
- `npm run zip-data` bundles the files for creating build host containers
165-
- `npm run check` checks for lint and format issues
166-
- `npm run docs` to generate documentation
177+
```bash
178+
cdk destroy --all --force
179+
```
167180
168181
## Security
169182

0 commit comments

Comments
 (0)