Skip to content

Commit 2dfd30d

Browse files
lgiommilgiommi
authored
S3 rebased (#16)
* Added feature of checking user group when creating a deployment * Fixed the case when using token with wlcg.groups claim instead groups * Create authorizeRequestedGroup function used in creating a deployment and listing the deployments for a specific group * Fixes after jacoco test * Added the force papameter in deployment deletion to skip IAM clients deletion * Fixed code after jacoco tests * Fixed tests and updated documentation * Fix comment in documentation * Try to fix jackson dependency * Fix indentation in pom.xml * Use macos for the VM where to run the github action * Use ubuntu-20.04 to run the github workflow * Enabled force parameter from IM * Fixed explanation of force parameter * Added PAAS DEP USER tag * Modified descritpion of force parameter * Not setting monitoring url is now allowed * Added .DS_Store in gitignore * Fix monitoring + ower string empty/null * remove default of monitoring.url in application.properties * Add new line at the end of application.properties * Modify the readme to add new information about features of the orchestrator's IAM client * Add functionality of s3 buckets management * Start working on creation/deletion buckets with reading secrets from vault * Added possibility to contact vault to read secrets * Fix code after jacoco tests * Refactoring of the code introducing S3Service class * Modify definition of constants * Fix IM test * Change projectKey and organization for sonarcloud * Introduced better handling of exceptions, and the S3ServiceException class * Fix exception handling * added and used getter for bucket_name, s3_url and s3_tosca_node_type * Move getters of tosca variables in ToscaService class * Enabled bucket versioning * Metedata should be written before eventual failures * Created a proper function to enable bucket versioning and put metadata writing after bucket creation and before enabling versioning * Orchestrator now writes aws_access_key and aws_secret_key in the template, and proper bucket_name * Implemented a regular expression to check the bucket name * Fix checking bucket name with Boolean.FALSE.equals * Update Jenkinsfile * Update Jenkinsfile * Remove github clone and checkout in Jenkinsfile * Added userGroup in the vault path where read secrets * Implemented deleting objects, obect versions and delete markers from buckets --------- Co-authored-by: Luca Giommi <[email protected]>
1 parent 190e2f3 commit 2dfd30d

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+950
-412
lines changed

‎.github/workflows/sonar.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,6 @@ jobs:
4747
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Needed to get PR information, if any
4848
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
4949
run: mvn -Dcheckstyle.skip -B org.sonarsource.scanner.maven:sonar-maven-plugin:sonar
50-
-Dsonar.projectKey=indigo-paas_orchestrator
51-
-Dsonar.organization=indigo-paas
50+
-Dsonar.projectKey=infn-datacloud_orchestrator
51+
-Dsonar.organization=infn-datacloud
5252
-Dsonar.host.url=https://sonarcloud.io

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
.DS_Store
12
*.pydevproject
23
.metadata
34
.gradle

‎Jenkinsfile

Lines changed: 45 additions & 203 deletions
Original file line numberDiff line numberDiff line change
@@ -1,235 +1,77 @@
1-
#!/usr/bin/groovy
2-
3-
@Library(['github.com/indigo-dc/[email protected]']) _
4-
51
pipeline {
2+
63
agent {
7-
label 'java'
4+
node { label 'jenkinsworker00' }
85
}
96

107
environment {
11-
dockerhub_repo = "indigodatacloud/orchestrator"
12-
dockerhub_image_id = ""
8+
DOCKER_HUB_CREDENTIALS = 'docker-hub-credentials'
9+
DOCKER_HUB_IMAGE_NAME = 'indigopaas/orchestrator'
10+
HARBOR_CREDENTIALS = 'harbor-paas-credentials'
11+
HARBOR_IMAGE_NAME = 'datacloud-middleware/orchestrator'
1312
}
1413

1514
stages {
16-
stage('Fetch code') {
17-
steps {
18-
checkout scm
19-
}
20-
}
2115

22-
stage('Style Analysis') {
23-
steps {
24-
MavenRun('checkstyle')
25-
}
26-
post {
27-
always {
28-
CheckstyleReport()
29-
dir("$WORKSPACE/target") {
30-
deleteDir()
31-
}
32-
}
33-
}
34-
}
35-
36-
stage('Unit testing coverage') {
37-
steps {
38-
MavenRun('cobertura')
39-
}
40-
post {
41-
success {
42-
CoberturaReport('**/target/site/cobertura/coverage.xml')
43-
JUnitReport()
44-
dir("$WORKSPACE/target") {
45-
deleteDir()
46-
}
47-
}
48-
}
49-
}
50-
51-
stage('Integration tests') {
52-
steps {
53-
MavenRun('integration-test')
54-
}
55-
post {
56-
success {
57-
JUnitReport()
58-
}
59-
}
60-
}
61-
62-
/*
63-
stage('Dependency check') {
16+
stage('checkout and compiling') {
6417
agent {
65-
label 'docker-build'
66-
}
67-
steps {
68-
checkout scm
69-
OWASPDependencyCheckRun("$WORKSPACE/orchestrator/src", project="Orchestrator")
70-
}
71-
post {
72-
always {
73-
OWASPDependencyCheckPublish()
74-
HTMLReport("$WORKSPACE/orchestrator/src",
75-
'dependency-check-report.html',
76-
'OWASP Dependency Report')
77-
deleteDir()
78-
}
79-
}
80-
}
81-
82-
83-
stage('Build Javadoc and REST documentation') {
84-
when {
85-
branch 'master'
86-
}
87-
steps {
88-
withCredentials([string(
89-
credentialsId: "indigo-github-token",
90-
variable: "GITHUB_TOKEN")]) {
91-
// git defaults
92-
sh 'git remote set-url origin "https://indigobot:${GITHUB_TOKEN}@github.com/indigo-dc/orchestrator"'
93-
sh 'git config user.name "indigobot"'
94-
sh 'git config user.email "<>"'
95-
// build docs
96-
sh 'git checkout gh-pages'
97-
sh 'git merge --ff -s recursive -X theirs --commit -m "Merge remote-tracking branch <origin/master>"'
98-
sh 'rm -rf "${WORKSPACE}/apidocs"'
99-
sh 'rm -rf "${WORKSPACE}/restdocs"'
100-
MavenRun('clean javadoc:javadoc package -P restdocs -Deditorconfig.skip=true')
101-
sh "mv ${WORKSPACE}/target/site/apidocs ${WORKSPACE}/apidocs"
102-
sh 'git add -A'
103-
sh 'git commit -am "Update documentation"'
104-
// push to gh-pages
105-
sh 'git push origin HEAD:gh-pages'
18+
docker {
19+
label 'jenkinsworker00'
20+
image 'maven:3.5.4-ibmjava-8'
21+
args '--privileged'
22+
reuseNode true
10623
}
10724
}
108-
}
109-
*/
110-
stage('Metrics') {
111-
agent {
112-
label 'sloc'
113-
}
11425
steps {
115-
checkout scm
116-
SLOCRun()
117-
}
118-
post {
119-
success {
120-
SLOCPublish()
26+
configFileProvider([configFile(fileId: 'maven-nexus-settings.xml', variable: 'MAVEN_SETTINGS')]) {
27+
sh 'mvn -s $MAVEN_SETTINGS editorconfig:format'
28+
sh 'mvn -s $MAVEN_SETTINGS clean install'
12129
}
12230
}
12331
}
12432

125-
stage('DockerHub delivery') {
126-
when {
127-
anyOf {
128-
branch 'master'
129-
branch 'releases/*'
130-
tag 'v*'
131-
}
132-
}
133-
agent {
134-
label 'docker-build'
135-
}
33+
stage('Build and tag Docker Image') {
13634
steps {
137-
checkout scm
13835
script {
139-
PROJECT_VERSION="""${sh([
140-
returnStdout: true,
141-
script: 'mvn -q -Dexec.executable=echo -Dexec.args=\'${project.version}\' --non-recursive exec:exec']).trim()
142-
}"""
143-
MavenRun('-DskipTests=true package')
144-
dockerhub_image_id = DockerBuild(
145-
dockerhub_repo,
146-
tag: PROJECT_VERSION,
147-
build_dir: 'docker')
148-
}
149-
}
150-
post {
151-
success {
152-
DockerPush(dockerhub_image_id)
153-
}
154-
failure {
155-
DockerClean()
156-
}
157-
always {
158-
cleanWs()
159-
}
160-
}
161-
}
36+
def dockerImage = docker.build("orchestrator:${env.BRANCH_NAME}", "-f docker/Dockerfile docker/")
16237

163-
stage('DockerHub delivery (for pull requests)') {
164-
when {
165-
changeRequest()
166-
}
167-
agent {
168-
label 'docker-build'
169-
}
170-
steps {
171-
checkout scm
172-
script {
173-
MavenRun('-DskipTests=true package')
174-
dockerhub_image_id = DockerBuild(dockerhub_repo,
175-
tag: env.CHANGE_ID,
176-
build_dir: 'docker')
177-
}
178-
}
179-
post {
180-
success {
181-
DockerPush(dockerhub_image_id)
182-
}
183-
failure {
184-
DockerClean()
185-
}
186-
always {
187-
cleanWs()
38+
sh("docker tag orchestrator:${env.BRANCH_NAME} ${HARBOR_IMAGE_NAME}:${env.BRANCH_NAME}")
39+
sh("docker tag orchestrator:${env.BRANCH_NAME} ${DOCKER_HUB_IMAGE_NAME}:${env.BRANCH_NAME}")
18840
}
18941
}
19042
}
191-
192-
stage('Notifications') {
193-
when {
194-
tag 'v*'
195-
}
43+
stage('Push to Docker Hub and Harbor') {
19644
parallel {
197-
stage('Notify DEEP') {
45+
stage('Push to Docker Hub') {
19846
steps {
199-
JiraIssueNotification(
200-
'DEEP',
201-
'DPM',
202-
'10204',
203-
"[preview-testbed] New orchestrator version ${env.BRANCH_NAME} available",
204-
"Check new artifacts at:\n\t- Docker image: [${dockerhub_image_id}|https://hub.docker.com/r/${dockerhub_repo}/tags/]",
205-
['wp3', 'preview-testbed', "orchestrator-${env.BRANCH_NAME}"],
206-
'Task',
207-
'mariojmdavid',
208-
['wgcastell',
209-
'vkozlov',
210-
'dlugo',
211-
'keiichiito',
212-
'laralloret',
213-
'ignacioheredia']
214-
)
47+
script {
48+
// Retrieve the Docker image object from the previous stage
49+
def dockerhubImage = docker.image("${DOCKER_HUB_IMAGE_NAME}:${env.BRANCH_NAME}")
50+
51+
// Login to Docker Hub
52+
docker.withRegistry('https://index.docker.io/v1/', DOCKER_HUB_CREDENTIALS) {
53+
// Push the Docker image to Docker Hub
54+
dockerhubImage.push()
55+
}
56+
}
21557
}
21658
}
217-
stage('Notify XDC') {
59+
60+
stage('Push to Harbor') {
21861
steps {
219-
JiraIssueNotification(
220-
'XDC',
221-
'XDM',
222-
'10100',
223-
"[preview-testbed] New orchestrator version ${env.BRANCH_NAME} available",
224-
"Check new artifacts at:\n\t- Docker image: [${dockerhub_image_id}|https://hub.docker.com/r/${dockerhub_repo}/tags/]",
225-
['WP3', 't3.2', 'preview-testbed', "orchestrator-${env.BRANCH_NAME}"],
226-
'Task',
227-
'doinacristinaduma',
228-
['doinacristinaduma']
229-
)
62+
script {
63+
// Retrieve the Docker image object from the previous stage
64+
def harborImage = docker.image("${HARBOR_IMAGE_NAME}:${env.BRANCH_NAME}")
65+
66+
// Login to Harbor
67+
docker.withRegistry('https://harbor.cloud.infn.it', HARBOR_CREDENTIALS) {
68+
// Push the Docker image to Harbor
69+
harborImage.push()
70+
}
71+
}
23072
}
23173
}
23274
}
23375
}
234-
} // stages
235-
} // pipeline
76+
}
77+
}

‎asciidoc/index.adoc

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -239,6 +239,11 @@ include::{snippets}/reset-deployment/http-response.adoc[]
239239

240240
A `DELETE` request is used to delete the deployment from the id.
241241

242+
=== Request parameters
243+
244+
[cols=",a"]
245+
include::{snippets}/delete-deployment/request-parameters.adoc[]
246+
242247
==== Example request
243248

244249
include::{snippets}/delete-deployment/curl-request.adoc[]

‎gitbook/how_to_deploy.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,7 @@ By default the REST APIs are not authenticated; if you want to enable the IAM in
143143
* **Description**: The CLUES OAuth2 client secret
144144

145145
Please make reference to the [IAM guide](https://indigo-dc.gitbooks.io/iam/content) for further information on how to register the Orchestrator as protected resource server.
146+
To allow orchestrator to create IAM clients, the IAM client of the orchestrator should also have `client_credentials` as grant type, and `iam:admin.read` and `iam:admin.write` as scopes (starting with IAM v1.8.2).
146147

147148
:warning: Even if the authentication is optional and disabled by default, you are highly encouraged to enable it, otherwise you will not be able to create deployments neither on OpenStack nor on OpenNebula.
148149

‎pom.xml

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@
7575
<elasticsearch.version>1.7.5</elasticsearch.version>
7676
<assertj.version>3.8.0</assertj.version>
7777
<openid-connect.version>1.3.3</openid-connect.version>
78-
<im-java-api.version>0.4.15</im-java-api.version>
78+
<im-java-api.version>0.4.17</im-java-api.version>
7979
<monitoring-pillar-domain.version>1.8.1-FINAL</monitoring-pillar-domain.version>
8080
<flowable.version>6.5.0</flowable.version>
8181
<alien4cloud.version>2.1.0-DEEP-1.3.1</alien4cloud.version>
@@ -119,11 +119,23 @@
119119
<url>https://repository.indigo-datacloud.eu/repository/maven-public</url>
120120
</repository>
121121

122+
<repository>
123+
<id>cnaf-sd-nexus-repository</id>
124+
<name>CNAF SD Nexus repository</name>
125+
<url>https://repo.cloud.cnaf.infn.it/repository/maven-public/</url>
126+
</repository>
127+
122128
</repositories>
123129

124130

125131
<dependencies>
126132

133+
<dependency>
134+
<groupId>software.amazon.awssdk</groupId>
135+
<artifactId>s3</artifactId>
136+
<version>2.20.52</version>
137+
</dependency>
138+
127139
<dependency>
128140
<groupId>com.google.guava</groupId>
129141
<artifactId>guava</artifactId>
@@ -208,6 +220,7 @@
208220
<dependency>
209221
<groupId>com.fasterxml.jackson.dataformat</groupId>
210222
<artifactId>jackson-dataformat-xml</artifactId>
223+
<version>2.8.11</version>
211224
</dependency>
212225

213226
<!-- Ignite -->

‎src/main/java/it/reply/orchestrator/Application.java

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,6 @@
1818
package it.reply.orchestrator;
1919

2020
import java.util.TimeZone;
21-
2221
import org.springframework.boot.SpringApplication;
2322
import org.springframework.boot.autoconfigure.SpringBootApplication;
2423
import org.springframework.boot.autoconfigure.web.ErrorMvcAutoConfiguration;

‎src/main/java/it/reply/orchestrator/config/WebAppInitializer.java

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,6 @@
1919

2020
import it.reply.orchestrator.config.filters.CustomRequestLoggingFilter;
2121
import it.reply.orchestrator.config.properties.OrchestratorProperties;
22-
2322
import org.springframework.boot.context.properties.EnableConfigurationProperties;
2423
import org.springframework.boot.web.servlet.FilterRegistrationBean;
2524
import org.springframework.context.annotation.Bean;

‎src/main/java/it/reply/orchestrator/config/properties/MonitoringProperties.java

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -18,12 +18,9 @@
1818
package it.reply.orchestrator.config.properties;
1919

2020
import java.net.URI;
21-
2221
import javax.validation.constraints.NotNull;
23-
2422
import lombok.Data;
2523
import lombok.NoArgsConstructor;
26-
2724
import org.checkerframework.checker.nullness.qual.NonNull;
2825
import org.springframework.boot.context.properties.ConfigurationProperties;
2926
import org.springframework.validation.annotation.Validated;
@@ -34,8 +31,6 @@
3431
@NoArgsConstructor
3532
public class MonitoringProperties {
3633

37-
@NotNull
38-
@NonNull
3934
private URI url;
4035

4136
@NotNull

0 commit comments

Comments
 (0)