- Moving to better .env for docker-compose.yml

- houskeeping for registrated gitlab-runner

Author: Springjunky

.env.template

@@ -0,0 +1,2 @@
+# This is the Environment for the docker-compos-file:
+# autogenerated ...do NOT edit

.gitignore

@@ -1,4 +1,4 @@
-docker-compose.yml
 docker-compose.yml-*
 *.jpi
 *.war
+.env

README.md

@@ -2,9 +2,11 @@
 
 ##### Tired of endless installation and configuration .... ?!
 
-My personal solution is a local Build Environment with Jenkins, Gitlab, (Sonar) and Nexus; ready in a few minutes.
+My personal solution is a local Build Environment with Jenkins, Gitlab / Gitlabrunner, (Sonar) and Nexus; ready in a few minutes.
 Your own lokal, personal, continous build enviroment (maybe in future releases I just call it lpcbe).
 
+# This is NOT for any cluster (Swarm / Kubernetes)
+
 ### System requirements
 * At least 8GB Memory with 3GB Swap and 10GB Disk-Space
 * docker version >= 17.06.0
@@ -15,7 +17,7 @@ Bring up your own build environment ... just do a
 ```
    git clone https://github.com/Springjunky/docker-local-build-environment.git
    cd docker-local-build-environment
-   sudo ./prepareCompose.sh
+   sudo ./setupEnvironment.sh
    docker-compose up --build -d
    docker-compose logs
 ```
@@ -39,7 +41,7 @@ Now you are ready to go with a little CI/CD Environment:
 ... not really, its all http .. don't worry about it! It's only local communication
 
 ##### security paranoia
-All the exposed ports are reachable from outer world because docker creates and deletes dynamically FORWARD rules with default policy ACCEPT on startup / shutdown containers wich have exported ports.
+All the exposed ports are reachable from outer world because docker creates and deletes dynamically iptables FORWARD rules with default policy ACCEPT on startup / shutdown containers wich have exported ports.
 
 To deny acccess from outer world the DOCKER-USER Chain (since docker 17.06) ist the medium of choice for your own rules (this is the first target in the FORWARD-Chain and never touched by docker).
 
@@ -97,23 +99,19 @@ echo "See logs with prefix DCKR-EXT-ACCESS-DENY:"
 * MAVEN_HOME is /opt/maven
 * JAVA_HOME is /usr/lib/jvm/java-8-openjdk-amd64
 * Blue Ocean is installed and works perfect with a GitHUB Account, not GitLab ... sorry, this is Jenkins.
-  You need to be logged in to use Blue Ocean
+  You need to be logged as a jenkins-user in to use Blue Ocean
 
 ###  Giltab
 
-* the docker-registry is at port 5555 (and secured with an openssl certificate ..thats part of
-  prepareCompose.sh), just create a project in gitlab and click at the  registry tab to show
+* the docker-registry from GitLab is at port 5555 (and secured with an openssl certificate ..thats part of
+  prepareEnvironment.sh), just create a project in gitlab and click at the  registry tab to show
   how to login to the project registry and how to tag your images
 * ssh cloning and pushing is at port 2222
 
 #### gitlab-runner
-The runner is a gitlab-multirunner image with a docker-runner (concurrent=1) , based on [gitlab/gitlab-runner][2]  The docker-compose section has an environment called
-REGISTER_MODE, it can set to KEEP or REFRESH
-* KEEP register at one time a runner and keep it during startups
-* REFRESH at every startup remove all old runners and register one new runner (the pipeline-history ist lost.)
-
-It takes a long time until gitlab is ready to accept a runner registration, if it fails, increase the REGISTER_TRYS
+The runner is a gitlab-multirunner image with a docker-runner (concurrent=1) , based on [gitlab/gitlab-runner][2]  at every startup any runner is removed and only ONE new runner ist registrated to avoid multiple runners  (the pipeline-history maybe lost.) docker-in-docker works :-)
 
+It takes a long time until gitlab is ready to accept a runner registration, if it fails, increase the REGISTER_TRYS in docker-compse.yml
 
 
 #### Jenkins and Gitlab

docker-compose.yml.template renamed to docker-compose.yml

@@ -1,9 +1,9 @@
 #####################################
 # Don't touch this file is generated
-# expecilly for: HOSTNAME
+# expecilly for: ${DC_HOSTNAME}
 ###################################
 #
-# Consider to Backup  BASE_DATA_DIR
+# Consider to Backup  ${DC_BASE_DATA_DIR}
 #
 version: "3"
 
@@ -18,7 +18,7 @@ services:
      ports:
        - "80:80"       #http://
        # SSH Bypassing into gitlab, if you want to change this edit nginx.conf also
-       - "2222:2222"   #ssh port of gitlab (ssh://git@myhostname:2222/scott/foo.git)
+       - "2222:2222"   #ssh port of gitlab (ssh://git@myHOST:2222/scott/foo.git)
        - "5555:5555"   #Gitlab Docker Registry do NOT use 5000, this is an internal PORT of the gitlab-ce Image
      networks:
        - devstacknetwork
@@ -27,64 +27,65 @@ services:
   jenkins-fat:
     build: jenkins-fat
     extra_hosts:
-      - HOSTNAME:HOSTIP
+      - ${DC_HOSTNAME}:${DC_HOSTIP}
     networks:
       - devstacknetwork
     volumes:
-      - BASE_DATA_DIR/jenkins:/var/jenkins_home
+      - ${DC_BASE_DATA_DIR}/jenkins:/var/jenkins_home
       - /var/run/docker.sock:/var/run/docker.sock
 
 # ------------------------------------------------------------------------------
   gitlab:
     image: 'gitlab/gitlab-ce:latest'
     extra_hosts:
-      - HOSTNAME:HOSTIP
+      - ${DC_HOSTNAME}:${DC_HOSTIP}
     networks:
       - devstacknetwork
     environment:
       GITLAB_OMNIBUS_CONFIG: |
-        external_url 'http://HOSTNAME/gitlab'
+        external_url 'http://${DC_HOSTNAME}/gitlab'
         gitlab_rails['initial_root_password'] = "gitlab4me"
         gitlab_rails['initial_shared_runners_registration_token'] = "s3cretToken4Runner"
         gitlab_rails['gitlab_shell_ssh_port'] = 2222
         # docker-registry config
-        registry_external_url 'https://HOSTNAME:5555'
+        registry_external_url 'https://${DC_HOSTNAME}:5555'
         registry_nginx['listen_port'] = 5555
         registry_nginx['listen_https'] = true
         # SSL config just for the docker-registry need
-        nginx['ssl_certificate'] = "/etc/gitlab/ssl/HOSTNAME.crt"
-        nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/HOSTNAME.key"
+        nginx['ssl_certificate'] = "/etc/gitlab/ssl/${DC_HOSTNAME}.crt"
+        nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/${DC_HOSTNAME}.key"
         registry_nginx['proxy_set_headers'] = {
         "X-Forwarded-Proto" => "https",
         "X-Forwarded-Ssl" => "on"
         }
     volumes:
-      - BASE_DATA_DIR/gitlab/config:/etc/gitlab
-      - BASE_DATA_DIR/gitlab/logs:/var/log/gitlab
-      - BASE_DATA_DIR/gitlab/data:/var/opt/gitlab
+      - ${DC_BASE_DATA_DIR}/gitlab/config:/etc/gitlab
+      - ${DC_BASE_DATA_DIR}/gitlab/logs:/var/log/gitlab
+      - ${DC_BASE_DATA_DIR}/gitlab/data:/var/opt/gitlab
 # ------------------------------------------------------------------------------
   nexus:
     image: sonatype/nexus3
     extra_hosts:
-      - HOSTNAME:HOSTIP
+      - ${DC_HOSTNAME}:${DC_HOSTIP}
     networks:
       - devstacknetwork
     environment:
       - NEXUS_CONTEXT=nexus
     volumes:
-      - BASE_DATA_DIR/nexus:/nexus-data
+      - ${DC_BASE_DATA_DIR}/nexus:/nexus-data
 
 # ------------------------------------------------------------------------------
   gitlabrunner:
     build: gitlabrunner
     network_mode: "host"
     extra_hosts:
-      - HOSTNAME:HOSTIP
+      - ${DC_HOSTNAME}:${DC_HOSTIP}
     environment:
-      - GITLAB_URL=http://HOSTNAME/gitlab
+      - GITLAB_URL=http://${DC_HOSTNAME}/gitlab
+      - HOSTNAME=${DC_HOSTNAME}
+      - HOSTIP=${DC_HOSTIP}
       - REGISTER_TOKEN=s3cretToken4Runner
-      - REGISTER_MODE=FRESH #KEEP register a new runner an keep it. FRESH unregister all runners and always install a new (pipeline-history lost!)
       - REGISTER_TRYS=60 # every 10 seconds a try to register the runner..gitlab takes a long time to startup
     volumes:
-      - BASE_DATA_DIR/gitlab-runner/config:/etc/gitlab-runner
+      - ${DC_BASE_DATA_DIR}/gitlab-runner/config:/etc/gitlab-runner
       - /var/run/docker.sock:/var/run/docker.sock

gitlabrunner/entrypointAutoregister

@@ -20,26 +20,21 @@ fi
 
 
 ###########################################################################################################
-echo "Try to autoregister runner at ${GITLAB_URL} with ${REGISTER_TOKEN} with $REGISTER_TRYS trys"
+echo "Try to autoregister runner at ${GITLAB_URL} with host ${HOSTNAME}:${HOSTIP} and token ${REGISTER_TOKEN} for $REGISTER_TRYS trys"
 
 grep docker-runner /etc/gitlab-runner/config.toml 2>/dev/null 1>/dev/null
 RUNNER_IS_PRESENT=$?
 
-if test $REGISTER_MODE = "KEEP" -a $RUNNER_IS_PRESENT -eq 0 ; then
-   echo "Keeping old runner"
-fi
-
 TRYS=$REGISTER_TRYS
 REGISTERED=1
 
-if test $REGISTER_MODE = "FRESH" -o  $RUNNER_IS_PRESENT -ne 0 ; then
     while test $TRYS -gt 0 -a $REGISTERED -gt 0 ; do
       TRYS=$(expr $TRYS - 1)
       gitlab-runner register -n --url=${GITLAB_URL} --registration-token=${REGISTER_TOKEN} \
       --executor=docker --name=docker-multi-runner \
       --run-untagged=true --locked=false --docker-image=docker:latest \
       --docker-privileged \
-      --docker-extra-hosts="HOSTNAME:HOSTIP"
+      --docker-extra-hosts="${HOSTNAME}:${HOSTIP}"
       REGISTERED=$?
       if [ $REGISTERED -ne 0 ] ; then
         echo "${GITLAB_URL} not ready $TRYS trys left, next try in 10 second for ${GITLAB_URL}"
@@ -56,10 +51,9 @@ if test $REGISTER_MODE = "FRESH" -o  $RUNNER_IS_PRESENT -ne 0 ; then
         --executor=docker --name=docker-multi-runner \
         --run-untagged=true --locked=false --docker-image=docker:latest \
         --docker-privileged \
-        --docker-extra-hosts="HOSTNAME:HOSTIP"
+        --docker-extra-hosts="${HOSTNAME}:${HOSTIP}"
       fi
    done
-fi
 
 # launch gitlab-ci-multi-runner passing all arguments
 exec gitlab-ci-multi-runner "$@"

prepareCompose.sh renamed to setupEnvironment.sh

@@ -5,7 +5,7 @@ if [ $(id -u) -gt 0 ] ;then
     exit 1
 fi
 
-echo "Prepare compose file and directorys"
+echo "Prepare .env for compose file and directorys"
 
 USER_DATA_DIR=$HOME/devstack-data
 HOSTNAME=$(hostname)
@@ -75,10 +75,10 @@ else
   exit 1
 fi
 
-if [ -f docker-compose.yml ]; then
-  FILE_NAME=docker-compose.yml-$(date +"%F-%H-%M-%S-%N")
-  cp docker-compose.yml $FILE_NAME
-  echo "previous docker-compose.yml saved as $FILE_NAME"
+if [ -f .env ]; then
+  FILE_NAME=.env-$(date +"%F-%H-%M-%S-%N")
+  cp .env $FILE_NAME
+  echo "previous .env saved as $FILE_NAME"
 fi
 # Copy preconfigs to host-volumes
 # sonar.properties
@@ -92,19 +92,27 @@ fi
 #Copy predefined Jobs and Configs
 cp -r preconfig/jenkins/* $USER_DATA_DIR/jenkins/
 
-# Set the right volume-names, hostname and host_ip in docker-compose.yml
-sed s#BASE_DATA_DIR#${USER_DATA_DIR}#g docker-compose.yml.template > docker-compose.yml
-sed -i s#HOSTIP#${HOSTIP}#g docker-compose.yml
-sed -i s#HOSTNAME#${HOSTNAME}#g docker-compose.yml
+# Set the right volume-names, hostname and host_ip in .env for docker-compose.yml
+echo "---------- generating .env file for docker-compose.yml "
+cat .env.template > .env
+echo "DC_HOSTNAME=${HOSTNAME}" >> .env
+echo "DC_HOSTIP=${HOSTIP}" >> .env
+echo "DC_BASE_DATA_DIR=${USER_DATA_DIR}" >> .env
+echo "---------- genarated file  ---------------------------- "
+cat .env
+echo "-------------------------------------------------------------------------------------------"
+
+#sed s#BASE_DATA_DIR#${USER_DATA_DIR}#g docker-compose.yml.template > docker-compose.yml
+#sed -i s#HOSTIP#${HOSTIP}#g docker-compose.yml
+#sed -i s#HOSTNAME#${HOSTNAME}#g docker-compose.yml
 
-# Gitlabrunner needs extra_hosts to clone stuff via hostname
-sed -i s#HOSTNAME#${HOSTNAME}#g gitlabrunner/entrypointAutoregister
-sed -i s#HOSTIP#${HOSTIP}#g gitlabrunner/entrypointAutoregister
+# Gitlabrunner needs extra_hosts to clone stuff via (outside) hostname
+# sed -i s#HOSTNAME#${HOSTNAME}#g gitlabrunner/entrypointAutoregister
+# sed -i s#HOSTIP#${HOSTIP}#g gitlabrunner/entrypointAutoregister
 
-chmod a+rw docker-compose.yml
 echo "-------------------------------------------------------------------------------------------"
 echo "-------------------------------------------------------------------------------------------"
-echo "docker-compose.yml created"
+echo "Evironment for docker-compose.yml created"
 echo "run "
 echo "docker-compose up --build -d "
 echo "docker-compose logs -f"