Skip to content

Commit fc5034b

Browse files
author
AlexF
committed
Initial commit
1 parent 5181a41 commit fc5034b

13 files changed

+332
-1
lines changed

Diff for: ‎README.md

+58-1
Original file line numberDiff line numberDiff line change
@@ -1 +1,58 @@
1-
# k8s_registry_redis
1+
# Docker registry with Redis cache based on Kubernetes
2+
3+
`image_retention.groovy` file - Jenkins pipeline.
4+
Setup proper value for DOMAIN variable in `image_retention.groovy`, i.e.:
5+
DOMAIN="your.docker-registry.com"
6+
7+
All k8s manifests are in subfolders.
8+
9+
## How to deploy
10+
11+
AWS ACM and Network Load Balancer are used.
12+
13+
Setup proper values for ACM and DOMAIN variables in deploy.sh
14+
15+
```
16+
# AWS ACM: arn:aws:acm:xxxxxxxxxxx
17+
export ACM="arn:aws:acm:xxxxxxxxxxxxx"
18+
# domain name for your docker registry without schema (http/https): your.docker-registry.com
19+
export DOMAIN="your.docker-registry.com"
20+
```
21+
22+
Run `bash deploy.sh`
23+
24+
## How to remove deployment
25+
26+
Run `bash undeploy.sh`
27+
28+
## Issues
29+
30+
Encountered issues related to using more than 1 replica and emptyDir using for Volumes (instead of PVC).
31+
32+
1) for Deployment with 2 replicas and emptyDir in Volumes - data consistency issue:
33+
34+
```
35+
- name: image-data
36+
emptyDir: {}
37+
```
38+
39+
During the docker push have retries and blob upload unknown:
40+
```
41+
with 2 replicas:
42+
f1b5933fe4b5: Pushing [==================================================>] 5.796MB
43+
blob upload unknown
44+
The push refers to repository [MYDOMAIN.com/alpine]
45+
f1b5933fe4b5: Retrying in 10 second
46+
```
47+
48+
2) for Deployment with 2 replicas and PVC, PVC accessMode should be ReadWriteMany (RWX) (ReadWriteOnce - RWO can be attached to one node only) - so, NFS, GlusterFS, Ceph, etc should be used:
49+
50+
```
51+
Events:
52+
Type Reason Age From Message
53+
---- ------ ---- ---- -------
54+
Normal Scheduled 27s default-scheduler Successfully assigned docker-registry/docker-registry-cron-1562070600-jpgr8 to ip-172-20-33-53.us-east-2.compute.internal
55+
Warning FailedAttachVolume 27s attachdetach-controller Multi-Attach error for volume "pvc-70a143c3-82a6-4c82-a45c-94fe607942c7" Volume is already used by pod(s) docker-registry-579469cd77-hv4zg
56+
57+
Warning FailedMount 95s kubelet, ip-172-20-33-53.us-east-2.compute.internal Unable to mount volumes for pod "docker-registry-cron-1562070000-l7dwz_docker-registry(2c8d33e3-9220-411a-8a6c-5afd3f7d3080)": timeout expired waiting for volumes to attach or mount for pod "docker-registry"/"docker-registry-cron-1562070000-l7dwz". list of unmounted volumes=[image-data]. list of unattached volumes=[image-data default-token-dxktg]
58+
```

Diff for: ‎common/namespace.yaml

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
name: docker-registry

Diff for: ‎deploy.sh

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/usr/bin/env bash
2+
3+
# AWS ACM: arn:aws:acm:xxxxxxxxxxx
4+
export ACM="arn:aws:acm:xxxxxxxxxxx"
5+
# domain name for your docker registry without schema (http/https): your.docker-registry.com
6+
export DOMAIN="your.docker-registry.com"
7+
8+
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
9+
envsubst < ./ingress-nginx/service-nlb.yaml.tmpl > /tmp/service-nlb.yaml
10+
kubectl apply -f /tmp/service-nlb.yaml
11+
12+
kubectl apply -f ./common/namespace.yaml
13+
14+
kubectl apply -f ./redis/deployment.yaml
15+
kubectl apply -f ./redis/service.yaml
16+
17+
kubectl apply -f ./registry/pvc.yaml
18+
kubectl apply -f ./registry/deployment.yaml
19+
kubectl apply -f ./registry/service.yaml
20+
envsubst < ./registry/ingress.yaml.tmpl > /tmp/ingress.yaml
21+
kubectl apply -f /tmp/ingress.yaml
22+
kubectl apply -f ./registry/cronjob.yaml

Diff for: ‎image_retention.groovy

+47
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
#!/usr/bin/env groovy
2+
3+
def CleanUpRegistry(REPOSITORY, SAVE_FIRST_N_IMAGES) {
4+
sh """#!/usr/bin/env bash
5+
set -x
6+
SAVE_FIRST_N_IMAGES=\$(($SAVE_FIRST_N_IMAGES + 1))
7+
ACCEPT_HEADER="Accept: application/vnd.docker.distribution.manifest.v2+json"
8+
DOMAIN="your.docker-registry.com"
9+
10+
ARRAY=()
11+
declare -A TIME_TAG_MAP
12+
TAGS=\$(curl -Ls https://\$DOMAIN/v2/$REPOSITORY/tags/list | jq -r '."tags"[]')
13+
if [ \${#TAGS[@]} -gt 0 ]; then
14+
for TAG in \${TAGS[@]}; do
15+
ID=\$(curl -Ls --header "\$ACCEPT_HEADER" GET https://\$DOMAIN/v2/$REPOSITORY/manifests/\$TAG | jq -r '.config.digest')
16+
UPL_TIME=\$(curl -Ls GET https://\$DOMAIN/v2/$REPOSITORY/blobs/\$ID | jq '.created' | sort | tail -n1 | sed 's/"//g')
17+
ARRAY+=(\$UPL_TIME)
18+
TIME_TAG_MAP[\$UPL_TIME]=\$ID
19+
echo "\$TAG | \$UPL_TIME | \$ID"
20+
done
21+
fi
22+
23+
if [ \${#ARRAY[@]} -gt 0 ]; then
24+
for i in \$(sort -rn < <(printf '%s\\n' \${ARRAY[@]}) | uniq | tail -n +\$SAVE_FIRST_N_IMAGES); do
25+
echo "For deletion: \$i --- \${TIME_TAG_MAP[\$i]}"
26+
echo "curl -Ls --header \"\$ACCEPT_HEADER" -X DELETE https://\$DOMAIN/v2/$REPOSITORY/manifests/\${TIME_TAG_MAP[\$i]}"
27+
done
28+
fi
29+
"""
30+
}
31+
32+
33+
node('ec2Slave') {
34+
try{
35+
parallel CleanUpRegistry1 : {
36+
stage ('CleanUpRegistry1'){
37+
CleanUpRegistry("alpine", 2)
38+
}
39+
}, CleanUpRegistry2 : {
40+
stage ('CleanUpRegistry2'){
41+
CleanUpRegistry("python", 1)
42+
}
43+
}
44+
} catch(err) {
45+
throw(err)
46+
}finally{}
47+
}

Diff for: ‎ingress-nginx/service-nlb.yaml.tmpl

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
kind: Service
2+
apiVersion: v1
3+
metadata:
4+
name: ingress-nginx
5+
namespace: ingress-nginx
6+
labels:
7+
app.kubernetes.io/name: ingress-nginx
8+
app.kubernetes.io/part-of: ingress-nginx
9+
annotations:
10+
# by default the type is elb (classic load balancer).
11+
service.beta.kubernetes.io/aws-load-balancer-type: nlb
12+
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "${ACM}"
13+
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
14+
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
15+
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
16+
spec:
17+
# this setting is to make sure the source IP address is preserved.
18+
externalTrafficPolicy: Local
19+
type: LoadBalancer
20+
selector:
21+
app.kubernetes.io/name: ingress-nginx
22+
app.kubernetes.io/part-of: ingress-nginx
23+
ports:
24+
- name: http
25+
port: 80
26+
targetPort: http
27+
- name: https
28+
port: 443
29+
targetPort: http
30+
---

Diff for: ‎redis/deployment.yaml

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: extensions/v1beta1
2+
kind: Deployment
3+
metadata:
4+
name: redis
5+
namespace: docker-registry
6+
spec:
7+
selector:
8+
matchLabels:
9+
app: redis
10+
replicas: 2
11+
template:
12+
metadata:
13+
labels:
14+
app: redis
15+
spec:
16+
containers:
17+
- name: master
18+
image: k8s.gcr.io/redis
19+
ports:
20+
- containerPort: 6379

Diff for: ‎redis/service.yaml

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
kind: Service
2+
apiVersion: v1
3+
metadata:
4+
name: redis
5+
namespace: docker-registry
6+
spec:
7+
selector:
8+
app: redis
9+
ports:
10+
- name: http
11+
protocol: TCP
12+
port: 6379
13+
targetPort: 6379

Diff for: ‎registry/cronjob.yaml

+32
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
apiVersion: batch/v1beta1
2+
kind: CronJob
3+
metadata:
4+
name: docker-registry-cron
5+
namespace: docker-registry
6+
spec:
7+
schedule: "*/30 * * * *"
8+
jobTemplate:
9+
spec:
10+
template:
11+
metadata:
12+
name: docker-registry-cron
13+
spec:
14+
containers:
15+
- name: docker-registry-cron
16+
image: registry:latest
17+
args:
18+
- /bin/registry
19+
- garbage-collect
20+
- --dry-run
21+
- /etc/docker/registry/config.yml
22+
envFrom:
23+
- configMapRef:
24+
name: registry
25+
volumeMounts:
26+
- name: image-data
27+
mountPath: /var/lib/registry
28+
restartPolicy: OnFailure
29+
volumes:
30+
- name: image-data
31+
persistentVolumeClaim:
32+
claimName: docker-registry

Diff for: ‎registry/deployment.yaml

+61
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
apiVersion: extensions/v1beta1
2+
kind: Deployment
3+
metadata:
4+
name: docker-registry
5+
namespace: docker-registry
6+
labels:
7+
app: docker-registry
8+
spec:
9+
replicas: 1
10+
selector:
11+
matchLabels:
12+
app: docker-registry
13+
template:
14+
metadata:
15+
labels:
16+
app: docker-registry
17+
spec:
18+
containers:
19+
- image: registry:latest
20+
name: docker-registry
21+
envFrom:
22+
- configMapRef:
23+
name: registry
24+
volumeMounts:
25+
# - name: files
26+
# mountPath: /etc/k8s
27+
- name: image-data
28+
mountPath: /var/lib/registry
29+
ports:
30+
- name: registry
31+
containerPort: 5000
32+
protocol: TCP
33+
imagePullPolicy: Always
34+
restartPolicy: Always
35+
volumes:
36+
# - name: files
37+
# configMap:
38+
# name: registry
39+
# items:
40+
# - key: HTPASSWD_FILE
41+
# path: htpasswd
42+
- name: image-data
43+
persistentVolumeClaim:
44+
claimName: docker-registry
45+
---
46+
kind: ConfigMap
47+
apiVersion: v1
48+
metadata:
49+
name: registry
50+
namespace: docker-registry
51+
data:
52+
REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
53+
REGISTRY_HTTP_SECRET: FPNxM9cwZDkxO5RkMDUzXjk6YjIyOWRmhjBlMWRhMjQG
54+
REGISTRY_STORAGE_CACHE_BLOBDESCRIPTOR: redis
55+
REGISTRY_REDIS_ADDR: redis:6379
56+
REGISTRY_STORAGE_DELETE_ENABLED: "true"
57+
# REGISTRY_AUTH: htpasswd
58+
# REGISTRY_AUTH_HTPASSWD_PATH: /etc/k8s/htpasswd
59+
# REGISTRY_AUTH_HTPASSWD_REALM: Private Registry
60+
# HTPASSWD_FILE: |
61+
# admin:xxxxxxxxxxxxxxxxxxxxxxx

Diff for: ‎registry/ingress.yaml.tmpl

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
apiVersion: extensions/v1beta1
2+
kind: Ingress
3+
metadata:
4+
name: docker-registry
5+
namespace: docker-registry
6+
annotations:
7+
nginx.ingress.kubernetes.io/proxy-body-size: "0"
8+
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
9+
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
10+
spec:
11+
rules:
12+
- host: ${DOMAIN}
13+
http:
14+
paths:
15+
- path: /
16+
backend:
17+
serviceName: docker-registry
18+
servicePort: 5000

Diff for: ‎registry/pvc.yaml

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
kind: PersistentVolumeClaim
2+
apiVersion: v1
3+
metadata:
4+
name: docker-registry
5+
namespace: docker-registry
6+
spec:
7+
accessModes:
8+
- ReadWriteOnce
9+
resources:
10+
requests:
11+
storage: 5Gi

Diff for: ‎registry/service.yaml

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
kind: Service
2+
apiVersion: v1
3+
metadata:
4+
name: docker-registry
5+
namespace: docker-registry
6+
spec:
7+
selector:
8+
app: docker-registry
9+
ports:
10+
- protocol: TCP
11+
port: 5000
12+
targetPort: 5000

Diff for: ‎undeploy.sh

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#!/usr/bin/env bash
2+
3+
kubectl delete namespaces docker-registry
4+
kubectl delete namespaces ingress-nginx

0 commit comments

Comments
 (0)