All Articles

Running a private docker registry inside kubernetes

I found this article from Peter Kohut really helpfull in getting this started. However, I wanted to use self singed certificates and also wanted the registry accessible on my virtual “bare-metal” cluster.

Prerequisites

  • A running kubernetes cluster
  • OpenSSL
  • NGINX Ingress Controller for Kubernetes

MySetup

  • Archlinux host with two Ubuntu 18.04.3 qemu guests running a kubernetes master and one node
  • NFS share on the host mounted to /data on both guests
  • Kubernetes version 1.17.0
  • Docker version 18.09.7

Goal

  • Having a private docker registry inside a kubernetes cluster
  • TLS termination in the kubernetes ingress
  • Make the registry available on my host machine
  • Use the registry inside kubernetes

Deploy the registry

I wanted the registry to support basic authentication and persist it on my host machines file system. To persist the data I decided to mount a /data directory via nfs and use hostPath volumes inside the deployment. Another possibility would have been to use the nfs volumes instead. The htpasswd for basic authentication is supplied as a secret.

Create the HTPASSWD secret

First of all use the docker registry image to create the htpasswd file

docker run --entrypoint htpasswd --rm registry:2 -Bbn admin admin | base64

This will give you a base64 encoded string to use in the secret

YWRtaW46JDJ5JDA1JFFleHBwdkJvTlBybHF5YUx5LlZ3V08xVUFuSVV2QU95eTFaSlRDRy95cHMxMDJhaXNpZDZPCgo=

In case you are wondering what is encoded in this string pipe it into base64 -d:

echo YWRtaW46JDJ5JDA1JFFleHBwdkJvTlBybHF5YUx5LlZ3V08xVUFuSVV2QU95eTFaSlRDRy95cHMxMDJhaXNpZDZPCgo= | base64 -d

Now we can put this in our secret to be created:

apiVersion: v1
kind: Secret
metadata:
  name: docker-registry
type: Opaque
data:
  HTPASSWD: YWRtaW46JDJ5JDA1JFFleHBwdkJvTlBybHF5YUx5LlZ3V08xVUFuSVV2QU95eTFaSlRDRy95cHMxMDJhaXNpZDZPCgo=

If you do not want to create the file yourself, feel free to either apply it directly or download it.

kubectl apply -f https://gitlab.com/inw/k8-dockerregistry/raw/master/registry-auth.yaml

Configure the registry

Next create a configMap, I’ve used the one Peter Kohut supplied in his article.

apiVersion: v1
kind: ConfigMap
metadata:
  name: docker-registry
data:
  registry-config.yml: |
    version: 0.1
    log:
      fields:
        service: registry
    storage:
      cache:
        blobdescriptor: inmemory
      filesystem:
        rootdirectory: /var/lib/registry
    http:
      addr: :5000
      headers:
        X-Content-Type-Options: [nosniff]
    auth:
      htpasswd:
        realm: basic-realm
        path: /auth/htpasswd
    health:
      storagedriver:
        enabled: true
        interval: 10s
        threshold: 3

As you can see the registry persists data on the filesystem in /var/lib/registry and uses the file /auth/htpasswd as source for authentication. The registry will listen on port 5000.

If you do not want to create the file yourself, feel free to either apply it directly or download it.

kubectl apply -f https://gitlab.com/inw/k8-dockerregistry/raw/master/registry-config.yaml

Deploy the registry

I’ve decided to use a deployment instead of a pod. However, one replica will be enough. 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: docker-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-reg
  template:
    metadata:
      labels:
        app: docker-reg
    spec:
      containers:
        - name: registry
          image: registry:2
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 5000
              protocol: TCP
          volumeMounts:
            - name: config
              mountPath: /etc/docker/registry
              readOnly: true
            - name: htpasswd
              mountPath: /auth
              readOnly: true
            - name: storage
              mountPath: /var/lib/registry            
      volumes:
        - name: config
          configMap:
            name: docker-registry
            items:
              - key: registry-config.yml
                path: config.yml
        - name: htpasswd
          secret:
            secretName: docker-registry
            items:
              - key: HTPASSWD
                path: htpasswd
        - name: storage
          hostPath:
            path: /data/dockerregistry
            type: Directory

In the template for the registry container we need to register the volumes and the volume mounts to use. We mount the registry-config.yml key from the configMap at /etc/docker/registry/config.yml and the docker-registry secret with the key HTPASSWD at /auth/htpasswd. Both files are mounted read only. Last but not least we bind /var/lib/registry to the host path /data/dockerregistry. If you want to use another volume type check out “Types of Volumes” in the official kubernetes documentation.

If you do not want to create the file yourself, feel free to either apply it directly or download it.

kubectl apply -f https://gitlab.com/inw/k8-dockerregistry/raw/master/registry-deployment.yaml

The registry pod shold now up an running. Make sure to check it with 

kubectl get po

If you experience problems check out the troubleshooting section.

Exposing the registry

Now its time to expose the registry as a service with a ClusterIP. This is quite straightforward.

apiVersion: v1
kind: Service
metadata:
  labels:
    app: docker-registry
  name: docker-registry
spec:
  ports:
  - name: http
    port: 5000
    protocol: TCP
    targetPort: 5000
  selector:
    app: docker-reg
  type: ClusterIP

If you do not want to create the file yourself, feel free to either apply it directly or download it.

kubectl apply -f https://gitlab.com/inw/k8-dockerregistry/raw/master/registry-service.yaml

If you like verify the service has endpoints by running kubectl get ep $servicename.

Creating the ingress with TLS termination

I’m using NGINX Ingress Controller for Kubernetes without ssl passthrough. Please refer to the installation guide for deploying the controller.

For all examples the domain registry.k8 is used.

Create the certificate and the secret

Use openssl to create a self signed certificate and a key:

openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out registry.crt -keyout registry.key

Make sure to supply registry.k8 as common name.

Now create a TLS-Secret for kubernetes by running

kubectl create secret tls registry-tls --key registry.key --cert registry.crt

as described here.

Configuring the ingress

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: docker-registry
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-body-size: 0m
spec:
  tls:
  - hosts:
    - registry.k8
    secretName: registry-tls
  rules:
  - host: registry.k8
    http:
      paths:
      - backend:
          serviceName: docker-registry
          servicePort: 5000

This ingress will be listening on ports 80 and 443 and send all traffic for registry.k8 to the docker-registry service on port 5000 - as configured in the configMap for the registry. The annotation nginx.ingress.kubernetes.io/proxy-body-size: 0m is needed to disable the max body size of the nginx ingress. Otherwise we would not be able to push docker images - or at least only very small ones.

If you do not want to create the file yourself, feel free to either apply it directly or download it.

kubectl apply -f https://gitlab.com/inw/k8-dockerregistry/raw/master/registry-ingress.yaml

Now run

kubectl get ing docker-registry

and note the ClusterIP of the ingress.

Configuring a reverse proxy

To get traffic into my cluster I’m using my master-node. Therefore I’ve added an entry to my /etc/hosts file containing registry.k8 $masternodeIP. Incomming traffic on my master-node is then handled by an nginx reverse proxy as a stream. Therefore I not have to worry about TLS termination on the reverse proxy.

To make this work edit the /etc/nginx and add include /etc/nginx/streams-enabled/*; as a top level entry. Streams have to be configured on the same level as http or mail. Now create a file named /etc/nginx/sites-available/reverse-proxy-stream.conf with the content of

error_log /var/log/nginx/error.log;

stream {
  
  upstream registry {
   server $INGRESS_CLUSTER_IP:443 max_fails=10 fail_timeout=10s;
  }  

  server {
        listen 5000;
        listen [::]:5000;

        proxy_pass registry;
  }
}

and link it into include /etc/nginx/streams-enabled/*;, run sudo nginx -t to verify your configuration an then restart the nginx service. Make sure to replace $INGRESSCLUSTERIP with the one you discovered above.

Now you should be able to open (https://registry.k8:5000/v2/_catalog) and authenticate with admin:admin - or whatever username and password you have chosen. Of course you can use whatever port you want the reverse proxy to listen to. Just make sure to remember it for the next and (nearly) last step. If you cannot reach the registry checkout the troubleshooting section.

Tell docker to trust the self signed certificates

Last but not least we must tell docker to trust our self signed certificates. On each docker installation you want to use your registry run

sudo mkdir -p /etc/docker/certs.d/registry.k8:5000
sudo cp registry.crt /etc/docker/certs.d/registry.k8\:5000/.

Docker will now trust your certificate for registry.k8:5000.

Login to your registry

docker login -u admin -p admin registry.k8:5000

Tag an image and push it

docker image tag $imagetopush registry.k8:5000/$imagetopush
docker push registry.k8:5000/$imagetopush

This should work. If not, refer to the troubleshooting section.

Tell kubernetes to use the registry

Wouldn’t it be nice to use your registry from inside your kubernetes cluster as well? Actually thats quite straightforward.

  • Make sure every node knows to point to your master-node when asked for registry.k8
  • Create a docker login secret kubectl create secret docker-registry regcred --docker-server=registry.k8:5000 --docker-username=admin --docker-password=admin

  • Use following container template:

    containers:
- name: mycontainer
image: registry.k8:5000/myimage
imagePullSecrets:
- name: regcred

    Further information can be found in the official documentation.

Troubleshooting

Registry pod has problems

  • check with kubectl describe po $podname for detailed errors.
  • check with kubectl logs $podname for detailed errors.

Most of the times I had a typo in a some config files misplacing the secret or the volumeMounts. Also some filesystem permissions may to be set to 777. There seems to be a problem with the umask in some cases using hostPath volumes. Just make sure the container is allowed to write to the host path.

Registry is not reachable from a browser

First check if your /etc/hosts lists registry.k8 and points to your kubernetes master and your registry pod is running. If this is the case check if your reverse proxy is configured correctly, e.g. points to the correct ClusterIP of the ingress. The next step is to check your ingress configuration, the service should point to your registry pod as endpoint. A very hand command to check if you can reach the registry is curl https://IP_ADDRESS:PORT/v2/_catalog -k. Just replace the IP_ADDRESS:PORT with either the domain name and port or the IP address/port combination of your ingress, service or pod. You should always get a JSON reply stating an unauthorized error. Just make sure you are on your kubernetes master if you check ingress, service or pod. Remember to use kubectl describe ... to get further information about a kubernetes resource.

Cannot push to the registry

Cannot connect

Make sure you have copied the certificate to /etc/docker/conf.d/$registry_domain:port/.. Next make sure the CN of the certificate is set to the registry domain by running openssl x509 -in certificate.crt -text -noout | grep CN. Point your browser to https://registry.k8:5000/v2/_catalog (or whatever you are using as registry domain) and check if the correct certificate is used. If this is not the case double check your ingress configuration and the used TLS secret.

Cannot upload

Make sure your ingress and reverse proxy do not limit the body size. Also make sure you tagged your image correctly. Do not forget to add the port, as I did first.

Published 1 Jan 2020

In my current life software developer, in a former one biochemist.Martin Schmidt on Twitter