Mongo SSL Auth on Kubernetes

9 min readJul 24, 2019

0 | Introduction

In this post, we will learn how to deploy MongoDB container in a Kubernetes cluster and configure authentication with SSL support.

Prerequisites

An operational Kubernetes cluster
kubectl
Understanding of Kubernetes concepts
Understanding of Mongodb and SSL certificates

The deployment described in that post was performed on Google Cloud Platform — Kubernetes Engine, but it should work on any other cloud provider.

Deployment
Test Connection
Create a MongoDB admin user
Enable Authentication
Generate TLS/SSL certificates
Use SSL certificates in the deployment
Expose MongoDB externally
Add client certificate
Cleanup
Useful Links

1 | Deployment

Let’s start with creating a Mongodb persistent storage, deployment and service.

Create a file my-mongo-pvc.yaml

# my-mongo-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-mongo-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Create a file my-mongo-deployment.yaml:

# my-mongo-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-mongo
spec:
  selector:
    matchLabels:
      app: my-mongo
      environment: my
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: my-mongo
        environment: my
    spec:
      hostname: my-mongo
      volumes:
        - name: my-mongo-persistent-storage
          persistentVolumeClaim:
            claimName: my-mongo-pvc
      containers:
        - name: my-mongo
          image: mongo:3.6
          imagePullPolicy: Always
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: my-mongo-persistent-storage
              mountPath: /data/db

Create a filemy-mongo-service.yaml

# my-mongo-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-mongo
spec:
  ports:
  - port: 27017
  selector:
    app: my-mongo
  clusterIP: None

Now, let’s create all these objects in the cluster with the command:

kubectl apply -f ./my-mongo-pvc.yaml \
  -f ./my-mongo-deployment.yaml \
  -f ./my-mongo-service.yaml

Now wait while deployment is creating the container:

kubectl get pod --watch

Once the container is in status Running we can log in to test our MongoDB deployment:

kubectl exec my-mongo-5c4f6c67cf-xmrnl \
  mongo -- --eval "db.adminCommand('ping')"

Note: we use the double dash -- after mongo to indicate that we want to pass the extra arguments to the mongo and not to the kubectl.

The output should look like this

MongoDB shell version v3.6.12
connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“306c6efe-ca85–44c8-b296–38924137fc34”) }
MongoDB server version: 3.6.12
{ “ok” : 1 }

Cool, our MongoDB server is running.

2 | Test connection

Now let’s test that we are able to connect to the MongoDB server inside the cluster. For that, we would need to create a new container based on MongoDb docker image to use as mongo client.

Create file my-mongo-client.yaml

# my-mongo-client.yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-mongo-client
  namespace: default
spec:
  containers:
  - name: my-mongo-client
    image: mongo:3.6
    command: ["tail", "-f", "/dev/null"]
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

We set the command totail -f /dev/null to prevent container from exiting, so we are able to SSH into it.

Now to create the container by executing

kubectl apply -f ./my-mongo-client.yaml

Once the container is started, run

kubectl exec -it my-mongo-client mongo -- --host=my-mongo

We should be able to see MongoDB shell and execute commands like show dbs or db.hostInfo().

3 | Create a MongoDB admin user

Now it is time to create MongoDB admin user. Admins users generally have to be created on admin database and assigned root role.

To learn more about User Management visit https://docs.mongodb.com/manual/tutorial/manage-users-and-roles/

First, we need to switch to the admin database:

> use admin

And then we will create theadmin superuser:

> db.createUser({ 
    user: "admin", 
    pwd: "Password1!", 
    roles: ["root"] 
  })

4 | Enable authentication

Now that we have created our admin user, we need to enable MongoDb authentication.

To do that we need to launch the MongoDB service with the --auth flag.
Add the following changes to our deployment

# my-mongo-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-mongo
spec:
  selector:
    matchLabels:
      app: my-mongo
      environment: my
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: my-mongo
        environment: my
    spec:
      hostname: my-mongo
      volumes:
        - name: my-mongo-persistent-storage
          persistentVolumeClaim:
            claimName: my-mongo-pvc
      containers:
        - name: my-mongo
          image: mongo:3.6
          args: ["--auth"] # this flag will enable authentication
          imagePullPolicy: Always
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: my-mongo-persistent-storage
              mountPath: /data/db

Apply changes with command

kubectl apply -f ./my-mongo-deployment.yaml

Wait until the container is restarted and let’s test the authentication

kubectl exec -it my-mongo-client \
  mongo -- --host=my-mongo --username=admin --password=Password1!

We should be logged in now, then we can view the authentication details

> db.runCommand({connectionStatus: 1})

5 | Generate SSL certificates

If we want to connect to our MongoDB instance over the untrusted network we do not want to send the username and password in plaintext. For that reason, we will configure SSL.

Because our service is not available on an external domain we won’t be able to sign our certificates with a well known Certificate Authority (they require the proof of domain ownership) so we will have to self-sign our certificates.

First, we will create a Certificate Authority (CA) root private key and certificate which we will use to sign the server certificate.

We will store our keys and certificates in the ssl directory

mkdir ssl && cd ssl

Generate a private key for the root authority

openssl genrsa -out root-ca.key 2048

Create a self-signed certificate

openssl req -x509 -new -nodes -key root-ca.key -sha256 \
  -days 1024 -out root-ca.pem

We will be asked information like country, city, company name, etc that will be stored in our certificate.

Now we’ll create the private key for our MongoDB instance

openssl genrsa -out mongodb.key 2048

Create a certificate signing request (CSR)

openssl req -new -key mongodb.key -out mongodb.csr

Once again we will be asked to input the information.

It is important to set Common Name to our MongoDB instance host my-mongo. Otherwise, certificate validation will fail when trying to connect.

The next step is to sign the CSR with our root-ca.key

openssl x509 -req -in mongodb.csr -CA root-ca.pem \
  -CAkey root-ca.key -CAcreateserial -out mongodb.crt \
  -days 700 -sha256

Finally, we will package the mongodb private key and certificate in a PEM file. This file will be used by the MongoDB instance to validate and encrypt SSL connections.

cat mongodb.key mongodb.crt > mongodb.pem

6 | Use SSL certificates in the deployment

In this step, we will configure MongoDB to use our newly created private key and certificate along with the root CA certificate.

Create file my-mongo-config.yaml

# my-mongo-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-mongo-config
  namespace: default
data:
  mongod.conf: |
    security:
      authorization: enabled
    net:
      ssl:
        mode: allowSSL
        PEMKeyFile: /etc/ssl/mongodb.pem
        CAFile: /etc/ssl/root-ca.pem
        allowConnectionsWithoutCertificates: true

This ConfigMap contains contents for the mongod.conf file which we will mount in the MongoDB container.

Here is a brief explanation of the configuration

mode enables SSL connections, change the value to requireSSL if we wish to disable unencrypted connections
PEMKeyFile sets path to the file that contains the TLS/SSL certificate and key
CAFile sets path to the root CA certificate
allowConnectionsWithoutCertificates makes it simple for us to connect without providing a certificate on the client side.

For more details about Mongo SSL configuration visit https://docs.mongodb.com/manual/tutorial/configure-ssl/

Create the ConfigMap by running

kubectl apply -f ./my-mongo-config.yaml

That configuration will allow us to enable SSL on MongoDB but first, we need to make the private keys and certificates available to the MongoDB container.

For that, we will use the Kubernetes Secrets.

Values in the secret need to be base64 encoded.

To get the values for the secret we will copy output from these commands into the secret.

cat ./ssl/root-ca.pem | base64 | tr -d '\n'
cat ./ssl/mongodb.pem | base64 | tr -d '\n'

If you’re on OSX you can pipe the output to pbcopy to copy the value in the clipboard.
Example: cat ./ssl/root-ca.pem | base64 | tr -d '\n' | pbcopy

Create file my-mongo-secret.yaml

# my-mongo-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-mongo-ssl-certs
type: Opaque
data:
  root-ca.pem: <base64 encoded root-ca.pem>
  mongodb.pem: <base64 encoded mongodb.pem>

Let’s create the secret

kubectl apply -f ./my-mongo-secret.yaml

To activate mongo configuration and mount our SSL certificates we need to adjust our deployment configuration

# my-mongo-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-mongo
spec:
  selector:
    matchLabels:
      app: my-mongo
      environment: my
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: my-mongo
        environment: my
    spec:
      hostname: my-mongo
      volumes:
        - name: my-mongo-persistent-storage
          persistentVolumeClaim:
            claimName: my-mongo-pvc
        - name: my-mongo-config
          configMap:
            name: my-mongo-config
        - name: my-mongo-ssl-certs
          secret:
            secretName: my-mongo-ssl-certs
      containers:
        - name: my-mongo
          image: mongo:3.6
          args:
            - --config
            - /etc/mongo/mongod.conf
          imagePullPolicy: Always
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: my-mongo-persistent-storage
              mountPath: /data/db
            - name: my-mongo-config
              mountPath: /etc/mongo
            - name: my-mongo-ssl-certs
              mountPath: /etc/ssl

In the volumes section, we specify that we want to use our my-mongo-config ConfigMap as well as my-mongo-ssl-certs Secret. In the sectionvolumeMounts we mount the ConfigMap to the/etc/mongo path, which will create /etc/mongo/mongod.conf file. Mounting secret to /etc/ssl will create us root-ca.pem and mongodb.pem files in that directory.

Let’s apply changes

kubectl apply -f ./my-mongo-deployment.yaml

After MongoDB container is restarted, check that certificates are where we are expecting them to be

kubectl exec my-mongo-68d5fb686c-j8bv5 sh -- -c 'ls /etc/ssl/*'

The output should look like this

/etc/ssl/mongodb.pem
/etc/ssl/root-ca.pem

To test the SSL connection we need to copy the root-ca.pem to my-mongo-client pod using kubectl command

kubectl cp ssl/root-ca.pem my-mongo-client:/

This will copy the certificate root-ca.pem to the root of the my-mongo-client container.

Finally, let’s connect using SSL

kubectl exec -it my-mongo-client \
  mongo -- --host=my-mongo --username=admin --password=Password1! \
  --ssl --sslCAFile=/root-ca.pem

7 | Expose MongoDB externally

If we need to open the MongoDB instance to the internet we can do so with Kubernetes service.

Note that it exposes our database to hacking attempts. Currently, the database is protected by password and passwords can be brute forced.

If you necessarily need to open your MongoDB to the internet — come up with a strong password, setup firewall rules to white list IPs and change the default MongoDB port from 27017 to distance yourself from passive scans.

Create my-mongo-external-service.yaml

# my-mongo-external-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-mongo-external
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app: my-mongo
  ports:
  - protocol: TCP
    port: 30000
    targetPort: 27017

Here we specify that service will listen on port 30000 and forward the traffic to our MongoDB instance.

If our cloud provider allows us to reserve a static IP, we can define it in the service with loadBalancerIP

Create the service

kubectl apply -f ./my-mongo-external-service.yaml

Wait while service is assigned an EXTERNAL-IP address

kubectl get svc

Now we are able to connect to the MongoDB instance from an external network

mongo --host=<your-external-ip> --port=30000 \
  --username=admin --password=Password1! \
  --ssl --sslCAFile=./ssl/root-ca.pem \
  --sslAllowInvalidHostnames \
  admin

Here the host is pointing to our service external IP address and the port is set to 30000.
Also, we supply the path to our root Certificate Authority root-ca.pem.

We will notice a warning

The server certificate does not match the hostname. 
Hostname: <your-external-ip> does not match CN: my-mongo

which will not allow us to connect unless the option sslAllowInvalidHostnames is specified.

When we connect from an external network MongoDB container reports it’s hostname as <your-external-ip> so it does not match my-mongo hostname which is saved in the certificate under Common Name

A possible solution would be to reserve a static IP address and add it to “Subject Alternative Names” when creating the CSR but this is outside of the scope of this material.
Hopefully, we know that we own the IP address so we can trust the server identity, our only goal is to have the connection encrypted.

Remember to configure firewall rules in your cloud provider to allow connections only from known IPs

8 | Add client certificate

To improve our security even further, we can issue a client certificate signed by our root CA. That will allow MongoDb to accept connections only from clients that provide a valid signed certificate.

We start by going into the ssl directory and creating client private key and certificate signing request.

openssl genrsa -out client.key 2048
openssl req -new -key client.key -out client.csr

Now we will sign it by our root Certificate Authority and create the client certificate client.crt

openssl x509 -req -in client.csr -CA root-ca.pem \
  -CAkey root-ca.key -CAcreateserial -out client.crt \
  -days 700 -sha256

Package client private key and certificate in a PEM file

cat client.key client.crt > client.pem

Now in order to connect we need to specify sslPEMKeyFile

mongo --host=<your-external-ip> --port=30000 \
  --username=admin --password=Password1! \
  --ssl --sslCAFile=./ssl/root-ca.pem \
  --sslAllowInvalidHostnames --sslPEMKeyFile=./ssl/client.pem \
  admin

Remember to remove allowConnectionsWithoutCertificates from my-mongo-config.yaml and restart the deployment to always require a client certificate.

9 | Cleanup

Hopefully, it was a good learning experience, now it is time to clean up

kubectl delete -f .

Thanks for reading!