Using Prometheus with GDK

This page details how to:

• Work with the Prometheus integration in GDK.
• Monitor the GDK with Prometheus.

Prometheus integration

Testing the Prometheus integration with the GitLab Development Kit requires some additional components. This is because the Prometheus integration requires a CI/CD deploy on Kubernetes. For simply viewing the metrics generated by the GitLab instance itself, see monitoring the GDK with Prometheus below.

Because of this, you need to either:

• Run a local Kubernetes cluster.
• Use a service like the Google Container Engine (GKE).

GKE is recommended for GitLab team members who intend to frequently develop against this setup. For infrequent use or while waiting for GKE access, minikube is a good option.

Setting it up locally with minikube is often easier, as you do not have to worry about Runners in GKE requiring network access to your local GDK instance.

Instructions for minikube

The following steps help you set up minikube locally.

Install kubectl if you do not have it

kubectl is required for minikube to function. You can also use homebrew to install it using brew install kubernetes-cli.

1. First, download it:

   ## For macOS curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s "https://storage.googleapis.com/kubernetes-release/release/stable.txt")/bin/darwin/amd64/kubectl"  ## For Linux curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s "https://storage.googleapis.com/kubernetes-release/release/stable.txt")/bin/linux/amd64/kubectl"

2. Then, add it to your path:

   chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/

Install minikube

For macOS with homebrew, run brew install minikube. You may need to brew link minikube if the link is not created automatically.

1. First, download it:

   ## For macOS curl -Lo minikube "https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64"  ## For Linux curl -Lo minikube "https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64"

2. Then, add it to your path:

   chmod +x ./minikube sudo mv ./minikube /usr/local/bin/

Install a virtualization driver

minikube requires virtualization. Install the appropriate driver for your operation system:

• macOS
• Linux

Start minikube

The following command starts minikube, running the first few containers with Kubernetes components.

Compatibility Note: We are not yet supporting Kubernetes 1.16, please use 1.15 the following until https://gitlab.com/gitlab-org/gitlab/issues/32721 is resolved.

For MacOS:

minikube start --vm-driver hyperkit --disk-size=20g --kubernetes-version=v1.15.4

For Linux:

minikube start --vm-driver kvm2 --disk-size=20g --kubernetes-version=v1.15.4

Open the Kubernetes Dashboard

Once minikube starts, open the Kubernetes dashboard to ensure things are working You can use this for future troubleshooting.

minikube dashboard

Configure GDK to listen to more than localhost

Follow the instructions to make your GDK accessible to other devices in the local network.

Edit the GitLab gitlab.yml file

We need to configure GDK to inform it of the real IP address of your computer. This is because GDK returns this information to the Runner and if it is wrong, pipelines fail.

1. Get your local IP address by running ifconfig or opening up Network Settings if on macOS. On Linux, you can also use ip addr show, or ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p'

2. Open gitlab/config/gitlab.yml and change the GitLab settings web server host line from either localhost or 127.0.0.1 to reflect the IP of the previous step:

   ## GitLab settings gitlab:  ## Web server settings (note: host is the FQDN, do not include http://)  host: 192.168.1.9  port: 3000  https: false

3. Save the file and run gdk restart to apply this change. You should now be able to access GitLab by the external URL (e.g., http://192.168.1.9:3000 not localhost), otherwise it may not work correctly.

Create a Project

Now that we have GDK running, we need to go and create a project with CI/CD set up. The easiest way to do this, is to simply import from an existing project with a simplified .gitlab-ci.yml.

Import https://gitlab.com/joshlambert/autodevops-deploy.git as a public project, to use a very simple CI/CD pipeline with no requirements, based on AutoDevOps. It contains just the deploy stages and uses a static image, since the GDK does not contain a registry.

Allow requests to the local network

We have CSRF protection in place on the cluster URL so if we try to connect minikube now, we get a Requests to the local network are not allowed error. The below steps disable this protection for use with minikube.

1. As root user, navigate to Admin Area (the little wrench in the top nav) > Settings > Network.
2. Expand the Outbound requests section, check the box to Allow requests to the local network from hooks and services, and save your changes.

Connect your cluster

1. In a terminal, run minikube ip to get the API endpoint of your cluster.
2. Next go back to your Kubernetes cluster dashboard. If it is not open, you can open one by running minikube dashboard.
3. At bottom of the page you find a list of secrets, with one named default. Click on it to view it. You need these values later.
4. In your GitLab instance, go to Operations > Kubernetes, and add a cluster. Select the option to add an existing cluster.
5. Enter any value for the Kubernetes cluster name.
6. For API URL, enter https://<MINIKUBE_IP>:8443 using the value from step 1.
7. For CA Certificate, paste in the value from your Kubernetes secret.
8. Similarly for Token, paste the value from the Kubernetes secret.
9. Save your changes.

Disable RBAC

AutoDevOps and Kubernetes app deployments do not yet support RBAC. To disable RBAC in your cluster, run the following command:

kubectl create clusterrolebinding permissive-binding \  --clusterrole=cluster-admin \  --user=admin \  --user=kubelet \  --group=system:serviceaccounts

Deploy Helm Tiller, Prometheus, and GitLab Runner

Back in the GDK on the cluster screen, you should now be able to deploy Helm Tiller. Once complete, also deploy a Runner and Prometheus.

If you get an error about an API token not yet being created, wait a minute or two and try again.

If installing Helm Tiller fails with ‘Kubernetes error’, you may have an existing configuration. To remove it:

kubectl delete configmap values-content-configuration-helm -n gitlab-managed-apps

Check that your Runner is connected

Once you have installed the Runner via the cluster screen (known as a GitLab managed app), we want to make sure it’s reachable. If it’s not, the jobs become stuck. To check it’s connected:

1. As an admin, navigate to Admin Area (the little wrench in the top nav) Overview > Runners.
2. To know your Runner is connected you should see: Runners currently online: 1, with the actual Runner listed below holding a valid IP such as 192.168.122.14 etc.
3. If you do not see the runner IP, then it is not connected and something has been missed from your network configuration and this is where you need to begin debugging. You can use minikube logs to see the pod logs for the managed runner which should give you an idea of whats going wrong.

Run a Pipeline to deploy to an Environment

Now that we have a Runner configured, we need to kick off a Pipeline. This is because the Prometheus integration only looks for environments which GitLab knows about and have a successful deploy. To do this, go into Pipelines and run a new Pipeline off the default branch.

You can validate the deploy worked by looking at the Kubernetes dashboard, or accessing the URL.

To retrieve the URL:

minikube service production

View Performance metrics

Go to Operations ➔ Environments then click on an Environment. You should see a new button appearing that looks like a chart. Click on it to view the metrics.

It may take 30-60 seconds for the Prometheus server to get a few sets of data points.

Configuring multiple minikube instances

Use the --profile or -p flag to define the minikube machine name. This allows multiple instances to run simultaneously. For instance, running a minikube instance for working in GitLab CE and GitLab EE at the same time can be accomplished by using all of the same commands outlined above with the additional --profile flag added:

For macOS:

minikube start --vm-driver hyperkit --disk-size=20g --profile ce-instance minikube start --vm-driver hyperkit --disk-size=20g --profile ee-instance

To get the CE instance IP:

minikube ip --profile ce-instance

To look at the EE instance dashboard:

minikube dashboard --profile ee-instance

Electing to use a specified machine name means appending the --profile flag and name to each minikube command you would like to execute. Without the flag, minikube assumes you mean the default instance named minikube. All machines are stored by default in ~/.minikube/machines.

Monitoring the GDK with Prometheus

If you just want to monitor GitLab with Prometheus in the GDK, the process is simpler:

1. Ensure you have created a loopback interface for the GDK. Update the default value of gdk.test:172.16.123.1 as necessary.

2. Enable Prometheus in gdk.yml and configure the extra hosts with your loopback configuration:

   prometheus:  enabled: true  extra_hosts:  - gdk.test:172.16.123.1

3. If you want to scrape Sidekiq metrics, update your gdk.yml as follows:

   gitlab:  rails:  rails_background_jobs:  sidekiq_exporter_enabled: true

4. Reconfigure the GDK with gdk reconfigure and restart with gdk restart.

5. After Prometheus has started (on port 9090 by default), you can see the scrape status at http://gdk.test:9090/targets. After the gitlab target has been scraped once, its metrics are ready for querying (for example, using Grafana).

Using Grafana with the GDK

To use Grafana with the GDK:

1. Enable Grafana in gdk.yml and configure the extra hosts with your loopback configuration:

   grafana:  enabled: true

2. Reconfigure the GDK with gdk reconfigure and restart with gdk restart.

3. Follow the prompt in the console to access Grafana, or go to gdk.test:4000. The initial username and password are admin:admin. To change the password, follow the prompt.

4. Go to http://gdk.test:4000/datasources to configure the Prometheus datasource and select Add data source.

5. Select Prometheus.

6. In the HTTP > URL text box, enter gdk.test:9090.

7. Select Save and test.

Grafana is now ready to use.

Third-party integrations

We support metric integrations from third parties, such as Grafana.

For instructions of how to use this functionality in GDK, see Accessing Prometheus via external services.