Mahdi Bagheri's Blog
Start Infrastructure Monitoring with Prometheus and Grafana

Start Infrastructure Monitoring with Prometheus and Grafana

Mahdi Bagheri's photo
Mahdi Bagheri

Introduction

To ensure the health, performance and availability of our systems we need a way to monitor and observe our machines in use. Prometheus and Grafana are two of the most widely used solutions to achieve this goal.

Prometheus is an open-source monitoring solution which can collect data from multiple endpoints, query data with its PromQL (Prometheus Query Language), evaluate the data and trigger alerts based on given rules.

Grafana is an open-source monitoring solution as well, but it doesn’t collect the data from the endpoints directly. It uses data sources like Prometheus that already scraped the necessary data and it allows us to create visually appealing dashboards in which we can see the collected data through which we can make informed decisions about the state of our machines. Possible data to display are for example different metric values or logs and Grafana provides also the ability to create alerts based on given rules.

In the following I will guide you through the steps on how to quickly start with a basic Prometheus and Grafana setup. And we will also collect our first data from the host machine on which Prometheus and Grafana are running with the help of the Node exporter and present the data inside a dashboard.

Prerequisites

To follow this process for setting up Prometheus and Grafana, it is recommended to have some basic experience with Linux and Docker.

I will perform these steps on an Ubuntu server, an I am assuming you have already set up your own Linux-based environment with Docker installed.

Folder Structure

The following shows the folder structure of the directories and files in use to quickly start a basic Prometheus and Grafana environment.

/opt/monitoring
|---.env
|---docker-compose.yaml
|---prometheus
|   |---prometheus.yaml
|
|---grafana
    |
    |---provisioning
        |    
        |---dashboards
        |   |---dashboards.yaml
        |   |---node-exporter-full-dashboard.json
        |   
        |---datasources
            |---datasource.yaml

In the presented folder structure, the monitoring stack is organized within the /opt/monitoring directory, where everything needed for the Prometheus and Grafana setup is stored. This structure is designed to automate much of the configuration, ensuring that you can easily replicate the setup on different machines without having to manually adjust these settings through the web interfaces.

Inside the monitoring directory there are two files .env, which holds the credentials for logging into Grafana and the docker-compose.yaml file, which is used to define and start the services via Docker Compose.

Within the monitoring directory there are two subdirectories: prometheus and grafana. The prometheus folder contains the prometheus.yaml file, which is the primary configuration file for Prometheus. It defines how and from where Prometheus scrapes metrics data. The grafana folder contains a dashboard .json file, as well as the the necessary configuration files dashboard.yaml, which specifies the dashboards that should be automatically loaded when Grafana starts and the datasource.yaml file through which Prometheus is already provided as a data source.

Docker Compose

Let’s start with the longest file first.

The docker-compose.yaml file defines three services: Prometheus, Grafana and the Node exporter.
They are configured so they can communicate through the same network named monitoring. Also two volumes are defined, one for Prometheus and one for Grafana, ensuring persistent data storage for each service.

Prometheus Service

We use the latest Prometheus Docker image and configure the service the automatically restart, unless stopped manually. In the volumes section, we mount the prometheus-data Docker volume to ensure persistent storage of Prometheus data, and we also provide our custom prometheus.yaml configuration file. In the command section, we specify the location of the Prometheus configuration file and set the storage path to /prometheus where Prometheus will store the collected time series data. The service exposes the port 9090 to the host machine and is connected to the monitoring network.

Grafana Service

We use the latest Grafana Docker image and configure the container to restart automatically, unless manually stopped. In the volumes section, we mount the grafana-data Docker volume to persist Grafana’s data and also mount the dashboard and datasources folders into the container containing the pre-downloaded dashboards and .yaml configuration files. In the environment section we set the admin username and password using environment variables stored in the .env file. Additionally we disable the option for sign-up, ensuring only the admin can create new users. The service exposes port 3000 to the host machine and is connected to the monitoring network.

Node exporter Service

We use the latest Node exporter Docker image and configure the container to restart automatically unless manually stopped. In the volumes section we grant the container read-only access to the hosts’s /sys, /proc, and / mount points, enabling it to collect system metrics. Through the command section we configure the Node exporter to gather metrics from these specific mount points. This service exposes the port 9100 to the host machine and is connected to the monitoring network.

services:
  prometheus:
    image: prom/prometheus:latest
    restart: unless-stopped
    volumes:
      - ./prometheus/prometheus.yaml:/etc/prometheus/prometheus.yaml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yaml'
      -  "--storage.tsdb.path=/prometheus"
    ports:
      - 9090:9090
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    restart: unless-stopped
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards
      - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources
    environment:
      - GF_SECURITY_ADMIN_USER=${GF_ADMIN_USER}
      - GF_SECURITY_ADMIN_PASSWORD=${GF_ADMIN_PW}
      - GF_USERS_ALLOW_SIGN_UP=false
    ports:
      - 3000:3000
    networks:
      - monitoring

  node-exporter:
    image: prom/node-exporter:latest
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.rootfs=/rootfs"
      - "--path.sysfs=/host/sys"
    ports:
      - 9100:9100
    networks:
      - monitoring



networks:
  monitoring:
    driver: bridge

volumes:
  prometheus-data:
  grafana-data:

.env

The .env file is very small consisting of two variables.

GF_ADMIN_USER=<username>
GF_ADMIN_PW=<password>

prometheus.yaml

The prometheus.yaml file is structured into two main sections: global and scrape_configs.

In the global section, we define configurations that apply globally across all jobs. In this case, we set both the scrape_interval and evaluation_interval to 15 seconds. This means Prometheus will check every 15 seconds for new data to scrape and will evaluate the collected data at the same interval.

The scrape_configs section defines the various target from which Prometheus will collect data. Each target is associated with a unique job name for easier identification. In this example the first job is named prometheus with the target localhost:9090, which points to the Prometheus service itself. The second job is named node-exporter-local, with the target node-exporter:9100, referring to the Node exporter service. Since Prometheus and Node exporter are both part of the same monitoring network, Prometheus can directly access the Node exporter’s endpoint.

For more detailed configuration options to fine-tune the prometheus.yaml, please refer to the Prometheus Documentation.

global:
  scrape_interval:  15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ['localhost:9090']
  - job_name: "node-exporter-local"
    static_configs:
      - targets: ['node-exporter:9100']

datasource.yaml

To avoid manually adding the Prometheus service as a data source in Grafana after startup, we can create a file called datasource.yaml and provide it to Grafana. This configuration automatically connects Grafana to Prometheus when it starts.

For more information on how to further adjust the datasource.yaml, please refer to the Grafana documentation.

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    basicAuth: false
    isDefault: true
    editable: true

dashboards.yaml

The dashbaord.yaml file is used to define how dashboards are provisioned in Grafana. Through this file we specify the location and settings for the dashboards that will be automatically loaded when Grafana starts.

For more information on how to adjust this configuration file to your needs please refer to the Grafana documentation at the Dashboards section.

apiVersion: 1

providers:
  - name: 'MyDashboards'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    alloUiUpdates: true
    editable: true
    options:
      path: /etc/grafana/provisioning/dashboards

Retrieving existing Dashboard(s)

On the following site you can download a pre-configured dashboard that is tailored to visualize the metrics collected by the Node exported. We save the .json file for this dashboard into the dashboards folder. Due to our Docker Compose configuration, Grafana will automatically detect and load the dashboard at runtime.

Start the services

Now that all the necessary files are prepared, you can start the services with a simple command:

docker compose up

Depending on your version of Docker Compose you might need to use a hyphen in the command:

docker-compose up

Wait a moment for the services to start, and then navigate to the Grafana web interface at localhost:3000.

Log in using the credentials defined in the .env file. Once logged in you should see the Prometheus data source listed under Connections → Data Sources, and the Node exporter dashboard should be available under Dashboards.

What’s next?

Your next steps could involve identifying additional areas from which you want to collect data. For example, you could integrate cAdvisor into your existing Docker Compose setup to gain insight into the performance and resource usage of your running containers. Additionally, you may want to explore the creation of alerts for specific events, ensuring that you’re notified when critical conditions arise, allowing you to act quickly and effectively.

#prometheusGrafanamonitoringDevops