Monitoring Stack Based on Prometheus¶

In this article

• Monitoring Stack Based on Prometheus. Main Features
• Features of the assembly
• Getting started after deploying Prometheus
• Prometheus Startup Screen
• Setting up a Telegram Bot
• Grafana
• Ordering a server with Prometheus using the API

Monitoring Stack Based on Prometheus. Main Features¶

• Metric Collection: Prometheus gathers metrics from various sources, including servers, containers, and services through Node Exporter, Blackbox Exporter, and cAdvisor. This encompasses both system metrics (CPU load, RAM usage, disk space) and application-specific metrics.
• Data Visualization: Integration with Grafana enables the creation of informative dashboards for visualizing collected metrics in the form of graphs, histograms, and other visualization methods.
• Alerting: The combination of Alertmanager and Prometheus provides a robust alerting system that triggers notifications when predefined threshold conditions are met. Alerts can be sent via various channels, including email, Telegram (via integration with a Telegram bot), and more.
• SSL Certificate Management: Nginx in tandem with Certbot automates the procurement and renewal of Let's Encrypt SSL certificates for web services.
• Service Availability Monitoring: Blackbox-Exporter enables active checks on the availability of various websites, APIs, and TCP services through scheduled queries.
• Docker Monitoring: cAdvisor provides detailed information about resource consumption by Docker containers within this environment.
• Open-Source Codebase: Prometheus is an open-source project with an active community of developers and users."

Features of the assembly¶

• Installation is possible on Ubuntu 20.04+, AlmaLinux 8/9, and RockyLinux 8/9;
• The installation time is approximately 15-20 minutes along with the operating system;
• Primary configuration files for Prometheus and Alertmanager are located in the /root/prometheus directory;
• The Telegram bot container for Prometheus uses the tienbm90/prometheus-bot:0.0.1 image, mounting two volumes:
  • /root/prometheus_bot/tg_bot for storing Telegram bot files;
  • /root/prometheus_bot/tg_bot/config.yml for storing the configuration file;
• The Telegram bot container uses port 9087 and always restarts upon shutdown;
• The Alertmanager container for Prometheus uses the prom/alertmanager:v0.24.0 image, connected to the prometheus_ bot container;
  • Uses port 9093 for accessing the web interface;
  • Mounts a volume ./alertmanager for storing configuration files and restarts upon manual shutdown;
• The Grafana container uses the grafana/grafana:10.1.9 image, mounting two volumes:
  • /data/grafana/datasources for storing data sources;
  • /var/lib/docker/volumes/grafana-data for storing Grafana data;
• Uses port 3000 for accessing the web interface and restarts upon manual shutdown;
• The Node Exporter container uses the prom/node-exporter image, mounting volumes /proc, /sys, and / for accessing system metrics;
  • Uses port 9100 for collecting metrics and always restarts upon shutdown;
• The cAdvisor container uses the gcr.io/cadvisor/cadvisor: image, mounting volumes /, /var/run, /sys, and /var/lib/docker/ for accessing Docker metrics;
  • Uses port 8080 for accessing the web interface and always restarts upon shutdown;
• The Blackbox Exporter container uses the prom/blackbox-exporter:v0.25.0 image, mounting a volume ./blackboxexporter/ for storing configuration files;
  • Uses port 9115 for accessing the exporter and always restarts upon shutdown;
• The Nginx and Certbot container uses the jonasal/nginx-certbot image, setting the environment variable CERTBOT_EMAIL;
  • Loads environment variables from the file /data/nginx/nginx-certbot.env;
  • Uses ports 80 and 443 for HTTP and HTTPS;
  • Mounts two volumes:
    • /var/lib/docker/volumes/nginx_secrets for storing SSL certificates;
    • /data/nginx/user_conf.d for storing user configurations for Nginx;
• Always restarts upon manual shutdown.

The web interface will be accessible at: https://{Server_ID_from_Invapi}.hostkey.in

Access URLs for Individual Applications:

• Alertmanager: http://<Server_IP_from_Invapi>:9093;
• Grafana: http://<Server_IP_from_Invapi>:3000. Upon first authentication, use credentials - admin/admin;
• Node Exporter: http://<Server_IP_from_Invapi>:9100;
• cAdvisor: http://<Server_IP_from_Invapi>:8080;
• Blackbox Exporter: http://<Server_IP_from_Invapi>:9115;

Getting started after deploying Prometheus¶

The service is managed in the Prometheus dashboard with a web interface, the link to which is located in the Invapi:

You can find the authorization data either in the Info >> 'Tags tab of the server control panel or in the email that was sent.

• Link: in the webpanel tag;
• Login and Password: in a letter to your e-mail upon receipt of the server;

Prometheus Startup Screen¶

The startup page of Prometheus presents an intuitive user interface containing numerous functional elements for effective monitoring and analysis of metric data. On this page, you will find a menu for navigation to key sections, settings for personalizing the interface, a field for entering queries to metric data, as well as panels for displaying query results in table or graph format.

Menu Items:

• Alerts - shows current alerts and notifications generated by Prometheus rules.
• Graph - allows visualizing and analyzing metric data using graphs.
• Status - displays the overall state of the Prometheus monitoring system and its components.
• Help - provides access to documentation and reference materials for Prometheus.

Settings:

• Use local time - display time on graphs in the user's local timezone.
• Enable query history - save query history for reuse.
• Enable autocomplete - enable auto-complete when entering queries.
• Enable highlighting - highlight syntax in queries.
• Enable linter - enable syntax checking of queries.

The line Expression (press Shift+Enter for newlines) with the Execute button is a field for entering queries to metric data, allowing multi-line queries and executing them by clicking the Execute button.

The Table and Graph buttons allow you to choose how to display the query results: as a table (Table) or graph (Graph).

Buttons:

• Add Panel - add a new panel to display the query result.
• Remove Panel - remove an existing panel.

Setting up a Telegram Bot¶

In this setup, there is a Telegram bot that can be immediately configured for use if needed. The bot's configuration files are located in the directory /root/prometheus_bot. A detailed instruction on setting up the bot can be found in the developers' documentation.

Grafana¶

Grafana is a software for visualizing and monitoring data in real-time. It can be used to display system monitoring metrics, IoT data, and log events. Supporting numerous data sources, including databases (e.g., MySQL, PostgreSQL), cloud provider services (e.g., AWS, Google Cloud), and monitoring systems (e.g., Prometheus). Grafana offers a flexible dashboard configuration for creating personalized interfaces, as well as a wide range of plugins and integrations with other tools.

The web interface will be available at: http://{Server_IP_from_Invapi}:3000.

Ordering a server with Prometheus using the API¶

curl -s "https://invapi.hostkey.com/eq.php" -X POST \
--data "action=order_instance" \
--data "token=" \
--data "deploy_period=" \
--data "deploy_notify=" \
--data "email=" \
--data "pin=" \
--data "id=" \
--data "os_id=" \
--data "root_pass=" \
--data "hostname=" \
--data "ssh_key=" \
--data "post_install_callback=" \
--data "post_install_script=" \
--data "reinstall_key=" \
--data "os_name=" \
--data "imitate_deploy=" \
--data "imitate_error=0" \
--data "own_os=" \
--data "jenkins_task=" \
--data "traffic_plan=" \
--data "invoice_relid=" \
--data "preset=" \
--data "location_name=" \