Home · About · Services · Industries · Projects · Products · Process · Blog · FAQ · Contact
Prometheus and Grafana monitoring for small teams: a practical setup A small, self-hosted Prometheus, Grafana and Alertmanager stack for a handful of servers: what to collect, the few alerts worth having, and how to keep it…
A disk fills up at 2am because a log file nobody rotates has been growing for months. The database stops accepting writes, the app returns errors, and the first anyone hears of it is a customer email the next morning. The server had plenty of warning signs. Nothing was watching them.
For a small team, the fix is not a large observability platform. A Prometheus and Grafana monitoring stack, with Alertmanager and two exporters, covers most of what a team running a few servers needs, and it fits on one small VPS.
The short answer: run Prometheus, Alertmanager and Grafana with Docker Compose on one small server. Install node_exporter on every machine and blackbox_exporter for HTTP and certificate checks. Start with five or six alerts that need a human, send them to one shared channel, and add an external heartbeat so you know if monitoring itself goes down.
What each piece does It helps to know the moving parts before installing anything, because each one fails in its own way.
Prometheus pulls (scrapes) metrics from HTTP endpoints on a schedule, stores them in a local time-series database and evaluates alert rules. It is the only part that holds data. Exporters expose metrics for things that do not speak Prometheus natively. node_exporter covers CPU, memory, disk and network on Linux hosts. blackbox_exporter probes URLs from the outside and reports status codes, latency and TLS certificate expiry. Alertmanager receives firing alerts from Prometheus, groups and de-duplicates them, and sends notifications to email, Slack, Microsoft Teams, Telegram or a webhook. Grafana draws dashboards from Prometheus queries. It is for looking at problems, not for finding them. Alerting belongs in Prometheus rules, kept in version control. Many services can expose Prometheus metrics themselves, sometimes after you switch it on: Traefik, MinIO, Grafana, and most application frameworks through a client library. PostgreSQL, MySQL, Redis and NGINX need a small exporter alongside them.
A compose file for the monitoring server We keep monitoring on its own small server rather than on the machines it watches. If a production host runs out of memory, you want the thing that tells you about it to still be alive. For a handful of servers, a VPS with 2 GB of RAM and some spare disk is enough. Prometheus's own documentation puts storage at roughly one to two bytes per sample, so a small fleet scraped every 30 seconds needs only a few gigabytes for a month of history.
services:
prometheus:
image: prom/prometheus:v3.5.0
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention.time=30d
- --storage.tsdb.retention.size=8GB
volumes:
- ./prometheus:/etc/prometheus:ro
- prom-data:/prometheus
ports:
- 127.0.0.1:9090:9090
restart: unless-stopped
alertmanager:
image: prom/alertmanager:v0.28.1
volumes:
- ./alertmanager:/etc/alertmanager:ro
restart: unless-stopped
blackbox:
image: prom/blackbox-exporter:v0.27.0
restart: unless-stopped
grafana:
image: grafana/grafana:12.1.1
environment:
GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin
GF_USERS_ALLOW_SIGN_UP: "false"
volumes:
- grafana-data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning:ro
secrets:
- grafana_admin
ports:
- 127.0.0.1:3000:3000
restart: unless-stopped
secrets:
grafana_admin:
file: ./grafana_admin.txt
volumes:
prom-data:
grafana-data:Pin image versions and upgrade on purpose; the tags above are examples, so check the current releases when you set this up. Prometheus 3.5 is a long-term support release, which suits a stack you want to leave alone. Both web ports bind to 127.0.0.1, so nothing is reachable from the internet. Reach Grafana through a reverse proxy with TLS and authentication, or over a private network such as Tailscale. The Prometheus UI has no login by default, so never expose it publicly.
Provision the Grafana data source from a file rather than clicking it together, so a rebuilt server comes back identical:
# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: trueCollecting from your servers On each server you want to watch, run node_exporter with the host's network and a read-only view of its filesystem. Bind it to the private or Tailscale address, not to 0.0.0.0, because its metrics reveal a fair amount about the machine.
# on each monitored server
services:
node-exporter:
image: prom/node-exporter:v1.9.1
command:
- --path.rootfs=/host
- --web.listen-address=${PRIVATE_IP}:9100
network_mode: host
pid: host
volumes:
- /:/host:ro,rslave
restart: unless-stoppedThen list the targets in prometheus.yml. Labels such as env and role are worth adding now. They let you write one alert that only pages for production, and filter dashboards without renaming anything later.
global:
scrape_interval: 30s
evaluation_interval: 30s
rule_files:
- /etc/prometheus/rules/*.yml
alerting:
alertmanagers:
- static_configs:
- targets: ['alertmanager:9093']
scrape_configs:
- job_name: node
static_configs:
- targets: ['10.0.0.11:9100', '10.0.0.12:9100']
labels: { env: production }
- targets: ['10.0.0.21:9100']
labels: { env: staging }
- job_name: websites
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets: ['https://example.com', 'https://api.example.com/health']
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox:9115The relabelling block is the standard blackbox pattern: Prometheus scrapes the exporter, and the exporter probes the real URL. The http_2xx module ships in blackbox_exporter's default configuration. Probing a health endpoint that touches the database tells you more than probing a static home page.
The alerts worth having first The most common way small-team monitoring fails is noise. Twenty alerts fire for things that fix themselves, people mute the channel, and the one real alert is missed. Start with a few rules where every firing means someone should act, and use the for clause so brief spikes do not page anyone.
# prometheus/rules/basics.yml
groups:
- name: basics
rules:
- alert: TargetDown
expr: up == 0
for: 3m
labels: { severity: critical }
annotations:
summary: "{{ $labels.instance }} ({{ $labels.job }}) is not responding"
- alert: SiteDown
expr: probe_success == 0
for: 2m
labels: { severity: critical }
annotations:
summary: "{{ $labels.instance }} is failing its HTTP check"
- alert: DiskFillingUp
expr: |
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 24 * 3600) < 0
and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes < 0.2
for: 30m
labels: { severity: warning }
annotations:
summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} will be full within a day"
- alert: LowMemory
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1
for: 10m
labels: { severity: warning }
- alert: CertificateExpiringSoon
expr: probe_ssl_earliest_cert_expiry - time() < 14 * 86400
for: 1h
labels: { severity: warning }
- alert: Watchdog
expr: vector(1)
labels: { severity: none }The disk rule predicts from the last six hours of growth instead of using a fixed threshold, so a disk sitting at 85% for a year stays quiet while one filling fast fires early. The certificate rule catches the renewal that silently stopped working, a common and entirely avoidable outage. Check rule files with promtool, which ships in the Prometheus image, then reload without a restart by sending Prometheus a SIGHUP:
docker compose exec prometheus promtool check rules /etc/prometheus/rules/basics.yml
docker compose kill -s SIGHUP prometheusRouting alerts so someone acts Send alerts to one shared channel the whole team can see, not to a personal inbox. Group them so a host going down produces one message, not ten. Then deal with the question every self-hosted setup has to answer: who tells you when the monitoring server is down?
# alertmanager/alertmanager.yml
route:
receiver: team
group_by: [alertname, instance]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers: [ 'alertname = "Watchdog"' ]
receiver: heartbeat
group_wait: 0s
group_interval: 1m
repeat_interval: 1m
receivers:
- name: team
slack_configs:
- api_url_file: /etc/alertmanager/slack_webhook
channel: '#alerts'
send_resolved: true
- name: heartbeat
webhook_configs:
- url: https://hc-ping.com/your-check-uuidThe Watchdog rule is always firing. Alertmanager forwards it roughly every minute to an external heartbeat service such as Healthchecks.io. The heartbeat route sets group_interval as well as repeat_interval, because Alertmanager only resends on group_interval ticks; leave it at 5m and the pings arrive every five minutes. Set the check period on the heartbeat service to match, with a few minutes of grace. If those pings stop, because Prometheus, Alertmanager or the whole server has died, the external service alerts you by email or phone. It takes ten minutes to set up and closes the biggest hole in a self-hosted stack.
Dashboards, logs and the hosted alternative For dashboards, import the community Node Exporter Full dashboard (ID 1860 on grafana.com) and one blackbox dashboard, then stop. Build your own only when a real incident shows you a question the existing ones cannot answer. Export dashboards you care about as JSON and keep them in the same repository as the rules.
Metrics tell you that something is wrong. Logs usually tell you why. When you are ready for logs, Loki with Grafana Alloy as the collector fits the same stack. Alloy replaced Grafana Agent, which has reached end of life, so do not start new setups on the old agent.
If nobody wants to look after a monitoring server, Grafana Cloud has a free tier that accepts metrics pushed from Alloy, and paid tiers beyond it. Check its current limits against your series count. The trade-off is the usual one: less to run, but your data and alert routing live with a vendor. Teams with a handful of servers and someone comfortable with Docker generally do well self-hosting. Teams without that person often do better on a hosted service or with someone running it for them.
A checklist before you call it done Monitoring runs on a separate server, with Prometheus and Grafana not exposed to the internet. Every production server runs node_exporter on a private address, and every public URL has a blackbox probe. Alerts exist for targets down, sites down, disk filling, low memory and certificates expiring. Alerts go to a shared channel, are grouped, and include enough detail to start investigating. A Watchdog heartbeat goes to an external service that alerts if it stops. Config, rules and dashboards are in version control and the stack can be rebuilt from them. You have tested each alert by actually breaking something on staging: stop a service, fill a disk. That last item is the one people skip. An alert you have never seen fire is an alert you do not know works. If the stack is running but alerts are noisy, ignored or untested, or if nobody on the team has time to own it, that is the point where it is worth getting help.
Infigini IT Services · 11 Kanwali Road, Dehradun 248001, India · info@infigini.com · +91 95684 87789