Overview
Infrastructure fails quietly. CPU saturates, memory leaks accumulate, and disks fill up long before anyone notices, and by the time a user reports the problem the damage is already done. The difference between a minor incident and an outage is almost always how quickly the right person finds out.
This guide walks through a lightweight, reproducible monitoring and alerting stack built on Grafana, Prometheus, and Docker. When a monitored metric crosses a threshold you define, Grafana sends an email to the addresses you specify, turning silent degradation into an actionable notification. The entire stack runs from a handful of config files and a single docker-compose up, so it is easy to version, review, and redeploy across environments.
Table of Contents
Why This Stack
Prometheus scrapes and stores time-series metrics, Node Exporter exposes host-level metrics (CPU, memory, disk, network), and Grafana provides dashboards plus a built-in alerting engine. Running all three under Docker Compose keeps the setup declarative and portable: the same three files reproduce the stack on any host, with no manual installation steps to drift out of sync. SMTP integration then delivers alerts straight to email, which requires no additional infrastructure and reaches on-call staff wherever they are.
Setup
1. Check your local IP address
Node Exporter runs on the host, so Prometheus needs the host's local IP as its scrape target.
$ ifconfig
# copy local ip address 4 at eno# section
# Put it into targets at prometheus.yml
2. Start up Grafana and Prometheus
With the three config files in place, bring the stack up in the background.
$ tree
.
├── docker-compose.yml
├── grafana.ini
└── prometheus.yml
$ docker-compose up -d3. Log in to Grafana
Open http://localhost:3000 and sign in with the default credentials admin / admin. Change the password on first login.
4. Identify the Prometheus data source
Confirm the containers are running and note the Prometheus container name.
$ docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------------------------------------
servermonitoring_grafana_1 /run.sh Up 0.0.0.0:3000->3000/tcp,:::3000->3000/tcp
servermonitoring_node-exporter_1 /bin/node_exporter Up 0.0.0.0:9100->9100/tcp,:::9100->9100/tcp
servermonitoring_prometheus_1 /bin/prometheus --config.f ... Up 0.0.0.0:9090->9090/tcp,:::9090->9090/tcpWhen adding the Prometheus data source in Grafana, use the container's internal address rather than localhost, since Grafana reaches Prometheus over the Docker network. In this case, set http://servermonitoring_prometheus_1:9090.
5. Configure email delivery (SMTP)
5-1. Get the Grafana container name/ID
# Find Grafana container
$ docker ps -a5-2. Issue an app password for Gmail
Gmail requires an app-specific password for SMTP rather than your account password. Generate one here: https://support.google.com/mail/answer/185833?hl
5-3. Edit grafana.ini
Copy the config out of the container, edit it, and copy it back.
$ sudo docker cp grafana_container_id:/etc/grafana/grafana.ini ./
$ sudo vi grafana.ini
$ sudo docker cp ./grafana.ini grafana_container_id:/etc/grafana/grafana.iniThe default SMTP section ships fully commented out:
#################################### SMTP / Emailing ##########################
[smtp]
;enabled = false
;host = localhost:25
;user =
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
;password =
;cert_file =
;key_file =
;skip_verify = false
;from_address = admin@grafana.localhost
;from_name = Grafana
# EHLO identity in SMTP dialog (defaults to instance_name)
;ehlo_identity = dashboard.example.com
# SMTP startTLS policy (defaults to 'OpportunisticStartTLS')
;startTLS_policy = NoStartTLS
# Enable trace propagation in e-mail headers, using the 'traceparent', 'tracestate' and (optionally) 'baggage' fields (defaults to false)
;enable_tracing = false
[smtp.static_headers]
# Include custom static headers in all outgoing emails
;Foo-Header = bar
;Foo = bar
[emails]
;welcome_email_on_sign_up = false
;templates_pattern = emails/*.html, emails/*.txt
;content_types = text/htmlEnable it and point it at Gmail's SMTP endpoint, substituting your address and app password:
#################################### SMTP / Emailing ##########################
[smtp]
enabled = true
host = smtp.gmail.com:587
user = example@gmail.com
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
password = """your_pw"""
;cert_file =
;key_file =
skip_verify = false
from_address = example@gmail.com
from_name = Grafana5-4. Mount grafana.ini in docker-compose.yml
So the SMTP config survives container restarts, mount it as a volume instead of relying on the in-container copy.
grafana:
image: grafana/grafana
volumes:
- ./grafana.ini:/etc/grafana/grafana.ini5-5. Restart and verify
Recreate the stack and confirm the SMTP settings are loaded.
$ docker-compose down
$ docker-compose up -d
$ docker-compose exec grafana cat /etc/grafana/grafana.ini | grep -A 20 "\[smtp\]"6. Send a test email
With SMTP configured, use Grafana's contact point test to send a verification email and confirm end-to-end delivery before relying on it for real alerts.
Tips and Gotchas
- Use the container name, not
localhost, for the data source. Inside the Docker network Grafana resolves Prometheus by container name;localhostpoints back at the Grafana container itself and will fail. - Gmail needs an app password. Standard account passwords are rejected for SMTP. Generate a dedicated app password and wrap it in triple quotes if it contains
#or;. - Mount grafana.ini as a volume. Editing the file inside the container works for a first pass, but the change is lost when the container is recreated. Volume-mounting makes the configuration persistent and version-controllable.
- Verify before you trust. Always send a test email and grep the loaded config after restart, so you discover misconfiguration during setup rather than during an incident.
Reference Configuration
- docker-compose.yml
/**
* Copyright 2024 Ryo M
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
version: '3'
services:
prometheus:
image: prom/prometheus
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- '9090:9090'
grafana:
image: grafana/grafana
ports:
- "3000:3000"
node-exporter:
image: quay.io/prometheus/node-exporter
ports:
- 9100:9100
volumes:
- ./proc:/host/proc
- ./sys:/host/sys
- ./rootfs:/rootfs- prometheus.yml
/**
* Copyright 2024 Ryo M
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
global:
scrape_interval: 15s
external_labels:
monitor: 'codelab-monitor'
scrape_configs:
- job_name: 'node'
scrape_interval: 5s
static_configs:
- targets: ['your_local_ip:9100']References
Official
Community
- https://amateur-engineer-blog.com/grafana-prometheus-dashboard
- https://qiita.com/shinfrom/items/46b6174879e97dcab568
- https://stackoverflow.com/questions/30172605/how-do-i-get-into-a-docker-containers-shell
Need help setting up monitoring and alerting for your infrastructure? Get in touch.