Grafana

󰃭 2024-10-24

Install and deploy the Grafana server

On Fedora/RHEL systems:

sudo dnf install -y grafana grafana-selinux chkconfig

On Debian systems:

sudo apt-get install -y apt-transport-https software-properties-common
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana

Reload the systemctl daemon, start and enable grafana.service:

sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server.service
sudo systemctl status grafana-server.service

Configure Grafana SELinux policy

This is not necessary on AlmaLinux 9, Rocky Linux 9, RHEL 9.

For some reason the grafana-selinux package does not provide what Grafana needs to cooperate with SELinux. It’s therefore necessary to use a third-party repository at https://github.com/georou/grafana-selinux to compile and install a proper SELinux policy module for Grafana.

# Clone the repo
git clone https://github.com/georou/grafana-selinux.git
cd grafana-selinux

# Copy relevant .if interface file to /usr/share/selinux/devel/include to expose them when building and for future modules.
# May need to use full path for grafana.if if not working.
install -Dp -m 0664 -o root -g root grafana.if /usr/share/selinux/devel/include/myapplications/grafana.if

# Compile and install the selinux module.
sudo dnf install -y selinux-policy-devel setools-console policycoreutils-devel
sudo make -f /usr/share/selinux/devel/Makefile grafana.pp
sudo semodule -i grafana.pp

# Add grafana ports
semanage port -a -t grafana_port_t -p tcp 3000

# Restore all the correct context labels
restorecon -RvF /usr/sbin/grafana-* \
		/etc/grafana \
		/var/log/grafana \
		/var/lib/grafana \
		/usr/share/grafana/bin

# Start grafana
systemctl start grafana-server.service

# Ensure it's working in the proper confinement
ps -eZ | grep grafana

username: admin
password: password (change this after)

Add Prometheus data source

Bar menu
Data sources
Add new data source
Choose Prometheus data source
Name: Prometheus
URL: http://localhost:9090
Save & test

Ensure the data source is working before continuing.

If you’re running Grafana on an SELinux host, set an SELinux boolean to allow Grafana to access the Prometheus port:

sudo setsebool -P grafana_can_tcp_connect_prometheus_port=1

Add Loki data source

Since Loki is running on hyperreal.coffee:3100, the Firewall’s internal zone on that host needs to allow connection to port 3100 from my IP address.

sudo firewall-cmd --zone=internal --permanent --add-port=3100/tcp
sudo firewall-cmd --reload

In the Grafana panel:

Bar menu
Data sources
Add new data source
Choose Loki data source
Name: Loki
URL: http://hyperreal.coffee:3100
Save & test

Ensure the data source is working before continuing.

Add Node Exporter dashboard

Visit the Grafana Dashboard Library.
Search for “Node Exporter Full”.
Copy the ID for Node Exporter Full.
Go to the Grafana panel bar menu.
Dashboards
New > Import
Paste the Node Exporter Full ID into the field, and press the Load button.

Add Caddy dashboard

Visit Caddy Monitoring on the Grafana Dashboard Library.
Copy the ID to clipboard.
Go to the Grafana panel bar menu.
Dashboards
New > Import
Paste the Caddy Monitoring ID into the field, and press the Load button.

Add qBittorrent dashboard

Visit qBittorrent Dashboard on Grafana Dashboard Library.
Copy the ID to clipboard.
Go to the Grafana panel bar menu.
Dashboards
New > Import
Paste the qBittorrent Dashboard ID into the field, and press the Load button.

Use HTTPS with Tailscale

sudo tailscale certs HOSTNAME.TAILNET.ts.net
sudo mkdir /etc/tailscale-ssl-certs
sudo mv *.key /etc/tailscale-ssl-certs/
sudo mv *.crt /etc/tailscale-ssl-certs/
sudo cp -v /etc/tailscale-ssl-certs/*.key /etc/grafana/grafana.key
sudo cp -v /etc/tailscale-ssl-certs/*.crt /etc/grafana/grafana.crt
sudo chown root:grafana /etc/grafana/grafana.key
sudo chown root:grafana /etc/grafana/grafana.crt
sudo chmod 644 /etc/grafana/grafana.key
sudo chmod 644 /etc/grafana/grafana.crt

Edit /etc/grafana/grafana.ini:

[server]
protocol = https
http_addr =
http_port = 3000
domain = HOSTNAME.TAILNET.ts.net
enforce_domain = false
root_url = https://HOSTNAME.TAILNET.ts.net:3000
cert_file = /etc/grafana/grafana.crt
cert_key = /etc/grafana/grafana.key

Techne