title: Loki

Loki

Rsyslog forwarding to Promtail and Loki

IMPORTANT
Running Loki and Promtail on the same host as Prometheus makes managing the firewall and network routes easier.

This is roughly what our network looks like:

Main Monitoring Node

Runs Prometheus, Promtail, Loki, and rsyslog.
Traffic must be allowed through the firewall on TCP port 514. If using Tailscale, ensure the ACLs are setup correctly.
It has an rsyslog ruleset that catches all forwarded logs through TCP port 514 and relays them to Promtail on TCP port 1514.
Promtail pushes the logs its receives via TCP port 1514 to the Loki client listening on TCP port 3100.

Regular Node 1

It has an rsyslog ruleset that forwards logs to the Main Monitoring Node on TCP port 514.
Is allowed to access TCP port 514 on the Main Monitoring Node.

Regular Node 2

It has an rsyslog ruleset that forwards logs to the Main Monitoring Node on TCP port 514.
Is allowed to access TCP port 514 on the Main Monitoring Node.

Install Rsyslog, Promtail, and Loki on the Main Monitoring Node

# Debian-based hosts
sudo apt install -y promtail loki rsyslog

# Fedora-based hosts
sudo dnf install -y promtail loki rsyslog

Edit /etc/promtail/config.yml.

server:
  http_listen_port: 9081
  grpc_listen_port: 0

positions:
  filename: /var/tmp/promtail-syslog-positions.yml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:1514
      labels:
        job: syslog
    relabel_configs:
      - source_labels: [__syslog_message_hostname]
        target_label: hostname
      - source_labels: [__syslog_message_severity]
        target_label: level
      - source_labels: [__syslog_message_app_name]
        target_label: application
      - source_labels: [__syslog_message_facility]
        target_label: facility
      - source_labels: [__syslog_connection_hostname]
        target_label: connection_hostname

Edit /etc/loki/config.yml.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

Edit /etc/rsyslog.d/00-promtail-relay.conf.

# https://www.rsyslog.com/doc/v8-stable/concepts/multi_ruleset.html#split-local-and-remote-logging
ruleset(name="remote"){
  # https://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html
  # https://grafana.com/docs/loki/latest/clients/promtail/scraping/#rsyslog-output-configuration
  action(type="omfwd" Target="localhost" Port="1514" Protocol="tcp" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted")
}


# https://www.rsyslog.com/doc/v8-stable/configuration/modules/imudp.html
module(load="imudp")
input(type="imudp" port="514" ruleset="remote")

# https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html
module(load="imtcp")
input(type="imtcp" port="514" ruleset="remote")

Ensure the firewall allows TCP traffic to port 514.

sudo firewall-cmd --permanent --zone=tailnet --add-port=514/tcp
sudo firewall-cmd --reload

Restart and/or enable the services.

sudo systemctl enable --now promtail.service
sudo systemctl enable --now loki.service
sudo systemctl enable --now rsyslog.service

Install and configure Rsyslog on Regular Node 1 and Regular Node 2

# Debian
sudo apt install -y rsyslog

# Fedora
sudo dnf install -y rsyslog

Enable and start the rsyslog service.

sudo systemctl enable --now rsyslog

Edit /etc/rsyslog.conf.

###############
#### RULES ####
###############

# Forward to Main Monitoring Node
*.* action(type="omfwd" target="<IP addr of Main Monitoring Node>" port="514" protocol="tcp"
    action.resumeRetryCount="100"
    queue.type="linkedList" queue.size="10000")

Restart the rsyslog service.

sudo systemctl restart rsyslog.service

In the Grafana UI, you should now be able to add Loki as a data source. Then go to Home > Explore > loki and start querying logs from Regular Node 1 and Regular Node 2.