Skip to content
Build I.T.(a.k.a The Homelab Club)

Syslog (w/ Grafana)

Syslog (w/ Grafana)

Guide on setting up syslog with Grafana Loki and Nginx

What you will need

This project assumes that you have already followed our Raspberry Pi installation project, and have the resulting configured Raspberry Pi. If you are installing on different hardware or a different OS, these steps may not all be the same for you.

System Preparation

Before continuing with the project, make sure your system is up to date.

Updating System
sudo apt update
sudo apt upgrade -y

Install a text editor of your choice. Here we use vim as an example.

Installing Vim
sudo apt install vim

Concepts

What is Log Management?

Log management is the process of centrally collecting, parsing, and store log data. It can be used to trace issues across hosts and automatically trigger alerts based on the logs.

What is Syslog?1/2

Standard for message logging to allow different applications to generate messages to be parsed and stored by other systems. Syslog typically runs on port 514 commonly on UDP but also can be TCP depending on how critical the logs are.

What is Syslog?2/2

The message contains the following:

  • Facility: what type of system generated the message
  • Severity: from 0 (Emergency) to 7 (Debug)
  • Message: contains two parts the tag and content
    • Tag: name of the program/process that generated the message
    • Content: the actual log message with event details

Grafana Loki and Alloy1/3

There are multiple different possible syslog collection and management solutions such as Splunk, Graylog, and ELK.

Grafana Loki Pros and Cons:

  • Pros:
    • Lower storage requirements as only the labels are indexed
    • Lower resource requirements during log ingestion due to less being indexed
    • Easy Grafana integration for visualization and connections to other Grafana tools

Grafana Loki and Alloy2/3

  • Cons:
    • Searches for a random word or broad searches may take longer and use more resources
    • Limited log querying capabilities
    • Too many labels can cause issues and increase resource requirements

Grafana Loki and Alloy3/3

Grafana Alloy sits between the application (Nginx) and log collection server (Grafana Loki). It collects and forwards logs from Nginx to Loki to be parsed, stored, and queried. Other tools such as Rsyslog or the Splunk Universal Forwarder, can perform similar log collection for other colleciton servers.

Package Installation1/2

We will need the following packages for this project:

  • Nginx: Webserver which will be used to generate logs
  • Grafana Alloy: Collect and forward logs
  • Grafana Loki: Store and parse logs
  • Grafana: visualization and querying
  • A few other linux packages to download the Grafana repo

Package Installation2/2

Follow the instructions on Grafana website to add the stable Grafana repository

Install the other Grafana packages:

Installing Loki and Alloy
sudo apt install grafana loki alloy

Setting up the services

Verify Loki

  1. Start the loki service
    Start Loki Service
    sudo systemctl start loki
  2. Check the status if reports an error go to step 4 otherwise skip to setting
    Check Loki Status
    systemctl status loki
  3. Check the journal for what the issue is
    Check Loki Journal
    sudo journalctl -xeu loki
  4. If the issue is failed to determine user credentials then create the loki user and go back to step 1
    Create Loki User
    # -M do not create home directory
    # -r create system account
    # -s /sbin/nologin disables normal login
    sudo useradd -M -r -s /sbin/nologin loki

Setting up Grafana

  1. Check grafana service

    Check Grafana Status
    systemctl status grafana-server

  2. If it is inactive then start the service

    Start Grafana Server
    sudo systemctl start grafana-server

  3. Go to Grafana through the browser going to localhost:3000 and log in with default credentials username admin and password admin

  4. Change the password to something you will remember

  5. Add the Loki source by going to connections -> add new connection -> Loki -> Add new data source

  6. Set url to http://localhost:3100 then scroll down to the bottom and click save and test

  7. Click explore on the left menu and go to loki to ensure that it is working

Configuring Alloy1/2

  1. Edit the Alloy configuration

    Edit Alloy
    sudo vim /etc/alloy/config.alloy

  1. Add the following configurations at the bottom of the file to have it take in and forward logs from Nginx

    Configure Alloy
    loki.source.syslog "nginx" {
        listener {
                address = "127.0.0.1:1514"
                protocol = "udp"
                syslog_format = "rfc3164"
                labels = {
                        job = "nginx",
                }
        }
        forward_to = [loki.write.local.receiver]
    }
    loki.write "local" {
        endpoint {
            url = "http://127.0.0.1:3100/loki/api/v1/push"
        }
    }

Configuring Alloy2/2

  1. Save and exit the file then check the configuration using Alloy validate

    Validate Alloy COnfiguration
    sudo alloy validate /etc/alloy/config.alloy

  2. If there are no errors restart Alloy

    Restart Alloy
    sudo systemctl restart alloy

Configuring Nginx

  1. Install Nginx

    Install Nginx
    sudo apt install nginx

  2. Edit the Nginx default site

    Edit Default Site
    sudo vim /etc/nginx/sites-enabled/default

  3. In the http block add the following to enable Nginx built-in logging syslog

    error_log syslog:server=127.0.0.1:1514;
    access_log syslog:server=127.0.0.1:1514;

  4. Save and exit then check if the configuration is valid

    Check Nginx Configuration
    sudo nginx -t

  5. Reload Nginx for it to pull the updated file

    Reload Nginx
    sudo systemctl reload nginx

  6. Go to the default site to generate an access log by going to localhost in the browser

View the Logs

  1. Go back to Grafana at localhost:3000 -> explore -> then if it is not already on Loki set it to Loki

  2. Switch from builder to code in the query and put in {job="nginx"} to get all the logs that match the job label nginx

  3. You should see something like below

    2026-11-10 13:05:53.499 unk 127.0.0.1 - - [10/Nov/2026:13:05:53 -0500] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:153.0) Gecko/20100101 Firefox/153.0"

  4. Click on the log then expand the subfields to see more of the data. Note the indexed label job which is what we used to query for the log.

  1. You can also use Wireshark to view the UDP packets sent after the HTTP request is made since the default does not encrypt the log messages

Up next

Wireguard
Switch to normal mode

Contents