Application logs, server logs, and infrastructure logs often contain the first clues when something breaks. The problem is not a lack of data, but the effort required to read and understand it.

Engineers usually scroll through thousands of lines, search for error codes, and try to connect events across time. This is slow and error-prone, especially during incidents.

A LogAnalyzer Agent solves this problem by acting like a calm, experienced engineer who reads logs for you and explains what is going on.

In this article, you’ll learn how to build such an agent using FastAPI, LangChain, and an OpenAI model.

We’ll walk through the backend, the log analysis logic, and a simple web UI that lets you upload a log file and get insights in seconds. We’ll also upload this app to Sevalla so that you can share your project with the world.

You just need some basic knowledge of Python and HTML/CSS/JavaScript to finish this tutorial.

Here is the full code for reference.

What We’ll Cover

• What a LogAnalyzer Agent Actually Does

• High-Level Architecture

• Designing a Prompt That Works

• Handling Large Log Files Safely

• Analyzing Logs with LangChain and OpenAI

• Building the FastAPI Backend

• Creating a Simple and Clean Web UI

• Running the Application Locally

• Deployment to Sevalla

• Conclusion

What a LogAnalyzer Agent Actually Does

A LogAnalyzer Agent takes raw log text as input and produces human-friendly analysis as output.

Instead of returning a list of errors, it explains the main failures, the likely root cause, and what to do next. This is important because logs are written for machines, not for people under pressure.

In this project, the agent behaves like a senior site reliability engineer. It reads logs in chunks, identifies patterns, and summarises them in simple language. The intelligence comes from a language model, while the reliability comes from careful handling of input and chunking.

High-Level Architecture

The system has three main parts.

The first part is a web UI built with plain HTML, CSS, and JavaScript. This UI allows a user to upload a text file and start analysis.

The second part is a FastAPI backend that receives the file, validates it, and coordinates the analysis.

The third part is the analysis engine itself, which uses LangChain and an OpenAI model to interpret the logs.

The flow is simple: the browser sends a log file to the backend. The backend reads the file, splits it into manageable pieces, and sends each piece to the language model with a clear prompt. The responses are combined and sent back to the browser as a single analysis.

Designing a Prompt That Works

The heart of any AI agent is the prompt. A weak prompt gives vague answers, while a strong prompt produces useful insights.

In this project, the prompt tells the model to act like a senior site reliability engineer. It asks for four things: main errors, likely root cause, practical next steps, and suspicious patterns.

Here is the prompt template used in the backend:

log_analysis_prompt_text = """
You are a senior site reliability engineer.
Analyze the following application logs.
1. Identify the main errors or failures.
2. Explain the likely root cause in simple terms.
3. Suggest practical next steps to fix or investigate.
4. Mention any suspicious patterns or repeated issues.
Logs:
{log_data}
Respond in clear paragraphs. Avoid jargon where possible.
"""

This prompt is simple but effective. It gives the model a role, a clear task, and constraints on the output style. Asking for clear paragraphs helps ensure the response is readable and useful for non-experts as well.

Handling Large Log Files Safely

Language models have input limits. You can’t send a large log file in one request and expect good results. To handle this, the backend splits the log text into smaller chunks. Each chunk overlaps slightly with the next to preserve context.

We’ll use the RecursiveCharacterTextSplitter from LangChain for this purpose. It ensures that chunks aren’t cut in awkward places and that important lines aren’t lost.

def split_logs(log_text: str):
    """Split log text into manageable chunks"""
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=2000,
        chunk_overlap=200
    )
    return splitter.split_text(log_text)

This approach allows the agent to scale to large files while staying within model limits. Each chunk is analyzed independently, and the results are later combined.

Analyzing Logs with LangChain and OpenAI

Once the logs are split, each chunk is passed through the language model using the prompt template. The model used here is a lightweight but capable option, configured with a low temperature to keep responses focused and consistent.

llm = ChatOpenAI(
    temperature=0.2,
    model="gpt-4o-mini"
)

The analysis function loops over all chunks, formats the prompt, invokes the model, and stores the result.

def analyze_logs(log_text: str):
    """Analyze logs by splitting and processing each chunk"""
    chunks = split_logs(log_text)
    combined_analysis = []

  for chunk in chunks:
          formatted_prompt = log_analysis_prompt_text.format(log_data=chunk)
          result = llm.invoke(formatted_prompt)
          combined_analysis.append(result.content)
      return "\n\n".join(combined_analysis)

This design keeps the logic easy to understand. Each chunk produces a small analysis, and the final output is a stitched together explanation of the whole log file.

Building the FastAPI Backend

FastAPI is a good choice for this project because it’s fast, simple, and easy to read. The backend exposes three endpoints. The root endpoint serves the HTML UI. The /analyze endpoint accepts a log file and returns the analysis. And the /health endpoint is used to check if the service is running and properly configured.

The analyze endpoint performs several important checks. It ensures that the file is a text file, verifies that it isn’t empty, and handles errors gracefully. This prevents unnecessary calls to the model and improves user experience.

@app.post("/analyze")
async def analyze_log_file(file: UploadFile = File(...)):
    """Analyze uploaded log file"""
    if not file.filename.endswith(".txt"):
        return JSONResponse(
            status_code=400,
            content={"error": "Only .txt log files are supported"}
        )

     try:
        content = await file.read()
        log_text = content.decode("utf-8", errors="ignore")
        if not log_text.strip():
            return JSONResponse(
                status_code=400,
                content={"error": "Log file is empty"}
            )
        insights = analyze_logs(log_text)
        return {"analysis": insights}
    except Exception as e:
        return JSONResponse(
            status_code=500,
            content={"error": f"Error analyzing logs: {str(e)}"}
        )

This careful handling makes the agent more robust and production-friendly.

Creating a Simple and Clean Web UI

A good agent isn’t useful if people can’t interact with it easily. The frontend in this project is a single HTML file with embedded CSS and JavaScript. It focuses on clarity and speed rather than complexity.

The UI allows users to choose a log file, see the file name, click an analyze button, and view results in a formatted area. A loading spinner provides feedback while the analysis is running. Errors are shown clearly, without technical noise.

The upload and analysis logic is handled by a small JavaScript function that sends the file to the backend using a fetch request.

async function uploadLog() {
    const fileInput = document.getElementById("logFile");
    const file = fileInput.files[0];

if (!file) {
        alert("Please select a log file first");
        return;
    }
    const formData = new FormData();
    formData.append("file", file);
    const response = await fetch("/analyze", {
        method: "POST",
        body: formData
    });
    const data = await response.json();
    document.getElementById("result").textContent = data.analysis;
}

This minimal approach keeps the frontend easy to maintain and adapt.

Running the Application Locally

To run this project, you need Python, a virtual environment, and an OpenAI API key. The API key is loaded from a .env file to keep secrets out of code. Once dependencies are installed, you can start the server using Uvicorn.

if __name__ == "__main__":
    import uvicorn
    port = int(os.getenv("PORT", 8000))
    uvicorn.run(app, host="0.0.0.0", port=port)

After starting the server, you can open the browser, upload a log file, and see the agent in action.

Deployment to Sevalla

You can choose any cloud provider, like AWS, DigitalOcean, or others, to host your service. I’ll be using Sevalla for this example.

Sevalla is a developer-friendly PaaS provider. It offers application hosting, database, object storage, and static site hosting for your projects.

Every platform will charge you for creating a cloud resource. Sevalla comes with a $20 credit for us to use, so we won’t incur any costs for this example.

Let’s push this project to GitHub so that we can connect our repository to Sevalla. We can also enable auto-deployments so that any new change to the repository is automatically deployed.

Log in to Sevalla and click on Applications → Create new application.

You can see the option to link your GitHub repository to create a new application. Use the default settings. Then click Create application.

Now we have to add our OpenAI API key to the environment variables. Click on the Environment variables section once the application is created, and save the OPENAI_API_KEY value as an environment variable.

Now we’re ready to deploy our application. Click on Deployments and click Deploy now. It will take 2–3 minutes for the deployment to complete.

Once done, click on Visit app. You’ll see the application served via a URL ending with sevalla.app. This is your new root URL. You can replace localhost:8000 with this URL and start using it.

Congrats! Your log analyzer service is now live. You can find a sample log in the GitHub repository which you can use to test the service.

You can extend this by adding other capabilities and pushing your code to GitHub. Sevalla will automatically deploy your application to production.

Conclusion

Building a LogAnalyzer Agent is a practical way to apply language models to real engineering problems. Logs are everywhere, and understanding them quickly can save hours during incidents. By combining FastAPI, LangChain, and a clear prompt, you can turn raw text into actionable insight.

The key ideas are simple: split large inputs, guide the model with a strong role and task, and present results in a clean interface. With these principles, you can adapt this agent to many other analysis tasks beyond logs.

Hope you enjoyed this article. Learn more about me by visiting my website.

Logs keep track of what’s happening inside your code so that when things go wrong, you don’t have to guess. They’re similar to print or console.log, but more powerful.

In this tutorial, I will use Python to create and walk you through some logging code examples.

Before we talk about logs, let’s understand the different error types you might use or encounter.

Types of Errors

When you’re building a production-level application, you need to display errors based on their severity. There are several error types, and the most important ones are:

• DEBUG: Detailed information, typically useful for diagnosing problems.

• INFO: General information about the program’s progress.

• WARNING: Something unexpected happened, but it’s not critical.

• ERROR: An error occurred, but the program can still run.

• CRITICAL: A very serious error that may stop the program from running.

What is Logging?

Now, let’s get straight to the point and understand what logging is.

In simple terms, logs or logging is the act of recording information about everything your program does. The recorded information could be anything, from basic details like which functions were called to more detailed ones like tracking errors or performance issues.

Why Do We Need Logging?

You might be thinking, "If logs are printing errors, info, and so on, I can just use print statements. Why do I need logging?" Well, print works, but logging gives you more control:

↳ It can store messages in a file.
↳ It has different levels (info, warning, error, and so on).
↳ You can filter messages based on importance.
↳ It helps in debugging without cluttering your code.

These are things print statements can't do effectively.

How to Add Logs in Python

In Python, the logging module is built specifically for logging purposes.

Let’s set up some logs to see how they work.

Step 1: Import the Logging Module

To start using logging, we need to import the module:

import logging

Step 2: Log Messages

Now, you can start logging messages in your program. You can use different log levels based on the importance of the message. As a reminder, those levels are (from least to most urgent):

• DEBUG

• INFO

• WARNING

• ERROR

• CRITICAL

Let’s log a simple message at each level:

logging.debug("This is a debug message")
logging.info("This is an info message")
logging.warning("This is a warning message")
logging.error("This is an error message")
logging.critical("This is a critical message")

When you run this, you’ll see a message printed to the console, similar to this:

You might wonder why you don’t see the DEBUG and INFO messages. The default logging level prevents this.

By default, the logging level is set to WARNING. This means that only messages with a severity of WARNING or higher will be displayed (that is, WARNING, ERROR, and CRITICAL).

Step 3: Set Up the Basic Configuration

To see the debug and info messages, we need to set the logging level DEBUG before running the code.

This means we need to configure the logs. So to do this, use the method basicConfig below:

logging.basicConfig(level=logging.DEBUG)

This basic configuration allows you to log messages at the DEBUG level or higher. You can change the level depending on the type of logs you want.

Now, all logs are printing:

Step 4: Log to a File

Now, let’s save these logs in a file so we can keep track of errors, as well as when they occurred. To do this, update the configuration:

logging.basicConfig(filename='data_log.log', level=logging.DEBUG, 
                    format='%(asctime)s - %(levelname)s - %(message)s')

Here:

• asctime – The time when the event occurred.

• levelname – The type of the log (for example, DEBUG, INFO).

• message – The message we display.

Now, when you run the program, the log file will generate and save your logs, showing the exact timing, error type, and message. Like this:

How to Use Loggers for More Control

If you’re working on a large project, you might want a utility logger that you can use anywhere in the code. Let’s create this custom logger.

First, we’ll update the basicConfig to add the filename, line number, and ensure it writes everything, even special characters:

logging.basicConfig(
        filename=log_file,
        level=logging.DEBUG,
        format='%(asctime)s - %(levelname)s - %(filename)s:%(lineno)d - %(message)s', 
        filemode='w',
        encoding='utf-8' 
    )

Explanation:

• encoding='utf-8' — Ensures special characters are logged.

• %(filename)s:%(lineno)d — Logs the filename and line number where the log was generated.

Now, let’s set up a custom console logger:

  console_handler = logging.StreamHandler()
  console_handler.setLevel(logging.DEBUG)
  console_formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(filename)s:%(lineno)d - %(message)s')  # Added line number
  console_handler.setFormatter(console_formatter)


   logging.getLogger().addHandler(console_handler)

This setup does the following:

• console_handler: Sends log messages to the console (stdout).

• console_formatter: Formats the log message with time, level, filename, line number, and the message.

• logging.getLogger().addHandler(console_handler): Adds the custom handler to the root logger, so the log messages are printed to the console.

Full Example Code

import logging
import os
from datetime import datetime

def setup_daily_logger():
    base_dir = os.path.dirname(os.path.abspath(__file__))
    log_dir = os.path.join(base_dir, 'logs')  
    os.makedirs(log_dir, exist_ok=True)


    current_time = datetime.now().strftime("%m_%d_%y_%I_%M_%p")
    log_file = os.path.join(log_dir, f"{current_time}.log")


    logging.basicConfig(
        filename=log_file,
        level=logging.DEBUG,
        format='%(asctime)s - %(levelname)s - %(filename)s:%(lineno)d - %(message)s', 
        filemode='w',
        encoding='utf-8' 
    )


    console_handler = logging.StreamHandler()
    console_handler.setLevel(logging.DEBUG)
    console_formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(filename)s:%(lineno)d - %(message)s')  # Added line number
    console_handler.setFormatter(console_formatter)


    logging.getLogger().addHandler(console_handler)


    return logging.getLogger(__name__)

What Happens Now?

Now, whenever you run the program, a new log file will be created in the logs folder. Each time the program is executed, a new log file with a unique timestamp will be generated.

Like this:

These logs will give you a clear picture of your program’s behavior and help with debugging.

I hope this article helped you get a clearer picture of logs and their importance in programming.

Practical Real-World Examples

Now that you understand what logs are and how to set them up in Python, let’s look at real-world use cases.

1. Bot: Scraping Korea’s Largest Property Website

Here’s an example of a bot designed to scrape Korea’s biggest property website.

• The logs show every step the bot takes, making it easier to track progress.

• If an error occurs at any step, it gets recorded in the log file.

• Even if the bot crashes, I can check the logs to pinpoint where things went wrong.

One of the methods in this bot’s class uses logging to track whether the bot correctly selects the province.

Here:

• If an error or warning occurs, it’s saved in the log file.

• Later, you can review the logs and find out exactly what happened

2. Bot: Scraping Facebook Groups

Now, let’s see how logging helps in a Facebook group scraper.

Error Tracking

• At one point, the bot failed due to an error.

• Since we had logging in place, the error was saved in the log file.

• This allows you to quickly find out what went wrong.

Here, you see the exact filename and line number where the error occurs.

Once we identified and fixed the issue, the bot started working again.

It captures every detail in the log, saving hours of debugging by pinpointing where errors occur.

Debugging Made Easy

• The logs recorded every detail of the bot’s execution.

• This can save you hours of debugging because you’ll know exactly where the error occurred.

Conclusion

Logging is one of those things no one thinks about until something breaks. But when it does, logs become your best friend.

Remember:

• Logging isn’t just for error tracking—it helps you monitor your program’s flow.

• Instead of guessing what went wrong, check the logs. The answer is usually right there.

Make sure to add logging to your code. You’ll thank yourself later!

Stay Connected - @syedamahamfahim 🐬

If you're in information technology, you'll likely agree that logging is important. It helps you monitor a system, troubleshoot issues, and generally provides useful feedback about the system’s state. But it’s important to do logging right.

In this handbook, I'll explain what the syslog protocol is and how it works. You'll learn about syslog's message formats, how to configure rsyslog to redirect messages to a centralized remote server both using TLS and over a local network, how to redirect data from applications to syslog, how to use Docker with syslog, and more.

Table of Contents

1. Prerequisites
2. Introduction
3. What is syslog?
   • Syslog protocol
   • Syslog daemons
   • Syslog message formats
     • RFC3164 format
     • RFC5424 format
   • Syslog log levels
   • Syslog facilities
4. How to Configure rsyslog to Redirect Messages to a Centralized Remote Server using TLS
   • Update rsyslog
   • Install dependencies
   • Configure the exporting rsyslog server
   • Only forward logs generated by certain programs
   • Specify correct domain names and certificate paths in your configuration
   • Install certbot certificates
   • Give access to certificates to rsyslog
   • Configure accepting rsyslog server
   • Ensure firewall is not blocking your traffic
   • Restart rsyslog
   • Test the configuration
   • How to store remote logs in a separate file
   • Performance considerations
5. How to Configure rsyslog to Redirect Messages to a Centralized Remote Server Over a Local Network
   • Export the server setup
   • Accept the server setup
   • Restart rsyslog and test
6. Other Possibilities for Log Forwarding
7. How to Redirect Data from Applications to syslog
   • Standalone host application and syslog
     • Redirecting logs to syslog when running in foreground
     • Redirecting logs to syslog when running in background with systemctl
     • Redirecting logs from existing log files
   • Docker and syslog
     • Configuring a single Docker container
     • Configuring a Docker service through docker-compose file
     • Configuring a default for every container through the Docker daemon
     • Enabling applications inside Docker to log to syslog directly
   • How to Use Logging Libraries for Your Programming Language to log to syslog
     • Node.js client
     • Python client
8. Conclusion

Prerequisites

In this guide we will discuss syslog and its associated concepts. While I will explain most of the topics we come across, you should have a foundational knowledge about the following:

• using the Linux terminal (such as navigating the directory tree, creating and editing files, changing file permissions, etc.)
• a basic understanding of networks (domain name, host, IP address, TLS/SSL, TLS certificate, private/public key, and so on).

Introduction

Every system/application might provide its logs in different formats. If you have to work with many such systems and maintain them, it's important to deal with logs in a centralized, manageable, and scalable way.

First of all, it's useful to gather all the logs from the applications on your machine into one place for later processing.

Having collected all the logs in one place, you can now move on to processing them. But what if your machine is just a single node out of a group of servers? In this case, local log processing gives you insights about this single node but certainly not all of them.

Now you may very well want to transfer all the gathered logs to a central server which parses all the records, discovers any issues and inconsistencies, fires alerts, and finally stores the logs for future forensic analysis.

Note the convenience of having a central point of access to all your logs. You don't have to run around from machine to machine, searching for appropriate information and manually overlaying different log files.

So, to achieve the above, you can leverage the syslog protocol and use a very popular syslog daemon called rsyslog to collect all the logs and forward them to a remote server for further processing in a secure and reliable fashion.

And that’s exactly the example that I want to present in this tutorial to showcase a common and important use case of syslog. I'll give those who are not familiar with it a first taste of the problems it can solve.

So we'll explore this scenario, with examples of redirecting logs from host applications, Docker containers, and Node.js and Python clients, in this article.

But first, you need to understand the terminology around syslog as it's often shrouded in myths, mystery, and filled to the brim with confusion. Well, maybe I'm being overly dramatic here, but you get my point: terminology is important.

What is syslog?

Nowadays there is a lot of uncertainty as to what the word syslog actually refers to, so let’s clear it up:

Syslog protocol

Syslog is a system logging protocol that is a standard for message logging. It defines a common message format to allow systems to send messages to a logging server, whether it is set up locally or over the network. The logging server will then handle further processing or transport of the messages.

As long as the format of your messages is compliant with this protocol, you just have to pass them to a logging server (or, put differently, a logging daemon which we will talk about shortly) and forget about them.

The transport of the messages, rotation, processing, and enrichment will, from that point on, be handled by the logging server and the infrastructure it connects to. Your application does not have to know or deal with any of that. Thus we get a decoupled architecture (log handling is separated from the application).

But the main point of the syslog protocol is, of course, standardization. First of all, it is much easier to parse the logs when all the applications adhere to some common standard that generates the logs in the same format (or more or less the same, but let’s not jump the gun just yet).

If your logs have a common format, it’s first of all easy to filter the records by a particular time window or by the respective log levels (also referred to as severity levels. For example: info, warning, error, and so on).

Secondly, you may have a lot of different applications that implement a logtransport themselves. In that case, you'd have to spend quite some time skimming through the docs, figuring out how to configure file logging, log rotation, or log forwarding for every application instead of just configuring it once in your syslog server and expecting all your applications to simply submit their logs to it.

Syslog daemons

Now that you understand that syslog is a protocol that specifies the common format of log messages produced by the applications, we can talk a bit about syslog daemons. They're essentially logging servers or log shippers designed to take in log records, convert them to syslog format (if they are not already converted), and handle data transformations, enrichment, and transport to various destinations.

One of the original older implementations of a syslog daemon for Linux was referred to simply as syslog (leading to much confusion) or sysklogd. Later, more modern and commonly used implementations such as rsyslog or syslog-ng emerged. These were also made for Linux specifically.

But if you are interested in a cross platform syslog daemon, which can also be used on MacOS, Android, or even Windows, you can take a look at nxlog.

In the later sections of this handbook, we will see multiple practical example of working with syslog. For this we will use rsyslog, which is a lightweight and highly performant syslog daemon with a wide range of features. It typically comes preinstalled on many Linux distributions (both Debian- and RedHat-based).

Rsyslog, like many other syslog daemons, listens to a /dev/log unix socket by default. It forwards the incoming data to a /var/log/syslog file on Debian-based distributions or to /var/log/messages on RedHat-based systems.

Some specific log messages are also stored in other files in /var/log, but the bottom line is that all of this can, of course, be configured to suit your needs.

Now you know about the true meaning of syslog and syslog daemons. But there is one important caveat. In many cases (and during the course of this guide as well) saying syslog colloquially refers to the syslog daemon as well as the infrastructure around it (Unix sockets, files in /var/log, as well as other daemons if the messages are forwarded across the network). So, saying “publish a message to syslog” means sending a message to the /dev/log Unix socket where it will be intercepted and processed by a syslog daemon according to its settings.

Syslog message formats

Syslog defines a certain format (structure) of the log records. Applications working with syslog should adhere to this format when logging to /dev/log. From there, syslog daemons will pick up the messages, and parse and process them according to their configuration.

There are two types of syslog formats: the original old BSD format which came from the early versions of BSD Unix systems and became a standard with RFC3164 specification, as well as a newer one from RFC5424.

RFC3164 format

This format consists of the following 3 parts: PRI, HEADER (TIMESTAMP, HOSTNAME), MSG (TAG, CONTENT). Here is a more concrete example (taken directly from RFC3164, by the way):

<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8

Let's see what's going on here:

• <34> (PRI) – priority of the log record which consists of the facility level multiplied by 8 plus the severity level. We will talk about facilities and severity levels soon, but in the example above we get: a facility number 4 (34 // 8 = 4) and a critical severity level (34 % 8 = 2).
• Oct 11 22:14:15 (TIMESTAMP) – timestamp in local time without the year and a millisecond or a timezone portion. It follows a format string “Mmm dd hh:mm:ss”
• mymachine (HOSTNAME) _ hostname, IPv4, or IPv6 address of the machine that the message originates from.
• su (TAG) – Name of the program or process that generated the message. Any non-alphanumeric character terminates the TAG field and is assumed to be a starting part of the next (CONTENT) field. In our case, it is a colon (“:”) character. But it could also have been just a space, or even square brackets with the PID (process id) inside, such as “[123]”.
• : 'su root' failed for lonvick on /dev/pts/8 (CONTENT) – An actual message of the log record.

As you can see, RFC3164 doesn’t provide a lot of structural information, and has some limitations and inconveniences such as a restricted timestamp or certain variability and uncertainty (for example, in the delimiters after the TAG field). Also, the RFC3164 format stipulates that only ASCII encoding is supported.

All the above is actually the result of RFC3164 not being a set-in-stone strict standard, but rather a best-effort generalization of some syslog implementations that already existed at the time.

RFC5424 format

RFC5424 presents an upgraded and more structured format which deals with some of the problems found in RFC3164.

It consists of the following parts: HEADER (PRI, VERSION, TIMESTAMP, HOSTNAME, APP-NAME, PROCID, MSGID), STRUCTURED DATA (SD-ELEMENTS (SD-ID, SD-PARAM)), MSG. Below is an example:

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

• <34> (PRI) – priority of the log record. Combination of severity and facility. Same as for RFC3164.
• 1 (VERSION) – version of the syslog protocol specification. This number is supposed to be incremented for any future specification that makes changes to the HEADER part.
• 2003-10-11T22:14:15.003Z (TIMESTAMP) – a timestamp with year, sub-second information, and timezone portions. It follows the ISO 8601 standard format as described in RFC3339 with some minor restrictions, like not using leap seconds, always requiring the “T” delimiter, and upper casing every character in the timestamp. NILVALUE (“-”) will be used if the syslog application cannot obtain system time (that is, it doesn’t have access to the time on the server).
• https://mymachine.example.com/ (HOSTNAME) – FQDN, hostname, or the IP address of the log originator. NILVALUE may also be used when the syslog application does not know the originating host name.
• su (APP-NAME) – device or application that produced the message. NILVALUE may be used when the syslog application is not aware of the application name of the log producer.
• - (PROCID) – implementation-dependent value often used to provide a process name or process ID of the application that generated the message. A NILVALUE should be used when this field is not provided.
• ID47 (MSGID) – field used to identify the type of message. Should contain NILVALUE when not used.
• - (STRUCTURED DATA) – provides sections with key-value pairs conveying additional metadata about the message. NILVALUE should be used when structured data is not provided. Examples: “[exampleSection@32473 iut="3" eventSource="Application" eventID="1011"][exampleSection2@32473 class="high"]”. In practice the STRUCTURED DATA part was rarely used and the metadata information was usually put into the MSG part that many applications structure as a JSON.
• BOM'su root' failed for lonvick on /dev/pts/8 (MSG) – actual message of the log record. The “BOM” at the beginning is an unprintable character which signifies that the rest of the payload is UTF8 encoded. If this character is not present, then other encodings like ASCII can be assumed by the syslog daemons.

RFC5424 has a more convenient format used for a timestamp and much more structural parts which you can use for specifying all sorts of metadata for the log messages.

Also, the specification was made to be extendable with the VERSION field. Even though I am not aware of any particular syslog specification extensions that make use of the latter and increment the version, the possibility is always there.

Finally, the new format supports UTF8 encoding and not just ASCII.

Notice that if a message directed to /dev/log does not follow one of the described syslog formats, it will still be processed by daemons such as rsyslog. Rsyslog will try to parse such records according to either its defaults or custom templates.

Templates are a separate topic on their own that needs its own article, so we will not be focusing on them now.

By default, rsyslog will treat such messages as unstructured data and process them as they are. It will try filling in the gaps like a timestamp field, severity level, and so on to the best of its ability and in accordance with its default parameters (for example, timestamp will become current time, log level will be “user”, and severity level will be “info”).

When inspecting the log records in /var/log/messages or /var/log/syslog (depending on the system) you will probably see a different format from those described above. For rsyslog it looks like this:

Feb 19 10:01:43 mymachine systemd[1]: systemd-hostnamed.service: Deactivated successfully.

This is just the format that rsyslog uses to display ingested messages that it saved to disk and not a standard syslog format. You can find this format in the rsyslog.conf file or in the official documentation under the name RSYSLOG_TraditionalFileFormat. But you can alway configure how rsyslog outputs its messages yourself using templates.

One important aspect to understand is that rsyslog processes messages as they come. It receives and immediately forwards them to the specified destinations or saves them locally to files, such as /var/log/messages. Once the messages are fully processed, rsyslog does not retain any metadata about them apart from what it stored to log files.

This means that if records in /var/log/messages are stored in the traditional rsyslog format presented above, they will not keep, for example, their initial PRI value. While PRI and other data are accessible to rsyslog internally when processing and routing messages, not all of this information is by default stored in the log files.

Syslog log levels

Syslog supports the following log levels referred to as severity levels as per syslogs’s terminology:

These levels allow you to categorize messages by the severity (importance) criteria, with emergency being the highest level.

Syslog facilities

Syslog facilities represent the origin of a message. You can often use them for filtering and categorizing log records by the system that generated them.

Note that syslog facilities (as well as severity levels, actually) are not strictly normative, so different facilities and levels may be used by different operating systems and distributions. Many details here are historically rooted and not always utility-based.

Note that the syslog protocol specification defines only the codes for facilities. The keywords may be used by syslog daemons for readability.

Now that we have seen the list of all the existing facilities, pay attention to the ones such as “security”, “authpriv”, “log audit”, or “log alert”. It is possible for an application to log to different facilities depending on the nature of the message.

For example, an application might typically log to the “user” facility, but once it receives an important alert, it might log to facility 14 (log alert). Or in case of some authentication/authorization notice, it may direct it to “auth” facility, and so on.

If you have a custom application and are wondering which facility would be best suited for it, you can use the “user” facility (code 1) or custom local facilities (codes 16-22).

The ultimate difference between user and local facilities is that the former is a more general one, which aggregates the logs from different user applications. But be aware that other software might just as well use one of the local facilities on your machine.

How to Configure rsyslog to Redirect Messages to a Centralized Remote Server using TLS

Let’s now look at a practical example that I mentioned at the beginning. This might not appear to be the most basic use case – especially for those who are not familiar with syslog daemons – but it's quite a common scenario. I hope it will help you learn a lot of useful things along the way.

Now I'll walk you through the steps you'll need to take to forward the syslog data from one server to another that will play the role of a centralized log aggregator. In this example, we will be sending logs as they flow in, using the TCP protocol with certificates for encryption and identity verification.

In the following examples, I assume that you have a centralized server for accepting the syslog data and one or more exporting servers that forward their syslog messages to that central accepting node. I'll also assume that all the servers are discoverable by their respective domain names and are running Debian-based or RedHat-based Linux distributions.

So, let's dive in and get started.

Update rsyslog

As rsyslog typically comes preinstalled on most common Linux distros, I won't cover the installation process here. Just make sure your rsylogd is up-to-date enough to take advantage of its wide range of features.

Run the following command across all your servers:

rsyslogd -v

And ensure that the version in the output is 6 or higher.

If this is not the case, run the following commands to update your daemon:

For Debian-based distributions:

sudo apt-get update
sudo apt-get install --only-upgrade rsyslog
sudo systemctl restart rsyslog

For RedHat-based:

sudo yum update rsyslog
sudo systemctl restart rsyslog

Or use dnf instead of yum for CentOS8/RHEL8:

Install dependencies

To handle the secure forwarding of the messages over the network using TLS, we will need to install the rsyslog-gnutls module. If you prefer to compile rsyslog from source, you will have to specify a respective flag when building. But if you use package managers you can simply run the following for every server:

For Debian-based distributions:

sudo apt-get update
sudo apt-get install rsyslog-gnutls
sudo systemctl restart rsyslog

For RedHat based:

sudo yum install epel-release
sudo yum install rsyslog-gnutls
sudo systemctl restart rsyslog

Configure the exporting rsyslog server

Now, we will create a rsyslog configuration file for the nodes that are going to be exporting their logs to the central server. In order to do so, create the configuration file in the config directory of rsyslog:

sudo touch /etc/rsyslog.d/export-syslog.conf

Ensure that the file is readable by the syslog user on Debian-based distributions (chown syslog:adm /etc/rsyslog.d/export-syslog.conf or chmod 644 /etc/rsyslog.d/export-syslog.conf). Note that on RedHat-based distros like CentOS rsyslog runs under root, so there shouldn’t be any permissions issues.

Now open the created file and add the following configuration:

# Set certificate files
global(
  DefaultNetstreamDriverCAFile="<path_to_your_ca.pem>"
  DefaultNetstreamDriverCertFile="<path_to_your_cert.pem>"
  DefaultNetstreamDriverKeyFile="<path_to_your_private_key.pem>"
)

# Set up the forwarding action for all messages
*.* action(
  type="omfwd"
  StreamDriver="gtls"
  StreamDriverMode="1"
  StreamDriverPermittedPeers="<domain_name_of_your_accepting_central_server>"
  StreamDriverAuthMode="x509/name"
  target="<domain_name_or_ip_of_your_accepting_central_server>" port="514" protocol="tcp"
  action.resumeRetryCount="100" # you may change the queue and retry params as you see fit
  queue.type="linkedList" queue.size="10000"
)

The above configuration will forward all the messages that are digested by rsyslog to your remote server. In case you want to achieve a more fine-grained control, refer to the subsection below.

Only forward logs generated by certain programs

If you want to forward messages for a certain program only, you can specify the following condition instead of *.* before action in the configuration above:

if $programname == '<your_program_name>' then
# ...right here goes your action and all the rest

If you want to specify more than one program name, add multiple conditions using or:

if ($programname == '<your_program_name1>' or $programname == '<your_program_name2>' $programname == '<your_program_name3>') then
# ...right here goes your action and all the rest

For more information refer to the RainerScript documentation for rsyslog here.

Specify correct domain names and certificate paths in your configuration

Now, let’s go back to our configuration file. It will use TLS (as you can see in StreamDriverMode="1") and forward all the data to target on port 514, which is a default port for syslog.

To make this configuration valid, you will need to replace <domain_name_of_your_accepting_central_server> and <domain_name_or_ip_of_your_accepting_central_server> with the respective domain name of your central accepting server (for example: my-central-server.company.com) as well as specify correct paths to certificates in the global section.

Note that, since on Debian-like distros rsyslog typically runs under the syslog user, you will have to ensure that the certificates themselves and all the directories in their path are readable and accessible by this user (for directories this means that both “r” and “x” permission bits must be set).

On RedHat-based systems, on the other hand, rsyslog often runs as root, so there is no need to tweak the file permissions.

To check under which user your rsyslog runs, run the following:

sudo ps -aux | grep rsyslog

And look at the left side at the username executing rsyslogd.

If you don’t have SSL certificates yet, read the next two subsections about installing certs with Let's Encrypt and providing access to rsyslog. If you already have all the needed certificates and permissions, you can safely skip these steps.

Install certbot certificates

First, you'll need to install certbot. For Debian-based systems, run the following:

sudo apt-get install certbot

If you get an error that the package is not found, run sudo apt-get update and try again.

For RedHat-based systems:

sudo yum install epel-release
sudo yum install certbot

Ensure that no server is running on port 80 and then run certbot in standalone mode, specifying your domain name with the -d flag to get your SSL certificates:

sudo certbot certonly --standalone -d <your_domain_name>
# For example: sudo certbot certonly --standalone -d my-sever1.mycompany.com

Follow the prompts of certbot, and in the end you will receive your SSL certificates that will be stored at /etc/letsencrypt/live/<your_domain_name>/.

Confirm that there are no problems during the certificate renewal process like this:

sudo certbot renew --dry-run

Certificates will be automatically renewed by certbot, so you don’t have to worry about manually updating them every time. If you installed certbot as described above, it will use a systemd timer or create a cron job to handle renewals.

Give access to certificates to rsyslog

If you are running a Debian-based system, then, as mentioned above, you have to grant the syslog user the necessary privileges to access certbot certificates and keys. This is because the /etc/letsencrypt/live directory with certbot-generated files is restricted to root user only.

So, we will copy the certificates and keys over to standard certs and keys locations. For Debian-based distributions, these are /etc/ssl/certs and /etc/ssl/private, respectively. Then we'll change the permissions of these files.

First, create a group that will have access to SSL certificates:

sudo groupadd sslcerts

Add the syslog user to this group:

sudo usermod -a -G sslcerts syslog

Add permissions and ownership for the /etc/ssl/private directory to the created group:

sudo chown root:sslcerts /etc/ssl/private
sudo chmod 750 /etc/ssl/private

Now, we'll create a script that will move certificate files from Let's Encrypt’s live directory to a /etc/ssl. Run the following:

sudo touch /etc/letsencrypt/renewal-hooks/deploy/move-ssl-certs.sh
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/move-ssl-certs.sh

Note that by creating the script in the /etc/letsencrypt/renewal-hooks/deploy directory, it will automatically run after every certificate renewal. This way, you won’t have to worry about manually moving certificates and granting permissions every time they expire.

Open the created file and add the following content, replacing <your-domain-name> with the domain of your machine which corresponds to the directory created by certbot in /etc/letsencrypt/live:

#!/bin/bash

# Define the source and destination directories
DOMAIN_NAME=<your-domain-name>
LE_LIVE_PATH="/etc/letsencrypt/live/$DOMAIN_NAME"
SSL_CERTS_PATH="/etc/ssl/certs"
SSL_PRIVATE_PATH="/etc/ssl/private"

# Copy the full chain and private key to the respective directories
cp "$LE_LIVE_PATH/fullchain.pem" "$SSL_CERTS_PATH/$DOMAIN_NAME-letsencrypt-fullchain.pem"
cp "$LE_LIVE_PATH/cert.pem" "$SSL_CERTS_PATH/$DOMAIN_NAME-letsencrypt-cert.pem"
cp "$LE_LIVE_PATH/privkey.pem" "$SSL_PRIVATE_PATH/$DOMAIN_NAME-letsencrypt-privkey.pem"

# Set ownership and permissions
chown root:sslcerts "$SSL_CERTS_PATH/$DOMAIN_NAME-letsencrypt-fullchain.pem"
chown root:sslcerts "$SSL_CERTS_PATH/$DOMAIN_NAME-letsencrypt-cert.pem"
chown root:sslcerts "$SSL_PRIVATE_PATH/$DOMAIN_NAME-letsencrypt-privkey.pem"

chmod 644 "$SSL_CERTS_PATH/$DOMAIN_NAME-letsencrypt-fullchain.pem"
chmod 644 "$SSL_CERTS_PATH/$DOMAIN_NAME-letsencrypt-cert.pem"
chmod 640 "$SSL_PRIVATE_PATH/$DOMAIN_NAME-letsencrypt-privkey.pem"

Now, execute the created script to actually move the certificates to /etc/ssl and give permissions to the syslog user:

sudo /etc/letsencrypt/renewal-hooks/deploy/move-ssl-certs.sh

Lastly, go into the rsyslog configuration file /etc/rsyslog.d/export-syslog.conf and change the certificate paths accordingly:

# Set certificate files
global(
DefaultNetstreamDriverCAFile="/etc/ssl/certs/<your_domain_name>-letsencrypt-fullchain.pem"
  DefaultNetstreamDriverCertFile="/etc/ssl/certs/<your_domain_name>-letsencrypt-cert.pem"
  DefaultNetstreamDriverKeyFile="/etc/ssl/private/<your_domain_name>-letsencrypt-privkey.pem"
)

Note that even though rsyslog typically runs as root on RedHat-based distributions, you may find that it’s not the case for your system.

If it's not, you can do the same permission manipulations as we did above. But the default recommended location for SSL certificates and keys might differ. For CentOS it’s /etc/pki/tls/certs and /etc/pki/tls/private. But you can also alway choose completely different locations if need be.

Configure accepting rsyslog server

Let’s now configure a central server that will accept the logs from the rest of the machines.

If you haven’t acquired SSL certificates for your server, refer to the section on installing certbot certificates.

If your server is Debian-based, refer to the section on giving access to certificates to rsyslog.

Now, similar to configuring the exporting server, create a rsyslog configuration file:

sudo touch /etc/rsyslog.d/import-syslog.conf

Open the file and add the following:

# Set certificate files
global(  DefaultNetstreamDriverCAFile="/etc/ssl/certs/<your_domain_name>-letsencrypt-fullchain.pem"
  DefaultNetstreamDriverCertFile="/etc/ssl/certs/<your_domain_name>-letsencrypt-cert.pem"
  DefaultNetstreamDriverKeyFile="/etc/ssl/private/<your_domain_name>-letsencrypt-privkey.pem"
)

# TCP listener
module(
  load="imtcp"
  PermittedPeer=["<your_peer1>","<your_peer2>","<your_peer3>"]
  StreamDriver.AuthMode="x509/name"
  StreamDriver.Mode="1"
  StreamDriver.Name="gtls"
)

# Start up listener at port 514
input(
  type="imtcp"
  port="514"
)

Note that you need to replace PermittedPeer=["<your_peer1>","<your_peer2>","<your_peer3>"] with an array of the domain names of your export servers, for example: PermittedPeer=["export-server1.company.com","export-server2.company.com","export-server3.company.com"].

Also don’t forget to double check and change your certificate paths in the global section as needed. Again, if you are on a RedHat-based system, you may simply reference certificates in Let's Encrypt’s live directory because of the root permissions.

Ensure firewall is not blocking your traffic

Make sure that the firewall on your central server doesn’t block incoming traffic on port 514. For example, if you are using iptables:

To check the rule already exists:

sudo iptables -C INPUT -p tcp --dport 514 -j ACCEPT

If previous command exits with an error, you can define an accepting rule with:

sudo iptables -A INPUT -p tcp --dport 514 -j ACCEPT # rules will apply immediately
iptables-save > /etc/iptables/rules.v4 # or use `iptables-save > /etc/sysconfig/iptables` for RedHat-based distributions

Restart rsyslog

Now, after you’ve added the appropriate configurations to all your servers, you have to restart rsyslog on all of them, beginning with your central accepting node:

sudo systemctl restart rsyslog

You can check if there are any errors after the rsyslog restart by executing the following:

sudo systemctl status rsyslog
sudo journalctl -u rsyslog | tail -100

The first command above will display the status of rsyslog, and the second one will output the 100 last lines of rsyslog’s logs. If you misconfigured something and your set up didn’t work, you should find helpful information there.

Test the configuration

In order to test whether your syslog redirection worked, issue the following command on the central node to start watching for new data coming in to syslog:

For Debian-based systems:

tail -f /var/log/syslog

For RedHat-based:

tail -f /var/log/messages

After that, go to each of your export nodes and run:

logger "Hello, world!"

You should see a “Hello, world!” message from each export server popping up in the syslog of your accepting machine.

If everything worked, then congrats! You have now successfully setup and verified syslog redirection over the network.

Note: press Ctrl+C to exit from the tail -f command executed on the central node.

In a later section, we will consider the same scenario but without certificates in case all your servers are located in a trusted local network. After that, we will finally explore how to redirect actual data from different applications to syslog.

How to store remote logs in a separate file

But wait a second – before moving on, let’s consider a small useful modification to our setup.

Let’s say you want to set up your central rsyslog in such a way that it will redirect remote traffic to a separate file instead of the typical /var/log/syslog or /var/log/messages.

To do this, make the following changes to your /etc/rsyslog.d/import-syslog.conf:

Add a ruleset property to the input object:

input(
  type="imtcp"
  port="514"
  ruleset="remote"
)

Then add the following line at the bottom of the file:

ruleset(name="remote") {
  if $hostname == '<your_remote_hostname>' then {
    action(type="omfile" file="/var/log/remote-logs.log")
    stop
  }
}

Change <your_remote_hostname> accordingly. You can also define multiple hostnames with an or as we have seen before. Also feel free to change the path to the output file (that is, /var/log/remote-logs.log) to suit your needs.

After that, restart rsyslog.

Performance considerations

Rsyslog is a very light and performant tool for managing and forwarding your logs over the network. Still, performing a TCP protocol and a TLS handshake to validate the certificates for every log message (or a batch of messages) comes with its costs.

In the next section, you'll learn how to perform TCP and UDP forwarding without TLS certificates. This will typically be a more performant way, but you should only use it in a trusted local network.

As for UDP, even though it’s more performant than TCP, you should use it only if potential data losses are acceptable.

If you don’t need a near real time log delivery, you might be better off storing all your logs in a single file (you can do this with rsyslog or by employing other tools or techniques). Then you can schedule a separate script, which will transfer this file to a central server when it reaches a certain size or when certain time intervals elapse.

In any case, before employing a particular solution, make sure you do a benchmark focusing on load testing your system to discover which approach works best for you.

How to Configure rsyslog to Redirect Messages to a Centralized Remote Server Over a Local Network

If your scenario doesn’t involve communications over an untrustworthy network, you might decide not to use certificates for forwarding your syslog records. I mean, TLS handshakes are costly after all!

Also, the configuration in this case which we'll discuss now will get quite a bit simpler. It will involve fewer steps, as our main concern when setting up syslog forwarding with TLS were SSL certificates and their file permissions.

Export the server setup

To configure your exporting server to forward syslog data using TCP without encryption, login to every exporting server and create an rsyslog configuration file:

sudo touch /etc/rsyslog.d/export-syslog.conf

Open this file and add the following configuration, replacing <your_remote_server_hostname_or_ip> with the hostname or IP of your central node, which must be discoverable on your network:

*.* action(
  type="omfwd"
  target="<your_remote_server_hostname_or_ip>"
  port="514"
  protocol="tcp"
  action.resumeRetryCount="100"
  queue.type="linkedList"
  queue.size="10000"
)

If you want to use the UDP protocol, you can simply change protocol=”tcp” to protocol=”udp”.

In case you are now wondering whether we could use UDP to forward the traffic in our previous example with certificates, then the answer is no. This is because a TLS handshake works over TCP but not UDP. At least it was originally designed this way, and even though there might exist certain variations and protocol modifications in the wild, they are very tricky and definitely out of the scope of this handbook.

Note that there is an alternative simpler but less flexible syntax for writing the above configuration.

For forwarding over TCP:

*.* @@<your_remote_server_hostname>:514

For forwarding over UDP:

*.* @<your_remote_server_hostname>:514

I am showing you these syntax variations because you may encounter them in other articles. These variations replicate the syntax of sysklogd daemon (yes, one of the first syslog daemon implementations which rsyslog is a backwards compatible fork of).

Accept the server setup

Create an rsyslog configuration file on your accepting server:

sudo touch /etc/rsyslog.d/import-syslog.conf

Open the file and add the following contents:

• To receive TCP traffic:

module(load="imtcp") # Load the imtcp module
input(type="imtcp" port="514") # Listen on TCP port 514

• To receive UDP:

module(load="imudp") # Load the imudp module for UDP
input(type="imudp" port="514") # Listen on UDP port 514

Legacy syntax alternatives of the config file for the receiving server would be the following:

For TCP:

$ModLoad imtcp # Load the imtcp module for TCP
$InputTCPServerRun 514 # Listen on TCP port 514

For UDP:

$ModLoad imudp # Load the imudp module for UDP
$UDPServerRun 514 # Listen on UDP port 514

Note, though, that even though you might sometimes encounter the legacy rsyslog syntax of receiving messages, it is not compatible with sysklogd.

To set up a listener in rsyslogkd, you would have to set a different special variable in the configuration file as described here. But going into details about syslogkd is outside the scope of this article.

Also, I don't recommend that you use the old syntax (neither does the author of rsyslog). It is presented purely for educational purposes, so that you know what it is in case you encounter it.

You can read more about rsyslog’s configuration formats here.

In case you use a firewall, check that its settings allow incoming UDP or TCP connections on port 514.

Restart rsyslog and test

Go to every machine starting with the accepting server and restart rsyslog. Then check that there are not errors in its logs:

sudo systemctl restart rsyslog
sudo journalctl -u rsyslog | tail -100

Other Possibilities for Log Forwarding

Rsyslog is a very powerful tool with a lot more functionality than we have covered so far. For example, it supports direct forwarding of the logs to Elasticsearch, which is a very performant log storage system. But that's a separate topic which deserves its own article.

For now, I will just give you a taste of how an example rsyslog-to-elasticsearch configuration might look like:

# Note that you will have to install "rsyslog-elasticsearch" using your package manager like apt or yum
module(load="omelasticsearch") # Load the Elasticsearch output module

# Define a template to constract a JSON message for every rsyslog record, sine Elasticsearch works with JSON
template(name="plain-syslog"
         type="list") {
           constant(value="{")
             constant(value="\"@timestamp\":\"")      property(name="timereported" dateFormat="rfc3339")
             constant(value="\",\"host\":\"")         property(name="hostname")
             constant(value="\",\"severity\":\"")     property(name="syslogseverity-text")
             constant(value="\",\"facility\":\"")     property(name="syslogfacility-text")
             constant(value="\",\"syslogtag\":\"")    property(name="syslogtag")
             constant(value="\",\"message\":\"")      property(name="msg" format="json")
           constant(value="\"}\n")
         }

# Redirect all logs to syslog-index of Elasticsearch which listens on localhost:9200
*.* action(type="omelasticsearch"
       server="localhost:9200"
       searchIndex="syslog-index"
       template="plain-syslog")

For more information refer to the docs.

How to Redirect Data from Applications to syslog

So far we have covered configuring a syslog daemon. But how do we actually push logs from real applications into a syslog?

Ideally, it would be best if your application already had a syslog integration and could be configured to send the logs to syslog directly.

But what if this is not the case? Well, it certainly is a pity, because manually redirecting stdout and stderr to syslog might come with its challenges and inconveniences. But don’t worry, it’s not that complicated! At least sort of.

Let’s take a look at different scenarios:

Standalone host application and syslog

First of all, let’s consider that you already have an application running locally on your host machine (no containerization). There are multiple ways to redirect its logs in this case.

Instead of using general example commands and constantly repeating “change in the command above accordingly” (which I am quite tired of at this point), I am going to install a real application and show the redirection with concrete examples.

The application of choice will be Mosquitto broker, since I am very fond of MQTT, but you can use whatever, as it’s just an example.

Oh, and by the way, Mosquitto does provide a direct integration with the syslog. It just requires a small change (log_dest syslog) in its configuration file. But we will not be looking into this, since our assumption is that we are working with an application incapable of interfacing with the syslog directly.

Here's how to install the broker on Debian-based systems:

sudo apt-get update
sudo apt-get install mosquitto

And here's RedHat-based installation:

sudo yum install epel-release
sudo yum install mosquitto

After the installation, Mosquitto might be automatically run in the background, so I stop it with sudo systemctl stop mosquitto.

Redirecting logs to syslog when running in foreground

You can run Mosquitto in the foreground and redirect all its logs to syslog using “info” level and local0 facility:

sudo mosquitto -c /etc/mosquitto/mosquitto.conf -v 2>&1 | sudo logger -t mosquitto -p local0.info

• The -c option specifies a non-default Mosquitto configuration file and may be omitted.
• -v specifies a verbose mode which produces more output.
• The -t flag provided to the “logger” command is a TAG representing the programname.

Note that the default facility of the logger tool is user and default severity level is notice.

Forwarding all the output into syslog with a common severity level is good and all, but it would make more sense to be able to distinguish at least between info and error messages.

To be able to distinguish those, you will have to write a custom bash script. Below you can see an example. Note that the strange looking part “> >(...)” is a Bash substitution feature.

#!/bin/bash

# Define your application command here
command="/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf -v"
programname="mosquitto"

# Use process substitution to handle stdout and stderr separately
{
    $command 2> >(while read line; do logger -p local0.error -t "$programname" "$line"; done) \
                             > >(while read line; do logger -p local0.info -t "$programname" "$line"; done)
}

To run the above script, just save it to a file, give it execute permissions with sudo chmod +x /path/to/your_script.sh, and run it with sudo ./your_script.sh.

Something to be aware of is that starting Mosquitto is not the most suitable command for the example above, since it redirects all its logging output to stderr by default. So you will only see messages with “error” severity in the syslog log files.

Now, here is an example of a bash script in case you want to determine severity level by parsing each application’s log record from stdout or stderr and base a severity level off of some specific substrings in each record (for example, “ERROR”, “WARN”, “INFO”, and so on):

#!/bin/bash

# Define your application command here
command="/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf -v"
programname="mosquitto"

# Execute command and pipe its stdout and stderr to a while loop
$command 2>&1 | while read line; do
    # Determine the severity level based on the content of the line
    if [[ "$line" == *"Error:"* ]]; then
        logger -t "$programname" -p user.err "$line" # Forward error messages as errors
    elif [[ "$line" == *"Warning"* ]]; then
        logger -t "$programname" -p user.warning "$line" # Forward warning messages as warnings
    else
        logger -t "$programname" -p user.info "$line" # Forward all other messages as info
    fi
done

Redirecting logs to syslog when running in background with systemctl

Many applications run as daemons (in the background). Oftentimes they can be started and managed using systemctl process management tool or similar.

If you want to redirect the logs of an application that runs as a systemctl daemon to syslog, follow the example below.

Here are the steps you'll need to perform when running Mosquitto broker in background:

Step 1: create a custom sh script:

sudo touch /usr/local/bin/mosquitto_with_logger.sh

Step 2: open the file and add the following content:

#!/bin/bash
/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf -v 2>&1 | logger -t mosquitto

Step 3: add execute permissions to the script:

sudo chmod +x /usr/local/bin/mosquitto_with_logger.sh

Step 4: create a systemd unit file:

sudo touch /etc/systemd/system/mosquitto_syslog.service

Step 5: open the file and add the following:

[Unit]
Description=Mosquitto MQTT Broker with custom logging
After=network.target

[Service]
ExecStart=/usr/local/bin/mosquitto_with_logger.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target

Step 6: reload systemd, enable the custom service to be run on system startup, and finally start it:

sudo systemctl daemon-reload
sudo systemctl enable mosquitto_syslog.service
sudo systemctl start mosquitto_syslog.service

Redirecting logs from existing log files

In case your application only logs to a file and you want to redirect these logs to syslog, see the following rsyslog configuration file that you can place in /etc/rsyslog.d/ with a .conf file extension:

module(load="imfile" PollingInterval="10") # Load the imfile module

# For info logs
input(type="imfile"
      File="/path/to/your/app-info.log"
      Tag="myapp"
      Severity="info"
      Facility="local0")

# For error logs
input(type="imfile"
      File="/path/to/your/app-error.log"
      Tag="myapp"
      Severity="error"
      Facility="local0")

# you can put your actions that will forward the data here
# or rely on the actions from the original rsyslog.conf file that imports this file

The configuration above assumes that you have separate files for info and error logs.

In principle, you could also forward all the contents from a single file by either assigning a common severity or by trying to parse out each new line in the file and guess its intended log level. This would require us to use rsyslog’s rulesets similar to the following:

module(load="imfile" PollingInterval="10") # Load the imfile input module

# Template for formatting messages with original severity and facility
template(name="CustomFormat" type="string" string="<%PRI%>%TIMESTAMP% %HOSTNAME% %msg%\n")

# Monitor a specific logfile
input(type="imfile"
      File="/path/to/your/logfile.log"
      Tag="myApp"
      Ruleset="guessSeverity")

# Ruleset to parse log entries and guess severity
ruleset(name="guessSeverity") {
    # Use property-based filters to check message content and route accordingly
    if ($msg contains "Error:") then {
        action(type="omfile"
               File="/var/log/errors.log" # Specify the log file for error messages
               Template="CustomFormat"
              )
    } else if ($msg contains "Warning:") then {
        action(type="omfile"
               File="/var/log/warnings.log" # Specify the log file for warning messages
               Template="CustomFormat"
              )
    } else {
        action(type="omfile"
               File="/var/log/info.log" # Specify the default log file for other messages
               Template="CustomFormat"
              )
    }
}

You should ensure that /path/to/your/logfile.log exists before applying the above configuration.

We used rulesets above which is another nice feature of rsyslog. You can read more on this in the official documentation.

However, the above configuration explicitly sets the destination for processed messages directing them to different files depending on their severity. If you want to forward the messages to the standard /var/log/messages or /var/log/syslog, you will have to specify it explicitly (and amend/add more templates to reflect the appropriate severity levels).

But what if you have many other rules in your main rsyslog config file? You don’t want to repeat them in the ruleset above and just want to parse out the severity level and pass the record on to rsyslog’s main configuration to handle the rest?

Unfortunately, I didn’t find a nice way of doing this. There is just one hacky approach to resubmit your record back to rsyslog using the logger utility and the omprog module. I will show this approach anyway for completeness, and since it’s a good way to see more rsyslog features. But be aware that it involves some overhead, since you'll basically be invoking rsyslog twice for every record.

To resubmit a record back to rsyslog, include the omprog module:

module(load="omprog")

And change the actions inside the if-else tree to the following:

action(type="omprog"
        Template="CustomFormat" # Optional property to format the message
        binary="/usr/bin/logger -t myApp -p local0.error"
      )

By the way, don’t forget to make sure that the log files are accessible to the user under which rsyslog runs.

I recommend that you keep all the parsing and log redirection logic in rsyslog config files. But if you don’t want to do so, and instead want to create a separate rsyslog configuration in your specific usecase, below you can find a bash script to do what we have done above.

The script tails a log file, parses each record to assign an appropriate severity level, and forwards these records to syslog:

#! /bin/bash

tail -F /path/to/log-file.log | while read line; do
  if [[ "$line" == *"Error:"* ]]; then
    logger -p local0.err "$line" # Forward error messages as errors
  else
    logger -p local0.info "$line" # Forward other messages as info
  fi
done

If you want to test whether the above script works, just create a log-file.log, run the script, and then issue echo "Error: this is a test error message" >> log-file.log in a separate terminal. After that you should see the error message in the rsyslog log file /var/log/messages or /var/log/syslog.

Running the script above directly will block your terminal and wait for its compilation. So for practical scenarios, you'll want to dispatch it to the background using, for example, setsid or other tools.

One important thing before we move on is that when testing the above scripts and configurations, be aware that your rsyslog might have a deduplication feature on. If this is the case, duplicate messages might not get processed. This is a legacy feature but chances are that it’s still in your configuration (mainly on Debian-based systems). Read more here.

In addition, you can drop unwanted messages.

Docker and syslog

Default logs generated to stdout/stderr by the applications running in Docker containers are stored in files under /var/lib/docker/containers (the exact path may depend on your system).

To access the logs for a particular container you can use docker logs <container name or container id>. But what if you want to redirect the stdout and stderr of your containerized applications into syslog directly? Then there are again multiple options. Below I will be using a Mosquitto broker container as an example.

Configuring a single Docker container

If you are starting a container using a docker run command, refer to the example below:

docker run -it -d -p 1883:1883 -v /etc/mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf --log-driver=syslog --log-opt syslog-address=udp://192.168.0.1:514 --log-opt tag="docker/{{.Name}}/{{.ID}}" eclipse-mosquitto:2

In this example:

• --log-driver=syslog specifies that the syslog driver should be used.
• --log-opt syslog-address=udp://192.168.0.1:514 specifies the protocol, address, and port of your syslog server. If you have a syslog server running locally and just want your logs to appear under /var/log on your local machine, then you can omit this option.
• --log-opt tag="docker-{{.Name}}-{{.ID}}" sets a custom TAG field for the logs from this container. {{.Name}} will resolve to the container name, while {{.ID}} to the container id. Note that you shouldn’t use slashes (“/”) here as rsyslog will not parse them and truncate the TAG parts which follow. But it will work with hyphens (“-”). This also implies that rsyslog tries its best to parse all the possible message formats and it might not always be what you expect. You can read more here.
• The rest of the flags like -it -d or -p and -v are container specific flags which specify the mode of the container, mapped ports, volumes and so on. You can read more about them in detail in this article.

Configuring a Docker service through docker-compose file

If you are using docker-compose instead of executing docker run directly, here is an example docker-compose.yml file:

version: '3'
services:
  mosquitto:
    image: eclipse-mosquitto:2
    logging:
      driver: syslog
      options:
        syslog-address: "udp://192.168.0.1:514"
        tag: "docker/{{.Name}}/{{.ID}}"
   ports:
     - 1883:1883
     - 8883:8883
     - 9001:9001
   volumes:
     - ./mosquitto/config:/mosquitto/config

Pay attention to directives driver, syslog-address, and tag, which are similar to the docker-run example.

Configuring a default for every container through the Docker daemon

If you don't want to specify log driver options in every docker-compose file or every time you use a “docker run” command, you can set the following configuration in /etc/docker/daemon.json which will apply it globally.

{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "udp://192.168.0.1:514",
    "tag": "docker/{{.Name}}/{{.ID}}"
  }
}

After that, restart docker with sudo systemctl restart docker or sudo service docker restart.

Enabling applications inside Docker to log to syslog directly

If you have an application which is able to forward its logs to syslog directly (such as Mosquitto) and you want to use it in a container, then you will have to map the local /dev/log to /dev/log inside the container.

For that, you can use the volumes section of docker-compose.yml or the -v flag of the docker run command.

How to Use Logging Libraries for Your Programming Language to log to syslog

Now, what if you are developing an application or need to create some custom aggregation script which forwards messages from certain apps or devices to syslog?

To give a simple real world example, you might want to build a control console for your IoT devices.

Let’s say you have a bunch of devices that connect to an MQTT broker. Whenever those devices generate log messages, they publish them to a certain MQTT topic.

In this case you might want to create a custom script that subscribes to this topic, receives messages from it and forwards them to syslog for further storage and processing. This way you will gather all your logs in one place with the ability to further visualize and manage them with tools such as Splunk, Elastic stack, and so on or run any statistics or reports on them.

Below, I am going to show you how to fire messages from your Node.js or Python application to syslog. This will enable you to implement your custom applications that work with syslog.

Note that the example scenario above gave a practical use case for what we will see in this section. But we will not explore it further, since it would require a bit more time and effort and might lead us off of the main point of this guide.

But if you are interested in managing something like the above, you can easily extend the scripts I show below by using the MqttJS library and connecting to the broker with Node.js as described here or using Paho MQTT Python client as shown in this tutorial.

Node.js client

Unfortunately, there aren’t many popular, maintainable, well-proven libraries for syslog logging with Node.js. But one good option you can use is a flexible general purpose library for logging called winston. It's quite usable in both small and larger scale projects.

When installing winston, you will also have to additionally install a custom transport called winston-syslog:

npm install winston winston-syslog

Here is a usage example:

const winston = require('winston');
require('winston-syslog').Syslog;

const logger = winston.createLogger({
  levels: winston.config.syslog.levels,
  format: winston.format.printf((info) => {
    return `${info.message}`;
  }),
  transports: [
    new winston.transports.Syslog({
      app_name: 'MyNodeApp',
      facility: 'local0',
      type: 'RFC5424',
      protocol: 'unix', // Use Unix socket
      path: '/dev/log', // Path to the Unix socket for syslog
    })
  ]
});

// Log messages of various severity levels
// When using emerg level you might get some warnings in your terminal
// But don't panic - this is expected, since it's the most sever level 
logger.emerg('This is an emerge message.');
logger.alert('This is an alert message.');
logger.crit('This is a crit message.');
logger.error('This is an error message.');
logger.warning('This is a warning message.');
logger.notice('This is a notice message.');
logger.info('This is an informational message.');
logger.debug('This is a debug message.');


logger.end();

Note that if you remove the format property from the object passed to createLogger, you will see a JSON payload consisting of a message and severity level for messages in syslog. That’s the default format of records parsed by winston-syslog.

Python client

In case of Python, you don’t even have to install any third party dependencies as Python already comes with two quite capable libraries: syslog and logging. You can use either one.

The former is tailored to working with syslog specifically, while the latter can also handle other log transports (stdout, file, and so on). It can also often be seamlessly extended to work with syslog for existing projects.

Here is an example of using the “syslog” library:

import syslog

# Log an single info message
# Triggers an implicit call to openlog() with no parameters
syslog.syslog(syslog.LOG_INFO, "Message an info message.")

# You can also set the facility
syslog.openlog(ident="MyPythonApp", facility=syslog.LOG_LOCAL0)

# messages with different severity levels and LOG_LOCAL0 facility
syslog.syslog(syslog.LOG_EMERG, "This is an emerge message.")
syslog.syslog(syslog.LOG_ALERT, "This is an alert message.")
syslog.syslog(syslog.LOG_CRIT, "This is a critical message.")
syslog.syslog(syslog.LOG_ERR, "This is an error message.")
syslog.syslog(syslog.LOG_WARNING, "This is a warning message.")
syslog.syslog(syslog.LOG_NOTICE, "This is an notice message.")
syslog.syslog(syslog.LOG_INFO, "This is an informational message.")
syslog.syslog(syslog.LOG_DEBUG, "This is a debug message.")

# Close the log if necessary (usually handled automatically at program exit)
syslog.closelog()

And here is an example of using the “logging” library. Note that “logging” has a predefined set of log levels which does not fully align with syslog severity levels (for example, some levels like “crit”, “emerg”, and “notice” are missing by default). You can, however, extend it when needed but we will keep it simple here. For more information refer here:

import logging
import logging.handlers

# Create a logger
logger = logging.getLogger('MyPythonApp') # Set application name
logger.setLevel(logging.INFO) # Set the default log level

# Create a SysLogHandler
syslog_handler = logging.handlers.SysLogHandler(address='/dev/log', facility=logging.handlers.SysLogHandler.LOG_LOCAL0)

# Optional: format the log message
# Set a format that can be parsed by rsyslog.
# The one presented below is a simplification of RFC3164
# Note that PRI value will be prepended to the beginning automatically
formatter = logging.Formatter("%(name)s: %(message)s") 
syslog_handler.setFormatter(formatter)

# Add the SysLogHandler to the logger
logger.addHandler(syslog_handler)

# Log messages with standard logging levels
logger.debug('This is a debug message.')
logger.info('This is an informational message.')
logger.warning('This is a warning message.')
logger.error('This is an error message.')
logger.critical('This is a critical message.')

Alternatively, you can use a non-standard “loguru” library or any other. Built-in libraries are quite powerful and sufficient for most of the use cases. But if you are already using a library like loguru in your project, you can extend it to work with syslog:

from loguru import logger
import logging
import logging.handlers

class SyslogHandler:
    def __init__(self, appname, address='/dev/log', facility=logging.handlers.SysLogHandler.LOG_USER):
        self.appname = appname
        self.handler = logging.handlers.SysLogHandler(address=address, facility=facility)
        self.loglevel_map = {
            "TRACE": logging.DEBUG,
            "DEBUG": logging.DEBUG,
            "INFO": logging.INFO,
            "SUCCESS": logging.INFO,
            "WARNING": logging.WARNING,
            "ERROR": logging.ERROR,
            "CRITICAL": logging.CRITICAL,
        }

    def write(self, message):
        # Extract the log level, message text, and other necessary information.
        loglevel = self.loglevel_map.get(message.record["level"].name, logging.INFO)
        logmsg = f"{self.appname}: {message.record['message']}"

        # Create a log record that the standard logging system can understand.
        record = logging.LogRecord(name=self.appname, level=loglevel, pathname="", lineno=0, msg=logmsg, args=(), exc_info=None)
        self.handler.emit(record)

    def flush(self):
        pass

# Configure Loguru to use the above defined SyslogHandler
appname = "MyPythonApp"
logger.add(SyslogHandler(appname), format="{message}")

# Now you can log messages and they will appear be directed to syslog
logger.info("This is an informational message sent to syslog.")

Conclusion

In this handbook, you learned all about syslog. We clarified the confusing terminology, explored its use cases, and saw a lot of usage examples.

The main points to take away are:

• Syslog is a protocol describing the common format of message exchange between applications and syslog daemons. The latter take on message parsing, enrichments, transport, and storage.
• People commonly colloquially refer to the infrastructure of syslog daemons, their configuration, log storage files, and accepting sockets as “syslog”. “Redirect logs to syslog” means redirect the logs to the /dev/log socket where they will be picked up by a syslog daemon, processed, and saved according to its configuration.
• There are two standard syslog formats: the obsolete RFC3164 and a newer RFC5424.
• Some well known syslog daemons include: sysklogd (Linux), rsyslog (Linux), syslog-ng (Linux), and nxlog (cross-platform).
• Rsyslog and other log daemons can forward logs from one server to another. You can use this to create a log collecting infrastructure with a central server processing all the logs coming from the rest of the nodes.
• Even though it incurs some overhead, it’s important to forward the logs using the TLS protocol in case they are transported over an untrustworthy network.
• The rsyslog daemon is a lightweight and powerful tool with many features. It can collect messages from different sources, including files and network. It can process this data using customizable templates and rulesets, and then either save it to disk or forward it elsewhere. Rsyslog can also directly integrate with Elasticsearch, among other capabilities.
• It is possible to forward the logs of an application to syslog even if it does not provide a native integration. You can do this for standalone host applications, containerized systems, or through an aggregation script written in a programming language of your choice.
• Output of the standalone apps (stdout and stderr) can be captured and redirected to the logger Linux utility tool. Docker provides a separate syslog driver for logging, while many programming languages have dedicated logging libraries.

Thanks for reading, and happy logging!

But fear not, fellow coder, for I bring you the trusty map to navigate the treacherous bug-infested waters of programming. Whether you're a self-learner aiming to land that dream job in tech or just tinkering for fun, here's how to become a debugging ninja.

Look for Errors in Your IDE

Your Integrated Development Environment (IDE) isn't just a fancy text editor – it's your first line of defense against bugs.

TypeScript, for example, is like that friend who points out the pothole you're about to step into – it helps catch errors early with its type-checking prowess.

Imagine you accidentally try to add a number to a string. TypeScript waves a big red flag, saving you from a facepalm moment later. It's one of the many reasons we adore TS.

Example: You declare let age: number = 'twenty';. TypeScript will frown upon this, telling you that 'twenty' is not a number. It's like having a guardian angel for your code.

Try and Isolate the Area

Before you start pulling out your hair, try to play detective and isolate where the crime scene is.

Is the bug lurking in the backend, hiding in the frontend, conspiring in the database, or chilling in the infrastructure?

When you're working locally, it's usually one of the first three suspects. And here's a hot tip: the network tab in your browser's developer tools is like your police scanner, helping you pinpoint the location of the distress call.

Example: Let's say you send out a request to GET /users and it returns a 500 status. That's the server telling you, "Mate, I've got problems." It's a backend issue. But if the call comes back with a 200 status and your UI is still playing hide and seek with the data, then the bug's hosting a party in your frontend. The network tab just handed you the address.

By narrowing down the location of your issue, you can focus your debugging efforts more efficiently. It's like knowing whether to raid the castle, the dragon's lair, or the dark forest. Happy hunting!

Look for Errors in the Browser Console

The browser console is your Sherlock Holmes magnifying glass for web projects. It uncovers clues hidden in plain sight. The console tab, on the other hand, is like tracking where the villain has been, helping you spot those pesky code misfires.

Example: Your React app isn't fetching data. A quick peek in the console tab shows an "undefined" error, and a line number. This is where your problem is at. Elementary, my dear Watson!

Add console.log() to Different Functions

Ah, the humble console.log(), the print statement that could. When in doubt, log it out. It's like dropping breadcrumbs through your code to see how far Little Red Riding Hood gets before she meets the Big Bad Bug.

Example: Unsure if your function is receiving the expected data? console.log('Data:', data) at the start of the function. No data? Now you know where the problem starts.

Use Try-Catch Blocks

Try-catch blocks are your safety net, allowing your code to perform daring feats without crashing your app. They let you gracefully handle errors by catching them before they wreak havoc. They also let you specify your own custom error messages for a given block of code, helping you to find the problem area.

Example: Wrap your API call in a try-catch. If the call fails, the catch block catches the error, allowing you to console.log it or display a friendly message to the user.

Heres what a try catch block looks like in JS:

  function displayUsers() {
    try {
      const users = getUsers();
    } catch (error) {
      console.log("oh crap");
    }
  }

Search Google or Use ChatGPT to Help With Error Messages

Stuck on an error message? Google and ChatGPT are your library and librarian. Just copy and paste the error into the search bar and watch a plethora of solutions unfold. It's like asking the hive mind: someone, somewhere, has had your problem before.

Example: Getting a "TypeError: Cannot read property 'map' of undefined"? A quick search reveals you might be trying to use .map() on something that's not an array. Oops!

Test Often

The mantra "test early, test often" will save you heaps of time. By testing small bits of code as you go, you catch bugs early, when they're easier to squash. It's like cleaning as you cook– it makes the final cleanup so much easier.

Example: Just added a new feature? Test it out before moving on. Does it work as expected? Great! No? Time to debug while the code is still fresh in your mind.

Try a Different Approach

If you're banging your head against the wall with a problem, maybe it's time to climb over it instead. Don't get too attached to your code. Be willing to refactor or even start from scratch if it means a cleaner, more elegant solution.

Example: If your code is more tangled than a bowl of spaghetti, stepping back and rethinking your approach might reveal a simpler, more efficient path.

Debugging is part art, part science, and entirely a test of patience. But with these strategies in your toolkit, you'll be squashing bugs with the best of them. Happy coding, and may your bug hunts be short and your code be clean!

Real Life Scenario

Let's take real life scenario. I have a React, Node, Postgres app that displays users in the browser. The code, as far as I know, should be working but I'm not seeing the users displayed on the frontend.

Step 1 – Check the Console

Let's have a nosey at the Chrome dev tools console and see what's going on.

"Look ma, errors!"

Ah, the plot thickens in the saga of "Why isn't this thing working?". Let's dive into the drama unfolding in your console and break down the breadcrumbs left behind by our mischievous friend, the bug.

First up, we have our leading clue: GET http://localhost:3000/api/users 500 (Internal Server Error). This line is the equivalent of a scream in the night in our detective story. It tells us that our backend is in distress, possibly tied to a nefarious SQL query or a rogue piece of logic in our API route.

The server's cry for help is loud and clear: "Internal Server Error." Classic move by the backend, really.

Now, our supporting cast makes their entrance with ResponseError: Response returned an error code. This is the big reveal. The issue isn't just a server having a bad day – it's a ResponseError caught red-handed by UsersApi.request, and even tells us where the error line is (UserApi.ts:83).

Step 2 – Check the backend Terminal

Our journey into investigating the bug has brought us to the backend, where we are greeted with this:

If you see this and your first instinct is to run away and hide, don't worry – that was mine too. But fear not! There are plenty of clues that point us to the issue.

When a backend error occurs, this is what's known as a stack trace – basically all the errors, info, line numbers, and so on that the compiler encountered in one big block of text. Thanks compiler!

What we do here is look for key words, recognisable files, or anything that's readable by humans. Did you spot anything? Let's dive deeper:

Digging deeper into the errors

The highlighted parts in yellow, indicate that there was an error in userController.ts, specifically at getAllUsers() function. If we read further, the highlighted parts in red point us to the error message:

Authentication failed against database server at `dpg-cn9kr28l6cac73a1a7eg-a.frankfurt-postgres.render.com`, 
the provided database credentials for `dmin` are not valid.\n\nPlease make sure to provide valid database credentials for the database server

Hurray! Now we know the error. We have spelled "admin" incorrectly in our database connection string, meaning the connection failed. Doh! After we fix this, the error is resolved:

Error resolved

Step 3: Verify the Fix

Now we've added a fix, we can verify by checking the browser to see if everything's working. In this case, checking the UI is enough to verify, but for more complex flows you can get that the API is returning the correct status code (in this case, 200)

Conclusion

I hope this article has shed some light on how you can debug your projects.

If you are looking for more debugging insights, and real industry level projects to build, you can check out by YouTube where we build and deploy full stack applications using React, Node, and some other cool tech. Hope to see you there!

Logging, safeguarding, and utilizing data is a massive part of any software development operation. And chronological records of events have become an invaluable tool for determining future decisions.

Today, we're fortunate enough to have the concept of system logs, which represents a collection of files.

These files have data crucial to software development and system operations. And it's important to properly organize these records for any tech uses cases, whether IT systems or software development.

What is Log Management?

Applications and programs have data logs, including information gathered over a given period. Logs are computer-generated files that record processes or activities within a software or operating system. These files keep track of any events that occur within a network or system. An event log is a file where these activities are recorded.

These can include error reports, messages, file transfers, and requests, which typically have time stamps. These time stamps give developers and IT professionals a better view of what and when an event occurred.

In short, log management refers to the practice of constantly gathering, processing, storing, analyzing, and synthesizing data.

All these processes are necessary for optimizing system performance, identifying technical issues, enhancing resource management, improving compliance, and tightening security.

Log management falls under the following categories:

• Collection: This includes any cluster data from applications, users, servers, endpoints, operating systems, or other significant sources within the company.
• Analysis: This tool analyzes the log collection extracted from the log server. Its purpose is to proactively identify security threats, bugs, and other issues.
• Monitoring: This tracks activities and events while providing time stamps for accurate documentation.
• Indexing: Commonly used to assist the IT team with searching, filtering, sorting, and analyzing all logs.
• Retention: This tool determines the duration of retained log data within the log file.
• Reporting: This feature automates audit logs reports concerning resource allocation, operational performance, and security and regulatory compliance.

How Do Log Management Systems Help?

By definition, a log management system or LMS is a software solution for gathering, sorting, and storing event logs and log data. Event logs and data come from several sources, and collecting and sorting them can take time.

Log management system software makes storing data in one centralized database easier for developers, IT teams, and security officers. This automates the process of file indexing, making each event searchable.

IT teams have easier access to data required to make resource allocation, security, and network integrity decisions. Log management tools help manage the high volumes of organization log data created across the organization, effectively streamlining activities within the network.

These tools assist in distinguishing:

• Which information and data need logging
• What format to use for logging
• What is the correct time to save log data
• What method to use for disposing or destroying expendable data

Why is Log Management Critical?

Image via PAOLO / Adobe Stock

The best log management strategies and systems provide real-time insights into system operations and integrity. Log management solutions are most effective when they give you the following:

• Merged data storage using a unified log aggregation
• Boosted security through real-time surveillance, minimized attack surface, improved detection, and quicker response times
• Better visibility and observable variables in every part of the enterprise using the same event log
• Quicker and more accurate troubleshooting ability using advanced network analytics
• Amplified customer experience using log data, predictive modeling,and data analysis

What Are the Common Challenges in Log Management?

Typically, when there is a proliferation of devices connected to a system, it causes a sudden outburst of data. That outburst, especially in cases where a system shifts to the cloud, creates more complexities in log management for organizations.

The best way to tell if a log management solution is effective is if it addresses the following challenges:

1. Latency

Cataloging within the log file can be very time-consuming and computationally expensive. It can cause delays between entering data into the system and including data in search visualizations and results. A good log management system should give low latency to achieve faster transactions.

2. Uniformity

Because a log management solution extracts data from various systems, applications, and hosts, data characteristics can vary. As they need to be collated into one system, these data must have the same format.

When dealing with uniformly presented data, information security and IT teams can readily generate insights and analysis to improve business services and operations.

3. Volume

Event logs can generate data at an incredible rate. For most companies, the sheer amount of data constantly produced takes enormous effort to organize.

Effective log management software must be able to handle the collection, formatting, analysis, and storage of a ridiculous amount of data. It must also be able to do all that while providing timely insights.

4. High IT Workload

Manual log management is incredibly labor-intensive, expensive, and time-consuming. Proper digital log management software assists in automating these tedious tasks to reduce the burden on IT professionals.

What to Look for in Log Management Systems

Considering the tremendous amounts of data generated in any organization today, manually managing logs is impossible for IT professionals. That’s why you should look for advanced log management software and systems. Specifically, the software is invaluable in automating key aspects such as data gathering, formatting, and analysis.

Here are the key things developers and IT organizations should look into before investing in log management software.

1. Automation prioritization to decrease IT workload

Log management can drain resources from the IT team because it is time-consuming. Most of these systematic data collection and analysis tasks can be automated using advanced tooling.

You want to avoid overstraining your IT department with time-consuming processes as much as possible. Sophisticated log management tools have automation tools to mitigate this problem.

For instance, if your system has a bug that could potentially crash critical processes. With IT professionals unburdened by tedious processes, they can easily detect and create a patch for the bug.

2. Unified systems to boost access and improve security

When you have log management centralized, it improves data access while strengthening an organization's security. Connecting and saving data in a central location allows for faster anomaly detection and action.

A centralized log management system helps companies stay ahead of vulnerabilities and mitigate any system damage. It is also another line of defense against cyber criminals trying to break into your system via those vulnerabilities and malware.

For example, one of your technologies has an outdated operating system that could result in a security vulnerability. With a centralized log, it is easier for IT professionals to detect this problem and quickly resolve it.

3. Flexibility and scalability by leveraging the cloud

Since the data landscape is consistently growing, a cloud-based solution makes the most sense for log management. Cloud-based log management tools are ideal for their scalability. It allows them to increase or decrease processing and storage based on their current needs.

In any business involving technology, data management and organization is paramount. An effective log management software is critical in managing event logs and streamlining operations.

A typical example of what slows them down is how a company scales goes back to operational tasks. When operational tasks are streamlined, it provides the required space to generate improvements to internal systems.

Log Management Systems Are Not Optional

As you have already guessed, log management systems are an absolute necessity for IT professionals. Without them, your IT team will always be swamped with operational tasks, and vulnerabilities will slip through the cracks.

These days it is no longer a question of if you have a log management system but what kind you are using.

Feature image via Unsplash (Campaign Creators).

In this tutorial, you'll learn the following:

• Introduction to logging
• Understanding log configurations
• Channel drivers for log files
• Formatting log messages

Log file image

Introduction to Logging

Laravel provides a log of what's happening in your application. The log service is built upon the Monolog open-source library.

The logging service is robust as it allows you to write log messages to files and send critical ones to teams on Slack (if configured), Socket, databases, and other web services.

The channel you wish to write your log information on is defined by the team, as there are a couple of channels supported by Laravel. Based on the severity of log information, the write information can also be done in multiple channels. You'll see how you can do this in the configuration section.

How to Configure Your Laravel Logs

Laravel log configuration is located in the config > logging.php file. Consider using a couple of log channels based on your preferences, such as stack, single, daily, syslogs, slack, papertail, and so on.

The channels are where you send log information. The default channel for every project is usually stack. You can change it by defining the LOG_CHANNEL in the env or specifying the channel name as the second parameter in the same logging.php file.

'default' => env('LOG_CHANNEL', 'stack')

The stack channel has a driver name set to stack. Channels set to single means you get all logs in a single log file. You can also use daily which means a new log is auto-generated every day. It is an array.

You can also use multiple channels, 'channels' => ['daily', 'slack'], and ignore_exception is a boolean (true, false).

I highly recommend using the daily channel, as this helps you keep track of daily logs by auto-generating a new log file every day (laravel-2023-01-15.log, laravel-2023-01-16.log and so on) without having to clear logs for the previous day.

The daily options keep you updated each day with log info in your log files for as long as you want. It also enables you to monitor frequent errors within the application if they occur on different days.

 'channels' => [        
    'stack' => [ 
        'driver' => 'stack',            
        'channels' => ['daily'],    
        'ignore_exceptions' => false,        
        ],
....],

Channel Drivers for Log Files

Here’s the list of the channel drivers Laravel offers:

1. Single: The Single driver ensures log information is sent to a single file. The driver sends logs to storage > logs > laravel.log by default.
2. daily: The driver ensures that logs are written daily. The beauty about this is that every day new log file is auto-generated. Compared to the Single driver, there's no need to clear up the previous day's log information frequently. But the drawback to this channel is that you might have several log files. Every week/month, you should clear up all unused files.

Within the logs directory, you often get logs like this:

Daily log file

3. slack: The slack driver ensures that all logs are sent to Slack. Slack needs to be configured to get user credentials (username, webhook) to help with error logging. This is super helpful as it allows your team to stay updated about what's happening right in a Slack channel.

4. syslog: Logs using this driver will send log reports to the system log. The location of this log driver is dependent on the server operating system.

5. errorlog: Logs set up to use this driver will send log reports to the error logs file setup on the web server operating system.

6. monolog: This driver provides support for all Monolog handlers.

7. custom: This driver helps create a custom channel based on user preference. It could be to a third-party service that needs log reports.

8. stack: The stack driver is responsible for creating multiple channels

9. null: All log messages get discarded by the driver.

How to Format Log Messages

If you need a refresher on how facades work or how to create one, you should refer to this article about how facades work in Laravel.

Laravel has a Log facade that helps with writing logs. Import the facade at the top of the file to use any log level.

<?php

use Illuminate\Support\Facades\Log; 


Log::info($message);

You can also choose to escape the Log facade, so you won't need to import anything. This is suitable if you're logging a single instance of log info.

<?php

\Log::info($message);

In a recent Laravel release, logging has greatly improved. You can do away with the Log facade while logging info and reference the info from within your Laravel application.

<?php

info($message);

Other logging levels used to write information include alert, emergency, critical, warning, error, notice, and debug.

Within a file, you can log any of the data types or messages and even format the output of text you wish to write to the log file.

How to format strings, booleans, and integers:

<?php

use Illuminate\Support\Facades\Log;


Log::warning('There is a warning'); 

Log::error(false);

Log::notice(500);

How to format to an array:

You can also format to an array. With the array function, a new array is created. So we can write an array to the log file by passing the log info to the array function. The json_decode converts the JSON object to a PHP object, and the true ensure it returns associative arrays (key and value pairs).

<?php

$person = '{"Peter":35, "John":37, "Yinka":43}'; 

$data = json_decode($person, true);

info($data);

How to format to an object:

You can also write JSON objects to the log file when working with logs. Use the json_encode to encode values into JSON format.

<?php 

$data = ["Peter"=> 35, "John"=> 37, "Yinka" => 43];

info(json_encode($data));

How to concatenate string with array or objects:

This is helpful when including a string to track the log information. You can do this using the concatenation operator (.)

<?php

$persons = ["Peter"=> 35, "John"=> 37, "Yinka" => 43];

info('The person info ' . json_encode($persons));

How to write to dedicated channels:

This method is helpful when you feel there's a need to write into specific channels aside from the default log channel. It would help if you had to specify the channel name when calling the Log facade.

<?php


\Log::channel('slack')->info('registeration successful');

The snippet above ensures the write operation is done on the Slack channel. Also, the stack method allows logging on multiple channels.

<?php


\Log::stack(['single', 'slack'])->info('registeration successful!');

You can learn more about custom channels via factories and monolog channel customization from the official documentation.

Wrapping up

In this article, you have learnt about logging, configuring logs in your Laravel application, available channel drivers, and how to write log files in different formats.

You should now have a better understanding of laravel logging. Keep learning, and Happy Coding!

You can find me on LinkedIn and Twitter.

Say you're working on your new great idea – a web or mobile app, and a back end server. Nothing too complicated so far. Until you realize that you need to stream data from your server to these clients.

Usually, when working on this, the first thing that comes to mind is to use one of the cool kids on the block, like WebSockets, SocketIO, or even a paid service that takes care of it for you.

But there's another method that's usually left out, and you might not have heard about it yet. It's called SSE, short for Server-Sent Events.

SSE has a special place in my heart because of its simplicity. It's lightweight, efficient, and very powerful.

To explain SSE in detail and how I use it, I will go over a small side project of mine that I think is an excellent showcase of SSE. I'll be using Typescript, Express, and RxJS, so get your environment ready and buckle up as we are about to dive into some code.

Before we get started, there is something that you should know about SSE. As its name suggests, Server-Sent Events is uni-directional from your server to the client. This may be a deal-breaker if your client needs to stream back data to the server. But this is not the case in many scenarios, and we can just rely on REST to send data to the server.

What's the Project?

The idea of this project is simple: I have a bunch of scripts running around on Raspberry Pis, droplets on Digital Ocean, and other places that are not easily accessible to me. So I want a way to print out logs and view them from anywhere.

As a solution, I would like a basic web app to push my logs and have a direct link to my session that I can open on any device or even share with others.

There are a couple of things to keep in mind before we proceed.

First, logs coming from my scripts are not that frequent, and the overhead of using HTTP is negligible for my use case. Because of this, I decided to publish my logs over a basic REST API and use SSE on the client-side to subscribe the incoming logs.

Logging Example

Second, this tool is mainly for quickly debugging things I'm working on. There are many production-ready and enterprise tools out there that I could use instead. But I wanted something very light and easy to use.

Let's Write Some Server-side Code

The server-side setup is straightforward. So let's start with a diagram to give you an idea of the setup before explaining everything in detail.

Server Diagram

If we think of our backend server as a pipeline, on one end we have a series of publishers – in our case, the scripts publishing logs. On the other end, we have some clients subscribing to these logs.

To connect these two ends, I will be using an RxJS Subject. It will allow me to publish anything from the publishers over REST and then subscribe to these events and forward the messages to the clients over SSE.

To get started, let's define our Log interface. To keep things simple, I will only define a content field that will hold our log information.

interface Log {
  content: string;
}

How to set up RxJS

Let's import RxJS, create a new Subject for our Logs, and define a function to publish our logs to this Subject.

Of course, we could export our Subject and directly call it from our router, but I prefer to abstract away the implementation and only provide the emit function to the rest of my code.

import { Subject } from 'rxjs';

// Log Subject
const NewLog$ = new Subject<Log>();

/**
 * Emit a new log to the RxJS subject
 * @param log
 */
export function emitNewLog(log: Log): void {
    NewLog$.next(log);
}

Finally, let's define a new route on our Express server that would accept new logs from our client and publish them to the emitNewLog method that we have just created.

app.post('/', (req: Request, res: Response) => {
  const content = req.body.content;
  const log: Log = { content: content };
  emitNewLog(log);
  return res.status(200).json({ ok: true });
});

We are now done with the publishing side. What's left is to define our SSE route, subscribe to the RxJS Subject, and deliver the logs to our client.

How to set up the SSE Route

Let's define a new route for our SSE connection. To enable SSE, we need to flush a couple of headers back to our client.

We want the ‘Connection’ set to ‘keep-alive’, ‘Cache-Control’ set to ‘no-cache’, and ‘Content-Type’ set to ‘text/event-stream’. This way our client will understand that this is an SSE route.

In addition, I have added ‘Access-Control-Allow-Origin’ for CORS and ‘X-Accel-Buffering’ set to ‘no’ to keep Nginx from messing with this route. Finally, we can flush the headers back to our client to kickstart the event stream.

app.get('/', (req: Request, res: Response) => {
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Connection', 'keep-alive');
  res.setHeader('Access-Control-Allow-Origin', '*');
  res.setHeader('X-Accel-Buffering', 'no');
  res.flushHeaders();
});

We can now start streaming data by writing something into our response.

SSE provides a text-based protocol that we can use to help our clients differentiate between the event types. Each one of our events should look like the following:

event: ${event name}\n
data: ${event data}\n\n

To make my life a bit easier, I have created a helper function to take care of serialization for us.

/**
 * SSE message serializer
 * @param event: Event name
 * @param data: Event data
 */
function serializeEvent(event: string, data: any): string {
  const jsonString = JSON.stringify(data);
  return `event: ${event}\ndata: ${jsonString}\n\n`;
}

We can now subscribe to the RxJS Subject we created earlier, serialize each new log, and write it as a NEW_LOG event to our connection.

app.get('/', (req: Request, res: Response) => {
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Connection', 'keep-alive');
  res.setHeader('Access-Control-Allow-Origin', '*');
  res.setHeader('X-Accel-Buffering', 'no');
  res.flushHeaders();

  NewLog$.subscribe((log: Log) => {
    res.write(serializeEvent('NEW_LOG', log));
  });

}

Finally, we have to make sure to unsubscribe from our observer when the SSE connection is closed. Putting all of these together, we should have something like this:

app.get('/', (req: Request, res: Response) => {
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Connection', 'keep-alive');
  res.setHeader('Access-Control-Allow-Origin', '*');
  res.setHeader('X-Accel-Buffering', 'no');
  res.flushHeaders();

  const stream$ = NewLog$.subscribe((log: Log) => {
    res.write(serializeEvent('NEW_LOG', log));
  });

  req.on('close', () => {
    stream$.unsubscribe();
  });
});

That’s it! We are done with our backend server and it’s time to move to the frontend code.

Write the Client Code

Subscribing to our SSE route on the browser is very straightforward. First, let’s move to our client code and create a new instance of the EventSource interface and pass our endpoint to the constructor.

const eventSource = new EventSource("/");

Then, we can add event listeners for the events we want to subscribe to (in our case, NEW_LOG) and define a callback method to handle our log.

eventSource.addEventListener(
   "NEW_LOG", (event) => {
       const log = JSON.parse(event.data);
       // use the data to update the UI
    }, false
);

And finally, we can close the connection whenever we are done listening to these events.

eventSource.close();

Conclusion

As you can see, Server-Sent Events make it very easy to stream content from the server to the client. They are specifically helpful because we get a built-in interface in most modern browsers, and we can easily poly-fill for those that do not provide the interface.

In addition, SSE automatically handles re-connect for us in case the client loses connection with the server. Therefore, it is a valid alternative to SocketIO and WebSockets in various scenarios where we need a uni-directional event streaming from the server.

If you are further interested in this project, I have added a couple of extra functionalities to the code that we just went over and a web GUI that you can check out here: LogSnag Console.

Console Demo

These logs provide information such as a trace stack, breadcrumbs, and (assuming this is a web application) browser data. This can help you triage issues and resolve bugs faster, with less investigative overhead.

How to Prepare Your Sentry Account

Begin by navigating to Sentry and clicking "Get Started". You will be taken to the account creation screen:

You can either sign up with OAuth or create separate credentials for Sentry. If you choose to create separate credentials, you'll need to enter an organization name now (this can be changed later). I used my username as my organization name.

Once you create your account, Sentry will take you through a tutorial to set up your first project. Click "I'm Ready" to be taken to the first step.

For our purposes, the NODE.JS option is the platform you should select. Then click "Create Project".

This takes you to the instructions for preparing the SDK to integrate with your codebase. Leave that page open as you will need your dsn value.

How to Use Sentry in Your Code

Your next step is to install the necessary Sentry packages:

npm install @sentry/node @sentry/integrations

The @sentry/node package is the core SDK for your Node.js project, and the @sentry/integrations package contains a tool you will use for mapping the file path.

Your Sentry tooling should be loaded as early as possible in your code flow. Ideally, this means you should initialize it within the entry point for your application (that is, index.ts).

Start by importing the packages:

import * as Sentry from "@sentry/node";
import { RewriteFrames } from "@sentry/integrations";

The first import pulls in the Sentry-Node tooling, and the second gives you access to the RewriteFrames integration. This integration allows you to adjust the pathing of the stack trace, which is necessary for properly pointing to your compiled JavaScript files.

Now you need to instantiate the Sentry monitor and provide the configuration:

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  tracesSampleRate: 1.0,
  integrations: [
    new RewriteFrames({
      root: global.__dirname,
    }),
  ],
});

Here you have passed a configuration object to the Sentry.init() method (which is used to instantiate and initialise the Sentry process). To break these options down:

• dsn is a unique URL used to connect your Sentry instance to your dashboard. We will explore this a bit later.
• tracesSampleRate determines the percent of events the monitor should send to the dashboard. A value of 1.0 sends 100% of the captured events – but if you find this to be too noisy you can reduce this number.
• integrations loads the integrations you want to use. In this case, you are loading the RewriteFrames option and setting the root path for your stack traces to global.__dirname (which resolves to the directory you run your application from).

Then, anywhere in your code base where you are logging errors (such as a try / catch block or a .catch() chain), add Sentry.captureException(error) (replacing error with the variable that represents your error object) to pass that error off to your Sentry monitor.

How to Connect Your Code to Your Dashboard

Back on that project setup page, you'll see a URL value for the dsn option in the configuration.

Your dsn should be treated as a secret and not shared with anyone. You can achieve this by adding it to your .env file (assign it to SENTRY_DSN to match with our configuration from the previous step).

The dsn tells Sentry where to send the captured errors, and the dashboard uses it to link those errors to your new project.

A note for front end projects:
Because you do not have access to a .env on the front end, you will need to expose your dsn publicly. We will cover how to handle this in the next step.

Once this is set up, you can click "View a sample event for this SDK" in the small print at the bottom of the Sentry page. This will generate a fake error event and take you to the dashboard.

How to Configure Your Sentry Dashboard

The Sentry website will offer you a quick tour of the dashboard, which you can follow if you would like, or you can skip it and continue with this article.

This view shows you the specific details for a captured error event. In this case, it is the sample event generated by Sentry from the previous step.

The top half offers information such as the browser data from the user that triggered the error, the error message, and the error type. The bottom half provides the stack trace and breadcrumbs (actions that took place to trigger this error) – both helpful for reproducing this error in triage.

At the very top you should see your project's name (which defaults to your organization name) and a gear. Click that gear to be taken to your project's settings.

Here you see some configurations for your project. The "Name" determines the name of your project. Changing the "Platform" affects how stack traces are rendered. You are welcome to experiment with the other settings as desired.

For front-end projects:
As mentioned earlier, you will need to expose your DSN publicly. You can set your webpage's URL in the "Allowed Domains" to prevent data being sent from any other source.

On the side bar are a few additional options. Selecting "Client Keys(DSN)" will take you to a page where you can copy your DSN again, if needed. You can also delete and regenerate it if you accidentally exposed it.

Selecting "Alerts" will allow you to configure how you receive notifications for error events. I have mine set to send to a Discord Webhook, but you can configure a number of integration options for receiving your alerts.

Finally you have the main side bar. Here you can configure your organization settings, including renaming your organization or creating additional organizations and projects.

Conclusion

You have now successfully integrated Sentry with your Node.js-Typescript project. You are now ready to start receiving error information, triaging issues, and improving your project's stability.

Feel free to experiment with Sentry's settings and features to personalize your experience to meet your needs. Happy Coding!

The only perfect code is code that has never been written. As a developer, you are bound to face errors and will be responsible for debugging them.

If you're coding in Python, you can always look at its error messages to figure out what's going on. But what if an error occurs that you have no idea what's breaking your code?

Something might be strangely wrong in the background, but you are unable to recognize it. You can always turn it off and on again – or even better, you can check the logs.

What is Logging?

If an error occurs or your app decides to work strangely, your log files will come in handy. You can traverse through them and find out where exactly the application is having problems and how you can replicate those problems.

By reproducing the problem, you can dig deeper and find a reasonable solution for the errors. Something which might otherwise take several hours to detect might just take few minutes to diagnose with the presence of log files.

How Logging Works in Django

Thankfully, Django has support for logging and most of the hard work has already been done by its developers. Django comes with Python's built-in logging module to leverage system logging.

The Python logging module has four main parts:

1. Loggers
2. Handlers
3. Filters
4. Formatters

Every component is explained meticulously in the Django Official Documentation. I don't want you to be overwhelmed with its complexity, so I'll explain every single part briefly:

1. Loggers

Loggers are basically the entry point of the logging system. This is what you'll actually work with as a developers.

When a message is received by the logger, the log level is compared to the log level of the logger. If it is the same or exceeds the log level of the logger, the message is sent to the handler for further processing. The log levels are:

• DEBUG: Low-level system information
• INFO: General system information
• WARNING: Minor problems related information
• ERROR: Major problems related information
• CRITICAL: Critical problems related information

2. Handlers

Handlers basically determine what happens to each message in a logger. It has log levels the same as Loggers. But, we can essentially define what way we want to handle each log level.

For example: ERROR log level messages can be sent in real-time to the developer, while INFO log levels can just be stored in a system file.

It essentially tells the system what to do with the message like writing it on the screen, a file, or to a network socket

3. Filters

A filter can sit between a Logger and a Handler. It can be used to filter the log record.

For example: in CRITICAL messages, you can set a filter which only allows a particular source to be processed.

4. Formatters

As the name suggests, formatters describe the format of the text which will be rendered.

Now that we have covered the basics, let's dig deeper with an actual example. Click here for the source code.

Please note that this tutorial assumes that you are already familiar with the basics of Django.

Project Setup

First, create a virtual environment called venv inside your project folder django-logging-tutorial with the command below and activate it.

``` bash mkdir django-logging-tutorial virtualenv venv source venv/bin/activate

Create a new Django project called django_logging_tutorial. Notice that the project folder name is with a dash while the project name is with an underscore (- vs _). We will also run a series of commands quickly to set up our project.

How to Configure Your Log Files

Let's first set up the settings.py file of our project. Heads up – notice my comments in the code which will help you understand this process better.

This code is also mentioned in the 3rd example of the official documentation and in most of our projects, it will serve just fine. I have slightly modified it to make it more robust.

``` python

LOGGING = { 'version': 1,

The version number of our log

'disable_existing_loggers': False,

django uses some of its own loggers for internal operations. In case you want to disable them just replace the False above with true.

A handler for WARNING. It is basically writing the WARNING messages into a file called WARNING.log

'handlers': { 'file': { 'level': 'WARNING', 'class': 'logging.FileHandler', 'filename': BASE_DIR / 'warning.log', }, },

A logger for WARNING which has a handler called 'file'. A logger can have multiple handler

'loggers': {

notice the blank '', Usually you would put built in loggers like django or root here based on your needs

'': { 'handlers': ['file'], #notice how file variable is called in handler which has been defined above 'level': 'WARNING', 'propagate': True, }, }, }

If you read my comments above, you might have noticed that the logger part was just blank. Which essentially means any logger.

Be careful with this approach as most of our work can be satisfied with in-built Django loggers like django.request or django.db.backends.

Also, for the sake of simplicity, I only used a file for storing the logs. Depending on your use case you might also choose to drop an email when CRITICAL or ERROR messages are encountered.

To learn more about this, I would encourage you to read the handler part of the docs. The docs might feel overwhelming at the start, but the more you get used to reading them the more you might discover other interesting or better approaches.

Don't worry if it's your first time working with documentation. There is always a first time for everything.

I've explained most of the code in the comments, but we still haven't touched upon propagate yet. What is it?

When propagate is set as True, a child will propagate all their logging calls to the parent. This means that we can define a handler at the root (parent) of the logger tree and all logging calls in the subtree (child) go to the handler defined in the parent.

It is also important to note that hierarchy is important here. We can also just set it up as True in our project as it won't matter in our case since there is no subtree.

How to Trigger Logs in Python

Now, we need to create some log messages so we can try out our configuration in settings.py.

Let's have a default homepage that just displays 'Hello FreeCodeCamp.org Reader :)' and every time someone visits the page we note down a WARNING message in our warning.log file as 'Homepage was accessed at 2021-08-29 22:23:33.551543 hours!'

Go to your app logging_example, and in views.py include the following code. Make sure you have added logging_example in the INSTALLED_APPS in setting.py.

``` python

from django.http import HttpResponse import datetime

import the logging library

import logging

Get an instance of a logger

logger = logging.getLogger(name)

def hello_reader(request): logger.warning('Homepage was accessed at '+str(datetime.datetime.now())+' hours!') return HttpResponse("

Hello FreeCodeCamp.org Reader :)

In the project's urls.py, add the following code so that when we access the homepage the right function is called.

``` python from django.contrib import admin from django.urls import path from logging_example import views

urlpatterns = [ path('admin/', admin.site.urls), path('',views.hello_reader, name="hello_reader") ]

Time for Some Testing

Finally, our simple setup is done. All we need to do now is to fire up our server and test our log.

Run the development server with this command:

``` bash python manage.py runserver

Now, go to your homepage 127.0.0.1:8000 where you will be greeted with the message we have coded. Now check your warning.log file in the path created. Sample output is shown below:

``` txt Homepage was accessed at 2021-08-29 22:38:29.922510 hours! Homepage was accessed at 2021-08-29 22:48:35.088296 hours!

That's it! Now you know how to perform logging in Django. If you have any questions, just drop me a message. I promise to help :)

If you found my article helpful and want to read more, please check out some Django tutorials at my blog Techflow360.com.

In my previous article, I talked about the importance of logs and the differences between structured and unstructured logging.

Logs are easy to integrate into your application, and they give you the ability to represent any type of data in the form of strings.

Metrics, on the other hand, are numerical representations of data. These are often used to count or measure a value and are aggregated over a period of time.

Metrics give us insights into the historical and current state of a system. Since they are just numbers, we can also use them to perform statistical analysis and predictions about the system’s future behaviour.

You can also use metrics to trigger alerts and notify you about issues in the system’s behaviour.

Logs vs. Metrics

How Logs and Metrics are Formatted

Logs are represented as strings. They can be simple text, JSON payloads, or key-value pairs (like we discussed in structured logging).

A typical log entry looks like this:

[2020-09-27T18:54:41,500+0530]-[ERROR]-[InventoryValidator]-[13] Exception in fetching product information - Product Not Available

Metrics are represented as numbers. They measure something (like CPU usage, number of errors, and so on) and are numeric in nature.

A typical metric looks like this:

{class=InventoryValidator, exception=Product Not Available, timestamp=1609306200}

The Resolution of Logs and Metrics

Logs contain high-resolution data. This includes complete information about an event and can be used to correlate the flow (or path) that the event took through the system.

In case of errors, logs contain the entire stack trace of the exception, which allows us to view and debug issues originating from downstream systems as well.

A log entry showing the stacktrace of an error

In short, logs can tell you what happened in the system at a certain time.

Metrics contain low-resolution data. This may include a count of parameters (such as requests, errors, and so on) and measures of resources (such as CPU and memory utilization).

A Metric showing number of hits to a service

In short, metrics can give you a count of something that happened in the system at a certain time.

The Cost of Logs and Metrics

Logs are expensive to store. The storage overhead of logs also increases over time and is directly proportional to the increase in traffic.

Metrics have a constant storage overhead. The cost of storage and retrieval of metrics does not increase too much with the increase in traffic. It is, however, dependent on the number of variables we emit with each metric.

Cardinality of Metrics

Metrics are identified by two key pieces of information:

• A metric name
• A set of key-value pairs called tags or labels

A permutation of these values provides the metric its cardinality. For example, if we are measuring the CPU utilization of a system with three hosts, the metric has a cardinality value of 3 and can have the following three values:

(name=pod.cpu.utilization, host=A)
(name=pod.cpu.utilization, host=B)
(name=pod.cpu.utilization, host=C)

Similarly, if we introduced another tag in the metric that determined the AWS region of the hosts (say, us-west-1 and us-west-2), we would now have a metric with a cardinality value of 6.

Types of Metrics

Golden signals

Golden signals are an effective way of monitoring the overall state of the system and identifying problems.

• Availability: State of your system measured from the perspective of clients (for example, the percentage of errors on total requests).
• Health: State of your system measured using periodic pings.
• Request Rate: Rate of incoming requests to the system.
• Saturation: How free or loaded the system is (foe example, the queue depth or available memory).
• Utilization: How busy the system is (for example, CPU load or memory usage). This is represented as a percentage.
• Error Rate: Rate of errors being produced in the system.
• Latency: Response time of the system, usually measured in the 95th or 99th percentile.

Resource metrics

Resource metrics are almost always made available by default from the infrastructure provider (AWS CloudWatch or Kubernetes metrics) and are used to monitor infrastructure health.

• CPU/Memory Utilization: Usage of the system’s core resources.
• Host Count: Number of hosts/pods that are running your system (used to detect availability issues due to pod crashes).
• Live Threads: Threads spawned in your service (used to detect issues in multi-threading).
• Heap Usage: Heap memory usage statistics (can help debug memory leaks).

Business metrics

Business metrics can be used to monitor granular interaction with core APIs or functionality in your services.

• Request Rate: Rate of requests to the APIs.
• Error Rate: Rate of errors being thrown by the APIs.
• Latency: Time taken to process requests by the APIs.

Dashboards and Alerts for Metrics

Since metrics are stored in a time-series database, it’s more efficient and reliable to run queries against them for measuring the state of the system.

You can use these queries to build dashboards for representing the historical state of the system.

A Wavefront dashboard with some important metrics

They can also be used to trigger alerts when there is an issue with the system (like an increase in the number of errors observed or a sudden spike in CPU utilization).

Due to their numeric nature, we can also create complex mathematical queries (such as X% of errors in last Y minutes) to monitor system health.

The biggest challenge, however, in handling metrics is deciding the right amount of cardinality that makes the metric useful while also keeping its costs under control.

Emitting too many metrics, or metrics with too many dimensions, can lead to an increase in storage and processing costs. You need to choose the minimum cardinality that is just enough to give a high level picture about the system.

How to Use Logs and Metrics

Both logs and metrics have their own pros and cons. However, in any production system, we need to use both logs and metrics together to effectively monitor the system and debug any issues.

Metrics are often the first line of sight into the health of a system. Let's take the example of an e-commerce application like Amazon. The most important metric for such a use-case is the total number of successful and failed orders.

On a normal day, the metric for number of failed orders would remain at zero or some very small number. If there is an issue in the system that causes orders to suddenly start failing, this metric will show an increase in count.

You can create an alert on a combination of two metrics - total orders and failed orders. This will allow you to send a notification when the percentage of failed orders increases beyond a certain threshold (say 5%).

Once you are notified about the failing orders, you can then refer to the logs to find the cause of the failures. The logs would contain the error messages leading to the failure, as well as the detailed stacktrace that can identify the root cause of the failure.

Conclusion

In this article, we saw the differences between metrics and logs, and how metrics can help us monitor the health of our system more efficiently. Metrics can also be used to create dashboards and alerts using monitoring software like Wavefront and Grafana.

It is also necessary to use both metrics and logs in coordination to accurately detect and debug issues.

Thank you for staying with me so far. Hope you liked the article. You can connect with me on LinkedIn where I regularly discuss technology and life. Also take a look at some of my other articles on Medium. Happy reading 🙂

Logging is one of the most important parts of software systems. Whether you have just started working on a new piece of software, or your system is running in a large scale production environment, you’ll always find yourself seeking help from log files.

Because of this, logs are the first thing developers look for when something goes wrong, or something doesn’t work as expected.

Logging the right information in the right way makes a developer's life so much easier. And in order to get better at logging, you need to know what and how to log.

In this article, we’ll take a look at the some basic logging etiquette that can help you get the most out of your logs.

What to Log and How Logging Works

Let’s start with an example of an e-commerce system and take a look at logging in its Order Management Service (OMS).

Suppose a customer order fails due to an error from Inventory Management Service (IMS), a downstream service that OMS uses to verify the available inventory.

Let’s assume that OMS has already accepted an order. But during the final verification step, IMS returns the following error because the product is no longer available in the inventory.

404 Product Not Available

What to Log

Normally, you would log the error in this way:

log.error(“Exception in fetching product information - {}”, ex.getResponseBodyAsString())

This will output a log in the following format:

[2020-09-27T18:54:41,500+0530]-[ERROR]-[InventoryValidator]-[13] Exception in fetching product information - Product Not Available

Well, there isn’t much information available in this log statement, is there? A log like this doesn’t serve much purpose because it lacks any contextual information about the error.

Can we add more information to this log to make it more relevant for debugging? How about adding the Order Id and Product Id?

log.error("Exception in processing Order #{} for Product #{} due to exception - {}", orderId, productId, ex.getResponseBodyAsString())

This will output a log in the following format:

[2020-09-27T18:54:41,500+0530]-[ERROR]-[InventoryValidator]-[13] Exception in processing Order #182726 for Product #21 due to exception - Product Not Available

Now this makes a lot of sense! Looking at the logs, we can understand that an error occurred while processing Order #182726 because Product #21 was not available.

How to Log

While the above log makes perfect sense for us humans, it may not be the the best format for machines. Let’s look at an example to understand why.

Suppose there is some issue in the availability of a certain product (say Product #21) due to which all orders containing that product are failing. You have been assigned the task to find all the failed orders for this product.

You happily do a grep for Product #21 in your logs and excitedly wait for the results. When the search completes, you get something like this

[2020-09-27T18:54:41,500+0530]-[ERROR]-[InventoryValidator]-[13] Exception in processing Order #182726 for Product #21 due to exception - Product Not Available

[2020-09-27T18:53:29,500+0530]-[ERROR]-[InventoryValidator]-[13] Exception in processing Order #972526 for Product #217 due to exception - Product Not Available

[2020-09-27T18:52:34,500+0530]-[ERROR]-[InventoryValidator]-[13] Exception in processing Order #46675754 for Product #21 due to exception - Product Not Available

[2020-09-27T18:52:13,500+0530]-[ERROR]-[InventoryValidator]-[13] Exception in processing Order #332254 for Product #2109 due to exception - Product Not Available

Not quite what you were expecting right? So how can you improve this? Structured logging to the rescue.

What is Structured Logging?

Structured logging solves these common problems and allows log analysis tools to provide additional capabilities. Logs written in a structured format are more machine-friendly, meaning they can be easily parsed by a machine.

This can be helpful in the following scenarios:

• Developers can search logs and correlate events, which is invaluable both during development as well as for troubleshooting production issues.
• Business teams can parse these logs and perform analysis over certain fields (for example, unique product count per day) by extracting and summarising these fields.
• You can build dashboards (both business and technical) by parsing the logs and performing aggregates over relevant fields.

Let’s use our earlier log statement and make a small change to make it structured.

log.error("Exception in processing OrderId={} for ProductId={} due to Error={}", orderId, productId, ex.getResponseBodyAsString())

This will output a log in the following format:

[2020-09-27T18:54:41,500+0530]-[ERROR]-[InventoryValidator]-[13] Exception in processing OrderId=182726 for ProductId=21 due to Error=Product Not Available

Now this log message can be easily parsed by the machine by using “=” as a delimiter to extract the OrderId, ProductId and Error fields. We can now do an exact search over ProductId=21 to accomplish our task.

This also allows us to perform more advanced analytics on the logs, such as preparing a report of all the orders that failed with such errors.

If you use a log management system like Splunk, the query Error=”Product Not Available” | stats count by ProductId can now produce the following result:

+-----------+-------+
| ProductId | count |
+-----------+-------+
| 21        | 5     |
| 27        | 12    |
+-----------+-------+

We could also use a JSON layout to print our logs in the JSON format:

{  
    "timestamp":"2020-09-27T18:54:41,500+0530"  
    "level":"ERROR"  
    "class":"InventoryValidator"  
    "line":"13"  
    "OrderId":"182726"  
    "ProductId":"21"  
    "Error":"Product Not Available"
}

It’s important to understand the approach behind structured logging. There is no fixed standard and it can be done in many different ways.

Conclusion

In this article, we saw the pitfalls of unstructured logging and the benefits and advantages offered by structured logging.

Log management systems such as Splunk are hugely benefited by a well structured log message and can offer easy search and analytics on log events.

The biggest challenge however, when it comes to structured logging, is establishing a standard set of fields for your software. This can be achieved by following a custom logging model or centralised logging which ensures that all developers use the same fields in their log messages.

Thank you for staying with me so far. Hope you liked the article. You can connect with me on LinkedIn where I regularly discuss technology and life. Also take a look at some of my other articles. Happy reading. 🙂

Logging is universally present in software projects and has many different forms, requirements, and flavors.

Logging is everywhere, from small 1-person-startups to large enterprises. Even a simple algorithmic programming question involves some logging along the way.

We rely so much on logging to develop, maintain, and keep our programs up and running. However, not much attention has been paid to how to design logging within our systems.

Often times, logging is treated as a second thought – it's only sprinkled into source code upon implementation like some magic powder that helps lighten the day-to-day operational abyss in our systems.

Are we all log baes?

Just like how any piece of code written will eventually become technical debt – a process that we can only slow down with great discipline – loggers rot at an unbelievable speed. After a while, we find ourselves fixing problems caused by loggers more often than the loggers give us useful information.

So how can we manage this mess with loggers and turn them into one of our allies rather than legacy ghosts haunting us from past development mistakes?

“State of The Art”

Before I dive deeper into my proposed solution, let’s define a concrete problem statement based on my observations.

So what exactly is logging? Here's an interesting and on-point one-liner that I found from Colin Eberhardt’s article:

Logging is the process of recording application actions and state to a secondary interface.

This is exactly how logging is woven into systems. We all seem to subconsciously agree that loggers don't belong to any particular layers of our systems. Instead, we consider them to be application-wide and shared amongst different components.

A much approved answer on StackOverflow

A simple diagram where logging has been fit into a system that is designed with clean architecture would look something like this:

We can safely say that logging itself is a subsystem within our application. And we can safely say that, without careful consideration, it often spirals out of control faster than we think.

While designing logging to be a subsystem within our applications is not wrong, the traditional perception of logging (with 4 to 6 levels of info, warn, error, debug, and so on) often makes developers focus on the wrong thing. It makes us focus on the format rather than the actual purpose of why we are writing logs.

This is one of the reasons why we log errors out without thinking twice about how to handle them. It's also why we log at every step of our code while ironically being unable to debug effectively if there is a production issue.

This is why I am proposing an alternative framework for logging and in turn, how we can design logging into our systems reliably.

The Good, The Bad, and The Ugly

This is a framework for how I think we should strategize our logging. It has three – and only three – categories or concerns for our logs.

First rule of logging: Don’t log

Overlogging is detrimental to our teams’ productivity and ability to handle business-as-usual operations.

There’re tons of reasons why we should not “Log whenever you can” as advised by some observability fanfare. Logging means more code to maintain, it incurs expense in terms of system performance, and more logging subjects us to more data privacy regulatory audits.

If you need more reasons to refrain from logging, check out this post by Nikita Sobolev or this post by Jeff Atwood.

Nevertheless, I’m not advising that you eliminate logs altogether. I think logging, used correctly, can significantly help us keep our systems running reliably.

I’m just proposing we start without logging and work our way up to identify places where we need to log, rather than “log everywhere as we might need to look at them”.

My rule of thumb for adding a line of logging is “if we can’t pin down an exact reason or a scenario when we will look at the log, don’t log".

With that being said, how can we safely introduce logging when it’s absolutely necessary? How should we structure our logs and format their content? What information is necessary to include in logs?

The Ugly

These are the first type of logs that I want to describe, and they are also the ones that I find least frequently. (If we find them too often, we might have bigger issues in our systems!)

“The Ugly” is the kind of log under catastrophic or unexpected scenarios that requires immediate action (like catastrophic errors that need an application restart). We can argue that, under these circumstances, it makes more sense to use alerting tools like Sentry.

Nevertheless, an error log might still be useful to provide ourselves with more context surrounding these errors which are not available in their stack trace. But they could help with reproducing these errors, like user inputs.

Just like the errors that they accompany, these logs should be kept to a minimum in our code and placed in a single location. They should also be designed/documented in the spec as a required system behavior for error handling. Also, they should be woven into the source code around where the error handling happens.

While format and level for “The Ugly” logs are completely preferential on a team-by-team basis, I would recommend using log.error or log.fatal before a graceful shutdown and restart of the application. You should also attach the full error stack trace and the function or requests’ input data for reproduction if necessary.

The Bad

“The Bad” is the type of logs that addresses expected, handled errors like network issues and user input validation. This type of log only requires developers’ attention if there’s an anomaly with them.

Together with a monitor set up to alert developers upon an error, these logs are handy to mitigate potential serious infrastructure or security problems.

This type of log should be spec-ed inside error handling technical requirements as well, and can actually be bundled if we are handling expected and unexpected errors in the same code location.

Based on the nature of what they are making “visible” for developers, log.warn or log.error can be used for “The Bad” logs given a team's convention.

The Good

Last but definitely not least, “The Good” is the type of log that should appear most often in our source code – but it's often the most difficult to get right. “The Good” kind of logs are those associated with the “happy” steps of our applications, indicating the success of operations.

For its very nature of indicating starting/successful execution operations in our system, “The Good” is often abused by developers who are seduced by the mantra “Just one more bit of data in the log, we might need it”.

Again, I would circle back to our very first rule of logging: “Don’t log unless you absolutely have to”. To prevent this kind of over-logging from happening, we should document “The Good” as part of our technical requirements complementing the main business logic.

On top of that, for every single one of “The Good” logs that are inside our technical spec, they need to pass the litmus test: are there any circumstances under which we would look at the log (be it a customer support request, an external auditor’s inquiry)? Only this way will log.info not be a dreaded legacy that obscures developers’ vision into our applications.

The Rest (That You Need to Know)

By now I assume you've noticed that the general theme of my proposed logging strategy revolves around clear and specific documenting of the log's purpose. It is important that we treat logging as part of our requirements, and that we're specific about what keywords and messages we want to tag in each log's context for them to be effectively indexed.

Only by doing that can we be aware of each and every log that we produce, and in turn, have a clear vision into our systems.

As logs are upgraded to first-class citizens with concrete technical requirements in our specs, the implications are that they would need to be:

• maintained and updated as the business and technical requirements evolve
• covered by unit and integration tests

This might sound like a lot of extra work to get our logs right. However, I argue that this is the kind of attention and effort logging deserves so it can be useful.

Serve our logs, and we will be rewarded splendidly!

A Practical Migration Guide

I reckon there’s no use for a new logging strategy (or any new strategies/frameworks for that matter) for legacy projects if there’s no way of moving them from their messy state to the proposed ideal.

So I have a three-step general plan for anyone who is frustrated with their system's logs and is willing to invest the time to log more effectively.

Identify The Usual Suspects

Since the idea is to reduce garbage logs, our first step is to identify where the criminals are hiding. With the powerful text editors and IDEs we have nowadays (or grep if you are reading this in the past through a window-to-the-future), all occurrences of logging can be easily identified.

A document (or spreadsheet if you would like to be organised) documenting all of these logging occurrences might be necessary if there are too many of them.

Convict Them Bad Actors!

After identifying all suspects, it’s time to weed out the bad apples. Logs which are duplicated or unreachable are low hanging fruits that we can immediately eliminate from our source code.

For the rest of our logging occurrences, it’s time to involve other stakeholders like the “inception” engineer who started the project (if that is possible), product managers, customer support, or compliance folks to answer the question: Do we need each one of these logs, and if so, what are they being used for?

Light At the End of the Tunnel

Now that we have a narrowed-down list of absolutely necessary logs, turning them into technical requirements with documented purpose for each one of them is essential to nail down a contract (or we can call it specification) for our logging subsystem. Ask yourself what to do when a log.error happens, and who are we log.info-ing for?

After this, it’s just a matter of discipline in the same way that we write and maintain software in general. Let's all work together and make logging awesome!

You can reach out to me on Twitter with any questions or comments.

Logzero is a Python package created by Chris Hager that simplifies logging with Python 2 and 3. Logzero makes it easier as a print statement to show information and debugging details.

If you are wondering what logging is, I recommend that you read the previous article I wrote about “How to Run Machine Learning Experiments with Python Logging Module”, especially the first 3 sections.

In that article, you will learn:

• What is Logging?
• Why logging is important.
• Applications of logging in different technology industries.

Logzero has different features that make it easier to use in Python projects. Some of these features are:

• Easy logging to console and/or file.
• Provides a fully configured standard Python logger object.
• Pretty formatting, including level-specific colors in the console.
• works with all kinds of character encodings and special characters.
• Compatible with Python 2 and 3.
• No further Python dependencies.

Installation

To install logzero with pip run the following:

pip install -U logzero

You can also install logzero from the public Github repo:

git clone https://github.com/metachris/logzero.git
cd logzero
python setup.py install

Basic Example

We will start with a basic example. In the python file, we will import the logger from logzero and try 4 different logging level examples.

#import logger from logzero
from logzero import logger

logger.debug("hello")
logger.info("info")
logger.warning("warning")
logger.error("error")

The output is colored so it's easy to read.

logzero output

As you can see each level has its own color. This means you can identify the level easily by checking the color.

Write logs to a file

Most of the time Python users tend to write logs in the file. When the system is running you can save logs in the file and review them for error checks and maintenance purposes. You can also set a file to save all the log entries in legzero.

We will import the logger and logfile from logezero. The logfile method will help us configure the log file to save our log entries.

Now your log entries will be logged into the file named my_logfile.log.

#import logger and logfile
from logzero import logger, logfile

#set logfile path
logfile('my_logfile.log')

# Log messages
logger.info("This log message saved in the log file")

The output in the my_logfile.log contains the logging level label (for info level labeled as “I”), date, time, python filename, line number and the message itself.

[I 200409 23:49:59 demo:8] This log message saved in the log file

Rotating a log file

You don't need to have a single log file saving all the log entries. This results in a massive log file that is intensive for the system to open and close.

You can use the maxBytes and backupCount parameters to allow the file to roll over at a predetermined size. When the size is about to be exceeded, the file is closed and a new file is silently opened for output. Rollover occurs whenever the current log file is nearly maxBytes in length. If either maxBytes or backupCount is zero, rollover never occurs.

In the example below, we have set the maxBytes to be 1000000 bytes (1 MB). This means that when the size exceeds 1MB the file is closed and a new file is opened to save log entries. The number of backups to keep is set to 3.

# Set a rotating logfile
logzero.logfile("my_logfile.log", maxBytes=1000000, backupCount=3)

Set a Minimum Logging Level

[Photo by Son Nguyen Kim](https://www.toptal.com/resume/son-nguyen-kim?hstc=753710.17be834d28ba29055621f0833fc6733b.1582400164835.1582400164835.1582400164835.1&hssc=753710.1.1582400164836&__hsfp=3618320745" rel="noopener nofollow noopener)

The logging level means to set the importance level of a given log message. You can also set a different log level for the file handler by using the loglevel argument in the logfile method.

In the example below, we set loglevel to be warning. This means all log entries below the warning level will not be saved into a log file.

#import logzero package
from logzero import logger, logfile
import logging

# You can also set a different loglevel for the file handler
logfile("my_logfile.log", loglevel=logging.WARNING)

# Log messages
logger.info("This log message saved in the log file")
logger.warning("This log message saved in the log file")

Set a custom formatter

How you want the log record to be formated is up to you. There are different ways you can format your log record. You can include the date, time and logging level in your format so that you know when the log was sent and at what level.

The example below shows how you can configure the format of the log records.

#import logzero package
import logzero
from logzero import logger, logfile
import logging

#set file path
logfile("my_logfile.log")

# Set a custom formatter
my_formatter = logging.Formatter('%(filename)s - %(asctime)s - %(levelname)s: %(message)s');
logzero.formatter(my_formatter)

# Log messages
logger.info("This log message saved in the log file")
logger.warning("This log message saved in the log file")

In the example above we have configured the log format by including filename, date, time, logging level name, and message.

This is the output in the my_logfile.log:

demo.py - 2020–04–10 00:51:44,706 - INFO: This log message saved in the log file
demo.py - 2020–04–10 00:51:44,707 - WARNING: This log message saved in the log file

Custom Logger Instances

Instead of using the default logger, you can also setup specific logger instances with logzero.setup_logger(..). You can configure and returns a fully configured logger instance with different parameters such as name, logfile name, formatter, maxBytes, backupCount, and logging level.

This is a working example of how to setup logging with a custom logger instance:

import logzero package
from logzero import logger, logfile, setup_logger
import logging

# Set a custom formatter
my_formatter = logging.Formatter('%(filename)s - %(asctime)s - %(levelname)s: %(message)s');


#create custom logger instance
custom_logger = setup_logger(
 name="My Custom Logger",
 logfile="my_logfile.log",
 formatter=my_formatter,
 maxBytes=1000000,
 backupCount=3,level=logging.INFO)

# Log messages
custom_logger.info("This log message saved in the log file")
custom_logger.warning("This log message saved in the log file")

In the example above we have set a custom logger instance called custom_logger with different configured parameter values.

Wrap up

In this article, you've learned the basics, along with some examples, of how to use the Logezero Python package. You can learn more about the features available in the documentation. Now you can start implementing the logzero package in your next python project.

If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post! I can also be reached on Twitter @Davis_McDavid

What is logging?

Let’s say you are developing a software product. It works remotely, interacts with different devices, collects data from sensors and provides a service to the user. One day, something goes wrong and the system is not working as expected. It might not be identifying the devices or not receiving any data from the sensors, or might have just gotten a runtime error due to a bug in the code. How can you know for sure?

Now, imagine if there are checkpoints in the system code where, if the system returns an unexpected result, it simply flags it and notifies the developer. This is the concept of logging.

Logging enables the developers to understand what the code is actually doing and how the work-flow is. A large part of software developers’ lives is monitoring, troubleshooting and debugging. Logging makes this a much easier and smoother process.

Visualisation of logs

_[source](https://www.datalabsagency.com/wp-content/uploads/2014/11/Interactive-Data-Visualisation-Service.png" rel="noopener" target="blank" title=")

Now, if you are an expert developer who has been developing and creating software for quite a while, then you would think that logging is not a big deal and most of our code is included with a **Debug.Log('____')** statement. Well, that is great but there are some other aspects of logging we can make use of.

Visualisation of specific logged data has the following benefits:

• Monitor the operations of the system remotely.
• Communicate information clearly and efficiently via statistical graphics, plots and information graphics.
• Extract knowledge from the data visualised in the form of different graphs.
• Take necessary actions to better the system.

There are a number of ways we can visualise raw data. There are a number of libraries in the Python and R programming languages that can help in plotting graphs. You can learn more about it here. But in this post, I am not going to discuss about above mentioned methods. Have you ever heard about the ELK stack?

ELK stack

E — Elasticsearch, L — Logstash, K — Kibana

Let me give a brief introduction to it. The ELK stack is a collection of three open source softwares that helps in providing realtime insights about data that can be either structured or unstructured. One can search and analyse data using its tools with extreme ease and efficiently.

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected. Elasticsearch lets you perform and combine many types of searches — structured, unstructured, geo, metric etc. It is built on Java programming language, which enables Elasticsearch to run on different platforms. It enables users to explore very large amount of data at very high speed.

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favourite “stash” (like Elasticsearch). Data is often scattered or siloed across many systems in many formats. Logstash supports a variety of inputs that pull in events from a multitude of common sources, all at the same time. Easily ingest from your logs, metrics, web applications, data stores, and various AWS services, all in continuous, streaming fashion. Logstash has a pluggable framework featuring over 200 plugins. Mix, match, and orchestrate different inputs, filters, and outputs to work in pipeline harmony.

Kibana is an open source analytics and visualisation platform designed to work with Elasticsearch. You use Kibana to search, view, and interact with data stored in Elasticsearch indices. You can easily perform advanced data analysis and visualise your data in a variety of charts, tables, and maps. Kibana makes it easy to understand large volumes of data. Its simple, browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time.

To get a better picture of the workflow of how the three softwares interact with each other, refer to the following diagram:

_[source](http://Howtodoinjava.com" rel="noopener" target="blank" title=")

Implementation

Logging in Python

Here, I chose to explain the implementation of logging in Python because it is the most used language for projects involving communication between multiple machines and internet of things. It’ll help give you an overall idea of how it works.

Python provides a logging system as a part of its standard library, so you can quickly add logging to your application.

import logging

In Python, logging can be done at 5 different levels that each respectively indicate the type of event. There are as follows:

• Info — Designates informational messages that highlight the progress of the application at coarse-grained level.
• Debug — Designates fine-grained informational events that are most useful to debug an application.
• Warning — Designates potentially harmful situations.
• Error — Designates error events that might still allow the application to continue running.
• Critical — Designates very severe error events that will presumably lead the application to abort.

Therefore depending on the problem that needs to be logged, we use the defined level accordingly.

Note: Info and Debug do not get logged by default as logs of only level Warning and above are logged.

Now to give an example and create a set of log statements to visualise, I have created a Python script that logs statements of specific format and a message.

import logging
import random

logging.basicConfig(filename="logFile.txt",
                    filemode='a',
                    format='%(asctime)s %(levelname)s-%(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S')
for i in xrange(0,15):
    x=random.randint(0,2)
    if(x==0):
        logging.warning('Log Message')
    elif(x==1):
        logging.critical('Log Message')
    else:
        logging.error('Log Message')

Here, the log statements will append to a file named logFile.txt in the specified format. I ran the script for three days at different time intervals creating a file containing logs at random like below:

2019-01-09 09:01:05,333 ERROR-Log Message
2019-01-09 09:01:05,333 WARNING-Log Message
2019-01-09 09:01:05,333 ERROR-Log Message
2019-01-09 09:01:05,333 CRITICAL-Log Message
2019-01-09 09:01:05,333 WARNING-Log Message
2019-01-09 09:01:05,333 ERROR-Log Message
2019-01-09 09:01:05,333 ERROR-Log Message
2019-01-09 09:01:05,333 WARNING-Log Message
2019-01-09 09:01:05,333 WARNING-Log Message
2019-01-09 09:01:05,333 ERROR-Log Message
2019-01-09 09:01:05,333 CRITICAL-Log Message
2019-01-09 09:01:05,333 CRITICAL-Log Message
2019-01-09 09:01:05,333 CRITICAL-Log Message
2019-01-09 11:07:05,333 ERROR-Log Message
2019-01-09 11:07:05,333 WARNING-Log Message
2019-01-09 11:07:05,333 ERROR-Log Message
2019-01-09 11:07:05,333 ERROR-Log Message
2019-01-09 11:07:05,333 WARNING-Log Message
2019-01-09 11:07:05,333 CRITICAL-Log Message
2019-01-09 11:07:05,333 WARNING-Log Message
2019-01-09 11:07:05,333 ERROR-Log Message

Setting up Elasticsearch, Logstash and Kibana

At first let’s download the three open source softwares from their respective links [elasticsearch],[logstash]and[kibana]. Unzip the files and put all three in the project folder.

Let’s get started.

Step 1 — Set up Kibana and Elasticsearch on the local system. We run Kibana by the following command in the bin folder of Kibana.

bin\kibana

Similarly, Elasticsearch is setup like this:

bin\elasticsearch

Now, in the two separate terminals we can see both of the modules running. In order to check that the services are running open localhost:5621 and localhost:9600.

After both the services are successfully running we use Logstash and Python programs to parse the raw log data and pipeline it to Elasticsearch from which Kibana queries data.

Step 2— Now let’s get on with Logstash. Before starting Logstash, a Logstash configuration file is created in which the details of input file, output location, and filter methods are specified.

input{
 file{
 path => "full/path/to/log_file/location/logFile.txt"
 start_position => "beginning"
 }
}
filter
{
 grok{
 match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level}-%{GREEDYDATA:message}"}
 }
    date {
    match => ["timestamp", "ISO8601"]
  }
}
output{
 elasticsearch{
 hosts => ["localhost:9200"]
 index => "index_name"}
stdout{codec => rubydebug}
}

This configuration file plays a major role in the ELK stack. Take a look at filter{grok{…}} line. This is a Grok filter plugin. Grok is a great way to parse unstructured log data into something structured and queryable. This tool is perfect for syslog logs, apache and other webserver logs, mysql logs, and in general, any log format that is generally written for humans and not computer consumption. This grok pattern mentioned in the code tells Logstash how to parse each line entry in our log file.

Now save the file in Logstash folder and start the Logstash service.

bin\logstash –f logstash-simple.conf

In order to learn more about configuring logstash, click [here].

Step 3 — After this the parsed data from the log files will be available in Kibana management at localhost:5621 for creating different visuals and dashboards. To check if Kibana is receiving any data, in the management tab of Kibana run the following command:

localhost:9200/_cat/indices?v

This will display all the indexes. For every visualisation, a new Index pattern has to be selected from dev tools, after which various visualisation techniques are used to create a dashboard.

Dashboard Using Kibana

After setting up everything, now it’s time to create graphs in order to visualise the log data.

After opening the Kibana management homepage, we will be asked to create a new index pattern. Enter index_name* in the Index pattern field and select @timestamp in the Time Filter field name dropdown menu.

Now to create graphs, we go to the Visualize tab.

Select a new visualisation, choose a type of graph and index name, and depending on your axis requirements, create a graph. We can create a histogram with y-axis as the count and x-axis with the log-level keyword or the timestamp.

Creating a graph

After creating a few graphs, we can add all the required visualisations and create a Dashboard, like below:

Note — Whenever the logs in the log file get updated or appended to the previous logs, as long as the three services are running the data in elasticsearch and graphs in kibana will automatically update according to the new data.

Wrapping up

Logging can be an aid in fighting errors and debugging programs instead of using a print statement. The logging module divides the messages according to different levels. This results in better understanding of the code and how the call flow goes without interrupting the program.

The visualisation of data is a necessary step in situations where a huge amount of data is generated every single moment. Data-Visualization tools and techniques offer executives and other knowledge workers new approaches to dramatically improve their ability to grasp information hiding in their data. Rapid identification of error logs, easy comprehension of data and highly customisable data visuals are some of the advantages. It is one of the most constructive way of organising raw data.

For further reference you can refer to the official ELK documentation from here — https://www.elastic.co/learn and on logging in python — https://docs.python.org/2/library/logging.html

A good logging mechanism helps us in our time of need.

When we’re handling a production failure or trying to understand an unexpected response, logs can be our best friend or our worst enemy.

Their importance for our ability to handle failures is enormous. When it comes to our day to day work, when we design our new production service/feature, we sometimes overlook their importance. We neglect to give them proper attention.

When I started developing, I made a few logging mistakes that cost me many sleepless nights. Now, I know better, and I can share with you a few practices I’ve learned over the years.

Not enough disk space

When developing on our local machine, we usually don’t mind using a file handler for logging. Our local disk is quite large and the amount of log entries being written is very small.

That is not the case in our production machines. Their local disk usually has limited free disk space. In time the disk space won’t be able to store log entries of a production service. Therefore, using a file handler will eventually result in losing all new log entries.

If you want your logs to be available on the service’s local disk, don’t forget to use a rotating file handler. This can limit the max space that your logs will consume. The rotating file handler will handle overriding old log entries to make space for new ones.

Eeny, meeny, miny, moe

Our production service is usually spread across multiple machines. Searching a specific log entry will require investigating all them. When we’re in a hurry to fix our service, there’s no time to waste on trying to figure out where exactly did the error occur.

Instead of saving logs on local disk, stream them into a centralized logging system. This allows you to search all them at the same time.

If you’re using AWS or GCP — you can use their logging agent. The agent will take care of streaming the logs into their logging search engine.

To log or not log? this is the question…

There is a thin line between too few and too many logs. In my opinion, log entries should be meaningful and only serve the purpose of investigating issues on our production environment. When you’re about to add a new log entry, you should think about how you will use it in the future. Try to answer this question: What information does the log message provide the developer who will read it?

Too many times I see logs being used for user analysis. Yes, it is much easier to write “user watermelon2018 has clicked the button” to a log entry than to develop a new events infrastructure. This is not the what logs are meant for (and parsing log entries is not fun either, so extracting insights will take time).

A needle in a haystack

In the following screenshot we see three requests which were processed by our service.

How long did it take to process the second request? Is it 1ms, 4ms or 6ms?

2018-10-21 22:39:07,051 - simple_example - INFO - entered request 2018-10-21 
22:39:07,053 - simple_example - INFO - entered request 2018-10-21 
22:39:07,054 - simple_example - INFO - ended request 2018-10-21 
22:39:07,056 - simple_example - INFO - entered request 2018-10-21 
22:39:07,057 - simple_example - INFO - ended request 2018-10-21 
22:39:07,059 - simple_example - INFO - ended request

Since we don’t have any additional information on each log entry, we cannot be sure which is the correct answer. Having the request id in each log entry could have reduced the number of possible answers to one. Moreover, having metadata inside each log entry can help us filter the logs and focus on the relevant entries.

Let’s add some metadata to our log entry:

2018-10-21 23:17:09,139 - INFO - entered request 1 - simple_example
2018-10-21 23:17:09,141 - INFO - entered request 2 - simple_example
2018-10-21 23:17:09,142 - INFO - ended request id 2 - simple_example
2018-10-21 23:17:09,143 - INFO - req 1 invalid request structure - simple_example
2018-10-21 23:17:09,144 - INFO - entered request 3 - simple_example
2018-10-21 23:17:09,145 - INFO - ended request id 1 - simple_example
2018-10-21 23:17:09,147 - INFO - ended request id 3 - simple_example

The metadata is placed as part of the free text section of the entry. Therefore, each developer can enforce his/her own standards and style. This will result in a complicated search.

Our metadata should be defined as part of the entry’s fixed structure.

2018-10-21 22:45:38,325 - simple_example - INFO - user/create - req 1 - entered request
2018-10-21 22:45:38,328 - simple_example - INFO - user/login - req 2 - entered request
2018-10-21 22:45:38,329 - simple_example - INFO - user/login - req 2 - ended request
2018-10-21 22:45:38,331 - simple_example - INFO - user/create - req 3 - entered request
2018-10-21 22:45:38,333 - simple_example - INFO - user/create - req 1 - ended request
2018-10-21 22:45:38,335 - simple_example - INFO - user/create - req 3 - ended request

Each message in the log was pushed aside by our metadata. Since we read from left to right, we should place the message as close as possible to the beginning of the line. In addition, placing the message in the beginning “breaks” the line’s structure. This helps us with identifying the message faster.

2018-10-21 23:10:02,097 - INFO - entered request [user/create] [req: 1] - simple_example
2018-10-21 23:10:02,099 - INFO - entered request [user/login] [req: 2] - simple_example
2018-10-21 23:10:02,101 - INFO - ended request [user/login] [req: 2] - simple_example
2018-10-21 23:10:02,102 - INFO - entered request [user/create] [req: 3] - simple_example
2018-10-21 23:10:02,104 - INFO - ended request [user/create [req: 1] - simple_example
2018-10-21 23:10:02,107 - INFO - ended request [user/create] [req: 3] - simple_example

Placing the timestamp and log level prior to the message can assist us in understanding the flow of events. The rest of the metadata is mainly used for filtering. At this stage it is no longer necessary and can be placed at the end of the line.

An error which is logged under INFO will be lost between all normal log entries. Using the entire range of logging levels (ERROR, DEBUG, etc.) can reduce search time significantly. If you want to read more about log levels, you can continue reading here.

2018-10-21 23:12:39,497 - INFO - entered request [user/create] [req: 1] - simple_example
2018-10-21 23:12:39,500 - INFO - entered request [user/login] [req: 2] - simple_example
2018-10-21 23:12:39,502 - INFO - ended request [user/login] [req: 2] - simple_example
2018-10-21 23:12:39,504 - ERROR - invalid request structure [user/login] [req: 1] - simple_example
2018-10-21 23:12:39,506 - INFO - entered request [user/create] [req: 3] - simple_example
2018-10-21 23:12:39,507 - INFO - ended request [user/create [req: 1] - simple_example
2018-10-21 23:12:39,509 - INFO - ended request [user/create] [req: 3] - simple_example

Logs analysis

Searching files for log entries is a long and frustrating process. It usually requires us to process very large files and sometimes even to use regular expressions.

Nowadays, we can take advantage of fast search engines such as Elastic Search and index our log entries in it. Using ELK stack will also provide you the ability to analyze your logs and answer questions such as:

1. Is the error localized to one machine? or does it occur in all the environment?
2. When did the error started? What is the error’s occurrence rate?

Being able to perform aggregations on log entries can provide hints for possible failure’s causes that will not be noticed just by reading a few log entries.

In conclusion, do not take logging for granted. On each new feature you develop, think about your future self and which log entry will help you and which will just distract you.

Remember: your logs will help you solve production issues only if you let them.