NGINX logging and log rotation

Logging

There are two important NGINX directives to enable logging; access_log and error_log. The simplest configuration can just specify log file paths and log level for the error_log:

http {
    ...
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log error;
    ...
}

The access_log logs every request, whereas the error_log directive is used for logging only error messages. If present, NGINX will happily log to these locations, and we can use general file tools to view or search the logs for any issues.

The error log levels are warn, error, crit, alert, and emerg. You probably want error for all errors or warn to catch warnings as well.

You can edit the log format with log_format directive. In the following example, we'll use a log format compression (you can choose your own name) that adds $gzip_ratio of the request:

http {
  log_format compression
    '$remote_addr - $remote_user [$time_local] '
    '"$request" $status $body_bytes_sent '
    '"$http_referer" "$http_user_agent" "$gzip_ratio"';

  server {
    gzip on;
    access_log /var/log/nginx/access.log compression buffer 32k;
    ...
  }
}

I also added a buffer of a certain size that indicates the logs' logs are written to the file. In general, stick to defaults unless you have a good reason not to.

The other variables like $remote_addr and $http_referer are quite self-descriptive, but as always, it's good to consult the NGINX documentation for more details and options.

Some might prefer to log in the JSON format for easier consumption from 3rd-party tools. Here is an example how this could look like:

log_format main '{'
                 '"remote_addr": "$remote_addr",'
                 '"remote_user": "$remote_user",'
                 '"time_local":  "$time_local",'
                 '"request":     "$request",'
                 '"status":      "$status",'
                 '"body_bytes_sent": "$body_bytes_sent",'
                 '"http_referer": "$http_referer",'
                 '"http_user_agent": "$http_user_agent"'
                 '}';

Similar to the system journal, NGINX logs will grow in size quite quickly. We already saw the [flush=time] option for the access_log directive that deletes the log file contents after a specific time. Unfortunately, this option will clear the logs completely and, therefore, not give us a chance to keep a few days of log data available for inspection. Also, it does not apply to error_log. We won't use the flush option to solve this need but introduce an external log rotation.

Log Rotation With logrotate

logrotate is a daemon designed to help with running programs that generate large log files. It allows rotation, compression, removal, and even mailing of discarded log files. It can be instrumented to rotate logs in a timely fashion or when it grows too large.

logrotate should be already installed on our system (as the logrotate RPM package) with a default configuration at /etc/logrotate.conf configuration file:

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4
...

# packages drop log rotation information into this directory
include /etc/logrotate.d

The file is self-explanatory. As with other programs, we can drop our configurations in the /etc/logrotate.d directory, and logrotate will use them. The syntax for the file is as follows:

/var/log/name/of/log/file.log {
    rotate 5
    monthly
}

/var/log/name/of/frequent/log/file.log {
    daily
}

This way, we can specify one or more log files and override (or add to) the default log rotation settings.

Most common directives we can specify in our logrotate configuration file:

• create/create $PERM $USER $GROUP: create the log file (with specific file permissions)
• daily: rotate log every day
• weekly: rotate log every week
• monthly: rotate log every month
• yearly: rotate log every year
• compress: enable compression
• dateext: include date in the log file name
• missingok: accept missing log file
• copytruncate: truncate the original log in place instead of creating a new one
• olddir: move old logs to this directory
• rotate $INTEGER: number of rotations a log file undergoes before it is removed (0 immediately removes files)

When we install NGINX from the Fedora or CentOS repository it already ships with its own logrotate settings:

$ sudo cat /etc/logrotate.d/nginx
/var/log/nginx/*log {
    create 0664 nginx root
    daily
    rotate 10
    missingok
    notifempty
    compress
    sharedscripts
    postrotate
        /bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null`
          2>/dev/null || true
    endscript
}

The postrotate directive specifies a script that runs after the rotation is done. In this case, it sends the nginx process the USR1 signal which is the official way of telling NGINX to re-open the logs again.

Having this file in place means our job is already done for us when it comes to NGINX log rotation.

A Cron job for logrotate is provided for us (/etc/cron.daily/logrotate) so we only have to ensure that a crond service is running. On newer Fedora systems, logrotate has its systemd service called logrotate that replaces the Cron job approach.