Howto: Use logrotate to manage log files

view full story

http://linuxers.org – Logrotate is a great utility to manage and adminster log files in a system. It provides you with a lot of options, namely, automatic rotation, compression and mailing of log files. Each log file can be rotated hourly, daily, weekly or monthly. In this article, I will show you how to manage your log files using logrotate. Any type of logs can be managed using logrotate. It runs as a daily cron job. These days most of the operating systems come with logrotate pre-installed so you probably won't have to deal with the installation. Lets start with looking at the files and directories used by logrotate. Files and Directories used by Logrotate The 2 important files in case of logrotate are /etc/logrotate.conf /var/lib/logrotate.status    or    /var/lib/logrotate/status The first file is where the whole configuration and default options of logrotate is stored. The second file contains the information about the last rotation times of different log files. The directory thats usually used by logrotate is /etc/logrotate.d The options specified in /etc/logrotate.conf are used as default for rotating and managing log files. But if we want to specify separate settings for a set of log files then we can add their config files in this directory /etc/logrotate.d. Although we can even use another directory to save config files by changing an option in /etc/logrotate.conf. Now, enough of the theory lets see how these config files really look and what all the different options in it mean. Logrotate Options Let me first show you some commonly used options of logrotate and then we will see how to configure various options in the config file. rotate <num> Log files are rotated "num" times before getting deleted or mailed. daily When used this means log files should be rotated daily weekly Log files should be rotated weekly monthly Log files should be rotated monthly notifempty Don't rotate the log if its empty compress Compress the files after rotating them delaycompress This options means to delay the compression of a log file to the next rotation cycle. This is used in combination with compress. missingok If the log file is missing, go on to the next one without issuing an error message. create <mode> <owner> <group> After rotation create a new empty file with the following properties. e.g create 640 root adm. Just "create" will ensure to inherit the properties of previous log files. Logrotate config files As stated above, /etc/logrotate.conf is where the default options to handle logs are specified. A typical log file will look something like this. # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp, or btmp -- we'll rotate them here /var/log/wtmp {     missingok     monthly     create 0664 root utmp     rotate 1 } /var/log/btmp {     missingok     monthly     create 0660 root utmp     rotate 1 } The options mentioned in the config file above means that the logs are going to be rotated "weekly" and only 4 copies of backlogs will be maintained. On every rotation "create" a new empty file with properties inherited from the previous logs. Compression of backlogs is disabled. The "include" directive is used to mention the directory in which the package specific log config files will be saved. Managing Package specific log files in /etc/logrotate.d Now, if there is an application which generates logs how would it be able to use logrotate to control and manage its logs. This is all done using its /etc/logrotate.d directory. This is where applications drop the information required by logrotate to manage their logs. e.g consider the example of apache logs. My apache logrotate config file looks something likes this (its /etc/logrotate.d/apache2). /var/log/apache2/*.log {     weekly     missingok     rotate 5     compress     delaycompress     notifempty     create 640 root adm     sharedscripts     postrotate         if [ -f "`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`" ]; then             /etc/init.d/apache2 reload > /dev/null         fi     endscript } Now, cross checking  the options from the table mentioned above the following properties of the logs can be inferred. Its going to be a weekly rotation 5 backlogs will be maintained in compressed format. But since we are using 'delaycompress' as well, the compression of a backlog will take place in the next rotation. No rotation will take place if file is empty. No error in case no log is found. On every rotation a new file will be created with owner root, group adm and permission 640 We haven't discussed anything about 'sharedscripts' or 'postrotate' yet. Lets begin with prerotate/endscript and postrotate/endscript. Prerotate and Postrotate Both of these options control the scripts that will need to be run either before or after rotating a log file. Lets take the above example. postrotate         if [ -f "`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`" ]; then             /etc/init.d/apache2 reload > /dev/null         fi endscript 'endscript' is mandatory to be mentioned in both prerotate and postrotate to specify the ending of the script. You would have also noticed the sharedscripts option in the apache2 logrotate config file. Now, if sharedscripts options is not specified, prerotate and postrotate will run the scripts on each log file which is rotated. But if you look carefully at the file logs mentioned in the apache2 logrotate conf file, its a wild card /var/log/apache2/*.log i.e. not just a single file but a bunch of files matching a pattern. These files in our case will be /var/log/apache2/access.log and error.log. So, for every rotation we will have 2 log files. But look at the script that we want to run after rotation. It reloads the apache server and there is probably no need to run this script twice. So, by using sharedscripts we are ensuring that the script doesn't run on every rotated log file but only once. I hope this howto will be helpful for a lot of you. If you have any doubts or want something to be corrected here, feel free to comment. (General)