ionice and daily backups
Wednesday, June 25th, 2008Anyone running a web server will either use a RAID for information protection, or will run a cron job that backups all the data daily
If you are running a budget server like mine, then most likely you’ll be doing daily backups. At 4 am everyday a backup process runs and stores a dump of the SQL and gzip of all the web folders including the SVN repository on a separate harddrive, which gets unmounted after the backup is done. The process takes around 2 hours to complete, during which the hard drives are all stressed and if the backup scripts were not carefully written they will hog any apache request due to slow disk access.
Because the primary reason of a webserver is to serve pages, I do two things to guarantee that any apache or database process will take precedence in resources when compared to the backup process:
1. The backup process runs with nice -n19 <backup>. This means that the process is running with a CPU idle priority, and will only use the CPU if no other process needs it. Using only nice will guarantee that the CPU is free, but programs like tar and gzip are big disk users and although they might not use a lot of CPU, a very small CPU usage is enough to generate big disk reads/writes. Large disk reads/writes usually will block other processes and keep them waiting for the disk resource, and will thus slow down apache, postgres and mysql and finally all pages accessed during the backup. In simple words, during backup if one tries to browse any of my websites when the backup process is only niced, the pages will load very slow.
2. This leads us to the second precaution, ionice -c2 -n7 nice -n19 <backup>. ionice is a tool that not so many people know that it exists. It basically prioritizes the disk access for different processes. Long tasks that access the disk a lot, and which are not that important or time critical should be given a lower priority compared to those that require fast turn around. ionice has three priority classes given by the parameter -c. Class 1 is idle priority, and only root can run processes in that class. Class 2 is best effort priority, within which there are 0-7 levels given by the -n parameter, 0 being highest priority and 7 being lowest. Class 3 is real time.
ionicing and nicing the backup process makes a huge difference in the speed and response of the webserver during the daily backup, especially if it was long. Try going over its man page. A good post about ionice is here.