Search This Blog


Tuesday, May 26, 2015

Useful sed tricks to customize configuration files

A typical software installation goes like this. You install the software using apt-get install or yum install.
Then, you manually edit the software's configuration file in order to satisfy your requirements. If you have to repeat the install on multiple machines, this quickly becomes tedious.

Instead of manually editing the file, I run a text manipulation command such as sed or awk to make the required changes. Then, I script the procedure by inserting the commands in a bash script file.

The scripting of configuration changes serves multiple purposes:

  • It is a permanent record of the configuration changes.

  • It is readily repeatable on the same or a different machine.

Below, I illustrate 2 sed tricks to make configuration changes to the Apache webserver. The target configuration file is /etc/apache2/apache2.conf.

Before you make any change, please first backup the original configuration file.

$ sudo cp /etc/apache2/apache2.conf /etc/apache2/apache2.conf.orig

Replacing first occurrence of a string

The default apache2.conf file contains the following line:

Timeout 300

Below is the sed command to change the first occurrence of Timeout in the file to 100.

$ sudo sed -i "0,/^Timeout\s/ s/^Timeout\s\+[0-9]\+/Timeout 100/" /etc/apache2/apache2.conf

The -i parameter tells sed to edit the file in place - that is, directly in apache2.conf.

0,/^Timeout\s/ specifies the range of lines over which the sed command is to be executed. In this example, the starting line is the first line (line 0). The finishing line is the line returned by a search for the word Timeout which appears at the beginning of a line (^) and followed by a whitespace (\s).

The line range parameter limits the change to only the first occurrence of Timeout in the file. If you leave out the line range, each occurrence of Timeout in the file will be modified. In many scenarios, leaving it out is OK because the parameter occurs only once in the configuration file.

For some configuration files, a parameter can occur multiples times, in different sections. Next, I illustrate how to limit the change to within a particular section in the configuration file.

Replacing a string within a target section

The MaxClients parameter occurs in 3 sections within the apache2.conf file:

  • mpm_prefork_module

  • mpm_worker_module

  • mpm_event_module

I want to change the MaxClients parameter within the mpm_prefork_module only.

The default mpm_prefork_module is like this:

<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0

Note that a section is delimited by the opening <IfModule> and closing </IfModule> statements.

The following sed command modifies the value of MaxClients to 18 within the mpm_prefork_module section.

$ sudo sed -i "/<IfModule mpm_prefork_module>/,\@</IfModule>@ s/MaxClients\s\+[0-9]\+/MaxClients 18/" /etc/apache2/apache2.conf

The line range is defined by the /<IfModule ... >/,\@</IfModule>@ clause in the above statement. The opening line in the line range is specified by a search for the <IfModule ... > pattern. The closing line is specified by the search pattern \@</IfModule>@.

An explanation of the closing line pattern is warranted. The slash (/) character is part of the search pattern for the closing line (</IfModule>). However, the slash is also the default delimiter for sed. Therefore, we must use a different delimiter (@) for the closing-line search pattern. Note that the first @ is escaped (\@).

The s/MaxClients.../MaxClients 18/ clause changes the value of MaxClients to 18.


The above are examples of how you can use sed to script common scenarios of changing configuration files. You can achieve the same result using other tools such as awk or perl. Please use the comment system to let us know your own examples.

If you are interested to learn more about sed, please read my earlier posts on the tool:

Tuesday, May 12, 2015

How to add and delete a user from a Linux group

Being in the right Linux group expedites many system administration tasks, and will save you time. For example, the Apache web server logs have the following file ownership and permissions.

# ls -al /var/log/apache2/*.log
-rw-r----- 1 root adm 882984 May 11 12:14 /var/log/apache2/access.log
-rw-r----- 1 root adm 418 May 11 01:55 /var/log/apache2/error.log

To read the Apache logs, you need root permissions. However, there is a shortcut that does not require you to run sudo. Note that adm - the admin group for Debian-based systems - is the group owner of the log files. So, if you become a member of adm, you don't need to sudo to read the log files.

To add peter to the adm group, execute any of the following commands:

  • $ sudo usermod -aG adm peter

  • $ sudo gpasswd -a peter adm

To verify that peter is now a member of the adm group, execute any of the following commands:

  • $ id -nG peter
    peter adm www-data

    You may be tempted, as I was, to not specify peter in the above command. Don't skip the parameter. Without the user parameter, you won't see the effect of the change in group membership - unless you log out and log back in. If you are running X, it means you have to log out of X, not just opening a new command shell window within the same X session.

  • $ groups peter
    peter : peter adm www-data

    Again, specify peter in the command. Otherwise, you must log out and then log back in before executing the command.

  • $ grep adm /etc/group

If you have made a mistake, and now want to remove peter from the adm group, run any of the following commands:

  • $ sudo gpasswd -d peter adm
    Removing user peter from group adm

  • $ sudo deluser peter adm
    Removing user `peter' from group `adm' ...

Besides the adm group, you should consider adding yourself to the www-data group. The Apache web server runs under the www-data user account on Debian systems. As a member of the www-data group, you can more easily modify web server files and directories.

An earlier post on group membership appears here.

Wednesday, April 29, 2015

How to schedule background jobs using crontab

The cron daemon is a great user tool to automate tasks that don't require human intervention. Users pre-specify jobs to run in the background at particular times, for example, every Monday, Wednesday and Friday at 2am.

To use cron, each user creates his own crontab ('cron table') file. The command to examine one's crontab file is crontab -l.

$ crontab -l
0 2 * * 1,3,5 /home/peter/backups/ 2>&1

The MAILTO line specifies the email address to which cron sends the output of command execution. Please refer to my earlier post on how to set up an SMTP server to forward your emails.

The second crontab line specifies that the script should be executed at 2am every Monday, Wednesday and Friday. The syntax may look complicated. Fortunately, you can use the on-line Crontab Generator to craft the crontab statements. If you want to learn the syntax, click here instead.

Create crontab

Your crontab file is initially empty. To create the file from scratch, run the crontab command and type in the crontab statements.

$ crontab

Alternatively, put the statements into a temporary file, say /tmp/cron, and run this command:

$ cat /tmp/cron | crontab -

Edit crontab

If you want to modify crontab contents after they are created, run this command:

$ crontab -e

The command opens the crontab file in your default text editor. It is the most versatile way to modify crontab. You can use it to create, modify, and delete crontab statements. Don't forget to save the file after you finish editing.

The downside for this edit command is the time and overhead of starting the text editor. You can append a new statement directly by using the command in the next section.

Add to crontab

When I was new to crontab, I made the mistake of trying to append a statement by running crontab without any argument. That actually replaced everything in the crontab file with the new input.

The correct command to append a new statement is:

$ (crontab -l; echo "30 04 * * 4 /home/peter/backups/ 2>&1") | crontab -

The trick is to run 2 commands in a subshell grouped by the round brackets. The first command, crontab -l, fetches the existing crontab statements. The echo command echoes the new statement to be appended. The collective output from both commands are piped to crontab standard input.

Empty crontab

To erase all crontab contents, execute the following command:

$ crontab -r


You may use crontab to schedule regular maintenance and backup tasks. Once it is set up, the crontab file tends to be static. But, if you ever need to add another task, or change the scheduled times, the commands introduced in this post will come in handy.

Tuesday, April 28, 2015

Free edX course on Introduction to Java Programming

If you want to learn Java programming, here is your perfect opportunity to take an on-line course.

EdX, the popular MOOC (Massive Open On-line Course) provider, is hosting a course on Introduction to Programming with Java.

This is part 1 of a series of Java courses, and will take 5 weeks to complete on-line.

I have never taken a formal programming course on-line. So, I can't advise on the effectiveness of such a course. But, I've taken non-programming-related edX courses before, and the experience was positive.

Do sign up today. The course starts on April 28.

And let us know in a comment of how you find the course.

Thursday, April 23, 2015

Configuring Monit: a free system monitoring and recovery tool

Why Monit?

One morning, I went on-line to check my WordPress website. Lo and behold, I saw this error: 'Error establishing a database connection.' My website had been down for 4 hours, luckily in the middle of the night.

I used a free website monitoring service called StatusCake. Sure enough, it did send me an email alerting me about this problem. But, sending an email at 2am was not helpful in solving the problem. What I really needed was a tool that not only detected when the database process went down, but would also restart the process without human intervention. Monit is such a tool.

For the rest of this post, I assume you want Monit to monitor a LAMP server (Linux, Apache2, MySQL, PHP).

Install Monit.

To install Monit on Debian or Ubuntu, execute this command:

$ sudo apt-get install monit

As part of the installation, a monit service is created:

$ sudo chkconfig --list | grep -i monit  
monit       0:off  1:off  2:on   3:on   4:on   5:on   6:off

Configure Monit

The main Monit configuration file is /etc/monit/monitrc. To edit it, you need sudo privileges.

$ sudo vi /etc/monit/monitrc

After you make a change to the file, follow these steps to bring it into effect:

  1. Validate configuration file syntax.

    $ sudo monit -t

    If no error is returned, proceed to next step.

  2. Restart Monit.

    $ sudo service monit restart

Global settings

The key global settings to customize are:

  • Test interval

    By default, Monit checks your system at 2-minute intervals. To customize the interval, change the value (from 120) in the following statement. The unit of measure is seconds.

    set daemon 120
  • Log file location

    You can specify whether Monit logs to syslog or a log file of your choice.

    # set logfile syslog facility log_daemon  
    set logfile /var/log/monit.log
  • Mail server

    Specify a mail server for Monit to send email alerts. I set up exim4 as an SMTP server on the localhost. For instructions, refer to my previous post.

    set mailserver localhost
  • Email format

    Hopefully, you won't receive many alert emails, but when you do, you want the maximum information about the potential problem. The default email format contains all the information known to Monit, but you may customize the format in which the information is delivered. To customize, use the set mail-format statement.

    set mail-format {  
        from:     monit@$HOST  
        subject:  monit alert --  $EVENT $SERVICE  
        message:  $EVENT Service $SERVICE  
                  at $DATE  
                  on $HOST 
                  Your faithful employee,  

    For a description of the set mail-format statement, click here.

  • Global alerts

    If any actionable event occurs, Monit sends an email alert to a predefined address list. Each email address is defined using the set alert statement.

    set alert root@localhost not on { instance, action }

    In the above example, root@localhost is the email recipient. Please refer to my earlier post about redirecting local emails to a remote email account.

    Note that an event filter is defined (not on { instance, action }). Root@local will receive an email alert on every event unless it is of the instance or action type. An instance event is triggered by the starting or stopping of the Monit process. An action event is triggered by certain explicit user commands, e.g., to unmonitor or monitor a service. Click here for the complete list of event types that you can use for filtering.

    By default, Monit sends an email alert when a service fails and another when it recovers. It does not repeat failure alerts after the initial detection. You can change this default behavior by specifying the reminder option in the set alert statement. The following example sends a reminder email on every fifth test cycle if the target service remains failed:

    set alert root@localhost with reminder on 5 cycles
  • Enabling reporting and service management

    You can dynamically manage Monit service monitors, and request status reports. These capabilities are delivered by an embedded web server. By default, this web server is disabled. To enable it, include the set httpd statement.

    set httpd port 2812 and
        use address localhost  
        allow localhost

    Note: I've only allowed local access to the embedded web server. The Useful Commands section below explains the commands to request reporting and management services.

Resource monitor settings

The following are the key resources to monitor on a LAMP server.

  • System performance

    You can configure Monit to send an alert when system resources are running below certain minimum performance threshold. The system resources that can be monitored are load averages, memory, swap and CPU usages.

    check system   
      if loadavg (1min) > 4       then alert 
      if loadavg (5min) > 2       then alert 
      if memory usage   > 75%     then alert 
      if swap usage     > 25%     then alert 
      if cpu usage (user)   > 70% then alert 
      if cpu usage (system) > 30% then alert 
      if cpu usage (wait)   > 20% then alert
  • Filesystem usage

    You can create a monitor which is triggered when the percentage of disk space used is greater than an upper threshold.

    check filesystem rootfs with path /
      if space usage > 90% then alert

    You may have more than 1 filesystem created on your server. Run the df command to identify the filesystem name (rootfs) and the path it was mounted on (/).

  • MySQL

    Instead of putting the MySQL-specific statements in the main configuration file, I elect to put them in /etc/monit/conf.d/mysql.conf. This is a personal preference. I like a more compact main configuration file. All files inside the /etc/monit/conf.d/ directory are automatically included in Monit configuration.

    The following statements should be inserted into the mysql.conf file.

    check process mysql with pidfile /var/run/mysqld/  
          start program = "/etc/init.d/mysql start"  
          stop program = "/etc/init.d/mysql stop"  
          if failed unixsocket /var/run/mysqld/mysqld.sock then restart  
          if 5 restarts within 5 cycles then timeout

    If the MySQL process dies, Monit needs to know how to restart it. The command to start the MySQL process is specified by the start program clause. The command to stop MySQL is specified by the stop command clause.

    A timeout event is triggered if MySQL is restarted 5 times in a span of 5 consecutive test cycles. In the event of a timeout, an alert email is sent, and the MySQL process will no longer be monitored. To resume monitoring, execute this command:

    $ sudo monit monitor mysql
  • Apache

    I put the following Apache-specific statements in the file /etc/monit/conf.d/apache.conf.

    check process apache2 with pidfile /var/run/
          start program = "/etc/init.d/apache2 start"
          stop program = "/etc/init.d/apache2 stop"
          if failed host port 80 protocol http request "/monit/token" then restart
          if 3 restarts within 5 cycles then timeout
          if children > 250 then restart
          if loadavg(5min) greater than 10 for 8 cycles then stop

    At every test cycle, Monit attempts to retrieve This URL points to a dummy file created on the webserver specifically for this test. You need to create the file by executing the following commands:

    $ mkdir /var/www/monit
    $ touch /var/www/monit/token 
    $ chown -R www-data:www-data /var/www/monit

    Besides testing web access, the above configuration also monitors resource usages. The Apache process is restarted if it spawns more than 250 child processes. Apache is also restarted if the server's load average is greater than 10 for 8 cycles.

Useful commands

To print a status summary of all services being monitored, execute the command below:

    $ sudo monit summary  
    The Monit daemon 5.4 uptime: 3h 48m 

    System ''                Running
    Filesystem 'rootfs'                 Accessible
    Process 'mysql'                     Running
    Process 'apache2'                   Running

To print detailed status information of all services being monitored, execute the following:

    $ sudo monit status
    The Monit daemon 5.4 uptime: 3h 52m 

    System ''
      status                            Running
      monitoring status                 Monitored
      load average                      [0.00] [0.01] [0.05]
      cpu                               0.0%us 0.0%sy 0.0%wa
      memory usage                      377092 kB [74.0%]
      swap usage                        53132 kB [10.3%]
      data collected                    Wed, 22 Apr 2015 13:21:47
    Process 'apache2'
      status                            Running
      monitoring status                 Monitored
      pid                               12909
      parent pid                        1
      uptime                            6d 15h 18m 
      children                          10
      memory kilobytes                  2228
      memory kilobytes total            335420
      memory percent                    0.4%
      memory percent total              65.9%
      cpu percent                       0.0%
      cpu percent total                 0.0%
      port response time                0.001s to [HTTP via TCP]
      data collected                    Wed, 22 Apr 2015 13:21:47

To unmonitor a particular service (e.g., apache2):

    $ sudo monit unmonitor apache2

To unmonitor all services:

    $ sudo monit unmonitor all

To monitor a service:

    $ sudo monit monitor apache2

To monitor all services:

    $ sudo monit monitor all


I'd recommend that you run Monit on your server in addition to signing up for a remote website monitoring service such as StatusCake. While the 2 services do overlap, they also complement each other. Monit runs locally on your server, and can restart processes when a problem is detected. However, a networking problem may go undetected by Monit. That is where a remote monitoring service shines. In the event of a network failure, the remote monitor fails to connect to your server, and will therefore report a problem that may otherwise go unnoticed.