Tuesday, May 26, 2015

Useful sed tricks to customize configuration files

A typical software installation goes like this. You install the software using apt-get install or yum install.
Then, you manually edit the software's configuration file in order to satisfy your requirements. If you have to repeat the install on multiple machines, this quickly becomes tedious.

Instead of manually editing the file, I run a text manipulation command such as sed or awk to make the required changes. Then, I script the procedure by inserting the commands in a bash script file.

The scripting of configuration changes serves multiple purposes:

  • It is a permanent record of the configuration changes.

  • It is readily repeatable on the same or a different machine.

Below, I illustrate 2 sed tricks to make configuration changes to the Apache webserver. The target configuration file is /etc/apache2/apache2.conf.

Before you make any change, please first backup the original configuration file.

$ sudo cp /etc/apache2/apache2.conf /etc/apache2/apache2.conf.orig

Replacing first occurrence of a string

The default apache2.conf file contains the following line:

Timeout 300

Below is the sed command to change the first occurrence of Timeout in the file to 100.

$ sudo sed -i "0,/^Timeout\s/ s/^Timeout\s\+[0-9]\+/Timeout 100/" /etc/apache2/apache2.conf

The -i parameter tells sed to edit the file in place - that is, directly in apache2.conf.

0,/^Timeout\s/ specifies the range of lines over which the sed command is to be executed. In this example, the starting line is the first line (line 0). The finishing line is the line returned by a search for the word Timeout which appears at the beginning of a line (^) and followed by a whitespace (\s).

The line range parameter limits the change to only the first occurrence of Timeout in the file. If you leave out the line range, each occurrence of Timeout in the file will be modified. In many scenarios, leaving it out is OK because the parameter occurs only once in the configuration file.

For some configuration files, a parameter can occur multiples times, in different sections. Next, I illustrate how to limit the change to within a particular section in the configuration file.

Replacing a string within a target section

The MaxClients parameter occurs in 3 sections within the apache2.conf file:

  • mpm_prefork_module

  • mpm_worker_module

  • mpm_event_module

I want to change the MaxClients parameter within the mpm_prefork_module only.

The default mpm_prefork_module is like this:

<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0

Note that a section is delimited by the opening <IfModule> and closing </IfModule> statements.

The following sed command modifies the value of MaxClients to 18 within the mpm_prefork_module section.

$ sudo sed -i "/<IfModule mpm_prefork_module>/,\@</IfModule>@ s/MaxClients\s\+[0-9]\+/MaxClients 18/" /etc/apache2/apache2.conf

The line range is defined by the /<IfModule ... >/,\@</IfModule>@ clause in the above statement. The opening line in the line range is specified by a search for the <IfModule ... > pattern. The closing line is specified by the search pattern \@</IfModule>@.

An explanation of the closing line pattern is warranted. The slash (/) character is part of the search pattern for the closing line (</IfModule>). However, the slash is also the default delimiter for sed. Therefore, we must use a different delimiter (@) for the closing-line search pattern. Note that the first @ is escaped (\@).

The s/MaxClients.../MaxClients 18/ clause changes the value of MaxClients to 18.


The above are examples of how you can use sed to script common scenarios of changing configuration files. You can achieve the same result using other tools such as awk or perl. Please use the comment system to let us know your own examples.

If you are interested to learn more about sed, please read my earlier posts on the tool:


Jay Cheroske said...

Is it possible to write the closing line pattern as /<\/IfModule>/ instead?

Peter Leung said...

Jay, the answer is no. You can't write the closing pattern using / as the delimiter. The reason is that / is part of what you are searching. That is why I suggest using @.

Jay Cheroske said...

I understood why you did it the way you did. I just thought that you could also simply escape the forward slash with a backslash, and that both approaches would work. This is what I'm talking about:


Peter Leung said...

Excellent reference on escape rules, Jay.

And yes, escaping the forward slash in the search string did work for me.



Jay Cheroske said...

I love sed. I stink at it, but it's helped me out of so many pickles. Another really cool tool that I've just started using is a port of mustache to posix shell, available here:


Makes setting up config files so easy. Much easier than using a heredoc.

Unknown said...

I need all the sed example from scratch to till date

Ram said...

Post the link which have all the sed and Awk command explanation with all examples