Search This Blog

Loading...

Tuesday, September 16, 2014

How to optimize PNG images

My previous post introduces some tools to optimize JPEG images. The focus of this post is on optimizing PNG images. Two complimentary tools will be presented: optipng, and pngquant. The former, lossless, and the latter, lossy.

optipng

optipng optimizes a JPEG file by compressing it losslessly.

The command to install optipng on Debian/Ubuntu is:

$ sudo apt-get install optipng

For Fedora/Centos/RedHat, execute:

$ sudo yum install optipng

To optimize a JPEG file named input.png:

$ optipng -o7 -strip all -out out.png -clobber input.png

Notes:

  • Output PNG file.

    By default, optipng compresses the PNG file in-place, hence overwriting the original file. To write the output to a different file, use the -out option to specify a new output file. If the specified output file already exists, the -clobber option allows it to be overwritten. The -clobber is useful if you are running the command more than once.

    Alternatively, replace -out out.png with the -backup option. As a result, optipng first backs up the original input file before compressing the input file in-place.

  • Meta data.

    The -strip all option removes all meta data from the image.

  • Optimization level.

    The -o option specifies the optimization level, which ranges from 0 to 7. Level 7 offers the highest compression, but also takes the longest time to complete. It has been reported that there is a marginal return of improved compression as you increase the optimization level. The results obtained from my own 1-image test confirm that. The tests show that the default optimization level of 2 is pretty good, and that higher levels do not offer a big increase in compression.

    Optimization level Compression time
    (Seconds)
    File Size
    (Bytes)
    % Reduction
    Original N/A 285,420 N/A
    0 0.03 285,012 0.14
    1 3.07 242,548 15.02
    2 5.77 242,548 15.02
    3 10.33 242,175 15.15
    4 17.54 241,645 15.34
    5 34.61 241,258 15.47
    6 35.86 241,645 15.34
    7 71.37 241,258 15.47

pngquant

pngquant uses lossy compression techniques to reduce the size of a PNG image. It converts a 32-bit PNG image to a 8-bit paletted image. More specifically, instead of storing each pixel as a 4-channel, 32-bit RGBA value, each pixel is stored as an 8-bit reference for mapping to a color in a palette. This 8-bit color palette is embedded in the image, and is capable of defining 256 unique colors. The trick then becomes how to reduce the total number of colors in an image without sacrificing too much perceivable quality.

To install pngquant on Debian/Ubuntu:

$ sudo apt-get install pngquant

Note that the pngquant version shipped on Debian Wheezy is obsolete (1.0), and not recommended by the official pngquant web site. The examples below were run on version 2.0.0.

To install pngquant on Fedora/Centos/Redhat:

$ sudo yum install pngquant

To optimize a PNG image:

$ pngquant -o output.png --force --quality=70-80 input.png

Notes:

  • Specify the output image file name using the -o option. Without it, the default output name is the same as the input except that the extension is changed (for example, input-fs8.png).
  • Without the --force option, pngquant will not overwrite the output file if it already exists.
  • Since the introduction of the --quality=min-max option in version 1.8, the number of colors is automatically derived based on the specified min and max quality values. The min and max values range from 0 to 100, 100 being the highest quality.

    pngquant uses only the least number of colors required to meet or exceed the max quality level (80 in the above example). If it cannot achieve even the min quality value (70), the output image is not saved.

Below summarizes the results of optimizing one randomly chosen PNG image. It is not intended to be scientific or conclusive. Rather, I hope to give you an idea of the scale of reduction that is possible.

Quality
min-max
Orig 70-90% 70-80%
File Size
(Bytes)
1,281,420 445,464 376,221
% Reduction - 65.2 70.6

The 2 programs - optipng and pngquant - are not mutually exclusive. You will get the most compression from running pngquant. But if you want to get the last possible 1% or so compression, you may first run pngquant, then optipng.

$ pngquant -o lossy.png --force --quality=70-80 input.png $ optipng -o7 -strip all -out output.png lossy.png

Friday, September 12, 2014

How to optimize JPEG images

Poor load time degrades the user's experience of a web page. For a web page containing large images, optimizing images can significantly improve the load time performance which leads to better user experience. Moreover, if a web site is hosted on a cloud service which charges for cloud storage, compressing images can be financially worthwhile. This post explains the optimization of JPEG images using 2 command-line programs: jpegtran and jpegoptim. My next post introduces tools to optimize PNG images.

jpegtran

jpegtran optimizes a JPEG file losslessly. In other words, it reduces the file size without degrading the image quality. By specifying options, you can ask jpegtran to perform 3 types of lossless optimization:

  • -copy none

    An image file may contain metadata that are useless to you. For example, the following figure shows the embedded properties from a picture taken by a digital camera. Properties such as the Camera Brand and Camera Model can be safely stripped from the picture without affecting image quality.

  • -progressive

    There are two types of JPEG files: baseline and progressive. Most JPEG files downloaded from a digital camera or created using a graphics program are baseline. Web browsers render baseline JPEG images from top to bottom as the bytes are transmitted over the wire. In contrast, a progressive JPEG image is transmitted in multiple passes of progressively higher details. This enables the user to see an image preview before the entire image is displayed in its final resolution.

    For large JPEG images, converting from baseline to progressive encoding often results in smaller file size, and faster, user-perceived load time.

  • -optimize

    This option optimizes the Huffman tables embedded in a JPEG image.

To install jpegtran on Debian/Ubuntu:

$ sudo apt-get install libjpeg-progs

To install jpegtran on Fedora/Centos/RedHat:

$ sudo yum install libjpeg-turbo-utils

To optimize a JPG file:

$ jpegtran -copy none -progressive -optimize SAM_0297.JPG > opt_0297.JPG

Below are the results after running the above command.

File Size Before
(Bytes)
File Size After
(Bytes)
Reduction
(%)
3,119,056 2,860,568 8.3

jpegoptim

jpegoptim supports both lossless and lossy image optimization.

To install the program on Debian/Ubuntu:

$ sudo apt-get install jpegoptim

To install on Fedora/RedHat/Centos:

$ sudo yum install jpegoptim

To specify the same 3 types of lossless optimization as explained above, execute this command:

$ jpegoptim --strip-all --all-progressive --dest=opt SAM_0297.JPG SAM_0297.JPG 4000x3000 24bit N Exiff [OK] 3119056 --> 2860568 bytes (8.29%), optimized.

Notes:

  • The --all-progressive option is a feature introduced in jpegoptim version 1.3.0. The version on Debian Wheezy is only 1.2.3, therefore the option is not available.
  • By default, jpegoptim compresses in place, overwriting the input JPEG image. If you don't want the program to write over the input file, specify an alternative directory using the --dest option.

jpegoptim can also compress an image file using lossy optimization techniques. Specify an image quality from 0 to 100, with 100 being the highest quality (and lowest compression). To compress with 90% image quality, execute:

$ jpegoptim --max=90 --dest=opt SAM_0297.JPG SAM_0297.JPG 4000x3000 24bit Exif [OK] 3119056 --> 2337388 bytes (25.06%), optimized.

The table below summarizes the % of reduction in file size as you decrease the image quality. There is a trade-off between file size and image quality. While reducing image size is a worthwhile goal, you don't want to end up with an image that is not "pretty" enough. You are the final judge of the lowest quality that is acceptable to you. To pick the image quality to use for a specific picture, experiment by incrementally decreasing the image quality (say by 10 each time), visually inspect the output image, and stop when the image quality is no longer acceptable.

Quality 100% 90% 80%
File Size
(Bytes)
3,119,056 2,337,388 1,356,131
% Reduction - 25.0 56.5

Tuesday, September 2, 2014

Create video screencasts using recordmydesktop

Why make a video screen capture, aka screencast? Some possible reasons are:

  • Create a software demo.
  • Capture a video game session for playback later.
  • Keep a record of what transpires on the computer screen.

I create screencasts mainly for recordkeeping. As a result, I don't expect to edit the output video, for example, to obfuscate sensitive data, or to insert call-outs to draw attention to specific screen areas.

The tool I use is the command-line program recordmydesktop. It is a basic, no-frills screencasting program which offers video and audio capture but no editing features.

To install recordmydesktop on Debian/Ubuntu:

$ sudo apt-get install recordmydesktop

Record entire screen

$ recordmydesktop --no-sound --delay 5 -o myfile.ogv

Notes on command options:

  1. --no-sound is specified because I don't want to record a soundtrack for the video.
  2. With --delay 5, actual recording is delayed for 5 seconds after the command is run. This short delay gives you time to properly setup the screen before recording starts. For example, you may want to minimize the window in which you execute recordmydesktop.
  3. By default, the output file is named out.ogv. You can change the file name using the -o option, but the file extension must be ogv. recordmydesktop only outputs ogv video files. If you want to upload the output video to YouTube, use a video converter to convert from ogv to one of the YouTube supported video formats - e.g., avi.

To stop the recording, press the Control-Alt-s keys. This action automatically starts the encoding process. Depending on the length of the video, encoding can take a while to complete. After you stop the recording, you cannot resume recording the screen.

To temporarily pause the recording, press Control-Alt-p. You can resume recording by pressing the same keystrokes again.

Record a region

$ recordmydesktop --fps 15 --width 1038 --height 777 -x 88 -y 96 --no-sound --delay 5 -o myfile.ogv

--fps specifies the frame rate, the number of frames to capture per second. To avoid choppy videos, use a sufficiently high frame rate. 15 fps is in general good enough.

To capture a rectangular region on the screen, you need to explicitly specify its location and dimensions. The --x and --y options define the offsets in number of pixels from the upper left corner of the screen. The --width and --height options specify the dimensions of the region in number of pixels.

To help you determine the offsets and the dimensions, use the xwininfo command.

  1. Open the target window.

    If the region you want to capture is an existing window (say a browser window), simply open the window.

    Otherwise, open a new window, say a bash terminal. Re-size the terminal window to have the same size as the capture region. Relocate the window to the capture region on the screen.

  2. Run xwininfo without any argument.
  3. Click inside the target window.
  4. Look for the following fields in the command output:
    $ xwininfo ... Absolute upper-left X: 88 Absolute upper-left Y: 96 ... Width: 1038 Height: 777 ...

GUI screencast programs

If you prefer a GUI-based app, you may use gtk-recordmydesktop (a front-end for recordmydesktop), or kazam.

gtk-recordmydesktop can be installed like this:

$ apt-get install gtk-recordmydesktop

With the GUI front-end, you no longer need to manually calculate the dimensions, or the x and y offsets of the capture region. Instead, simply click and drag to define a rectangular region.

In summary, recordmydesktop is a nifty screencasting tool which allows you to video-capture your screen with or without sound. However, it does not have the editing features to add text call-outs or obfuscate sensitive screen contents - features which are critical for creating video tutorials. If you are aware of a video capture tool that does both recording and editing, please make a comment.

Tuesday, August 26, 2014

Building a firewall for a Debian web server

This post addresses how to configure the Linux firewall to protect a Debian-based web application server. While there are GUI tools for the job, we will focus on the command-line tool iptables.

The scenario is that you have just installed Debian (or Ubuntu) on a server connected to the Internet. This will be used as a web server hosting your WordPress blog. I assume that you already have Apache, and WordPress installed. Please refer to my earlier post for instructions on how to install WordPress on Debian.

Basic Requirements

Before we build the firewall, let's write down the basic requirements - the types of traffic the machine will accept and those it will drop.

  • Accept all outbound traffic (from server to the Internet).
  • Accept inbound ssh logins.
  • Accept inbound Web requests.
  • Accept inbound ping requests.
  • Log firewall-specific warnings.

Build firewall

Please follow the order of the steps below. The procedural order is designed to minimize the chance of locking yourself out by mis-configuring the firewall.

  1. Log in to the server either on the physical console or remotely via ssh.

    The physical console is better because then you don't need to worry about being locked out. However, it is not always possible to access the console because the machine may be sitting afar in a data center.

  2. Examine the current firewall configuration.
    $ sudo iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination

    Note the policies for INPUT, FORWARD, and OUTPUT. The typical default firewall is configured to accept all traffic, both inbound and outbound.

  3. Flush the firewall.

    Flush only if:

    • your firewall is not 'clean' - it has existing rules, and
    • the INPUT policy is ACCEPT.

    If the INPUT policy is not ACCEPT, you can make it so like this:

    $ sudo iptables -P INPUT ACCEPT

    To flush the firewall:

    $ sudo iptables -F

    Now, we are ready to add the firewall rules, one by one. Note that they comprise the basic rules to satisfy our stated requirements. Not included are specific rules to thwart common Internet attacks.

  4. Add rule # 1.
    $ sudo iptables -I INPUT 1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

    The above rule accepts any incoming traffic that is part of, or related to, an existing connection. If you are currently logged in via a remote ssh session, this rule prevents you from being locked out. So, it is important that you create it first.

  5. Add rule # 2.
    $ sudo iptables -I INPUT 2 -m conntrack --ctstate NEW -p tcp --syn --dport 80 --j ACCEPT

    This rule accepts all new incoming WordPress connections to port 80.

  6. Add rule # 3.
    $ sudo iptables -I INPUT 3 -m conntrack --ctstate NEW -p tcp --syn --dport 22 --j ACCEPT

    Rule # 3 accepts all new incoming ssh sessions to port 22.

  7. Add rule # 4.
    $ sudo iptables -I INPUT 4 -p icmp --icmp-type echo-request -m limit --limit 2/second -j ACCEPT

    This rule accepts incoming ping echo requests at the maximum rate of 2 requests per second.

  8. Add rule # 5.
    $ sudo iptables -I INPUT 5 -m limit --limit 2/min -j LOG --log-prefix "INPUT:DROP:" --log-level 6

    All incoming traffic that are not accepted by any prior rule get logged at a maximum rate of 2 entries per minute. The default log file is /var/log/messages. For easy identification, the log entries are prefixed with the string 'Input:Drop:'.

  9. Change default INPUT and FORWARD policies to DROP.

    With the policy change, all incoming traffic not explicitly accepted by any of the above rules are dropped.

    $ sudo iptables -P INPUT DROP $ sudo iptables -P FORWARD DROP

Your basic firewall is complete. You can view the newly created firewall rules via the following:

$ sudo iptables -L Chain INPUT (policy DROP) target prot opt source destination ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT tcp -- anywhere anywhere ctstate NEW tcp dpt:httpflags: FIN,SYN,RST,ACK/SYN ACCEPT tcp -- anywhere anywhere ctstate NEW tcp dpt:sshflags: FIN,SYN,RST,ACK/SYN ACCEPT icmp -- anywhere anywhere icmp echo-request limit: avg 2/sec burst 5 LOG all -- anywhere anywhere limit: avg 2/min burst 5 LOG level info prefix "INPUT:DROP:" Chain FORWARD (policy DROP) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination

Save firewall

The firewall that you just created will not persist. If you reboot the server, the firewall will revert to the default configuration - accept all traffic - before you made all the above modifications. To save the firewall changes permanently,

  1. Install iptables-persistent package.
    $ sudo apt-get install iptables-persistent
  2. Make an explicit save.

    After you finish modifying the firewall, you need to explicitly save the firewall configuration in the file /etc/iptables/rules.v4 using the command below.

    $ sudo iptables-save > /etc/iptables/rules.v4

Custom log file

The above firewall logs all dropped incoming traffic to the general system log file /var/log/messages. To avoid cluttering the file, I recommend sending the iptables-related log entries to a separate file, say /var/log/iptables.log. This is possible because iptables-related log entries are prefixed with a custom identifier - 'INPUT:DROP:'.

  1. Create a rsyslog rule to redirect firewall log entries.

    A new file /etc/rsyslog.d/10-iptables.conf is created to hold the rsyslog rule.

    $ cat > /etc/rsyslog.d/10-iptables.conf :msg, contains, "INPUT:DROP:" -/var/log/iptables.log & ~

    The first line in the file specifies that if a log entry contains the custom identifier, it is sent to /var/log/iptables.log.

    The second line skips forward to the next log entry, thereby preventing double logging into /var/log/messages.

  2. Restart rsyslog daemon.
    $ sudo service rsyslog restart

Tuesday, August 19, 2014

Using double dash to specify pesky command line parameters

Many Linux commands accept command-line options and positional parameters. The grep command searches for a pattern in a given file. Its 2 positional parameters are the pattern and the filename. It accepts options such as -i which specifies that the search is case insensitive.

An interesting scenario is when a positional parameter has a value starting with a dash ('-'). This makes the parameter indistinguishable from an option. For example, you try to grep the string -tea in a given file.

$ grep -i '-tea' test.txt grep: invalid option -- 't' Usage: grep [OPTION]... PATTERN [FILE]... Try 'grep --help' for more information.

The -tea is interpreted as an option. Consequently, the command failed.

To solve the problem, insert the double dash (--) on the command line to mark the end of all options. Everything that come after the -- marker are interpreted as positional parameters.

$ grep -i -- '-tea' test.txt Beverages -tea and coffee- are served.

Anther example scenario for using the double dash is the deleting of a file with a pesky file name. Suppose you want to delete a file named '-a'.

$ rm -a rm: invalid option -- 'a' Try `rm --help' for more information.

Enter -- to mark where the positional parameters start.

$ rm -- -a