How to migrate single site WordPress to multisite

The scenario

I had an existing WordPress website, say example1.com, that was hosted on a DigitalOcean VPS running LEMP(Linux, NGINX, MySQL, PHP). The website supported HTTPS using a TLS certificate issued by Let's Encrypt.

I wanted to start a new WordPress website, say example2.com. Barring a miracle, example2.com would initially have minimal traffic.

I decided that the new site would run on a virtual host on the same VPS, using WordPress's multisite feature.

WordPress multisite

Hosting multiple websites/domains on the same VPS can be a double-edged sword. Suffice to say, the advantage is economy of scale, and the disadvantage, putting all one's eggs in one basket.

The multisite model in WordPress can be summarized as '1 instance, 1 database'. The multiple websites share the same WordPress DocumentRoot directory (/var/www/example1) and the same WordPress mySQL database.

Within the single database, site-specific information is stored in tables identified by the blog IDs. For instance, the wp_posts table for example1.com retains the same name in multisite. However, the corresponding table for example2.com is named wp_2_posts (the 2 in the name refers to the official blog ID).

Multisite introduces a new level of complexity in administration. Seeing multisite in action is the best way to know what you are getting into before actually migrating your production website.

Trialing the migration

Setting up a separate VPS with the same configuration as the production system is the best option for conducting a trial migration. Notwithstanding, I opted for a poor man's platform to test the migration, my home workstation.

I won't be able to completely replicate at home the production environment. Most notably, no HTTPS for the home machine because there won't be TLS certificates.

To reflect the change from HTTPS to HTTP, I modify 2 WordPress administrative options, siteurl and home. Run the following SQL commands under mySQL:

update wp_options set option_value='http://example1a.com' where option_name = 'siteurl';
update wp_options set option_value='http://example1a.com' where option_name = 'home';

Note that instead of reusing the same names for example1.com and example2.com, I renamed them to say example1a.com and example2a.com respectively. In addition, I configure local DNS on my home workstation to map example1a.com and example2a.com to localhost's IP address. The reason is that I can access both the production and the trial websites at the same time. Add the following lines to the /etc/hosts file:

127.0.1.1   example1a.com
127.0.1.1   example2a.com

The rest of this post will detail the steps to convert WordPress from hosting a single site to hosting multiple sites.

Configuring system

1. Configure DNS.
   

   Register example2.com with a domain name registry and add the appropriate DNS records.

2. Obtain Let’s Encrypt TLS certificate for second domain.
   

   I assume certbot is already installed, certbot.timer enabled, and port 443 open.
   

   Although it is possible to bundle multiple domains, example1.com and example2.com in a single certificate, it is recommended that you create separate certificates for unique domain names.
   

   $ sudo certbot certonly --webroot -n --agree-tos -m sysadmin@example2.com -w /var/www/example1.com -d example2.com 
   

   
   

   -m: the email address for the certificate contact.
   

   -w: the DocumentRoot for example2.com which is the same as example1.com.
   

   -d: the domain.

3. Install certificate.
   

   Link the private key and the certificate generated by Let’s Encrypt to their respective expected TLS locations.
   

   $ sudo ln -s /etc/letsencrypt/live/example2.com/privkey.pem /etc/ssl/private/key2.pem 
   $ sudo ln -s /etc/letsencrypt/live/example2.com/fullchain.pem /etc/ssl/certs/cert2.pem
   

   
   

   Note that the names key2.pem and cert2.pem must be different from their counterparts for example1.com. Make a note of their names as you will need them later.

4. Configure website.
   

   Create /etc/nginx/sites-available/example2.com.conf by copying example1.com.conf and making the necessary changes.
   

   My skeleton example2.com.conf file looks like the following. The highlighted lines are relevant to the migration per se.
   

   server {
       listen 80;
       return 301 https://$host$request_uri;
   }
   server { 
       listen 443 ssl; 
       ssl_certificate     /etc/ssl/certs/cert2.pem;
       ssl_certificate_key /etc/ssl/private/key2.pem;
       root /var/www/example1.com; 
       server_name example2.com *.example2.com;
       index index.html index.php;
       location / {
           try_files $uri $uri/ /index.php?q=$uri&$args;
       }
       location ~ \.php$ {
         fastcgi_pass unix:/var/run/php/php7.3-fpm_example1.com.sock; 
       }
       location ~* /(?:uploads|files)/.*\.php$ {
         deny all;
       }
       location = /robots.txt {
         allow all;
         log_not_found off;
         access_log off;
       }
       add_header X-XSS-Protection "1; mode=block";
       add_header X-Content-Type-Options nosniff;
       add_header X-Robots-Tag none;
       add_header X-Download-Options noopen;
       add_header X-Permitted-Cross-Domain-Policies none;
       add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload";
       add_header Referrer-Policy no-referrer;
       add_header X-Frame-Options "SAMEORIGIN";
   }

Notes:

• The location of the certificate (cert2.pem) and key (key2.pem) need to be specified.
• The DocumentRoot location is the same as example1.com.
• The relevant server names are specified for this website(example2.com)
• PHP handling is listening to the same socket as example1.com (/var/run/php/php7.3-fpm_example1.com.sock).
5. Enable website.
   

   $ sudo ln -s /etc/nginx/sites-available/example2.com.conf /etc/nginx/sites-enabled/example2.com.conf

6. Reload NGINX.
   

   Test the syntax of the file changes before actually reloading the configuration files.
   

   $ sudo nginx -t
   $ sudo systemctl reload nginx
   

Configuring WordPress

1. Install wp_cli.
   

   Although one can handcraft the necessary lines in the WordPress configuration file (/var/www/example1.com/wp-config.php), I’d recommend using the command-line program [wp-cli](https://wp-cli.org/). To install, run this command sequence:
   

   $ curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
   $ chmod +x wp-cli.phar
   $ sudo mv wp-cli.phar /usr/local/bin/wp

2. Convert to multisite.
   

   $ wp core multisite-convert --subdomains --path=/var/www/example1.com
   

   
   

   The above command can be run while the website is up because it only statically inserts the following lines into wp-config.php.
   

   define( 'WP_ALLOW_MULTISITE', true );
   define( 'MULTISITE', true );
   define( 'SUBDOMAIN_INSTALL', true );
   $base = '/';
   define( 'DOMAIN_CURRENT_SITE', 'example1.com' );
   define( 'PATH_CURRENT_SITE', '/' );
   define( 'SITE_ID_CURRENT_SITE', 1 );
   define( 'BLOG_ID_CURRENT_SITE', 1 );
   

3. Patch ‘blocked cookie’ bug.
   

   Unless the bug is patched, login to your new website (example2.com) is prevented. The error message from Firefox is ‘Cookies are blocked or not supported by your browser. You must enable cookies to use WordPress.’
   

   To patch, edit /var/www/example1.com/wp-config.php, and insert the following define statement anywhere above the “That’s all” comment line.
   

   define('COOKIE_DOMAIN', false);
   ...
   /* That's all, stop editing! Happy publishing. */
   

4. Restart the PHP-FPM and NGINX daemons.
   

   $ sudo systemctl restart php7.3-fpm
   $ sudo systemctl restart nginx
   

Creating new site

1. Login to the original WordPress website example1.com using the URL(https://example1.com/wp-login.php).
   

   The ID to use to login is the same admin ID for example1.com. In multisite, this admin ID is promoted to super-admin status, capable of administering all domains within the network.


2. Click My Sites and select Network Admin and then Sites.

3. Click Add New.

4. Enter the required data, and click Add Site.

The Site Address is where one is supposed to enter the URL except it expects a subdomain, such as example2 which it will then concatenate with the primary domain to become example2.example1.com, not what I really wanted as in https://example2.com/. So for now, I simply play along by entering example2, and I will change it later. If you know a better way, please let me know in a comment.

The Admin Email can be that of an existing user, say the super-admin, or a new user.

5. Again, click My Sites and select Network Admin and then Sites.

6. Hover over example2.example1.com and click Edit.

7. Enter the correct Site Address, https://example2.com/, and click Save Changes.

Now, the new website is created and ready for you to edit. Browse to https://example2.com/wp-login.php and login as the super-admin user.

Gromit-MPX: a nifty videoconference screen annotation tool

The rise of the COVID-19 pandemic propels videoconferencing to the stratosphere of user adoption. Almost overnight, the previously unknown app Zoom became a household technology name. Technology behemoths like Google and Microsoft scrambled to beef up their own videoconferencing products to match Zoom's success.

Zoom allows the meeting presenter to share their desktop with other participants. Google Meet and Skype also have that screen sharing feature. What Zoom offers, as of today, but not Google Meet or Skype, is the ability to annotate the shared screen in real time.

Undoubtedly, Google and Microsoft will eventually incorporate screen annotation in their respective products, but for the time being, gromit-mpx is a viable stopgap solution.

With gromit-mpx, presenters can annotate their desktop using free-hand drawing. It is true that Zoom as well as several third-party open-source annotation apps such as ardesia and pylote give presenters more bells and whistles, for instance, to draw geometrical shapes such as solid or dashed lines and to enter text. Yet the no-frills gromit-mpx is tailor-made for videoconferencing because of its non-obtrusive, hotkey-based mode of operation.

In contrast to Zoom and pylote, gromit-mpx does not have a toolbar, thus saving valuable screen space. In lieu of a toolbar, gromit-mpx functionalities can be activated using hotkeys(see the table below). The inconspicuous use of hotkeys is generally less obtrusive to the presentation than the clicking of the mouse on a protruding toolbar.

Hotkey combo	Corresponding action
F9	Toggle drawing
Shift-F9	Clear screen
Ctrl-F9	Toggle visibility
Alt-F9	Quit app
Click	Draw with red pen (default)
Shift-Click	Blue pen
Ctrl-Click	Yellow pen
Wheel-button click	Green pen with arrow
Right click	Eraser

Installation

To install gromit-mpx on Debian or Ubuntu, enter:


# apt install gromit-mpx


Conclusion

If a videoconference presenter has the most basic requirement for an annotation tool, for instance, to draw meeting participants' attention to an area of the screen, gromit-mpx fits the bill well. Its handy hotkeys make annotation more seamless and speedy than the clunky toolbar used by more feature-complete apps, even Zoom.

Joplin vs Orgzly as note-taking to-do apps

This post evaluates 2 note-taking, to-do list managers: Joplin and org-mode/Orgzly. Both are free, open-source, cross-platform software.

As an avid emacs user, I have always used org-mode on my Linux desktop to take notes and compile my to-do lists. So much so, I held out as long as possible before I switched to another tool that could actually run on Android. Org-mode as an emacs tool did not support Android at the time. Painful as it was, I replaced org-mode with the cross-platform tool Joplin.

Joplin served me very well on both Linux and Android platforms…until I came across an Android app named Orgzly. Org-mode and Orgzly share the same plain text file format, and according to Orgzly documentation, 'files generated by Orgzly might differ in the amount of white space … Any other difference would be considered a serious bug.' File compatibility means that you can edit your tasks and notes using org-mode on Linux and Orgzly on Android.

One key difference between org-mode and Orgzly is how you edit the shared underlying files. Using org-mode, you edit the text files directly inside emacs the text editor. In contrast, you use Orgzly's GUI for editing.

Below, I compare Joplin and org-mode/Orgzly.

Portability

You can run Joplin in 3 ways:

1. as a desktop app on Linux, Windows or macOS,
   
2. as a mobile app on Android or iOS, and
   
3. as a terminal program on Linux, FreeBSD, macOS or Windows.
   

I used both its Linux desktop version as well as the Android mobile version, and had no problem vouching for Joplin.

Orgzly runs on Android only (no iOS version yet). For non-mobile platforms, you will need to run org-mode within the emacs editor. Although it has been done before, converting to emacs just to use org-mode may be an overkill for most people.

On portability, Joplin has a definite advantage over org-mode/Orgzly.

Installability

Mobile versions of Joplin can be installed via the respective Google and Apple app stores. Installing it on a desktop (Linux, Windows, macOS) is just as convenient. Joplin is available to download from the standard repository of many Linux distributions including Debian. You can also download the AppImage version on Joplin's website.

Orgzly can be installed on Android using either Google Play or F-Droid, the repository for free and open-source Android apps. Org-mode, the desktop counterpart, is now part of emacs: installing emacs automatically installs org-mode.

It is a tie.

Data import/export

Unless you are using a note-taking to-do app for the first time, you will want to easily import any data from your existing app into the new app. Conversely, to prevent vendor lock-in to any 1 app, you want to be able to export data in a format that other apps can easily import. One such format is ENEX, the file format for Evernote, the app with arguably the largest installed user base.

Joplin can import ENEX files, but cannot export to the same. In recompense, you can import and export data in Markdown, PDF and JSON formats.

Orgzly currently only supports the import of org-mode files, and does not support any import or export of third-party file formats.

The clear winner is Joplin.

User-friendliness

Both Joplin and Orgzly are minimalistic (even spartan in the case of Orgzly) but highly functional in their user interface design.

To their credit, both offer dark mode, i.e., the ability to set the background to dark.

Joplin has a slight edge over Orgzly in aesthetics.

Winner: Joplin.

Cloud storage

Both Joplin and Orgzly support data storage on popular cloud platforms. Cloud storage enables you to access your tasks and notes from multiple devices and platforms.

Some cloud platforms provide custom API that client apps such as Joplin and Orgzly can use to make connections. There is a cost to using API: the client app needs to write custom code for each cloud service.

The advent of the WebDAV protocol has created a level playing field for apps that need to exchange data over the Internet. Client apps only need to write the WebDAV interface once to support access to all WebDAV-compliant cloud services.

Joplin supports Dropbox and OneDrive via native API, and Nextcloud via the WebDAV protocol. Note that with OneDrive, upload files are restricted to a maximum size of 4 MB, which is relatively low if you are attaching large multimedia files.

Orgzly supports Dropbox through native API and Nextcloud and other cloud services through WebDAV.

A tie.

Encryption

As discussed above, both apps can store data in the cloud. Data is regularly transmitted in the cloud to keep client apps synchronized.

Joplin data transmission is encrypted end-to-end; Orgzly, unencrypted.

With Joplin, your privacy is protected: not even Joplin developers or your cloud host such as Dropbox can access your encrypted data.

This may well be the killer feature that swings the pendulum all the way in favor of Joplin for most people.

Project viability

The 2 projects are very similar on this point – both stable and actively maintained by a team of one major developer. So, don't expect major new features every few months.

For mature applications like note-taking to-do list managers, that may actually be a benefit.

According to Google Play, both apps are downloaded roughly the same number of times(50,000+). Their numbers pale in comparison to the behemoth Evernote (100,000,000+). However, both projects have reached a critical mass in their respective user communities.

A tie.

Conclusion

Both Joplin and org-mode/Orgzly are more than capable to do the basic job. But Joplin is the eminently obvious overall winner … unless you are a hardcore emacs fan.

How to remove passwords from PDF files

Recently, a financial institution emailed me a password-protected PDF file. Handling such PDFs was a nuisance. First, I had to call them to obtain the password. Second, because that was a file I'd like to access in the future, I needed to record the password, unless … I could somehow remove the password from the PDF file.

This post outlines several ways to remove a password from a PDF.

pdftk

pdftk is my go-to tool for manipulating PDFs. To save a password-protected PDF into a new file, without the password, simply execute a command like this:

$ pdftk MyInput.pdf input_pw PASSWORD output MyOutput.pdf
WARNING: The creator of the input PDF:
   MyInput.pdf
   has set an owner password (which is not required to handle this PDF).
   You did not supply this password. Please respect any copyright.

You can safely ignore the warning message.

An encrypted PDF file can have up to 2 passwords, the user password and the owner password, with the latter having more privileges. Either password will let you perform the operation, although pdftk prefers the owner password if the PDF has one. If you did not create the original PDF, most likely, the password given to you was the user password. Hence the warning.

Security conscious users would balk at specifying the plain-text password on the command line. Specifying the do_ask parameter allows you to enter the password via standard input.

$ pdftk MyInput.pdf output MyOutput.pdf do_ask

qpdf

An alternative solution is to use qpdf. For instance,

$ qpdf --decrypt --password=PASSWORD -- MyInput.pdf MyOutput.pdf

Note that the marker -- is used to separate the options from the input and output filenames.

To hide the password from the command line, specify the @- argument, which enables you to enter arguments via standard input. When prompted, enter --password=PASSWORD.

$ qpdf --decrypt @- -- MyInput.pdf MyOutput.pdf

Alternatively, you can specify the password inside a file, for instance, @/home/peter/arguments.txt. Note that the filepath is appended to the single character @. The file contains the line --password=PASSWORD.

$ qpdf --decrypt @/home/peter/arguments.txt -- MyInput.pdf MyOutput.pdf

pdftops/ps2pdf

This solution is more involved than the first 2: first convert the PDF to Postscript, and then back to PDF. I include it here to show an alternative approach, and it is probably not something you will actually do.

1. To convert it to Postscript:
   

   
   $ pdftops -upw PASSWORD MyInput.pdf MyInput.ps
   
   

   Note that -upw refers to the user password. If you have the owner password instead, replace -upw with -opw.
   
2. To save it back to PDF:
   

   
   $ ps2pdf MyInput.ps MyOutput.pdf
   
   

inxi: the Swiss Army knife for displaying Linux sysinfo

Do one thing and do it well - the Unix philosophy

inxi is the antithesis of the above venerable Unix philosophy. Many excellent tools exist for providing aspects of system and hardware information —lsb_release, uname, lshw, lscpu, lspci, lsusb, dmidecode, uptime, free, ip, parted, acpi, etc. Some of those tools may even report more details than inxi, but there is a definite advantage for using inxi—with just 1 command, you can see at a glance a machine's overall hardware and system configuration and real-time status.

System administrators and technical support engineers work with many machines, often at the same time. inxi enables them to quickly get a broad system configuration overview and assess the current machine status before doing maintenance or troubleshooting.

The tool organizes the machine data into the following categories:

• System (hostname, kernel, 64-/32-bit, desktop…)
• Machine (model, serial #, BIOS…)
• CPU (model, speed…)
• Graphics
• Audio devices
• Network devices
• Drives
• Partitions
• USB devices
• Sensors (temperatures, fan speed…)
• Repositories
• Real-time status (# processes, uptime…)

Installation

To install inxi on Debian buster,

# apt update && apt install inxi


Dependency checking

inxi calls numerous helper programs to do the actual work, not all of them may be pre-installed. Run the following inxi command (as non-root user) to test what is potentially missing on your system:


# inxi --recommends
-------------------------------------------------------
Test: recommended system programs:
...
ipmitool: -s IPMI sensors (servers)........... Missing
ipmi-sensors: -s IPMI sensors (servers)....... Missing
...
The following recommended system programs are missing:
Program: ipmitool ~ Install package: ipmitool
Program: ipmi-sensors ~ Install package: freeipmi-tools
...
-------------------------------------------------------

Note that the checking does not take into consideration whether your system actually supports the use of the helper programs. For instance, the 2 missing programs above (ipmitool and ipmi-sensors) only apply to servers. In this example, the target machine is not a server and does not support IPMI, so I did not install the recommended programs.

You must judge the merits of installing each helper program reported missing.

Usage

Root or non-root

You can run inxi as either root or a regular user. Certain output is restricted to root only, e.g., the motherboard serial number and detailed RAM data.

Basics

Although you can run inxi without any argument to get basic CPU and memory information, I'd recommend running it with -F.



-F is for Full, and is a shorthand for specifying all uppercase letter arguments (with some exceptions).

For instance, specifying -F automatically includes -P, but not -p. The uppercase argument -P shows partition information for the basic partitions: /, /boot, /home, etc. The lowercase letter argument -p includes snap partitions created when installing software using snap.


# inxi -Fxxxz


inxi output may contain IP addresses, MAC addresses, serial numbers—data that can uniquely identify the target system. If privacy is an issue, specify the -z flag to filter out private data from the report. Note that the default is to display the aforementioned data.

You can dial up the level of details in inxi output using the -x flag. Optionally specify up to 3 increasing levels of details: -x, -xx, and -xxx.

Advanced

If you want more details than what the -F option gives you, you can specify additional arguments to focus on specific aspects of your system.

# inxi -Fxmip -t --usb


The following is my favourite subset of the available arguments.

-d

inxi -F only displays data about hard disk drives. To include optical/DVD drives, specify -d.

-i

Default -F output hides the IP addresses for your network interfaces. To display IP, add -i.

-m

The -m argument reports data about individual memory slots.

-p

-F only reports standard partitions (/etc, /home, /opt…) and swap partitions. -p displays all mounted partitions, including partitions created by snap.

-r

This argument reports software package repository information.

-t

By default, -t displays the top 5 memory- and CPU-using processes. You can restrict to CPU or memory only, and adjust the number of processes reported. For instance, to display the top 10 memory-using processes, specify -tm10; top 10 CPU-using processes, -tc10.

Separate -t from other arguments(or add it to the end of an argument chain); otherwise inxi may return a syntax error.

# inxi -Fxmip -tc10


--usb

--usb displays USB device information.

Make it pretty

You can choose a color theme for inxi output. The argument is -c followed by a value between 0 and 42 inclusive, corresponding to the color theme.



Using the -c95 argument, you can preview the list of available color themes and then select one for the current inxi command.

# inxi -Fx -c95


Beware of this find command gotcha

find is a basic useful command that Linux users run all the time. The command searches a file system from a given starting location, and returns all matches based on input filters that you provide as arguments.

The Gotcha

The gotcha is when you try to narrow the search by pruning a sub-directory from the search (including the directory itself and everything under it). For instance, suppose you want to find all files under the directory /data that are owned by root, excluding the sub-directory /data/keepit and all files underneath.



My first attempt at the solution results in the following find command.

find /data -path /data/keepit -prune -o -user 0


The -o argument specifies the logical 'or' operator. The expression on the left,  '-path /data/keepit -prune'  indicates where to prune the search. The idea is that when the search reaches /data/keepit, the -prune argument causes the search to not descend further into the sub-directory. Furthermore, -prune always returns true. Hence, the whole expression returns 'true', without having to evaluate the expression on the right of -o.

The expression right of -o tests for root ownership (root is user 0).

I was befuddled to learn that running the above command returns /data/keepit (but not its descendants). If the search is snipped at /data/keepit, why is the sub-directory itself included in the output? Besides, /data/keepit is not owned by root.

Being unaware of this behavior could lead to some unintended and very bad consequences as files named in the find output are often piped to the xargs command for further processing.

The Explanation

Before I present my solution, let's discuss why the point of pruning, i.e., the sub-directory named in -path, is actually included in the output.

The primary purpose of find is to search for file matches. Yet, it can have side effects through actions you specify on the command line. In addition to -print/-print0, there is also the -exec action. Unless you explicitly specify an action, the find command assumes the default action is -print.

The above example has no explicit -print or -exec action, therefore, the  action defaults to print all file matches. This explains why /data/keepit, a match for -path, is in the output. Its descendants, on the other hand, were excluded because of pruning.

The Solution

My solution is to specify -print explicitly on the command line.

find /data -path /data/keepit -prune -o -user root -print


Lo and behold. When you run the above command, /data/keepit is no longer part of the output.

By specifying the -print action explicitly, the find command no longer defaults  to printing out each file match. Instead, it will only print a file match if it is explicitly requested.

Summary & Conclusion

The pruning logic of the find command is quite confusing. Reading its man page offers some help, but may generate more questions than answers. I hope that this article is of help. But, I recommend that before you use the -prune feature on your production data, test it on some dummy data first.

You have been forewarned.

ts: epitome of the Unix philosophy

Do one thing and do it well - the Unix philosophy

In this new age of Linux bloatware (hello, systemd), it is exhilarating to discover small gems like ts, a command-line tool that prepends a timestamp to each output line.

How is this useful?

I run scripts all the time—bash scripts, Ansible playbooks, etc–to automate system administration tasks. Many longer-running scripts that I run output statements in real-time to report what they are doing. For example, running an Ansible playbook will automatically output the name of the individual task as it is being executed. By default, however, no timestamps are displayed for the tasks.


$ ansible-playbook -b -i hostsfile myPlaybook.yml
PLAY [localhost] ****************************************************************

TASK [Gathering Facts] **********************************************************
ok: [localhost]

TASK [Disabe Caps Lock] *********************************************************
ok: [localhost]

TASK [Install X apps] ***********************************************************
ok: [localhost] => (item=autokey-gtk)
ok: [localhost] => (item=gnucash)

TASK [Install 64-bit texamker - Debian] *****************************************
changed: [localhost]
...snipped...
PLAY RECAP **********************************************************************
localhost : ok=8 changed=2 unreachable=0 failed=0


Often, I do want to display the timestamps for logging or troubleshooting purposes. An unusually short (or long) execution may signal something is amiss.

Granted, you can use the Ansible-specific profile_tasks plugin to profile your tasks. But, I propose ts as a quick-and-dirty solution: just pipe Ansible output to ts like the following.


$ ansible-playbook -b -i hostsfile myPlaybook.yml |ts
Mar 11 11:44:05 PLAY [localhost] ****************************************************************
Mar 11 11:44:05
Mar 11 11:44:05 TASK [Gathering Facts] **********************************************************
Mar 11 11:44:05 ok: [localhost]
Mar 11 11:44:06
Mar 11 11:44:06 TASK [Disable Caps Lock] *********************************************************
Mar 11 11:44:06 ok: [localhost]
Mar 11 11:44:06
Mar 11 11:44:06 TASK [Install X apps] ***********************************************************
Mar 11 11:44:10 changed: [localhost] => (item=autokey-gtk)
Mar 11 11:44:14 changed: [localhost] => (item=gnucash)
Mar 11 11:44:14
Mar 11 11:44:14 TASK [Install 64-bit texamker - Debian] *****************************************
Mar 11 11:44:19 changed: [localhost]
...snipped...
Mar 11 11:44:30 PLAY RECAP **********************************************************************
Mar 11 11:44:30 localhost : ok=8 changed=3 unreachable=0 failed=0


Optional ts arguments

By default, the ts command inserts the absolute timestamp into each output line. You can use the -s argument to replace the absolute timestamp with the elapsed duration since the start of execution.

$ ansible-playbook -b -i hostsfile myPlaybook.yml |ts -s
00:00:01 PLAY [localhost] ****************************************************************
00:00:01
00:00:01 TASK [Gathering Facts] **********************************************************
00:00:01 ok: [localhost]
00:00:02
00:00:02 TASK [Disable Caps Lock] *********************************************************
00:00:02 ok: [localhost]
00:00:02
00:00:02 TASK [Install X apps] ***********************************************************
00:00:06 changed: [localhost] => (item=autokey-gtk)
00:00:10 changed: [localhost] => (item=gnucash)
00:00:10
00:00:10 TASK [Install 64-bit texamker - Debian] *****************************************
00:00:15 changed: [localhost]
...snipped...
00:00:26 PLAY RECAP **********************************************************************
00:00:26 localhost : ok=8 changed=3 unreachable=0 failed=0



Another useful argument to know is -i. With this argument, each output line displays the elapsed time since the previous output line. You don't need to do the mental math to calculate how long a task took.

$ ansible-playbook -b -i hostsfile myPlaybook.yml |ts -i
00:00:00 PLAY [localhost] ****************************************************************
00:00:00
00:00:00 TASK [Gathering Facts] **********************************************************
00:00:01 ok: [localhost]
00:00:00
00:00:00 TASK [Disable Caps Lock] *********************************************************
00:00:00 ok: [localhost]
00:00:00
00:00:00 TASK [Install X apps] ***********************************************************
00:00:04 changed: [localhost] => (item=autokey-gtk)
00:00:04 changed: [localhost] => (item=gnucash)
00:00:00
00:00:00 TASK [Install 64-bit texamker - Debian] *****************************************
00:00:05 changed: [localhost]
...snipped...
00:00:00 PLAY RECAP **********************************************************************
00:00:00 localhost : ok=8 changed=3 unreachable=0 failed=0


In summary, ts is a fast and easy way to add timestamps to script or command output.

Snaps as self-contained, auto-updating, universal software packages

One of Linux's unique selling points is that users can choose from a variety of Linux distributions, each with its own features and advantages. However, a byproduct of the proliferation of distributions is that Linux developers are burdened with extra labor to package and deploy software in multiple incompatible package formats, such as RPM and DEB, using different package managers.

There have been several attempts to address this obstacle of deploying software across multiple Linux distributions. The latest such initiative, spearheaded by none other than the venerable Canonical Ltd, the Ubuntu developer, is Snapcraft. Snapcraft is the developer tool for packaging software in the universal snap format.

The rest of this post explains how to use snaps, from a user rather than developer's perspective.

[2018-05-23 update] A snap from the official Snap store was found to contain hidden cryptocurrency mining code, and was since removed from the store. The news highlight the fact that the mere existence of a snap in the Snap store does not guarantee its integrity. The lesson is to only use snaps from trusted sources featured in the Snap store, such as the original software author, an official maintainer, or a trusted community source.

What are snaps?

Snaps are universal software packages that can be deployed across the major Linux distributions and architectures including IoT. Although snaps is a format championed by the creator of Ubuntu, it is supported on all major Linux distributions including Debian, Ubuntu, Mint, Fedora, Gentoo, ArchLinux, Manjaro, and OpenSUSE.

Besides being distribution-agnostic, snaps are also self-contained. Linux users are all too familiar with "the dependency hell", ie, the occasional extreme frustration experienced in software installation due to dependency issues. A snap is self-contained in that it bundles the required runtime libraries inside the package.

Once you install a snap on your system, it will be auto-updated to run the latest release. You however can manually roll back to a previous version of the software if you so desire.

In summary, snaps are universal, self-contained, auto-updating Linux software packages.

Install snapd

To be able to run snap packages on your Linux system, you must first install snapd, the service responsible for running and managing snaps. The following command will install snapd on a Debian system.

$ sudo apt install snapd

The Snap store

You can search the relatively small but growing online Snap store for snaps to install. Alternatively, you can search using the command-line interface:

$ snap find firefox
Name     Version   Developer  Notes  Summary
firefox  59.0.2-1  mozilla    -      Mozilla Firefox web browser

Why would you download the firefox snap when firefox is readily available as a DEB, via the Debian standard repository? Similarly, chromium and libreoffice belong to the same category of software. The answer is that, in most cases, snaps offer a much more recent version of the software than that from the native Linux package manager. For instance, Debian 9 ("Stretch") packages FireFox 52 in its standard repository whereas you can get Firefox 59 as a snap.

The Snap store features some snaps that are not available from the standard repositories of major Linux distributions. Examples are ghostwriter and vidcutter. Instead of building such software manually, you can download and deploy their corresponding snaps from the Snap store.

Below is a non-exhaustive list of software available in snaps that I personally find useful(or fun).

• Chromium
• Firefox
• Ghostwriter-casept (a Markdown editor)
• LibreOffice
• Minecraft
• Nextcloud
• OBS Studio (for screencasting and live video streaming)
• Opera
• Skype
• Slack
• Solitaire
• Spotify
• VidCutter (a video editor)

To find out more about a particular snap, execute the following command:

$ snap info minecraft
name:      minecraft
summary:   Minecraft is a game about placing blocks and going on adventures.
publisher: snapcrafters
license:   Proprietary
description:   A game about placing blocks while running from skeletons
snap-id: aJQRf6WPQq04DH0TB2HdTB6K9rf6I1yX
channels:                
  stable:    latest (11) 148MB -
  candidate: latest (11) 148MB -
  beta:      latest (11) 148MB -
  edge:      latest (11) 148MB -

How to install a snap

Installing a snap is as easy as:

$ sudo snap install solitaire

If you don't have root privileges, you can still install snaps by first signing in to the Ubuntu Snap store. You will need an Ubuntu One account (which is free).

$ sudo snap login <your-email-address>
$ snap install solitaire

To list the snaps already installed on your system:

$ snap list
Name       Version    Rev   Tracking  Developer  Notes
core       16-2.32.1  4327  stable    canonical  core
skype      8.18.0.6   23    stable    skype      classic
solitaire  1.0        2     stable    1bsyl      devmode

To purge a snap from your local system:

$ sudo snap remove solitaire

Note that the above command will delete all snap-specific data and settings.

How to run a snap

Installing a snap in most cases will automatically create a shortcut on your desktop menu system. For instance, you will find an entry for the solitaire snap in the Games sub-menu.

Alternatively, you can always run the snap command explicitly on the command line.

$ snap run <your snap command>

The caveat with the command-line approach is that the snap argument may not always be obvious. For instance, with the solitaire snap, the corresponding name to use for the run command is solitaire.1bsyl, not solitaire. You can find out the specific name to use by examining snap's bin directory:

$ ls /snap/bin/
skype  solitaire.1bsyl
$ snap run solitaire.1bsyl

Scanning HTTPS for Mixed Content

Back in 2014, Google raised the awareness of using HTTPS ("Secure HTTP") by making its use a ranking signal in Google search algorithms. HTTPS essentially establishes secure encrypted connections to the cloud. Google further raised the stake of not using HTTPS by announcing that, beginning in July 2018, the Google Chrome browser with the release of Chrome 68 will mark all HTTP websites as being insecure. The consequence of not converting to HTTPS is that site visitors will be persuaded by the warning message to bounce from your website.

Even before the impending drop dead date, Chrome and other popular web browsers such as Firefox and Edge have been warning visitors to HTTP-connected sites with an informational message.

Web administrators had taken heed and converted their websites to HTTPS, many taking advantage of the free SSL certificates issued by Let's Encrypt. However, if you have successfully converted to HTTPS, your work may not be done. You still need to verify that your website is properly recognized as being secure. You want to see the padlock icon displayed next to the web page's URL in the browser window.

To many administrators' surprise, even a properly converted HTTPS website may still be marked as being insecure. This is most likely due to the website's mixed content. For a web page to be deemed secure, everything loaded by that page must be encrypted by HTTPS. A web page with mixed content loads both encrypted as well as non-encrypted contents such as images, videos, stylesheets and scripts.

While it is possible to manually spot mixed web content on a web page, checking a non-trivial website requires automation. Mixed Content Scan is a command-line web crawler which scans for mixed content. The rest of this post explains how to install and use the tool.

Installation

Mixed Content Scan is a batch PHP application. To install the tool, use composer, a PHP package dependency manager. For the latest instructions on how to install composer, please refer to this link. Note that the said procedure installs composer in the current directory. Optionally, move the executable to a globally accessible directory using the following command.

$ sudo mv composer.phar /usr/local/bin/composer

To install Mixed Content Scan:

$ composer global require bramus/mixed-content-scan:~2.8

The Mixed Content Scan executable is placed in ~/.config/composer/vendor/bramus/mixed-content-scan/bin.

Scanning for mixed content

To scan a website for mixed content, simply provide its URL as an argument to Mixed Content Scan:

$ cd ~/.config/composer/vendor/bramus/mixed-content-scan/bin
$ ./mixed-content-scan https://shadowofyourwings.com/

By default, the tool outputs the scan report on the terminal("standard output"). Alternatively, you can specify an output file using the --output parameter as follows:

$ cd ~/.config/composer/vendor/bramus/mixed-content-scan/bin
$ ./mixed-content-scan --output <some/file/path> https://shadowofyourwings.com/

You can also use the --ignore parameter to specify a file which contains URL patterns that the tool will ignore and not scan. The example site I use is a WordPress website. The scanning tool comes with a sample ignore file for WordPress which is located in ~/.config/composer/vendor/bramus/mixed-content-scan/bin/ignorepatterns/wordpress.txt.



$ cd ~/.config/composer/vendor/bramus/mixed-content-scan/bin
$ ./mixed-content-scan --ignore=~/.config/composer/vendor/bramus/mixed-content-scan/bin/ignorepatterns/wordpress.txt https://shadowofyourwings.com/
[2018-02-16 16:53:18] MCS.NOTICE: Scanning https://shadowofyourwings.com/
[2018-02-16 16:53:18] MCS.ERROR: 00000 - https://shadowofyourwings.com/
[2018-02-16 16:53:18] MCS.WARNING: http://gmpg.org/xfn/11
[2018-02-16 16:53:19] MCS.ERROR: 00001 - https://shadowofyourwings.com/about
[2018-02-16 16:53:19] MCS.WARNING: http://shadowofyourwings.com/wp-content/uploads/2017/05/peterLeung.jpg
[2018-02-16 16:53:19] MCS.WARNING: http://gmpg.org/xfn/11
[2018-02-16 16:53:20] MCS.ERROR: 00002 - https://shadowofyourwings.com/contact
[2018-02-16 16:53:20] MCS.WARNING: http://gmpg.org/xfn/11
... <output snipped> ...
[2018-02-16 16:53:38] MCS.NOTICE: Scanned 26 pages for Mixed Content


Mixed Content Scan numbers each page scanned, starting from 00000. In the above example, the About page (00001) has been flagged as having mixed content. The sources of mixed content as loaded by that page are twofold:

1. Vulnerable image file.
   The peterLeung.jpg file is being loaded via the insecure HTTP connection. The fix is simple: go to the WordPress administration web page, and change HTTP to HTTPS on the About web page.
2. Theme header profile
   The header of the default twentyseventeen WordPress theme contains a reference to http://gmpg.org/xfn/11. The code is in <document root>/wp-content/themes/twentyseventeen/header.php.
   
   Although the scanner reports its occurrence as a violation, browsers generally do not flag this as a mixed content error. This error can be safely ignored.

A review of 3 best-of-breed Markdown editors

As a technology blogger, I write HTML documents that are hosted on different platforms such as WordPress, Drupal, and Blogger. I like to compose HTML using the Markdown markup language. Unfortunately, the HTML editors bundled with the aforementioned platforms do not support Markdown natively. It is true that you can download Markdown plugins for WordPress and Drupal. But, at the end, I still find the HTML editors to be too intrusive for a writer such as myself to stay focused and productive.

Fortunately, there are many good special-purpose Markdown editors out there. My web authoring process involves first composing the document using a Markdown editor, and then copying and pasting the output HTML into the Content Management System(CMS). Below, I evaluate 3 open-source Markdown editors: justmd, Remarkable, and ghostwriter.

I will evaluate each editor from two sometimes conflicting viewpoints, that of a geek and a writer. As a geek, I side with editors that have many bells and whistles. But, as a writer, I prefer editors that help me create, often by filtering out as much distraction as possible, and forcing me to focus on the next word, phrase, sentence to put on the page.

justmd

justmd is a minimalist, bare-bones Markdown editor. When you open justmd, you will see a single window with 2 window panes of equal size, located side-by-side. One pane is where you enter the Markdown text; the other is the HTML preview pane. Although you can change the overall size of the encompassing window, you cannot change the ratio of the 2 panes. The geek in me cannot help but cringe at the discovery. After all, it is common among Markdown editors (including Remarkable and ghostwriter) to have separate input and preview windows that you can independently resize and even hide. Conversely, the writer in me gives justmd a big shout-out for its austere simplicity. You just open the app, and immediately start writing, without having to adjust the size of any window component. Writers will find justmd more conducive to writing than many editors that are much more customizable.

Minimalist as it is designed to be, justmd, as a Markdown editor, is not feature complete in its current status. The following features, which I deem to be very important for writers, are still missing in justmd:

• Spellchecker.
• Word count.
• Auto save.

This post was written entirely using justmd, and the overall experience was very positive. The lack of a spellchecker and word counter did not hamper the writing at all. On the contrary, it enhances my productivity by breaking the bad habits of constantly checking the word count and looking out for spelling errors in the midst of writing. Most Content Management Systems are capable of spellchecking and word counting. So, those tasks can be deferred until later, after you paste the HTML into the CMS.

Finally, I comment on the ease of installing justmd. None of the 3 editors being reviewed here are pre-packaged in the official repository of a major Linux distribution. Having said that, installing justmd is as easy as 1-2-3.

1. Download compressed tarball from justmd website.
2. Uncompress the tarball using command tar -zxvf justmd-linux-x64-v1.1.1.tar.gz.
3. Create shortcut to justmd binary.

Remarkable

Featurewise, Remarkable is middle-of-the-road, between justmd and ghostwriter. It has word counting, but no spellchecking. Like justmd, both input and preview functions coexist as panes side-by-side in a single window, but you can stack them vertically or horizontally, and you can resize each pane proportionally within the window.

Now, as a writer, I find Remarkable's user interface too colorful, too distracting. Specifically, its overly generous use of color for syntax highlighting and icon design is detrimental to the primary writing task. With color, less is more.

You can download the Remarkable package in .deb or .rpm format from its Linux download page. Users of Debian, Ubuntu, Fedora, SUSE, and Arch systems will find installation straightforward.

ghostwriter

ghostwriter is the most mature and feature complete of all 3 MarkDown editors. It offers spellchecking, word counting, auto saves, and much more.

Two unique features are especially noteworthy to writers: Hemingway and Focus modes. In Hemingway mode, two particular keyboard keys are disabled, namely, the delete and the backspace keys. The rationale is to increase productivity by delaying document editing as much as possible. In Focus mode, only the portion of the document you are working on is made prominent, and the rest fades out. You can configure the focus to be the current sentence, the current single or 3 lines, or the current paragraph.

Despite the rich feature set, the ghostwriter user interface is surprising clean and uncluttered.

The input and live preview functions reside in separate windows that you can resize and move around individually. Keen observers will definitely notice there is a real-time lag between actual text input and the update of the live preview. This is not a bug in the program. On the contrary, ghostwriter is programmed to only update the live preview when you stop typing(for a fraction of a second). The technical reason given by the developers is that the delay smoothens the jitters in synchronizing the rendering of large files. I can see many writers actually support this design decision because attention should be primarily focused on the writing, not the rendering, of the document.

Recall that the overall objective for using a MarkDown editor is to generate HTML code to insert into a CMS. With justmd and Remarkable, you need to first export to a HTML file, and then import the file (or copy and paste its contents) into the CMS. On the other hand, ghostwriter provides a shortcut Copy HTML button which is discreetly tucked away at the bottom right of the window. The button is a minor feature in the overall design scheme, but has a disproportionally high value to end users. Clicking the button copies the HTML code in its entirety into the clipboard. Importing the HTML into the CMS simply involves pasting the contents of the clipboard.

ghostwriter provides packages for Ubuntu , Fedora, openSUSE, and Arch Linux AUR. If you run Ubuntu or any of its derivatives such as Linux Mint, ghostwriter can be installed after adding a PPA repository and updating the local cache.

sudo add-apt-repository ppa:wereturtle/ppa
sudo apt update
sudo apt install ghostwriter

If ghostwriter is not pre-packaged for your distro, e.g., Debian, you can follow the on-line instructions to build the executable yourself. Depending on the particular distro and release, be prepared to spend some considerable time as you may run into the proverbial Linux dependency hell.

Feature comparison

Features	justmd	Remarkable	ghostwriter
Cross-platform	Linux(x64), Windows(x64), macOS	Linux, Windows	Linux, Windows
Linux installation	Downloadable executables	Downloadble packages for Debian, Ubuntu, Fedora, openSUSE, Arch	Downloadble packages for Ubuntu, Fedora, openSUSE, Arch
Export to HTML, PDF	Yes	Yes	HTML, PDF, Word, ODT
Spellchecker	No	No	Yes
Auto save	No	No	Yes
Word count	No	Character, word, line counts	Character, word, line, sentence, paragraph, page counts
Live preview	Fixed window proportion	Hidable, variable proportion	Separate resizable window (no dual panel)
GitHub-flavored syntax	Support for tables	Yes (tables, strikethrough, emphasis, etc)	Yes (tables, strikethrough, emphasis, etc)

Summary & conclusion

A writer's working style is intrinsically idiosyncratic. A writing environment that is distraction-free to one person may not be stimulating enough for another. Yet, ghostwriter is the clear winner of the 3 editors because it strikes a balance between clean design and feature richness. However, if ghostwriter is not pre-packaged for your Linux distro (say Debian), justmd and Remarkable are definitely worthwhile alternatives.

Sharing folders between VirtualBox host and guest machines

This post is the 4th installment of the VirtualBox series. The focus is on how to share folders between the host and the guest OSes. Part 1 of the series shows how to install VirtualBox on a Debian host, and how to create a FreeBSD virtual machine. Part 2 outlines the post installation tasks, including installing the universal VirtualBox extension pack. Part 3 shows how the guest OS can access USB drives mounted on the host.
The method outlined in this blog post works for Linux and Windows guest OSes but not FreeBSD. You can set up NFS or Samba to share a folder between a Linux host and a FreeBSD guest.

Prerequisites

To share a folder using the following procedure, the OS-specific guest additions package must be pre-installed. Part 1 of this series includes instructions for installing the guest additions package for the FreeBSD guest OS.

Procedure

1. Declare the shared folder on host OS.
   Open the Oracle VM VirtualBox manager on your host machine, select the target guest OS, and click Settings. Note that your guest machine does NOT need to be powered off.
   
2. Select Shared Holders, and click the Plus button to add a shared folder. Note that you can have more than 1 shared folder.
   
3. Specify the path for the shared folder.
   Click the down arrow next to the Folder Path parameter, select Other, and navigate to the target shared folder.
   
4. Configure the shared folder.
   The Folder name field refers to the folder name on the guest OS, which defaults to the folder name you specified in the last step. You should note the folder name in order to identify the full folder pathname on the guest OS. For example, if the shared folder on the host machine is /home/peter/Music, the corresponding folder name defaults to /media/sf_Music on the Ubuntu guest OS, and \\vboxsvr\Music on the Windows guest OS.
   
   The Auto-mount checkbox should be enabled. Optionally, you can also mount the folder manually, but then you must modify access permissions in order to make the folder writable by non-root users.
   
   Unless the shared folder is for one-off use only, you should enable the Make Permanent checkbox.
5. Power recycle the guest machine.
6. Grant folder access permission to non-root users.
   This step only applies to Linux guest OS(not Windows). Login to the guest OS, and add regular users to the vboxsf group using the following command.
   
   sudo usermod -aG vboxsf <someuserID>

Related posts

• Part 1 of VirtualBox series - on installation.
• Part 2 - post-installation.
• Part 3 - mounting of USB drives.

Ubuntu: how to reset lost administrative password

After an extended vacation, I came home to discover to my horror that I could no longer login to my seldom-used Ubuntu laptop. The reason was embarrassingly simple: I forgot my password. My muscle memory (or finger memory) did not help while I was frantically typing my usual passwords but to no avail. I own the administrative account on that system. So, I had no one else to turn to for help. Luckily, I was able to login another machine and google how to reset the administrative password on Ubuntu.

Prerequisites

There are 2 preconditions for using the procedure to reset the administrative password.

1. Physical access to machine.
   You need to access the system console in order to interrupt loading of the OS.
2. The root password was disabled.
   By default, Ubuntu disables the password of the root account by assigning it a value which cannot possibly match any encrypted value. During the installation of Ubuntu, you were asked to create the first user. That initial user, by default, belongs to the sudo group which means that the user can be elevated to perform system administration functions. The administrative password which we are going to reset refers to the password of the initial user.
   The procedure assumes that you did not manually assign root a valid password. If root has a valid password and you know it, then you can simply sign on as root and reset the administrative password using the password command. If the root password was also forgotten, this procedure does not apply because you need to enter that password to drop to root shell prompt in this procedure.

How to reset administrative password

1. Power recycle.
   After the BIOS screen appears, press down the left shift key to enter GRUB. This step can be quite finicky, and you may need to repeat it several times until you get the timing just right.
   
2. Scroll down to Advanced options for Ubuntu, and press Enter.
   
3. In the ensuing screen, scroll down to the top Recovery mode line, and press Enter.
   If the Linux kernel image had been upgraded on the machine before, you would see multiple recover mode lines on the screen. Select the recovery mode line that corresponds to the latest Linux image(that is nearest to the top).
   
4. In the Recovery Menu screen, scroll down to root, and press Enter.
   
5. Press Enter again.
   If you have previously assigned root a password, you would be prompted to enter it at this step. Otherwise, just press Enter to continue.
   
6. Remount filesystem.
   After all the hard work, you are now at the root shell prompt. The filesystem at this point is read-only. Remount the file system to add write permission.

   # mount -o rw,remount /
   

7. Reset administrative password.
   Use the passwd command to change the password for the administrative user.

   # passwd <adminuser>
   

   Press Control-D to return to the recovery menu.
   
8. Select resume.
   
9. Exit recovery.
   Press OK to exit recovery mode and continue booting.