Wednesday, May 29, 2019

Beware of this find command gotcha



find is a basic useful command that Linux users run all the time. The command searches a file system from a given starting location, and returns all matches based on input filters that you provide as arguments.

The Gotcha


The gotcha is when you try to narrow the search by pruning a sub-directory from the search (including the directory itself and everything under it). For instance, suppose you want to find all files under the directory /data that are owned by root, excluding the sub-directory /data/keepit and all files underneath.



My first attempt at the solution results in the following find command.

find /data -path /data/keepit -prune -o -user 0


The -o argument specifies the logical 'or' operator. The expression on the left,  '-path /data/keepit -prune'  indicates where to prune the search. The idea is that when the search reaches /data/keepit, the -prune argument causes the search to not descend further into the sub-directory. Furthermore, -prune always returns true. Hence, the whole expression returns 'true', without having to evaluate the expression on the right of -o.

The expression right of -o tests for root ownership (root is user 0).

I was befuddled to learn that running the above command returns /data/keepit (but not its descendants). If the search is snipped at /data/keepit, why is the sub-directory itself included in the output? Besides, /data/keepit is not owned by root.

Being unaware of this behavior could lead to some unintended and very bad consequences as files named in the find output are often piped to the xargs command for further processing.

The Explanation


Before I present my solution, let's discuss why the point of pruning, i.e., the sub-directory named in -path, is actually included in the output.

The primary purpose of find is to search for file matches. Yet, it can have side effects through actions you specify on the command line. In addition to -print/-print0, there is also the -exec action. Unless you explicitly specify an action, the find command assumes the default action is -print.

The above example has no explicit -print or -exec action, therefore, the  action defaults to print all file matches. This explains why /data/keepit, a match for -path, is in the output. Its descendants, on the other hand, were excluded because of pruning.

The Solution


My solution is to specify -print explicitly on the command line.

find /data -path /data/keepit -prune -o -user root -print


Lo and behold. When you run the above command, /data/keepit is no longer part of the output.

By specifying the -print action explicitly, the find command no longer defaults  to printing out each file match. Instead, it will only print a file match if it is explicitly requested.

Summary & Conclusion

The pruning logic of the find command is quite confusing. Reading its man page offers some help, but may generate more questions than answers. I hope that this article is of help. But, I recommend that before you use the -prune feature on your production data, test it on some dummy data first.

You have been forewarned.

Monday, March 11, 2019

ts: epitome of the Unix philosophy

Do one thing and do it well - the Unix philosophy

In this new age of Linux bloatware (hello, systemd), it is exhilarating to discover small gems like ts, a command-line tool that prepends a timestamp to each output line.

How is this useful?


I run scripts all the time—bash scripts, Ansible playbooks, etc–to automate system administration tasks. Many longer-running scripts that I run output statements in real-time to report what they are doing. For example, running an Ansible playbook will automatically output the name of the individual task as it is being executed. By default, however, no timestamps are displayed for the tasks.


$ ansible-playbook -b -i hostsfile myPlaybook.yml
PLAY [localhost] ****************************************************************

TASK [Gathering Facts] **********************************************************
ok: [localhost]

TASK [Disabe Caps Lock] *********************************************************
ok: [localhost]

TASK [Install X apps] ***********************************************************
ok: [localhost] => (item=autokey-gtk)
ok: [localhost] => (item=gnucash)

TASK [Install 64-bit texamker - Debian] *****************************************
changed: [localhost]
...snipped...
PLAY RECAP **********************************************************************
localhost : ok=8 changed=2 unreachable=0 failed=0


Often, I do want to display the timestamps for logging or troubleshooting purposes. An unusually short (or long) execution may signal something is amiss.

Granted, you can use the Ansible-specific profile_tasks plugin to profile your tasks. But, I propose ts as a quick-and-dirty solution: just pipe Ansible output to ts like the following.


$ ansible-playbook -b -i hostsfile myPlaybook.yml |ts
Mar 11 11:44:05 PLAY [localhost] ****************************************************************
Mar 11 11:44:05
Mar 11 11:44:05 TASK [Gathering Facts] **********************************************************
Mar 11 11:44:05 ok: [localhost]
Mar 11 11:44:06
Mar 11 11:44:06 TASK [Disable Caps Lock] *********************************************************
Mar 11 11:44:06 ok: [localhost]
Mar 11 11:44:06
Mar 11 11:44:06 TASK [Install X apps] ***********************************************************
Mar 11 11:44:10 changed: [localhost] => (item=autokey-gtk)
Mar 11 11:44:14 changed: [localhost] => (item=gnucash)
Mar 11 11:44:14
Mar 11 11:44:14 TASK [Install 64-bit texamker - Debian] *****************************************
Mar 11 11:44:19 changed: [localhost]
...snipped...
Mar 11 11:44:30 PLAY RECAP **********************************************************************
Mar 11 11:44:30 localhost : ok=8 changed=3 unreachable=0 failed=0


Optional ts arguments

By default, the ts command inserts the absolute timestamp into each output line. You can use the -s argument to replace the absolute timestamp with the elapsed duration since the start of execution.

$ ansible-playbook -b -i hostsfile myPlaybook.yml |ts -s
00:00:01 PLAY [localhost] ****************************************************************
00:00:01
00:00:01 TASK [Gathering Facts] **********************************************************
00:00:01 ok: [localhost]
00:00:02
00:00:02 TASK [Disable Caps Lock] *********************************************************
00:00:02 ok: [localhost]
00:00:02
00:00:02 TASK [Install X apps] ***********************************************************
00:00:06 changed: [localhost] => (item=autokey-gtk)
00:00:10 changed: [localhost] => (item=gnucash)
00:00:10
00:00:10 TASK [Install 64-bit texamker - Debian] *****************************************
00:00:15 changed: [localhost]
...snipped...
00:00:26 PLAY RECAP **********************************************************************
00:00:26 localhost : ok=8 changed=3 unreachable=0 failed=0



Another useful argument to know is -i. With this argument, each output line displays the elapsed time since the previous output line. You don't need to do the mental math to calculate how long a task took.

$ ansible-playbook -b -i hostsfile myPlaybook.yml |ts -i
00:00:00 PLAY [localhost] ****************************************************************
00:00:00
00:00:00 TASK [Gathering Facts] **********************************************************
00:00:01 ok: [localhost]
00:00:00
00:00:00 TASK [Disable Caps Lock] *********************************************************
00:00:00 ok: [localhost]
00:00:00
00:00:00 TASK [Install X apps] ***********************************************************
00:00:04 changed: [localhost] => (item=autokey-gtk)
00:00:04 changed: [localhost] => (item=gnucash)
00:00:00
00:00:00 TASK [Install 64-bit texamker - Debian] *****************************************
00:00:05 changed: [localhost]
...snipped...
00:00:00 PLAY RECAP **********************************************************************
00:00:00 localhost : ok=8 changed=3 unreachable=0 failed=0


In summary, ts is a fast and easy way to add timestamps to script or command output.