Roundup - Prevent shell bugs. (And: Are you a model Unix citizen?)

I don't always test my shell scripts, but when I do, I prefer roundup.

The majority of shell scripts I write are a couple of commands chained together with few switches, if any. These I test by hand, because a TDD approach would be heavy-handed.

At some point, hand-testing and eyeballing results for an arbitrary combination of switches and environmental conditions becomes a meticulous and time-consuming task. This makes it seriously tempting to move to a language less suitable for shell operations simply for access to serious testing tools.

That would be a mistake, because the pain is not shell itself, but its lack of a solid testing utility.

You would be correct in saying shell scripts can be tested with Ruby, Perl, and Python's testing tools through a series of system calls. But that brings us back to the point above. Shell is better at that.

I searched for pure, idiomatic shell code that could test mine, only to come up empty handed. The best maintained system didn't feel or look like shell to me. It also required my tests source special files in a special order and lacked simple features all correct shell scripts need; for instance, exiting with a non-zero status on failure.

So I wrote one.

Its name is roundup. Allow me to jump to a quick and dirty example before getting to something more meaty.

Example 1

grep-1-test.sh:


describe "grep(1)"

it_displays_usage() {
  usage=$(grep 2>&1 | head -n 1)
  test "$usage" = "Usage: grep [OPTION]... PATTERN [FILE]..."
}

it_will_fail_this_test() {
  echo foo | grep -q bar
}

Run:


$ roundup grep-1-test.sh
grep(1)
  it_displays_usage:                               [PASS]
  it_will_fail_this_test:                          [FAIL]
    + echo foo
    + grep -q bar
=========================================================
Tests:    2 | Passed:   1 | Failed:   1

Explained

The first test passes because it exited with a status of 0. The failing test did not.

If you're familiar with set -x then you recognize the output following the failing test's name. It's the trace of that test's run. Roundup highlights the failing line to bring your attention directly to the problem.

NOTE: By default, roundup will display the summary with color if stdout is a tty device (i.e. a Terminal). If you want the color for use with something like bcat, pass the --color argument to roundup.

Everything Is An Assertion

Each command executed can be thought of as an assertion in roundup because each test is run with -e set. The following example will demonstrate this. When formal comparisons are needed (ala assertEqual), here is my goto answer:

POSIX shell comes pre-packaged with two fantastic "assertion" utilities: test(1) and expr(1). With these and roundup's trace output, you get expected ... but was ... and more, for free!

Example 2

This is a test for far(1). A find and replace utility using ack and perl.

#!/usr/bin/env roundup

describe "far(1)"

before() {
    __DIR__="$PWD"
    rm -rf .sandbox
    mkdir -p .sandbox
    cd .sandbox

    echo echo foo > a.sh
    echo echo bar > b.sh
    echo echo foo > c.sh
}

after() {
    rm -rf "$__DIR__/.sandbox"
}

it_displays_usage_when_no_arguments() {
    far 2>&1 | grep -q "usage: far \[ACK_OPTIONS\] FIND REPLACE \[FILES\]"
}

it_exists_non_zero_when_no_arguments() {
    ! far
}

it_replaces_all_occurances_of_a_word() {
    far foo baz
    grep -qv foo *.sh
}

it_takes_ack_arguments() {
    ! far --no-sh foo baz
    grep -q foo *.sh
}

it_takes_specific_files_as_arguments() {
    far foo baz a.sh
    grep -qv foo a.sh
    grep -q  foo c.sh
}

The result:

far(1)
  it_displays_usage_when_no_arguments:             [PASS]
  it_exists_non_zero_when_no_arguments:            [PASS]
  it_replaces_all_occurances_of_a_word:            [PASS]
  it_takes_ack_arguments:                          [PASS]
  it_takes_specific_files_as_arguments:            [PASS]
=========================================================
Tests:    5 | Passed:   5 | Failed:   0

Also, follow the source links here to see roundup testing itself.

Testing command line interfaces

Use roundup to test command line interfaces too. If writing tests in shell for a Unix command is hard, then it's probably not an an ideal Unix shell citizen.

An exercise for the reader

Redis comes with redis-cli. Both of these tools are not written in shell, but you interact with them in one. Most of their CLI's are easy to test; but some are not for various reasons.

NOTE: I choose redis for this example because it's something I use very often, love using, and know its rich features will give readers enough substance for this exercise.

Below is the start of a redis-cli test plan. Use it to test as many of these as you can. Then answer these questions:

Here it is: redis-cli-test.sh

Enjoy it. It's bonus.