strum

A test runner for the .tspec format. A .tspec file declares commands to run and what they should produce — exit code, stdout, stderr, files on disk — and strum runs them and reports pass or fail.

It is aimed at testing compilers, command line tools and build pipelines: things whose behaviour is observable from the outside as process results and filesystem side effects.

Build

# once
cc build.c -o build

# from then on
./build && ./out/strum
#or
./build -r

flag	meaning
-d, --debug	-ggdb -fsanitize=address -Og instead of -O3
-s, --silent	only warnings and errors
-n, --dry-run	print the compile commands, run nothing
-r, --run	run out/strum after a successful build
-o, --outdir	output directory (default ./out/)
-c, --compiler	compiler to use (default clang)

Positional arguments are forwarded to the runner under --run, and the runner's exit code becomes the build's exit code:

./build -d -r                 # build, then run the whole suite
./build -r tests/parser       # build, then run one directory

strum.c needs C11 for _Static_assert. Clang and GCC accept it under -std=c99 as an extension, but -std=c11 is what the build uses.

Usage

strum [FLAGS] [target...]

A target is a .tspec file or a directory to search recursively. With no target, strum looks for tests/; if that directory does not exist there is simply nothing to run. A target named on the command line that does not exist is an error.

flag	meaning
-v, --verbose	show every command and assertion, not just failures
-q, --quiet	warnings and errors only
-l, --log-level	trace, debug, verbose, info, warn, error, fatal
-c, --color	auto (default), always, never
-n, --noescape	Do not escpae blob content
--gitignore	Generate .gitignore entry for .side_effects folder
-V, --version	print the version
-h, --help	print usage

Exit codes

code	meaning
0	everything passed
1	at least one test failed
2	a file could not be run — malformed, or a setup error
3	usage error

Output

A passing test is one line. A failing test prints its commands and the assertion that failed, indented underneath:

[STRUM] INFO:  filesystem/test.tspec::side_effects_is_wiped_before_each_run   pass
[STRUM] ERROR: assertions/test.tspec::stdout_matches_exactly                  fail

NOTE: lines are diagnostics for the mistakes that are easy to make and hard to see: an off-by-one byte count, a forgotten trailing newline, a directive missing its space, a path resolved against the wrong directory, an executable that could not be started and why.

Writing tests

The format is specified in spec/tspec_specification.txt. The short version:

:test pedantic_enabled

:command compile
:blob executable 3
cc
:blob args 45
main.c -o .side_effects/main -I../..
:int return 0

:command run
:blob executable 20
.side_effects/main
:int return 0
:blob stdout_contains 7
success

:file_contains 21 .side_effects/out.log 6
hello

Blob fields declare an exact byte count and the content follows on the next line, so arbitrary bytes need no escaping. The count excludes the newline that terminates the payload. If the value you are matching itself ends in a newline — as most command output does — count it and leave a blank line after the payload. Getting this wrong is the most common mistake; strum tells you the number to use.

Conventions

• One .tspec file per directory. Two would fight over the same .side_effects, and strum refuses to run them.
• Commands run with their .tspec file's directory as the working directory. Paths in assertions are resolved against it.
• Anything a command generates belongs in .side_effects/, which is deleted and recreated before each run. strum --gitignore >> .gitignore adds the right pattern.
• Commands get /dev/null on stdin. :blob stdin is reserved by the format and rejected.
• A failing command skips the rest of its own test; later tests still run, and a file that fails to parse does not stop other files.
• Files are discovered in sorted order, so a run over an unchanged tree reports in the same order every time.

Tests

tests/ is strum's own suite, written in .tspec and run by strum:

./build -r

Each directory covers one area — parser, streaming, diagnostics, exit_codes, and so on. Tests that need to inspect strum's own behaviour invoke ../../out/strum on a fixture in fixtures/. Fixtures use the .tspec.in extension so discovery, which only picks up *.tspec, does not try to run them directly.

The binary path is baked into those fixtures. If you change the output directory, update ../../out/strum in tests/*/test.tspec to match.

Limits

Compile-time, in strum.c. Memory use is fixed and predictable; there is no allocator in the run path.

constant	default	bounds
TSPEC_MAX_BLOB_SIZE	8192	one declared blob value
TSPEC_MAX_LINE_SIZE	512	one control line
TSPEC_MAX_ASSERTIONS	16	assertions per command
TSPEC_MAX_ARGV	64	arguments per command
STRUM_MAX_SPEC_FILES	512	files discovered in one run
STRUM_REPORT_SIZE	65536	buffered detail for one test

Tests and commands per file are unbounded — the file is streamed, and only one command is held in memory at a time. Command output is unbounded too: it is matched as it arrives rather than buffered, so a command may print gigabytes and still be compared byte for byte.

Roadmap

Future:

• v1.3.0: Record mode
• v1.2.0: Paralel tpsec execution
• v1.1.0: Windows support

Completed:

• v1.0.0: First release
  • Can run and evaluate .tspec files

Credits

Heavily inspired by Porth by Alexey Kutepov (rexim), and specifically by its test.py — the idea of recording a command's expected output into a binary-safe file with explicit size prefixes, and replaying it as a test, comes from there.

Uses logcie for logging and optly.h for argument parsing, both vendored in thirdparty/.

License

MIT.