Skip to content

test(ci): add CLI smoke and image format round-trip tests; run ctest in CI - #64

Merged
mcoliver merged 11 commits into
mainfrom
test/ci-smoke-suite
Sep 30, 2026
Merged

mcoliver merged 11 commits into
mainfrom
test/ci-smoke-suite

Conversation

@mcoliver

@mcoliver mcoliver commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

First pass at #60: run the existing tests in CI, and add end-to-end tests of the shipped command-line tools and image formats. Also fixes an intermittent crash the new tests found.

Stacked on #63. This PR's base is fix/windows-deps-and-oiio-write, because the format round-trip test needs the OIIO writer fixes. Merge #63 first; GitHub will then retarget this PR to main.

Crash fix: TwkMediaLibrary::Library worker thread race

utvio (and anything else that creates a media library, including the GUI) aborted about 1 in 40 runs, more often under load, with:

boost::lock_error: mutex lock failed in pthread_mutex_lock: Invalid argument
  TwkMediaLibrary::Library::threadMain()
  • Cause: m_taskThread was declared before m_taskMutex, m_taskCond, m_taskStop and m_taskQueue. Members are constructed in declaration order, so the worker thread started and locked a mutex that didn't exist yet.
  • Second bug: threadMain() also reset m_taskStop = false. If the destructor ran before the thread started, that erased the stop request and join() hung forever.
  • Fix: declare the thread last, remove the reset, and only join() a joinable thread.
  • Result: 0 failures in 900 parallel utvio runs. Before the fix it crashed within about 80.

New tests (src/test/CliSmokeTest, standard Python only)

  • cli-smoke: each bundled CLI (utvio, utvls, utvpkg, utvpush, utvshell, utvprof, py-interp, openutv-diagnostics, openutv-check-updates) must start, find its runtime dependencies, and exit with the expected output within a timeout.
    • On Windows it uses the .cmd wrappers, the way users run the tools.
    • Windows loader error dialogs are suppressed, so a missing DLL fails fast instead of hanging CI.
    • Qt tools run offscreen except on macOS.
    • This is the fix: utvpkg fails to find dependencies on Windows 11 #58 failure mode.
  • image-format-roundtrip: every image format utvio -formats advertises as writable gets a generated test image written to it and read back. Colored blocks in three corners are checked for orientation, channels and values, with a looser tolerance for lossy codecs.

Bugs found by running the tests in CI, fixed here

  • utvio on GPU-less Macs (VMs, CI, render nodes): it required a hardware pixel format and exited, even for -version. It now falls back to Apple's software renderer with a warning. UTV_SOFTWARE_GL=1 forces the software renderer.
  • utvio on Linux without an X display: segfaulted on an unchecked XOpenDisplay()/glXChooseVisual(). It now exits with a clear error. GLX context creation failures are reported too.
  • Linux on Mesa: glGenFencesNV & co. were linked directly, so binaries failed to load without NVIDIA's libGL. They're now looked up at runtime.
  • Existing tests replaced LD_LIBRARY_PATH via cmake -E env. They now prepend to it instead (--modify ...path_list_prepend).

Known issues found (tracked, not fixed here)

CLI

  • openutv-check-updates --help prints help. Previously any argument ran a live update check that could open a dialog.

CI

  • ctest --output-on-failure now runs after the macOS and Linux builds in branch-build, dev-build and release. Linux uses QT_QPA_PLATFORM=offscreen.
  • The Windows ctest step is now blocking (it had continue-on-error: true), with a timeout long enough for the round-trip test.
  • Locally on macOS: ctest passes 120/120 in about 12 s.

CI status

Branch CI passes on macOS, Windows and Linux (run 36659762676). The known issues above are reported as xfail or excluded with links, so they stay visible without blocking.

Not in this PR (rest of #60)

UI automation, testing the installed/signed package on clean machines, and movie/audio format round-trips.

🤖 Generated with Claude Code

…structed

m_taskThread was declared before m_taskCond, m_taskMutex, m_taskStop and
m_taskQueue. Members are constructed in declaration order, so the worker
thread started and locked m_taskMutex before the mutex existed. This
intermittently aborted every tool that creates a media library (utvio,
utv) with:

  boost::lock_error: mutex lock failed in pthread_mutex_lock: Invalid argument

Reproduced in ~1 of 40 serial utvio runs and within ~80 parallel runs;
0 failures in 900 parallel runs after the fix.

- Declare m_taskThread last so it starts after the members it uses.
- Don't reset m_taskStop in threadMain(): if the destructor ran before
  the thread started, the reset discarded the stop request and join()
  hung forever.
- Only join() a joinable thread (the protected default constructor never
  starts one).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Michael Oliver <mcoliver@gmail.com>
Previously any argument, including --help, ran a live update check that
could open a GUI dialog. Matches the --help added to openutv-diagnostics.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Michael Oliver <mcoliver@gmail.com>
…in CI

Refs #60. The existing 118 CTest tests were never run on macOS or Linux
CI, and the Windows step had continue-on-error: true.

New tests (src/test/CliSmokeTest, Python stdlib only):
- cli-smoke: every bundled CLI (utvio, utvls, utvpkg, utvpush, utvshell,
  utvprof, py-interp, openutv-diagnostics, openutv-check-updates) must
  start, resolve its runtime dependencies and exit with the expected
  output within a timeout. Uses the Windows .cmd wrappers when present,
  suppresses Windows loader error dialogs, and runs Qt tools offscreen
  off macOS. Catches #58-style dependency failures.
- image-format-roundtrip: for every image format `utvio -formats`
  advertises as writable, convert a generated test image to that format
  and back, then check orientation, channels and values at known sample
  points. Catches #59 (advertised formats that cannot be written),
  flipped/flopped output, truncated files and codec regressions.

CI:
- Run `ctest --output-on-failure` after the macOS and Linux builds in
  branch-build, dev-build and release (Linux with QT_QPA_PLATFORM=offscreen).
- Make the Windows ctest step blocking and raise its timeout for the
  round-trip test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Michael Oliver <mcoliver@gmail.com>
…lable

FBOVideoDevice required an accelerated CGL pixel format and exit(-1)'d
without one, so utvio could not run at all (even -version / -formats) on
headless macOS VMs, CI runners or render nodes without a GPU:

  ERROR: choosing pixel format: invalid pixel format

Retry with the existing Apple software renderer attributes and warn,
instead of exiting. UTV_SOFTWARE_GL=1 forces the software renderer.
Verified: all CLI checks and all 34 image format round-trips pass with
UTV_SOFTWARE_GL=1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Michael Oliver <mcoliver@gmail.com>
- Linux: staged binaries are built without RPATHs, so ctest could not
  load any of them (including the pre-existing py-interp, QFontTest and
  FastMemcpyTest tests). Use the same LD_LIBRARY_PATH build.sh sets up
  (stage lib + linuxbrew lib).
- Windows: stage Mesa's opengl32sw.dll as opengl32.dll before ctest,
  matching the installed layout. Without it utvio got the GDI OpenGL 1.1
  stub on the GPU-less runner and crashed (0xC0000005) when rendering.
- cli_common: exit code -1 (0xFFFFFFFF) on Windows is a plain exit(-1),
  not an NTSTATUS crash (utvprof -help exits -1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Michael Oliver <mcoliver@gmail.com>
- Linux: Qt, OpenColorIO and OpenEXR live in linuxbrew keg dirs
  (opt/<formula>/lib), not the linked prefix lib, and the staged binaries
  reference unversioned sonames. Add every keg lib dir to LD_LIBRARY_PATH
  for ctest (workaround until #66 fixes Linux launch).
- format_roundtrip: support per-platform known failures reported as
  xfail/XPASS. Windows heic/heif/hif (no HEVC encoder) and fits (read
  back fails) are tracked in #67.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Michael Oliver <mcoliver@gmail.com>
FBOVideoDevice passed the result of XOpenDisplay() straight to
DefaultRootWindow()/DefaultScreen() and dereferenced glXChooseVisual()'s
result, so utvio segfaulted (even for -version/-formats) whenever no X
display was available, e.g. headless render nodes and CI runners.

Exit with a clear error naming DISPLAY and suggesting xvfb-run, and
likewise when no suitable GLX visual exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Michael Oliver <mcoliver@gmail.com>
- Existing tests (FastMemcpyTest, QFontTest, ApplicationTest,
  AudioRendererTest, LoadingSharedLibrariesTest) replaced LD_LIBRARY_PATH
  via `cmake -E env`, discarding the caller's library path. Use
  `cmake -E env --modify LD_LIBRARY_PATH=path_list_prepend:...` instead.
- Linux CI: run ctest under xvfb-run so utvio's offscreen GLX context
  has a display.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Michael Oliver <mcoliver@gmail.com>
TwkGLF's NV fence sync object called glGenFencesNV & co. directly (via
GL_GLEXT_PROTOTYPES). libGL implementations without GL_NV_fence -- Mesa,
i.e. AMD/Intel drivers and software rendering -- don't export those
symbols, so binaries linking TwkGLF failed to load:

  symbol lookup error: ...: undefined symbol: glGenFencesNV

Resolve the five entry points with glXGetProcAddressARB and only use the
NV fence implementation when GL_NV_fence is advertised and all entry
points resolve; otherwise fall back to the stub. GL_ARB_sync remains the
preferred path, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Michael Oliver <mcoliver@gmail.com>
…reen in CI

- FBOVideoDevice: report a clear error when glXCreateContext or
  glXMakeCurrent fails, instead of failing later with "frame buffer
  incomplete: status = 0".
- CI: xvfb-run defaults to an 8-bit screen with no 32-bit GLX visual;
  run ctest on a 1920x1080x24 screen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Michael Oliver <mcoliver@gmail.com>
Linux binaries that link the rendering libraries can't load against
Mesa's libGL (direct GL extension symbols), and utvio's offscreen FBO
fails under llvmpipe. Tracked in #68.

- Linux ctest excludes LoadingSharedLibrariesTest, ApplicationTest,
  AudioRendererTest and image-format-roundtrip until #68 is fixed; all
  other Linux tests are blocking.
- cli-smoke reports utvio on Linux as xfail (XPASS once fixed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Michael Oliver <mcoliver@gmail.com>
@mcoliver
mcoliver changed the base branch from fix/windows-deps-and-oiio-write to main September 30, 2026 15:34
@mcoliver
mcoliver marked this pull request as ready for review September 30, 2026 16:16
@mcoliver
mcoliver merged commit 999e8ba into main Sep 30, 2026
13 of 14 checks passed
@mcoliver mcoliver mentioned this pull request Sep 30, 2026
1 task done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant