feat: decode video in hardware in the media player

[devopvoid]

Important

Linux has not been built or run. macOS and Windows were run on hardware; the Linux commit was written on a Mac, which has no Linux compiler. The CI jobs compile it for the first time, and it needs a run on a machine with an NVIDIA GPU: see Needs testing.

A MediaPlayer, and a MediaFileSource, can be asked to decode H.264 and VP9 video on the media engine or GPU of the machine instead of the processor. It is off unless asked for.

MediaFileSource source = new MediaFileSource(path, true);

// or, for a player of your own
MediaPlayer player = new MediaPlayer(new MediaReader(path), videoSource, null, true);

boolean hardware = player.isHardwareDecoding();

Platform	Hardware decoder	Status
macOS	VideoToolbox	tested
Windows	Direct3D 11	tested
Linux x86-64 and ARM64	NVDEC, on NVIDIA GPUs	not built

API

• MediaPlayer(reader, videoSource, audioSource, boolean hardwareDecoding), and MediaFileSource constructors with the flag. The existing constructors are unchanged and mean software.
• MediaPlayer#isHardwareDecoding() tells what is in use. That can be less than was asked for, and it can turn false while playing, but never true again.

Behavior

• The pictures are the same as software decoding gives. A decoded picture is read back from the hardware into system memory as NV12 and goes through the I420 conversion that already exists for formats WebRTC does not take.

• What it saves is processor time, not the copy. Measured on an Apple M2 with synthetic media (noisy frames, so decoding is not trivial), process CPU time per frame:

  Stream	Software, all cores	Hardware
  H.264 1080p	11.6 ms	1.43 ms
  H.264 4K	35.7 ms	4.15 ms
  VP9 1080p	10.2 ms	1.48 ms

  About seven to eight times less. On the clock, multi-threaded software is two to three times faster per frame, which does not matter at playback speed. These are figures for one machine, not a promise; the guide says so.

• Fallback to software, without the player noticing more than isHardwareDecoding() turning false:

  • a codec with no hardware decoder (VP8, MPEG-4, MJPEG, and H.265, which stays in software),
  • a stream FFmpeg does not offer the hardware for, such as VP9 profile 1 (4:4:4),
  • a decoder that fails. Until the hardware has produced its first frame, the packets sent to it are kept (up to 32) and decoded again in software if it fails, so none is lost. After the first frame, a failure goes on from the next key frame.
    The fallback lives in VideoDecoder: an error from it would end playback.

Native side

• VideoDecoder is the only place that knows. It picks the hardware device type of the platform (VideoToolbox, D3D11VA, CUDA), sets a get_format callback that takes the hardware pixel format where it is offered, and brings hardware frames into system memory. MediaPlayer gets SetHardwareDecoding and IsHardwareDecoding, and the JNI create a flag and a getter.
• FFmpeg build (dependencies/ffmpeg/CMakeLists.txt), H.264 and VP9 only:
  • macOS: --enable-videotoolbox, h264_videotoolbox, vp9_videotoolbox
  • Windows: --enable-d3d11va, h264_d3d11va, vp9_d3d11va, h264_d3d11va2, vp9_d3d11va2. The decoders offer the D3D11 pixel format, which the *_d3d11va2 hwaccels decode into, only when the older *_d3d11va ones are compiled in as well (h264_slice.c, vp9.c); without them the player never decoded in hardware on Windows. FFmpeg loads d3d11.dll and dxgi.dll itself when a device is created.
  • Linux x86-64 and ARM64: --enable-ffnvcodec --enable-nvdec, h264_nvdec, vp9_nvdec. FFmpeg loads libcuda.so.1 and libnvcuvid.so.1 with dlopen, so the libraries load on a machine with no NVIDIA driver.
    The list of components changed, so the next build of every platform rebuilds FFmpeg once. The WebRTC caches are not affected.
• dependencies/ffnvcodec (new): FFmpeg needs the headers of nv-codec-headers for NVDEC and finds them with pkg-config. The four headers dynlink_cuda.h, dynlink_cuviddec.h, dynlink_nvcuvid.h and dynlink_loader.h are taken unchanged from tag n12.0.16.1, the version of nvEncodeAPI.h that webrtc-jni vendors for NVENC and one FFmpeg n8.1 accepts (ffnvcodec >= 12.0.16.1 ffnvcodec < 12.1). nvEncodeAPI.h is copied from the NVENC directory, since configure checks for it too. The pkg-config file is ours; CMake fills in the path. The licenses of the files are collected in LICENSE and installed into the Linux platform jars under META-INF/licenses/ffnvcodec. The build needs pkg-config on Linux, which the runners have, and says so where it is missing.
• Left out on purpose: DXVA2, on Direct3D 9: a GPU of the last decade decodes H.264 and VP9 through Direct3D 11, and with D3D11VA tried first DXVA2 would only have been reached on a machine without Direct3D 11. 32-bit ARM, for which FFmpeg's own configure disables NVDEC (it allows it on x86 and little-endian aarch64 Linux only); VA-API, because FFmpeg links libva and it would become a requirement of the library; H.265 and AV1.

Testing

• HardwareDecodingTest (new, 7 tests), with media made for it: the existing assets are VP8 and MS-MPEG4, for which no platform has a decoder. media-test-h264.mkv is H.264 High from VideoToolbox's encoder, media-test-vp9.webm is VP9 from libvpx, media-test-vp9-444.webm the same in profile 1. 320x240, 15 fps, two seconds, with a key frame every second.
  • h264MatchesSoftware, vp9MatchesSoftware: the file is played with and without the flag, and the CRC of the luma of every frame is the same.
  • seeksInHardware: the decoder is flushed by a seek and goes on in hardware.
  • codecWithoutHardwareFallsBack: the VP8 asset, no hardware decoder, plays in software.
  • streamTheHardwareDoesNotTakeFallsBack: VP9 4:4:4, asked for hardware, plays in software with the same pictures.
  • fileSourceTakesTheOption, softwareIsTheDefault.
    A machine without a hardware decoder skips the tests that need one; -Dwebrtc.test.hardwareDecoding=true makes them fail instead.
• On an Apple M2 (macOS 14.5), with that property, all 53 tests of the module pass, the existing ones included.
• On Windows 11 with an AMD Radeon RX 9070 XT, with that property: HardwareDecodingTest 7/7 pass, every frame of H.264 and VP9 matching software, and so do the other tests of the module.
• The test media were wrapped in Matroska with a small script that is not part of the repo; how they were made is in the Javadoc of the test.

Not verified yet:

• Everything on Linux: that FFmpeg configures and builds with the new options, the pkg-config hand-over in the ARM64 cross build, decoding, and the comparison with software. On Linux, configure and the header set were checked on a Mac against FFmpeg's real configure with a stand-in for pkg-config: the version constraint is accepted and the four headers compile together. Whether FFmpeg then builds the NVDEC code for Linux could not be seen there.
• The macOS x86-64 cross build (CI cross compiles it; only arm64 was built here).
• Windows on Intel and NVIDIA GPUs, and Windows on ARM64 (CI builds it, without a GPU decoder to run on).
• The case where the hardware fails after FFmpeg has offered it. There was no stream to provoke it; the related case, no hardware format offered, is tested.
• Real content (a camera, a screen) and RTSP with packet loss; the figures are from synthetic media.
• CI runners are virtualized or have no GPU, so the hardware tests are skipped there, or fall back, which should not turn them red.
• Whether drivers on other hardware give exactly the same pictures. The tests compare every frame; a driver that differs would show up there.

Needs testing

On Linux with an NVIDIA GPU, and on Windows with an Intel or NVIDIA GPU:

mvn -pl webrtc-jni,webrtc,webrtc-java-media verify -Dtest=HardwareDecodingTest -Dsurefire.failIfNoSpecifiedTests=false -Dwebrtc.test.hardwareDecoding=true

This fails unless the player decodes in hardware, and the frames are compared with software. The module's tests only run from the reactor, with webrtc-jni in it: started alone they use the artifacts in ~/.m2, which can be from other builds, and -am does not bring in webrtc-jni, which webrtc does not depend on.

Docs

docs/guide/media/media-files.md has a new section, "Decoding in Hardware": the flag, the platforms, the fallback, and the measurements as measurements.