D3D12: DXIL tessellation through the Metal Shader Converter's emulation

[bahacan16]

Problem

Ghost of Tsushima (D3D12, DXIL shaders) creates 12-16 graphics pipelines with DXIL hull and domain shaders, all of them its water (ls_Main_techWaterMain, ps_Main_techWaterBlend, ps_WaterHeight_techWaterHeight, ...). The native runtime only builds tessellation pipelines for DXBC shaders (ml1083, DXMT's emulation), so for these it logs newRenderPipelineState failed (..., tessellation) and returns E_FAIL from CreateGraphicsPipelineState. The game reports "CreateGraphicsPipelineState failed" and the water is never drawn.

Cause

• device_CreateGraphicsPipelineState has no path for DXIL hull/domain stages (mad_tess_build is DXBC-only) and returns E_FAIL for any hull+domain pipeline it cannot build (geometry-shader pipelines already become placeholders since ml1138).
• The converter service does not ask the Metal Shader Converter for hull or domain reflection, and winemetal's local geometry-emulation pipeline builder has no tessellation variant.

Change

• Pipelines (mad_dtess_convert): a pipeline whose VS, HS and DS are all DXIL is converted with the converter's geometry/tessellation emulation, the way IRRuntimeNewGeometryTessellationEmulationPipeline expects it: the VS library (object variant, plus the synthesized stage-in function; the layout is passed even when empty, because a water grid built from SV_VertexID still needs a stage-in function), the hull library (irconverter_hull_shader + irconverter_tessellator) and the domain library (irconverter_dxil_domain_shader + the passthrough geometry shader for the tessellator's output primitive). The hull/domain pair is checked as IRRuntimeValidateTessellationPipeline does (interface sizes and counts, threadgroup shapes, factor range); anything missing logs why and keeps the pipeline a placeholder. gs_emu = 2 marks such a pipeline.
• Draws (mad_ts_draw): direct DrawInstanced / DrawIndexedInstanced with a patch-list topology whose control-point count matches the hull shader follow IRRuntimeDraw[Indexed]PatchesTessellationEmulation: draw info at 5, draw params at 4, vertex-buffer table at 6, the top-level argument buffer also at kIRArgumentBufferHullDomainBindPoint (3), and 15360 bytes of object threadgroup memory. Indirect draws on these pipelines stay skipped, as they are for every geometry-emulation pipeline today.
• Placeholder: a hull+domain pipeline that neither tessellation path can build now returns a placeholder whose draws are skipped (as ml1138 does for geometry shaders) instead of E_FAIL.
• Converter service (madeira_ir_unix.mm): hull/domain reflection (IRShaderReflectionCopyHullInfo / CopyDomainInfo, bound optionally) is returned through new ret_hs_* / ret_ds_* fields at the end of madeira_ir_convert_args; an emulated hull/domain stage with an empty reflection name gets the converter's fixed name so the conversion does not fail. The DXIL cache stores the new fields (MAD_DXC_VERSION 2, so old entries miss); tests/native/dxil_cache_test.c checks the round trip.
• winemetal (separate PR to willfaust/dxmt, dxmt.patch): WMTGeometryEmulationInfo's reserved[6] become hull_library, domain_library, max_tessellation_factor, tessellation (same size and offsets); the local geometry-emulation builder builds the tessellation variant when tessellation is set; a mesh draw with reserved[1] == 0x7e55 sets the object threadgroup memory length from reserved[0] first.
• madeira.cfg dxil-tess = 0 turns the DXIL path off (pipelines become placeholders). Catalog row regenerated.

Evidence

Ghost of Tsushima, iPhone 17 Pro Max, iOS 27:

• E_FAIL for these pipelines: the game logged "CreateGraphicsPipelineState failed". With the placeholder alone it carried on, and every run logged 12-16 placeholder hull+domain pipelines, all water; no water on screen.
• With the DXIL path: the water pipelines build ([winemetal] DXIL tessellation pipeline OK, no REFUSED) and a session draws ~1,900 DXIL tessellation draws (DXIL tessellation: N drawn in the ml1050 report).

Notes / risks

• Known artifact on the indirect path, not included here. The fork also drew indirect DXIL tessellation draws (a helper compute kernel turning ExecuteIndirect records into mesh-dispatch arguments). In Ghost of Tsushima those are the lit smoke particles (ls_SetColor / ps_SetColor_MultiLight), and once they were drawn the smoke rendered as hard-edged, uniformly dark rotated squares over the fires. Builds that drew only the direct (water) draws and skipped the indirect ones showed no squares in the same scene; turning the whole DXIL path off also removed them. The cause inside the converter's emulation was not found, so this PR ports only the direct-draw path; indirect draws on these pipelines are skipped and counted as before. The fork's tessellation-factor cap for those particles (a performance workaround) and an experimental "patch input topology" switch are not included either.
• Merge order: the willfaust/dxmt part must land (and the submodule pin move) before this builds -- madeira_d3d12.c uses the new WMTGeometryEmulationInfo fields, and app/libdxmt_unix.a needs the winemetal change to build the pipelines.
• madeira_ir_convert_args grows at the end; the PE DLL and the app's converter service must come from the same build (as today).
• If a game's DXIL tessellation misbehaves (fault, hang), dxil-tess = 0 returns its hull+domain pipelines to placeholders.
• Only built here: madeira_d3d12.c compiles for arm64ec (llvm-mingw 20260421) against the patched winemetal.h with no new warnings; the Objective-C++ service and winemetal were not compiled here (no Apple SDK), but are the code the fork ran on device. DXIL cache host test passes.

---

🤖 Generated with Claude Code

Claude-Session: https://claude.ai/code/session_0189oLHghpaYKLk4f786a6bc