Audio: FMOD Studio + Vercidium Audio system (phases 1-15) and Windows fixes - #31
Merged
Merged
Conversation
…pt-in) Adds two independent, opt-in audio subsystems and fixes several latent bugs found while wiring them up. WHAT - FMOD Engine as an alternative playback backend (--fmod, LUX_ENABLE_FMOD). AudioEngine/AudioSource/AudioListener now compile against either miniaudio (default) or FMOD Core, selected by #ifdef within each .cpp, so call sites never see the backend and the default build has no FMOD dependency. AudioEngine::Update() pumps FMOD::System::update() once per frame from Application::Run; it is a no-op under miniaudio. - RaytracedAudioScene (--raytraced-audio, LUX_ENABLE_RAYTRACED_AUDIO): wraps the Vercidium Audio SDK behind a Pimpl. One VAWorld per Scene, created on OnRuntimeStart, mirroring static MeshCollider geometry and tracking one emitter per AudioSourceComponent. Its per-source occlusion/reverb results feed FMOD's Channel::set3DOcclusion / setReverbProperties. Built without the SDK every method is a no-op, matching the DiscordSocial pattern. Both SDKs are proprietary and gitignored; their EULAs forbid redistributing the SDK itself, so they are fetched manually. Configure.py warns when a feature is enabled without its SDK present. BUGS FIXED ALONG THE WAY - AudioSourceComponent/AudioListenerComponent were never serialized, and a legacy-schema guard hard-rejected any scene containing them. Both now round-trip through SceneSerializer. - The same two components were missing from AllComponents/DuplicateComponents, so Scene::Copy silently dropped them on Play - audio could never work at runtime regardless of backend. - AttenuationModel was emitted as uint8_t; yaml-cpp writes unsigned char as a *character*, so saving a scene with an audio source wrote a raw control byte and corrupted the .luxscene. Now uint32_t both directions. - VA emitters default to zero rays of every type, so sources produced no occlusion or reverb at all. Ray counts are now set explicitly. - GetResult() segfaulted inside the SDK when querying an emitter created in the same frame, before its first vaWorldUpdate. Now guarded. - Stop() destroyed emitters the world still owned pending their reverb tail (vaWorldRemoveEmitter answers VA_PENDING_REMOVAL), double-freeing them in vaWorldDestroy. Teardown now only destroys what it actually removed. - Linux-Build.sh regenerated makefiles without the feature flags, silently producing a build with the features compiled out. It now forwards LUX_PREMAKE_OPTIONS. VERIFICATION Core, Editor and Lux-Runtime all build clean with --fmod --raytraced-audio. FMOD initialises against a real device; scene loads; VA world creation and static-geometry mirroring verified correct in-editor (mirrored AABB matches the source transform exactly); per-emitter results reach the playback layer (Valid=true). Every VA and FMOD call site was additionally compiled against the real vendored headers in standalone harnesses. KNOWN ISSUES - --raytraced-audio should stay OFF by default - Creating a VAWorld inside the engine process corrupts the heap on Play. Isolated: FMOD-only is clean, and the identical VA calls are clean in a standalone harness, so it is an interaction, root cause not yet found. - VA occlusion output does not respond to geometry (identical values with no wall and with a 100x75m wall), reproducible standalone. Open with Vercidium. - Windows FMOD paths in Dependencies.lua are unverified placeholders; only the Linux package has been fetched. - Static geometry is mirrored once at Play start and does not follow moving colliders; all geometry is hardcoded to VAMaterialConcrete. Requires project regeneration (new files + premake changes). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DsM8FJLGY8SzZqumBdRKh5
Three strands of work on the audio stack. Ray-traced acoustics (Vercidium) -------------------------------- Rebuilt to the topology Vercidium's docs describe: the listener is the only ray caster, casting all five ray types (including the ambient pair we were leaving at zero), and every source is registered as one of its targets. Adding a source now costs a target on the listener's existing ray budget rather than a second budget of its own. The simulation's full EAX/I3DL2 reverb set is read (vaEmitterGetEAX) and mapped onto FMOD_REVERB_PROPERTIES - previously one field of ~25 was used. Every value is clamped to FMOD's documented range because FMOD rejects the whole struct if one field is out, silently keeping the old reverb. Occlusion is applied as two bands rather than one scalar: the low-frequency gain scales the channel's volume (how much gets through) and the relative high-frequency loss drives the occlusion filter (how much more the highs are cut). Feeding HF straight into set3DOcclusion double-counts the loss and makes an occluded source inaudible instead of muffled. The frame is now explicitly two-phase - WaitForResults, then mutate and read, then OnUpdate - because the simulation is asynchronous and results were being read while workers were still writing them. Stop() also waits unconditionally before draining: vaWorldGetThreadsRunning reads false for work that has been queued but not started, which skipped the drain entirely. Measured end to end: reverb decay tracks room size (0.10s open field, 0.32s small room, 1.39s large hall) and reaches FMOD accepted. Occlusion does not respond to geometry at all - a Vercidium defect, with a self-contained repro under docs/vercidium-repro/ that includes reverb from the same emitter as a control, proving the rays do reach the geometry. 3D visualisation and the Audio Debugger --------------------------------------- Visualisation rays are delivered on Vercidium's worker threads and are valid only inside the callback, so they are copied out under a lock and copied again by the renderer. A ray that hits nothing still occupies its slot, filled with a far out-of-world sentinel, so bounces are classified by world containment and each polyline stops at its first miss. New AudioDebugPanel (View -> Audio Debugger) renders both halves of the stack, with the reverb the simulation produced and the values FMOD actually received side by side. Deliberately free of LUX_ENABLE_FMOD / LUX_ENABLE_RAYTRACED_AUDIO, which are Core-only defines: each backend names itself through its stats struct instead. FMOD Studio pipeline -------------------- Studio is linked alongside Core and owns it - Studio::System::initialize creates the core system, so Shutdown releases only Studio (releasing both is a double free) and Update calls Studio then Core. Studio's update does NOT recompute Core's 3D attenuation; measured with Channel::getAudibility, Studio alone leaves audibility frozen at the geometry a channel started with, which is indistinguishable from spatialisation being off. AudioBankBuilder shells out to fmodstudiocl the way ScriptBuilder shells out to dotnet, and is not behind LUX_ENABLE_FMOD - building banks needs the Studio application, not the SDK. Banks rebuild on Play when the .fspro is newer, and load strings-bank-first (it carries the path table; loading it late makes every event lookup fail unhelpfully). .fspro and .bank are asset types so the Content Browser can show and activate them - activation opens FMOD Studio. Both the browser and the asset registry treat a Studio project directory as opaque: it is dozens of GUID-named XML files plus gitignored build output, and importing the banks would put regenerable files into the tracked registry. Known gap: the binary runtime project format does not carry the Studio settings and runtime export does not copy banks, so an exported game ships without audio. That belongs with the runtime export work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DsM8FJLGY8SzZqumBdRKh5
Sources can now reference an authored FMOD Studio event instead of a raw
audio file. When an event is set it supersedes the file: spatialisation,
attenuation, cones, doppler and randomisation are authored in the event, and
the component's config fields for those no longer apply.
Events are referenced by GUID, not by path. The path is stored alongside it
purely as a label and is refreshed from the loaded banks on display - never
used to resolve. Renaming or moving an event in FMOD Studio changes its path
but not its GUID, so a path-referencing scene would go silently mute the
first time a designer reorganises the project, with no error anywhere.
AudioEventInstance wraps FMOD::Studio::EventInstance. Two details worth
knowing: IsPlaying counts SUSTAINING, because an event holding at a sustain
point is audible even though its state is neither STARTING nor PLAYING; and
the destructor stops IMMEDIATE rather than ALLOWFADEOUT, because it runs
during teardown where a fade would outlive the thing being torn down.
Scene owns instances per entity. A null entry in that map is meaningful - it
records an event that could not be resolved, so the "not in any loaded bank"
warning is logged once rather than on every frame. Instances are released
both per entity and with the runtime, since they hold Studio resources.
Ray-traced acoustics reach events as named parameters (Occlusion,
ReverbSend) rather than as a filter applied by the engine. That is the
philosophical difference between the two paths: on the legacy path the
engine decides what occlusion does to a sound, while an event is told what
was measured and its author decides what it means. An event declaring
neither parameter is simply unaffected.
The inspector gains an event picker listing what the loaded banks describe,
with the GUID and 3D/oneshot shown on hover. An assigned event missing from
the loaded banks is flagged rather than reading as "nothing assigned".
The legacy raw-file path is retained and marked as such. The project's
.fspro has no authored sounds yet, so removing it now would leave the engine
playing nothing at all.
Verified end to end: a 3D event authored through fmodstudiocl's scripting
API and built into the master bank is enumerated by the engine on project
open ("Loaded 2 bank(s) describing 1 event(s)"), which exercises bank
enumeration, GUID formatting and the list the picker and serializer share.
Note for anyone adding to this: AudioEventInstance.cpp is a new translation
unit, so the projects need regenerating - and regeneration drops the
--fmod / --raytraced-audio options, which must be passed again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DsM8FJLGY8SzZqumBdRKh5
…path
Adds one authored event to the sample project's FMOD Studio project so the
engine's event path can actually be exercised: without an event in a bank,
the picker is empty, no GUID can be resolved and nothing about the Studio
integration is testable.
The event is 3D (created with a spatializer on its master track) and
assigned to the Master bank, but has no sound on its timeline - what it
exists to prove is bank enumeration, GUID resolution, the inspector picker
and scene serialization, none of which need it to be audible.
Created through fmodstudiocl's scripting API rather than by hand, so it is
reproducible:
workspace.addEvent("LuxTestEvent", true)
ev.relationships.banks.add(masterBank)
Separate from the code change so it can be dropped on its own once the
project has real authored audio.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DsM8FJLGY8SzZqumBdRKh5
The Studio configuration added with the bank pipeline reached ProjectConfig and the serializer but never the settings UI, so pointing a project at its .fspro meant hand-editing the .luxproj YAML. Setting up audio was therefore undocumentable as an editor workflow, which is how the gap surfaced. Project Settings -> Audio now exposes the Studio project path, bank output directory, rebuild-on-play and live update, alongside the existing streaming threshold. Below them is live state rather than settings: how many banks and events are currently loaded, and buttons to build banks or open the project in FMOD Studio without entering Play. Both report clearly when they cannot work - a missing .fspro, or fmodstudiocl not being installed - since neither is recoverable from inside the editor and the fix is an environment variable. Live update is labelled as taking effect on the next project open, because the audio engine is initialised from Project::SetActive and the flag is read there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DsM8FJLGY8SzZqumBdRKh5
…should Two related passes over AudioSourceComponent. Picking an event ---------------- Events now record which bank describes them, captured during enumeration - the only place that relationship is known without asking FMOD again. The inspector picker becomes two stages, bank then event, with a search inside the dropdown, because one flat list is fine for five events and unusable for five hundred. Strings banks are excluded from the bank list: they carry the path table rather than events, so they would offer a permanently empty list. The bank is stored on the component as a hint, exactly like the event path - refreshed from the loaded banks on display and never used to resolve. The GUID still resolves the event on its own, whichever bank it turns out to live in. Filter and search state live on the panel rather than the component, since they are a view preference, not something to serialize into scenes. Trimming the component ---------------------- AudioSourceConfig carried fifteen fields, most of which an FMOD Studio event now owns. Keeping them was worse than useless: visible, editable, and silently ignored the moment an event was assigned. Removed Spatialization, AttenuationModel, RollOff, Min/MaxGain, Min/MaxDistance, the three cone angles and DopplerFactor, along with the AttenuationModelType enum and the setters that fed them. What remains is volume, pitch, play-on-awake, and looping for the legacy path. The playlist goes with them - AudioData, the four component helpers, its runtime storage and about forty-five lines of index juggling in Scene. A multi-instrument does the same job in Studio, with weighting and no-repeat. Removing the struct also exposed an OnComponentAdded<AudioData> specialization that had always been dead, since AudioData was never a component. In their place, ParameterOverrides: name/value pairs applied once when the instance is created, so two entities can share one event and still sound different. Scripts will drive parameters continuously through the C# API; these are the authored starting point. Behaviour changes worth knowing ------------------------------- Spatialization defaulted to false, which quietly made every raw source 2D unless someone ticked it. The legacy path is now always 3D with FMOD's default rolloff - tuning raw-file attenuation is precisely what an event should be doing instead. Old scenes still carry the removed keys. The deserializer ignores them deliberately, with the reason recorded: reading them would resurrect settings that no longer reach the mixer. Scenes keep playing, with default attenuation rather than whatever was tuned in the component. Paused now defaults to true so play-on-awake fires once for events. That required OnRuntimeStart to clear it when it starts a raw source - otherwise the per-frame path stays armed and restarts the source every time it finishes, turning a one-shot into an unintended loop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DsM8FJLGY8SzZqumBdRKh5
…lines Physics colliders previously drew through the selection wireframe pass, which has depth testing off so selection stays visible through geometry. Colliders inherited that and floated in front of the walls containing them. They now have their own pass with depth testing on and depth writes off, reusing the pre-depth output; the on-top behaviour is still available through the existing Show Physics Colliders On Top option, which routes back to the wireframe pass. The render graph declares the collider depth read so the dependency is real rather than incidental. Grid and wireframe pipelines gain BackfaceCulling off and DepthWrite off, which is what those passes actually want - they overlay rather than contribute occlusion. Selection outlines sample the jump-flood mask and distance buffers with a point sampler instead of a linear one. Those textures hold mask classes and distance vectors, not colour: interpolating them blends values across the selection boundary, which is meaningless and shows up as a frayed outline. The composite's alpha ramp is inverted to match, so the edge falls off outward. The jump-flood ping-pong reuses one pass across iterations, so its input is now bound inside the render queue rather than at record time - otherwise the first draw of an iteration can observe the previous iteration's binding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DsM8FJLGY8SzZqumBdRKh5
Transform gizmos operated on the entity's local transform, so dragging a child moved it in its parent's space rather than the world the handles were drawn in. They now take the world matrix and convert the edit back through the parent. They also use the un-reversed projection: ImGuizmo does its own depth maths and does not expect a reversed-Z matrix. Scale snap joins translation and rotation snap, bound through the same editor-preferences path so it persists. The editor camera built its view matrix with a fixed world-up vector, which is degenerate when looking straight down or straight up - exactly the top/bottom orthographic views. It now uses the camera's own up direction, which stays perpendicular at every orientation. Grid visibility is restored from settings on startup instead of always defaulting on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DsM8FJLGY8SzZqumBdRKh5
Imports fart-01.wav into the FMOD Studio project and places it on LuxTestEvent's timeline, with the encoding setting FMOD generated for it. The event was created empty, which was enough to verify the engine side - bank enumeration, GUID resolution, the inspector picker and scene serialization all work without the event being audible. Making a sound is the one part of the chain that was still unproven. Also includes SampleProject.fspackage, FMOD's exported project archive. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DsM8FJLGY8SzZqumBdRKh5
Adds the PostProcess block the serializer now writes for every scene. No authored change - the scene was saved by the editor and picked up defaults that did not exist when it was first written. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DsM8FJLGY8SzZqumBdRKh5
The complete design for LuxEngine's audio, from the current state to a general-purpose shipping system. A planning document, not a description of what exists - Architecture-LuxEngine.md remains the authority on that. Records the decisions it is built on so the reasoning is inspectable rather than implicit: general-purpose rather than genre-specific, further investment in Vercidium, all four subsystems at equal depth, a full runtime C# API, the legacy raw-file path deleted once events work, a seam left for networking, comprehensive accessibility, and Windows plus eventual consoles. Sixteen parts covering principles, architecture and ownership, the core runtime with function signatures, components, ray-traced acoustics, ambience and reverb zones, surfaces and physics audio, interactive music, dialogue and subtitles, accessibility, the C# API, editor tooling, performance budgets, platforms and shipping, a fifteen-phase roadmap, and six open questions left explicitly undecided so they are not settled by accident. Includes the known-broken items rather than only the aspirations: runtime export ships silent, VA occlusion is blocked upstream, and the premake feature flags are dropped on every regeneration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DsM8FJLGY8SzZqumBdRKh5
Validate bank build inputs and report build failures through the editor.
Preserve event lifetime and playback state across source operations and teardown. Includes FMOD-only implementations and callback support used by phase 4.
Support indexed, weighted listeners and attenuation targets, with prefab reference remapping and a reusable entity picker. Update VA listener integration.
Expose bank loading, components, one-shots, owned instances, parameters and mixer controls. Marshal callbacks on the main thread and invalidate handles safely during reload. Regenerate projects for the new native source.
Keep overlapping phase changes together: bank revisions versus instance generations, runtime source controls, listener synchronization, component persistence, prefab overrides, and editor picking. Documents the resulting FMOD/VA architecture. This integration and the backend build commit complete the preceding phase commits; intermediate commits were not built independently.
Require both SDKs, deploy shared libraries for Editor and Runtime, and read raw-file metadata through FMOD. Raw AudioSource compatibility remains through FMOD Core; this does not complete the planned runtime export or full legacy API removal. Regenerate projects. Combined tree previously verified with Linux Release Core/Editor/Runtime builds and real FMOD/VA headless tests.
Provide a camera/listener, scripted event emitter, mesh collider geometry, bank paths and step-by-step FMOD setup. Includes the authored sample event replacement and asset registry updates. Game assembly build, scene references, managed lifecycle and native Coral loading verified; no visual or audible editor verification.
Persist the current Audio Debugger layout separately from engine and sample changes.
Add a bounded bank manifest to runtime format 17 while preserving reads of older formats. Export validates bank output, copies banks and required FMOD/VA libraries, and preserves asset-relative bank paths. Runtime loads the exact bank manifest before scenes and scripts and rejects failed loads. Update the export UI, setup guide, architecture reference, and Windows Studio DLL deployment. Regenerate Premake projects for AudioBankManifest.cpp. Verified Linux Release Core, Editor, and Lux-Runtime builds by artifact timestamps. Headless tests cover relocated FMOD playback, bank failures, library copying, manifest validation, and actual project serializer v17 roundtrip/v16 compatibility. Windows execution and interactive playback remain untested.
ReverbSend was present on both events but drove nothing: no event had a send, so there was no level to automate and the value the engine writes each frame was discarded. Adds a send to the existing Reverb return on Fart and MusicAmbiance, with its level automated from ReverbSend: -80 dB (silent) at 0, 0 dB at 1. The send sits after the fader, which is where Studio appends it. That makes it post-fader, so the reverb follows the fader and the spatializer's distance attenuation, and it also sits after the occlusion EQ - a source behind a wall sends a muffled signal to reverb while VA raises how much of it returns. Drag it left of the EQ in the deck if the reverb should stay unmuffled. Banks rebuilt. Both events now carry the full chain the engine expects: local Occlusion and ReverbSend parameters, occlusion automating the EQ cutoff, and ReverbSend automating the send level. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fart's band A filter type was 7 while MusicAmbiance's was 1. Automating the cutoff of a band that is not a lowpass moves the filter corner without muffling, so occlusion would have been inaudible on that event even with the automation in place. Set to 1, matching MusicAmbiance, which was authored for occlusion deliberately. The scripting API does not expose the filter-type enum labels, so the known-good event is the reference rather than the numeric meaning. Banks rebuilt; the occlusion automation is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The event was authored, wired for occlusion and reverb, and built into the banks, but nothing in the scene played it - the demo had a single audio source, the Fart emitter. Adds an emitter at [-2.5, 1.6, -5] with Play On Awake on. That sits inside the building: between the back wall at z=-6.85 and the front wall at z=0, above the floor and below the roof at y=2.85, and clear of the Fart emitter at [0, 1, -3] so the two are distinguishable while walking around. It reuses the sphere mesh the Fart emitter already references, so no new asset reference is introduced. Play On Awake is on here, unlike the Fart emitter, whose playback is driven by FmodAudioDemo. Ambience should simply be running when the scene starts. Verified: scene loads in the editor with no missing-event error, which is what the engine reports when a source GUID does not resolve. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All six colliders were Concrete, so VA's per-surface material system was running but had nothing to show: every surface absorbed and transmitted identically. Back Wall -> Brick dense, strong muffling Front Wall -> Plaster softer, more transmission Floor -> Wood Roof -> WoodThin Cube -> Metal bright and reflective Cube -> Carpet the absorptive opposite The two walls differ deliberately. They are what stands between the listener and the emitters when you walk around outside, so which side of the building you are on now changes how the sound is occluded - which is the point of the demo. Tag names checked against AcousticMaterial.h; an unrecognised name deserializes to Default silently, so a typo would have been invisible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The building was a floor, a back wall, a roof and two side walls named
"Cube", with a front wall covering 6 of the 14 unit span - the facade was
more opening than wall. Everything was untextured default grey.
Geometry:
- The two "Cube" entities are 0.3 x 2.8 x 7 side walls; named accordingly.
- The front wall becomes a full facade: Front Wall Left and Right with a
1.2 m doorway between them and a lintel above, closed to 2.1 m.
- A rug inside, which gives the Carpet acoustic tag an actual surface.
This also sharpens the acoustics demo. With 8 units of open front, sound
escaped freely and occlusion barely registered; a single doorway means
strong occlusion outside with the opening as the leak path, which is what
the VA integration is there to show.
Materials: six CC0 sets from ambientCG at 1K JPG (15 MB installed), chosen
to match the acoustic tags so what you see is what you hear - brick walls,
plaster facade, wood floor, roofing tiles, carpet rug. The roof's acoustic
tag moves from WoodThin to Ceramic to match its new tiles. UV tiling is set
per material against its surface size so nothing is stretched, and normals
use the GL convention: the engine decodes with no Y flip.
Verified: 23 textures load, no asset or material errors.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A playing one-shot (e.g. a non-looping music event) was released and its play intent cleared when the listener left its max distance, so it never came back. Started one-shots now keep their instance while culled and end on their own; one-shots requested while culled are still discarded. Culled looping events now advance their saved timeline by the scene timestep (x pitch), wrapped over the event's timeline length, so re-entry lands where playback would have been instead of where it was culled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The engine calls none of the functions 1.9.0 removed or renamed (RefreshRayCount, HalfSpherePrimitive, WorldIsIndoors), links the production build so the new out-of-process debug window does not apply, and the Linux library still needs glibc 2.43. 1.9.0 does not fix occlusion: the docs/vercidium-repro program prints identical results on 1.8.0 and 1.9.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Audio phase 12 enabled ImGui in the standalone runtime for the accessibility menu and caption overlay. ImGuiRenderer clears its target to magenta before drawing, and in the runtime that target is the swapchain RuntimeLayer has just blitted the finished game frame into, so every exported game showed a solid magenta window. A RenderDoc capture confirmed the frame was correct up to the ImGuiRenderer pass. ImGuiLayer::SetClearMainViewport lets the application keep what it has already drawn; the editor keeps the clear, the runtime turns it off. Dist runtimes no longer enable ImGui at all, so shipping builds have no built-in accessibility menu or caption overlay. Application::m_ImGuiLayer is now initialised to null, since it stays unset when ImGui is off. The sample project's runtime export is now windowed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Shutdown: MaterialAsset's file-scope s_SRGBAlbedoTextureCache held Ref<Texture2D> past Renderer::Shutdown and the device teardown; the CRT destroyed it at exit and vkDestroyImage ran against a dead VkDevice (AV in nvoglv64 / vulkan-1, and heap corruption in the editor). Captured stack: atexit -> s_SRGBAlbedoTextureCache -> Texture2D -> Image2D::Release -> nvrhi Texture dtor -> nvoglv64. Release it in Renderer::Shutdown alongside the mip-gen pipeline cache. Minimize: a 0x0 surface made OnResize destroy the swapchain without creating a new one, but the render thread still had one frame queued that acquired on it -> vkAcquireNextImageKHR(VK_NULL_HANDLE) in the driver. Keep the old swapchain while the surface is 0x0 and retry until restored. Present now skips itself when the acquire failed, replacing the loop-local frameBeginSuccess that the render-thread lambdas read by reference after it went out of scope. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add a runtime toggle under Application Settings > Viewport > Gamepad with a device picker, so a specific controller (e.g. a DualSense among several) can drive the editor UI. The stock GLFW backend only reads GLFW_JOYSTICK_1, so ImGuiLayer now keeps the backend off the pad and feeds the chosen joystick itself; navigation is suspended during Play so the game receives the pad. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The existing controller API exposed raw GLFW joystick indices, which differ between devices (a DualSense and an Xbox pad disagree on axis numbering), and had no deadzone. Add a standard-layout gamepad API on top of glfwGetGamepadState: GamepadButton/GamepadAxis enums, down/pressed/released, analog axes with a scaled radial deadzone and 0..1 triggers, and "first connected gamepad" as the default device. Exposed to C# via Lux.Input. FlyCamera now supports a controller alongside keyboard and mouse: left stick moves, right stick looks, triggers/bumpers rise and fall, L3 latches sprint. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GLFW is input-only, so send the DualSense's USB output report ourselves through a minimal raw-HID layer built on system APIs (Windows SetupAPI/HID, Linux hidraw) - no new dependency. Scripts get Resistance, Weapon and Vibration trigger effects via Input.SetGamepadTriggerEffect; effects are cleared when Play stops and on exit so a trigger never stays stiff. USB only by design: any Bluetooth output report switches the pad into its enhanced input mode, which DirectInput (and so GLFW) cannot read until it reconnects. Bluetooth pads are skipped with a one-time warning. Controllers are now classified by GUID into a GamepadFamily. FlyCamera adds light trigger resistance on the up/down triggers, heavier while sprinting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add Input.RumbleGamepad / StopGamepadRumble with per-slot durations, routed by GamepadFamily using OS APIs only: - Windows: DualSense and DualShock 4 via their USB HID output reports (HID::DeviceGroup, shared with the trigger code); Xbox via XInput. - Linux: evdev FF_RUMBLE for every pad whose kernel driver supports it, matched to GLFW joysticks by device name. Rumble stops when Play stops and on exit. FlyCamera gives a short thump when gamepad sprint engages. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scene::ReleaseRuntimeAudio reaches RaytracedAudioScene::DestroyEmitter from outside the WaitForResults/OnUpdate window: script-driven entity destruction runs in m_PostUpdateQueue after vaWorldUpdate has kicked the next batch, and the event-cleared and AddComponent<AudioSourceComponent> paths run before the frame's join. vaEmitterRemoveTarget / vaWorldRemoveEmitter / vaEmitterDestroy then mutated the listener's target list and freed an emitter while Vercidium's workers could still be tracing it - the use-after-free Stop() already guards against. vaWorldWait first; inside the window the batch is already joined, so it returns immediately. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fade-out is still audible, and the dialogue, music stinger and physics voice directors retire instances once !IsPlaying(), so they must keep a fading event alive. SetProgrammerSound refusing a fading event is intended: it needs one that has never started. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every exported game links both SDKs, so their EULAs apply to it: FMOD's free commercial tier limits and mandatory in-game credit, Vercidium Audio's per-game Commercial Licence and broad definition of commercial use, and the ban on shipping either SDK inside an engine or toolset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Input.GetGamepadType (Xbox / PlayStation / Nintendo / Unknown) from the vendor ID, else the mapping or device name, for button prompts. - C# Input.GamepadConnected / GamepadDisconnected events, raised during Play by diffing the connected set each runtime frame; pads present at Play start raise nothing and handlers are cleared when Play stops. - Input::LoadGamepadMappings layers an SDL_GameControllerDB file over GLFW's built-in one: Resources/gamecontrollerdb.txt at startup and <project>/gamecontrollerdb.txt on project activation. - DualSense lightbar and player LEDs, DualShock 4 lightbar, via the same USB HID reports; written only when changed and restored to the default blue when Play stops and on exit. - Fix Input::Update never polling the last joystick slot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The accessibility mixer and the performance monitor both lock bus:/.
Studio bus locks are not counted, so whichever locked second failed
with FMOD_ERR_ALREADY_LOCKED ("Performance monitor lock budget bus" in
every runtime log) and that bus lost its voice counts and meters; a
direct unlock from one holder would also have released the other's
group.
AudioEngine::LockBusChannelGroup/UnlockBusChannelGroup count holders
per bus and only call FMOD on the first lock and last unlock. Both
systems now go through them; UnloadAllBanks forgets the counts after
both holders have reset.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FixedRenderWidth/Height were saved in the .luxproj but never written to Project.luxruntime, so exported games always used the 1920x1080 default. Runtime format 24 appends both at the end of the scene-renderer block; older exports keep the defaults. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Exports wrote over the previous <Game>-<Platform> folder, so files a past export or test run left behind shipped again: shaders deleted months ago, a Mono-era folder, archives and logs. The export now deletes that folder first, but only when it holds Assets/Project.luxruntime from a previous export. Any other non-empty folder at that path stops the export with an error instead of being overwritten, and a failed delete (a running game holding files) is reported rather than exporting on top of a half-cleared folder. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AUDIO_SYSTEM_PLAN's "where we are" ledger still described the branch before phase 3 (no C# API, silent exports, minimal listener). It now records what phases 3-15 built, that VA 1.9.0 still does not deliver occlusion, that the physics-ray fallback is unbuilt, that Dist has no built-in accessibility overlay, and which editor extras are missing; the roadmap marks phases 3-15 done and links each phase guide. AUDIO_ACCESSIBILITY now says Dist exports have no F10 menu or caption overlay, so a Dist game must draw captions from the C# events. Also moves the Architecture doc's "2.10 Audio" heading back below the gamepad-lights paragraph, which had landed inside the audio section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
No content change: the registry refreshes file timestamps and asset status, and the scene is re-serialized with entities reordered and 3.8 written as 3.79999995. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- run.py: each config block in the generated Core/Makefile now opens with nested ifeq/endif for CC/CXX/AR, so cutting at the first endif lost DEFINES and INCLUDES. Cut at the next config instead. - AudioTestHost.h: the accessibility mixer and performance monitor lock buses through AudioEngine's reference count since 34c4d4a. Add the same Lock/UnlockBusChannelGroup pair to the test host so the tests link, and clear the counts when the Studio system is released. All native, managed and SDK-layout audio tests pass on Linux. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
@coderabbitai full review |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the miniaudio backend with FMOD Studio + Vercidium Audio (both required) and builds the full audio system from
docs/AUDIO_SYSTEM_PLAN.mdin 15 phases: Studio events and banks, listeners, C# API, runtime export of banks, acoustic materials, zones/snapshots, surfaces and physics audio, interactive music, localized dialogue and subtitles, accessibility, dynamic acoustic geometry and portals, voice budgets and validation, and desktop platform profiles. Vercidium Audio is on 1.9.0 (5a344a78).Also on this branch (not audio)
This is one PR by choice. Reviewers should know it also carries:
docs/MATERIAL_EDITOR_PLAN.md).4f67e691) with SPDX headers, and a README rewrite for Windows and Linux.GamepadConnected/GamepadDisconnectedevents, SDL_GameControllerDB mappings (engine and per-projectgamecontrollerdb.txt), DualSense/DualShock 4 lightbar and player LEDs, and a selectable controller for ImGui gamepad navigation in the editor./plan-le,/profile,/shader-debug, the ImGui correctness rule, the self-contained product principle, anddocs/BEAM_EDITOR_PLAN.md.Why
The old audio path was raw-file playback with no authoring pipeline, no occlusion or reverb, no C# API, and silent exported games. This gives an authored FMOD Studio pipeline with ray-traced acoustics, a complete scripting surface, and working runtime export.
Fixes found during verification
update()failed every frame with Live Update on.AudioPerformanceenabled metering on a DSP Studio owns; every laterStudio::System::updatereturnedFMOD_ERR_BADCOMMAND. The monitor now meters through its own pass-through fader DSP at index 1 and releases it before bank unload.TreePop) in the Audio Debugger and Project Settings.859a72bb).ImGuiRenderercleared the swapchain the runtime had just blitted the game frame into.ImGuiLayer::SetClearMainViewport(false)keeps the frame; the editor keeps the clear.38d4a841). A file-scopeRef<Texture2D>cache inMaterialAssetoutlived the device and was destroyed atatexit; it is now released inRenderer::Shutdown.38d4a841). A 0x0 surface destroyed the swapchain while the render thread still had a frame queued, sovkAcquireNextImageKHRran onVK_NULL_HANDLE. The old swapchain is kept until the window is restored.b0adeb71). ScriptEntity.Destroy()andAddComponent<AudioSourceComponent>reachedDestroyEmitteroutside theWaitForResults/vaWorldUpdatewindow. It now joins the in-flight batch first.34c4d4a). The accessibility mixer and the performance monitor both lockbus:/; Studio locks are not counted, so the second holder failed withFMOD_ERR_ALREADY_LOCKEDand lost its meters. Locks are now reference-counted inAudioEngine.d8775d8) inInput::Update.99192c3a). Started one-shots keep their instance while culled, and culled loops advance their timeline so re-entry lands in the right place.Known limitations
docs/vercidium-repro/prints identical results on 1.8.0 and 1.9.0.Verification
fmod.dll,fmodstudio.dll,vaudionative.dllpresent next to Editor and Lux-Runtime.LuxSampleProject, FMOD Studio initializes with Live Update, zero[error]lines, clean shutdown (exit 0).da9556e): headless FMOD/VA regression tests all pass:tests/audio/run.py(13 native suites: physics audio, music, dialogue, accessibility, geometry, voice budgets, platform profiles, serialization, file streams, asset pack),run_managed.py(3 C# suites), andrun_sdk_layout.py(Windows + Linux).da9556efixes the runner itself, which had drifted from the current premake Makefile layout and the bus-lock refcount (34c4d4a).Review
/send-prrule list applied as automated checks over all added lines, plus manual review of UI scope paths and component completeness. The diff is too large for a full line-by-line read. A targeted review then covered the riskiest audio code: the FMOD callback thread, the Jolt contact listener, the C# bindings, VA world threading and teardown,AudioEnginelifecycle and bank-manifest validation, and legacy miniaudio-era scene loading. It found the VA emitter race above and nothing else must-fix.Automated review: Copilot reviewed up to
a51e7012; its three findings were fixed in5c7dd6cand resolved. A fresh Copilot pass onda9556ewas blocked by quota, and CodeRabbit and ultrareview both refuse a diff this size (668 files), so commits aftera51e7012had no bot review.Consider-tier, left as is:
static std::stringUI buffers in the Dialogue Lines editor (ProjectSettingsWindow.cpp).Regeneration
Required: many files added/removed. Run
scripts\Win-GenProjects.bat --last. FMOD Engine SDK must be inCore/vendor/FMOD/(orLUX_FMOD_SDK) and Vercidium Audio inCore/vendor/VA_RAY/(orLUX_VA_SDK); generation fails with the exact missing file otherwise.Doc updates
.claude/docs/Architecture-LuxEngine.md(§ 2.10 Audio, metering ownership, ImGui clear/Dist rule),Threading.md(audio budgets),Conventions.md(ImGui correctness),Building.md(audio SDKs),CLAUDE.md/AGENTS.md(product principle, skills), editor audio docs, and phase docs underdocs/AUDIO_*.md.🤖 Generated with Claude Code