support for real-time streaming

[that-9uy]

support for real-time streaming

[timonharz]

have a look at my fork at https://github.com/timonharz/Orpheus-FastAPI
its the same setup and supports a streaming endpoint

[that-9uy]

@timonharz what's your latency like to get the first audio chunk back? It's takes 4sec on my rtx 3090. is there a ways to get it to 200 ms?

[timonharz]

@that-9uy Are you using my fork? I'm seeing around 200ms latency using the streaming endpoint. You might want to check GPU utilization with nvtop or nvidia-smi—it should be hitting at least 75% during generation. If it's significantly lower, something's likely misconfigured. I'm still not satisfied with the generation speed; it feels unusually slow for a 3B parameter model. What t/s are you at?

[that-9uy]

@timonharz the utilization is around 85% and around 160 t/s, I'm working on a project and I want to have end to end latency less than 1000ms (i.e SST -> LLM -> TTS)

[timonharz]

@that-9uy 85% utilization is really good. have you tweaked any values to achieve 85%? I have a similiar use case as you, where I need ultra low latency at about 100ms. I am only at 70 t/s right now using a Nvidia L4 gpu. Have you modified anything to achieve 85%?

[timonharz]

@Lebofly thanks for telling me! my fork does not support the full functionality of the original repo. the project I am working on needs extremely fast streaming inference and thats what I am focusing on

[Lex-au]

@Lebofly thanks for telling me! my fork does not support the full functionality of the original repo. the project I am working on needs extremely fast streaming inference and thats what I am focusing on

Streaming is a bandaid—it gives the illusion of lower latency by reducing time-to-first-token, but it doesn’t fix the root cause. The real bottleneck is the audio codec Orpheus has locked us into. SNAC wasn’t designed for low-latency applications; in fact, it explicitly trades off latency for compression efficiency and token quality.

Part of this comes from its non-causal encoder—SNAC can look both forward and backward in time over the input, which is great for fidelity but inherently introduces delay. Worse still, its hierarchical token structure is strictly sequential: each finer-scale quantiser depends on coarser levels before decoding can even begin, compounding latency through token delay.

By contrast, something like HiFi-GAN is far more robust in low-latency scenarios, thanks to its causal or streaming-capable architecture. SNAC just isn’t built for that.

So no matter how many tweaks we layer on, there’s only so much headroom left—beyond that, it’s either a question of throwing enterprise-grade hardware with more workers and vector ops at the problem, or changing the codec entirely. And from what I understand, the latter isn’t on the table.

[timonharz]

@Lex-au i am not really deep in the matter. So we can't change the encoder to a HiFi-GAN, because it expects a sequence of mel-spectrogram frames (shape ≈ [ T, 80 ]) instead of the 3-level 7-token SNAC codebook we’re receiving now - is that right?

[Lex-au]

@Lex-au i am not really deep in the matter. So we can't change the encoder to a HiFi-GAN, because it expects a sequence of mel-spectrogram frames (shape ≈ [ T, 80 ]) instead of the 3-level 7-token SNAC codebook we’re receiving now - is that right?

You're pretty much on the money—HiFi‑GAN expects mel‑spectrogram frames (≈ [T, 80]) from an acoustic model, whereas Orpheus streams SNAC codebook IDs across four levels. Since there’s no learned bridge between those two representations, swapping in HiFi‑GAN would require either retraining Orpheus to emit mels, or building a codes-to-mels adapter—both of which are non-trivial. It’s essentially two different languages with no translator.

That’s why, for now, we’re focused on squeezing out latency wins from SNAC rather than changing the codec outright.

Also worth noting: when Canopy advertised "low-latency streaming (~200ms to ~25–50 ms)", that referred to streaming text into the model for faster first-token output—not actual audio synthesis. So while it gives the appearance of low latency, it’s still an illusion. The real-time factor between input and final audio output hasn’t meaningfully changed.

[NoSavedDATA]

@timonharz

I get an error on your docker-compose. Besides, CUDA_VISIBLE_DEVICES was duplicated and was also throwing an error.

binds, affinity = merge_volume_bindings(

File "/usr/lib/python3/dist-packages/compose/service.py", line 1548, in merge_volume_bindings
old_volumes, old_mounts = get_container_data_volumes(
File "/usr/lib/python3/dist-packages/compose/service.py", line 1579, in get_container_data_volumes
container.image_config['ContainerConfig'].get('Volumes') or {}
KeyError: 'ContainerConfig'

I corrected this by copying the original repo docker-compose-gpu.yml. However, now I get an error server-side when executing stream_test.py:

orpheus-fastapi | await func()
orpheus-fastapi | File "/app/venv/lib/python3.10/site-packages/starlette/responses.py", line 264, in stream_response
orpheus-fastapi | chunk = chunk.encode(self.charset)
orpheus-fastapi | AttributeError: 'bytearray' object has no attribute 'encode'

client-side:

Audio stream error: Error querying device -1

[Lohith-R-Gowda]

Hey guys, I am trying to use the quantized orpheus model for a realtime usecase. But I have been facing issues on streaming the tokens part. If anyone was able to crack it, please guide me what approach you took.
Thank you