feat(libsy): prepare requests for routed candidates - #463
Conversation
16dfb7c to
fb27c6d
Compare
|
One compatibility point I want to call out clearly before this merges: this PR makes the Rust This change is permanent; #464 does not restore struct-literal construction. It is not a break from a tagged Switchyard release because Custom libsy hosts should also replace manual model rewriting with The intermediate state after this PR is still usable: the native server and existing Stage prompt configuration continue to work. #464 adds the native |
Signed-off-by: Alex Fournier <afournier@nvidia.com>
99b3b46 to
d821351
Compare
Signed-off-by: Alex Fournier <afournier@nvidia.com>
d821351 to
5861240
Compare
Summary
Moves target-prompt policy into libsy, where the routing decision and ordered fallback candidates are known.
This is 2 of 3 for SWITCH-1253. It builds on #455, which provides the provider-safe request operation.
Before
A direct libsy host received a
RoutingOutcomeand prepared fallbacks itself, commonly by copying the published request and replacing the model:That works while every candidate receives the same request. It cannot safely select a different prompt per candidate, and preparing the first candidate too early would make a fallback inherit the first target's prompt.
Routing-time calls had the same split: libsy could ask a host to call a model, but the host had no candidate-aware request operation.
After
The host asks libsy to prepare the candidate it is about to call:
For routing-time calls, the same contract is available on
CallModel:The built-in Rust client uses the corresponding Rust methods. For fallback, libsy starts from the request before a target prompt was applied, stamps the new model, and applies only the new target's prompt. Custom Rust, Python, Relay, and other hosts no longer need to reproduce this policy.
What changes
with_target_prompts(...)policy in libsy.RoutingOutcome::request_for(...)andCallModel::request_for(...), with Python bindings for both.switchyard-llm-clientand the direct Python libsy host example/test path.Ownership and async behavior
Arc; requests do not copy the map per call.request_for(...)is called.await.API compatibility
The Python changes and the
CallModelmethods are additive.RoutingOutcomeitself landed onmainafter the last tagged release. This PR adds private preparation state and marks the struct#[non_exhaustive], so code tracking unreleasedmainthat constructs it with a struct literal or destructures every field will need to use its constructors/public fields instead.RoutingOutcome::route_to(...),RoutingOutcome::answered(...), field reads, and existing tagged APIs remain available.Not in this PR
targets.*.system_promptconfigurationThose are isolated in #464.
Validation
Algorithm.run_streamhosting covers prompted first and fallback candidates.Suggested review order
crates/libsy/src/core/algorithm.rs—RoutingOutcome,Driver, and candidate request contractscrates/libsy/src/core/target_prompts.rs— immutable prompt lookup policycrates/libsy-llm-client/src/run.rs— built-in retry/fallback consumercrates/libsy/src/algorithms/stage.rs— legacy Stage prompt compatibilitycrates/switchyard-py/src/libsy_bindings.rsandtests/test_libsy_minimal_bindings.py— direct Python host APIStack
targets.*.system_prompt, compatibility, docs, and integration testsThis PR's unique change is one signed commit,
fb27c6d5(18 files, +518/-151). GitHub currently compares the draft withmain, so it also displays PR1 below that commit. After #455 merges, this branch will be rebased onto the updatedmainto leave only the libsy layer in the displayed diff.