Skip to content

feat(expressive): expose expressive mode and add expressive agent example #2245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rosetta-livekit-bot wants to merge 1 commit into from
Closed

feat(expressive): expose expressive mode and add expressive agent example #2245

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/expressive-moods-speak.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents': minor
---

Expose expressive session options, normalize expression metadata into moods, and expand the expressive TTS provider support.
6 changes: 2 additions & 4 deletions agents/src/constants.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,8 @@ export const ATTRIBUTE_TRANSCRIPTION_SEGMENT_ID = 'lk.segment_id';
/**
* The expression (delivery/emotion) the agent used for a transcription segment, surfaced so
* the frontend can react to it, when expressive markup is stripped from the transcript. The
* value is a JSON object `{"value": ...}` carrying the segment's leading expression — the
* `<expression>` tag for Inworld or the `<emotion>` tag for Cartesia, e.g.
* `{"value": "speak happy"}`. A JSON object (rather than a bare string) so the shape can
* gain fields later without breaking parsers.
* value is a JSON object carrying the provider's leading expression and its normalized mood,
* e.g. `{"expression":"speak happy","mood":"happy"}`.
*/
export const ATTRIBUTE_TRANSCRIPTION_EXPRESSION = 'lk.expression';
export const ATTRIBUTE_PUBLISH_ON_BEHALF = 'lk.publish_on_behalf';
Expand Down
67 changes: 67 additions & 0 deletions agents/src/tts/_mood.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
// SPDX-FileCopyrightText: 2026 LiveKit, Inc.
//
// SPDX-License-Identifier: Apache-2.0
import { MOOD_KEYWORDS } from './_mood_data.js';

export type AgentMood =
| 'excited'
| 'happy'
| 'playful'
| 'curious'
| 'surprised'
| 'hopeful'
| 'empathetic'
| 'sad'
| 'angry'
| 'anxious'
| 'calm';

export const MOOD_PRIORITY: AgentMood[] = [
'angry',
'sad',
'anxious',
'surprised',
'playful',
'empathetic',
'excited',
'curious',
'hopeful',
'happy',
'calm',
];
export const DEFAULT_MOOD: AgentMood = 'calm';

function matchesWord(text: string, keyword: string): boolean {
let start = 0;
while (true) {
const at = text.indexOf(keyword, start);
if (at === -1) return false;
if (at === 0 || !/\p{L}/u.test(text[at - 1]!)) return true;
start = at + 1;
}
}

export function matchMood(label: string): AgentMood;
export function matchMood(label: string, fallback: AgentMood): AgentMood;
export function matchMood(label: string, fallback: null): AgentMood | null;
export function matchMood(
label: string,
fallback: AgentMood | null = DEFAULT_MOOD,
): AgentMood | null {
Comment on lines +6 to +50

@devin-ai-integration devin-ai-integration Bot Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 New public helpers in the mood module ship without documentation comments

The newly added exported type, constants and function (AgentMood, MOOD_PRIORITY, DEFAULT_MOOD, matchMood at agents/src/tts/_mood.ts:6-50) carry no TSDoc, which the repository's contribution rules require for every new interface/method addition.

Impact: Generated API docs omit the mood normalization surface, and reviewers/consumers have no stated contract for the fallback behaviour.

Rule reference

CONTRIBUTING.md: "If writing new methods/interfaces/enums/classes, document them. This project uses TypeDoc for automatic API documentation generation, and every new addition has to be properly documented."

The same omission applies to SessionConfig and parseSessionConfig in examples/src/expressive_agent/protocol.ts:54-60.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

const text = label.toLowerCase();
let best: AgentMood | null = null;
let bestScore = 0;
for (const mood of MOOD_PRIORITY) {
const score = Object.entries(MOOD_KEYWORDS[mood]).reduce(
(total, [keyword, weight]) => total + (matchesWord(text, keyword) ? weight : 0),
0,
);
if (score > bestScore) {
best = mood;
bestScore = score;
}
}
Comment on lines +54 to +63

@devin-ai-integration devin-ai-integration Bot Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Delivery labels that describe curiosity are reported as empathy instead

A label whose descriptive words tie in weight is resolved by a fixed priority list (matchMood at agents/src/tts/_mood.ts:54-63) that ranks empathy above curiosity, so "gently curious, welcoming" is published as empathetic and the PR's own expectation for that label does not hold.

Impact: Frontends reading the published mood get the wrong feeling for common mixed labels, and the new mood test in this PR fails.

Score computation and tie-break mechanism

Scoring is a sum over keyword prefix matches. For the lowercased label gently curious, welcoming:

  • empathetic matches BOTH gentle (weight 1, agents/src/tts/_mood_data.ts:189) and gently (weight 1, agents/src/tts/_mood_data.ts:190) because matchesWord only requires the match to start at a word start and does not require a word end — the single word "gently" is therefore counted twice → score 2.
  • curious matches curious (weight 2) → score 2.

MOOD_PRIORITY lists empathetic (index 5) before curious (index 7) and the loop updates only on score > bestScore, so the first mood with the maximum score wins → empathetic.

agents/src/tts/mood.test.ts:17 asserts matchMood('gently curious, welcoming') === 'curious', so this case is a failing test as well as a questionable classification. Either the overlapping gentle/gently stems must be de-duplicated (keep only the gentl stem) or the priority/tie-break must match the upstream Python ordering.

Prompt for agents 
matchMood in agents/src/tts/_mood.ts sums keyword weights per mood and breaks ties by the order of MOOD_PRIORITY (first mood with the strictly-highest score wins). Two problems interact: (1) MOOD_KEYWORDS.empathetic in agents/src/tts/_mood_data.ts contains both the stem 'gentle' and the longer form 'gently', and matchesWord only anchors at a word start (no word-end check), so a single occurrence of the word 'gently' contributes 2 points instead of 1; (2) MOOD_PRIORITY ranks 'empathetic' ahead of 'curious'. As a result matchMood('gently curious, welcoming') returns 'empathetic', while agents/src/tts/mood.test.ts:17 expects 'curious'. Verify against the upstream Python _mood.py/_mood_data.py whether the keyword table should contain only one stem ('gentl') and/or whether MOOD_PRIORITY ordering differs, and align the port so the shipped test passes and overlapping stems cannot double-count.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

return best ?? fallback;
}

export { MOOD_KEYWORDS } from './_mood_data.js';
Loading