When tool call includes content, it is not committed or used in the follow up response

[martin-purplefish]

OpenAI can return content along side a tool call:

    {
      "role": "assistant",
      "content": [
        {
          "text": "Oh, just a moment while I check that for you... \n\nLet me find out the current weather in NYC by querying our weather service.",
          "type": "text"
        }
      ],
      "tool_calls": [
        {
          "id": "call_Gwqkvbb3r0oRJnj3zzrIEL1R",
          "type": "function",
          "function": {
            "name": "get_weather",
            "arguments": "{\"location\":\"New York, NY\",\"unit\":\"f\"}"
          }
        }
      ]
    }

When this happens, there are a few problems:

1. The agent's audio plays, BUT the audio is not committed to the log.
2. There's no agent_speech_committed event.
3. (I think) the audio gets played AFTER the function call, rather than before it.

[martin-purplefish]

To expand a bit, what I'm seeing is this:

1. OpenAI returns a tool call with content:

       {
      "role": "user",
      "content": [
        {
          "text": "What'\''s the weather in NYC right now?",
          "type": "text"
        }
      ]
    },
    {
      "role": "assistant",
      "content": [
        {
          "text": "Oh, just a moment while I check that for you... \n\nLet me find out the current weather in NYC by querying our weather service.",
          "type": "text"
        }
      ],
      "tool_calls": [
        {
          "id": "call_Gwqkvbb3r0oRJnj3zzrIEL1R",
          "type": "function",
          "function": {
            "name": "get_weather",
            "arguments": "{\"location\":\"New York, NY\",\"unit\":\"f\"}"
          }
        }
      ]
    }

2. The agent plays the audio "Oh, just a moment while I check that for you... Let me find out the current weather in NYC by querying our weather service." and calls tool. However this message is not added to the chat context.

3. The agent then generates a response to the tool data, BUT without the text from the original tool call in the context. So we get a response like "Aha, let me take a look at that for you... The current weather in New York City is 72°F." that also has its audio play.

4. Only the message in 3. has agent_message_committed called.

5. The final chat context has the message from 3. but not from 2.

[theomonnom]

I'm not sure about the chat_ctx part, the collected_text should correctly be next to tool calls:

agents/livekit-agents/livekit/agents/pipeline/pipeline_agent.py

Line 795 in 74f00c3

ChatMessage.create_tool_calls(tool_calls_info, text=collected_text)