Issues pipeline chunk_size (1500) exceeds the TEI embedding limit (1000) — tail of long chunks is unsearchable

[kmr-rohit]

Problem

The GitHub issues ingestion pipeline chunks issue content up to 1500 characters, but truncates every chunk to 1000 characters before sending it to the TEI embeddings service:

• docs-agent-mcp/pipelines/issues-pipeline.py#L481 — chunk_size: int = 1500
• docs-agent-mcp/pipelines/issues-pipeline.py#L301 — max_tei_chars = 1000, applied as r["content_text"][:max_tei_chars]

So for any chunk longer than 1000 chars, the last ~500 characters are stored in Milvus and shown to users, but were never embedded — that text is invisible to semantic search.

[google-oss-prow]

@kmr-rohit: The label(s) kind/bug, area/pipelines cannot be applied, because the repository doesn't have them.

Details

In response to this:

Problem

The GitHub issues ingestion pipeline chunks issue content up to 1500 characters, but truncates every chunk to 1000 characters before sending it to the TEI embeddings service:

• docs-agent-mcp/pipelines/issues-pipeline.py#L481 — chunk_size: int = 1500
• docs-agent-mcp/pipelines/issues-pipeline.py#L301 — max_tei_chars = 1000, applied as r["content_text"][:max_tei_chars]

So for any chunk longer than 1000 chars, the last ~500 characters are stored in Milvus and shown to users, but were never embedded — that text is invisible to semantic search.

Why it matters

This fails silently: no error, no log, just degraded recall. It hurts precisely the long, detailed issue threads that are most valuable for troubleshooting, and issues_rag is our largest collection (~4.3k chunks).

It is also an inconsistency — the docs and code pipelines already default to chunk_size = 1000. The issues pipeline is the odd one out.

Proposed fix

Clamp chunk splitting to the TEI input limit so a chunk's embedding always covers its full stored text:

• Clamp to the existing MAX_TEI_INPUT_CHARS constant (pipelines/utils.py) rather than a new magic number, in both issues_utils.py and the mirrored KFP component in issues-pipeline.py
• Lower the default chunk_size to 1000 to match docs/code
• Add a regression test

Follow-up (important)

This fixes the pipeline, not the data already in Milvus. Existing issues_rag chunks were built at 1500. A re-index of issues_rag is required after this merges for the fix to reach production data.

Acceptance criteria

• No chunk can exceed the TEI embedding limit, even if a larger chunk_size is passed
• Regression test covers the clamp
• issues_rag re-indexed after merge

/kind bug
/area pipelines

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chasecadet]

/lgtm