Skip to content

Add content part support in prep for multi-modal embedding models - #27

Merged
chrisboulton merged 1 commit into
mainfrom
mm-embed
Jul 24, 2026
Merged

Add content part support in prep for multi-modal embedding models#27
chrisboulton merged 1 commit into
mainfrom
mm-embed

Conversation

@chrisboulton

@chrisboulton chrisboulton commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

The shape of this matches media for SeqAppend - from the modelsocket perspective it's something like:

{
  "request": "seq_command",
  "cid": "embed-1",
  "seq_id": "seq-123",
  "data": {
    "command": "embed",
    "inputs": [
      {
        "content": [
          {
            "text": "Find products similar to this bicycle"
          },
          {
            "media": {
              "uri": "https://example.com/bicycle.png",
              "mime_type": "image/png",
              "detail": "high"
            }
          }
        ]
      }
    ],
    "input_type": "image",
    "dimensions": 1024,
    "normalize": true,
    "truncate": false
  }
}

Summary by CodeRabbit

  • New Features

    • Added support for embedding content that includes text and media.
    • Media inputs can now reference a URI or embedded content, with optional metadata such as MIME type, detail, and hash.
    • Added reliable serialization and deserialization for embedding requests.
  • Bug Fixes

    • Improved consistency in handling media inputs across supported operations.
  • Chores

    • Updated the release version to 0.5.1.
    • Added macOS system files to the ignored files list.

Release 0.5.0 with vision support
@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 7b49f2f0-7513-4f7f-9bba-a6b565ea59cf

📥 Commits

Reviewing files that changed from the base of the PR and between cf831ce and b6f5b08.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (3)
  • .gitignore
  • Cargo.toml
  • src/protocol.rs

📝 Walkthrough

Walkthrough

The protocol now shares media input modeling between append and embed requests, supports ordered text/media embedding content, and adds round-trip tests for URI and blob media. The crate version and repository ignore rules are also updated.

Changes

Embedding protocol support

Layer / File(s) Summary
Media and embedding content contract
src/protocol.rs
A shared MediaInput type backs append and embed media aliases, while EmbeddingContentPart supports ordered text and media content.
Embedding serialization validation
src/protocol.rs
Round-trip tests cover simple embedding inputs and content containing URI- or blob-backed media.
Package and repository metadata
Cargo.toml, .gitignore
The crate version changes to 0.5.1, and .DS_Store is ignored alongside .venv.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested reviewers: zackangelo

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding content part support for multimodal embedding models.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch mm-embed

Comment @coderabbitai help to get the list of available commands.

@chrisboulton
chrisboulton merged commit 9451a59 into main Jul 24, 2026
16 checks passed
@chrisboulton
chrisboulton deleted the mm-embed branch July 24, 2026 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants