Skip to content

feat(ffi): add OTLP log and metric bindings - #783

Draft
bbednarski9 wants to merge 1 commit into
mainfrom
bbednarski/otel-signals-ffi-go
Draft

feat(ffi): add OTLP log and metric bindings#783
bbednarski9 wants to merge 1 commit into
mainfrom
bbednarski/otel-signals-ffi-go

Conversation

@bbednarski9

@bbednarski9 bbednarski9 commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Overview

Expose the core OTLP log and metric contracts through the C ABI and experimental Go binding.

This PR is independently rebased on main after #780. It contains only the C FFI/Go layer and has no branch dependency on the Python, Node.js, or dynamic-plugin/docs follow-up PRs. Review and merge it on its own readiness.

  • I confirm this contribution is my own work, or I have the right to submit it under this project's license.
  • I searched existing issues and open pull requests, and this does not duplicate existing work.

Details

  • Add checked C representations for log severity and metric measurements.
  • Add extended mark emission, metric emission, and independent OTLP log and metric subscriber lifecycle functions.
  • Regenerate the public C header.
  • Add Go event options, EmitMetric, direct subscriber APIs, and version 4 observability plugin configuration.
  • Preserve nil versus explicit-empty histogram boundaries and retain parent handles safely across cgo.
  • Add FFI validation, lifecycle, serialization, and Go export tests.

Validation:

  • FFI unit suite: 91 passed.
  • FFI integration suite: 83 passed.
  • Full Go suite: passed.
  • cargo fmt --all -- --check and Go formatting checks passed.

Breaking changes: none; existing C entry points remain available.

Where should the reviewer start?

Start with crates/ffi/src/api/scope.rs and crates/ffi/src/api/observability.rs, then review the Go-facing API in go/nemo_relay/nemo_relay.go.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

@coderabbitai

coderabbitai Bot commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 2f755a06-a350-45e3-86a4-ec370e46cc0c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size:XL PR is extra large Feature a new feature lang:go PR changes/introduces Go code lang:rust PR changes/introduces Rust code labels Aug 13, 2026
@bbednarski9 bbednarski9 mentioned this pull request Aug 13, 2026
Draft
2 tasks
@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown

Fern docs preview: https://nvidia-preview-pull-request-783.docs.buildwithfern.com/nemo/relay/dev

This was referenced Aug 13, 2026
Merged
Draft
Draft
@bbednarski9
bbednarski9 force-pushed the bbednarski/otel-signals-node branch from 1308886 to 2c3e3da Compare August 13, 2026 19:17
@bbednarski9
bbednarski9 force-pushed the bbednarski/otel-signals-ffi-go branch from 3bde200 to 8d9ab26 Compare August 13, 2026 19:17
@bbednarski9
bbednarski9 changed the base branch from bbednarski/otel-signals-node to main August 13, 2026 20:01
@github-actions github-actions Bot added size:XXL PR is very large lang:js PR changes/introduces Javascript/Typescript code lang:python PR changes/introduces Python code and removed size:XL PR is extra large labels Aug 13, 2026
@bbednarski9
bbednarski9 force-pushed the bbednarski/otel-signals-ffi-go branch from 8d9ab26 to 20f45e4 Compare August 13, 2026 20:05
@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown

License Diff

Compared against origin/main.

Lockfile license changes

Lockfile License Changes

Rust

Added

  • None

Removed

  • None

Updated/Changed

  • None

Node

Added

  • None

Removed

  • None

Updated/Changed

  • None

Python

Added

  • None

Removed

  • None

Updated/Changed

  • None
Status output 
[license-diff] selected languages: rust, node, python
[license-diff] generating current inventory
[license-diff] current: generating Rust inventory
[license-diff] current: Rust inventory complete (448 packages)
[license-diff] current: generating Node inventory
[license-diff] current: Node inventory complete (367 packages)
[license-diff] current: generating Python inventory
[license-diff] current: Python inventory complete (105 packages)
[license-diff] current inventory complete
[license-diff] checking out base ref origin/main into a temporary worktree
[license-diff] base: generating Rust inventory
[license-diff] base: Rust inventory complete (448 packages)
[license-diff] base: generating Node inventory
[license-diff] base: Node inventory complete (367 packages)
[license-diff] base: generating Python inventory
[license-diff] base: Python inventory complete (105 packages)
[license-diff] base inventory complete
[license-diff] removing temporary base worktree
[license-diff] comparing inventories
[license-diff] rendering Markdown output
[license-diff] done

@bbednarski9
bbednarski9 force-pushed the bbednarski/otel-signals-ffi-go branch from 20f45e4 to a27fea8 Compare August 13, 2026 21:03
@bbednarski9
bbednarski9 force-pushed the bbednarski/otel-signals-ffi-go branch from a27fea8 to 08d0917 Compare August 13, 2026 21:22
@bbednarski9
bbednarski9 force-pushed the bbednarski/otel-signals-ffi-go branch from 08d0917 to bdcb8e3 Compare August 13, 2026 21:28
@bbednarski9
bbednarski9 force-pushed the bbednarski/otel-signals-ffi-go branch from bdcb8e3 to 6e6044f Compare August 13, 2026 21:37
@bbednarski9
bbednarski9 force-pushed the bbednarski/otel-signals-ffi-go branch from 6e6044f to 0523d86 Compare August 13, 2026 21:52
@bbednarski9
bbednarski9 force-pushed the bbednarski/otel-signals-ffi-go branch from 0523d86 to ec22b35 Compare August 13, 2026 22:18
@bbednarski9
bbednarski9 force-pushed the bbednarski/otel-signals-ffi-go branch from ec22b35 to c373db5 Compare August 13, 2026 22:56
@bbednarski9
bbednarski9 force-pushed the bbednarski/otel-signals-ffi-go branch from c373db5 to 119d459 Compare August 13, 2026 23:06
rapids-bot Bot pushed a commit that referenced this pull request Aug 14, 2026
@bbednarski9
feat(observability): add core OTLP log and metric pipelines (#780)
bdbbb99 
#### Overview

Add the Rust source-of-truth APIs and runtime support for independent OTLP log and metric pipelines.

This is stack PR 1 of 5.

##### Stack navigation

1. **[#780 — Core runtime, shared event model, config v4, and CLI](#780) — this PR**
2. [#781 — Python and PyO3 bindings](#781)
3. [#782 — Node.js and N-API bindings](#782)
4. [#783 — C FFI and Go bindings](#783)
5. [#779 — Dynamic native/gRPC plugins and consolidated docs](#779)

**Position:** 1 of 5 · **GitHub base:** `main` · **Logical predecessor:** `main` · **Layer-only diff:** [compare branches](main...bbednarski/otel-signals-core) · **Next:** [#781](#781)

All five PRs target `main`. Their branches are cumulative: this PR includes every preceding layer until those PRs merge and this branch is rebased onto the updated `main`. The GitHub **Files changed** tab therefore shows the cumulative diff. Use the layer-only comparison above to review only the code introduced by this layer.

Review and merge in order: [#780](#780) → [#781](#781) → [#782](#782) → [#783](#783) → [#779](#779). After each merge, rebase the next branch onto the updated `main`; its PR remains targeted at `main` and its cumulative diff contracts to the remaining layers.

- [x] I confirm this contribution is my own work, or I have the right to submit it under the project's license.
- [x] I searched existing issues and open pull requests, and this does not duplicate existing work.

#### Details

- Add typed log severity, metric measurements, metric envelopes, optional mark `data_schema` and `severity`, and the Rust `metric` API.
- Keep `MetricMeasurement` as the serde-facing wire DTO. Each metric envelope is parsed atomically into `ValidatedMetricMeasurement` values before it reaches the OTLP metrics exporter.
- Make validation local to the parsed domain types: `InstrumentName` enforces the OTel instrument-name grammar and case-insensitive canonical key; `MetricValue` carries only `U64`, `I64`, or finite `F64`; `HistogramBoundaries` checks finite, strictly increasing, bounded buckets; and `MetricAttributes` owns scalar or homogeneous primitive-array parsing.
- Build `InstrumentDescriptor` from the typed fields and enforce supported `kind × value` combinations once. Envelope consistency groups descriptors by canonical name and requires stable kind and unit; description and histogram boundaries are retained as non-identifying advisory fields.
- Classify reserved metric marks before signal export. An invalid envelope is rejected atomically with a runtime diagnostic; a valid envelope carries only typed measurements into the metric registry and recorder.
- Use the validated descriptor to construct cached OTLP instruments and convert typed attributes directly to OpenTelemetry values. The recorder contains no JSON validation branches; impossible `kind × value` pairs are guarded as internal invariants.
- Export sanitized non-metric marks as structured OTLP logs with severity filtering and scope correlation.
- Add observability config version 4, signal-specific endpoint derivation and validation, lifecycle handling, diagnostics, layering, generated schema, and CLI editor support.
- Continue accepting version 3 as trace-only and preserve existing trace behavior for non-metric marks.
- Include minimal internal Python, Node.js, and FFI test bridges so this base commit remains workspace-buildable; their public APIs are reviewed in later stack PRs.

### Metric data-model architecture

```mermaid
flowchart LR
  subgraph Wire["Untrusted wire / serde boundary"]
    JSON["Metric mark JSON"]
    Envelope["MetricEnvelope"]
    WireMeasurement["MetricMeasurement<br/>name · kind · value_type · JSON value<br/>unit · description · boundaries · JSON attributes"]
    JSON --> Envelope --> WireMeasurement
  end

  subgraph Parse["Single parsing and validation boundary"]
    Convert["ValidatedMetricMeasurement::try_from(&MetricMeasurement)"]
    Name["InstrumentName<br/>OTel grammar + canonical name"]
    Value["MetricValue<br/>U64 | I64 | F64(FiniteF64)"]
    Descriptor["InstrumentDescriptor<br/>name · kind · unit · description · boundaries"]
    Bounds["HistogramBoundaries<br/>finite · strictly increasing · ≤ limit"]
    Attributes["MetricAttributes<br/>BTreeMap&lt;String, AttributeValue&gt;"]
    AttrValue["AttributeValue<br/>scalar or homogeneous typed array"]

    WireMeasurement --> Convert
    Convert --> Name
    Convert --> Value
    Convert --> Bounds
    Bounds --> Descriptor
    Name --> Descriptor
    Convert --> Attributes
    Attributes --> AttrValue
  end

  subgraph EnvelopePolicy["Envelope-level policy"]
    Parsed["Vec&lt;ValidatedMetricMeasurement&gt;"]
    Consistency["Canonical-name descriptor consistency<br/>kind + unit + value type<br/>(description/boundaries advisory)"]
    Convert --> Parsed --> Consistency
  end

  subgraph Export["OTLP exporter: typed inputs only"]
    Classify["MetricMarkClassification::Valid"]
    Registry["Instrument registry / cached OTLP instrument"]
    Record["record_measurement<br/>matches typed MetricValue"]
    OTLP["OpenTelemetry metrics export"]

    Consistency --> Classify --> Registry --> Record --> OTLP
    Attributes --> Record
    Descriptor --> Registry
    Value --> Record
  end

  Invalid["MetricMarkClassification::Invalid<br/>parse/validation error"]
  Convert -. failure .-> Invalid
  Consistency -. failure .-> Invalid
```

Validation:

- `cargo fmt --all`
- `cargo clippy --workspace --all-targets -- -D warnings`
- Focused metric-model and OTLP metrics tests, plus the full Rust workspace, Python, Node.js, and Go/FFI validation matrix, passed during implementation.
- `uv run pre-commit run --all-files` passed except the repository's `python-worker-proto-check`, which requires the unavailable `just` executable.

Breaking changes: none for version 3 trace configuration or existing trace subscriber APIs.

#### Where should the reviewer start?

Start with `crates/types/src/api/event.rs`: it contains the wire DTO, parsed-domain types, and atomic envelope parser. Then review `crates/core/src/observability/otel_signal.rs` for classification and `crates/core/src/observability/otel_metrics.rs` for typed instrument registration and recording. Configuration versioning and endpoint derivation live in `crates/core/src/observability/plugin_component.rs`.

#### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

- Relates to: none


## Summary by CodeRabbit

* **New Features**
  * Added OpenTelemetry log and metric exporting with configurable endpoints, transports, batching, filtering, resource metadata, and delivery controls.
  * Added validated metric events with measurements, attributes, histograms, limits, and temporality options.
  * Added typed log severity and data schema support for emitted events.
  * Observability configuration now defaults to version 4 while retaining version 3 trace-only compatibility.

* **Bug Fixes**
  * Improved handling and diagnostics for invalid events, export failures, queue drops, and shutdown errors.
  * Preserved metadata and severity when emitting tool and LLM events.

Authors:
  - Bryan Bednarski (https://github.com/bbednarski9)

Approvers:
  - Eric Evans II (https://github.com/ericevans-nv)
  - Maryam Najafian (https://github.com/mnajafian-nv)
  - Will Killian (https://github.com/willkill07)

URL: #780
@bbednarski9
bbednarski9 force-pushed the bbednarski/otel-signals-ffi-go branch from 119d459 to 6de23c3 Compare August 15, 2026 05:12
@github-actions github-actions Bot added size:XL PR is extra large and removed size:XXL PR is very large labels Aug 15, 2026
@bbednarski9
feat(ffi): add OTLP log and metric bindings
d417357 
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
@bbednarski9
bbednarski9 force-pushed the bbednarski/otel-signals-ffi-go branch from 6de23c3 to d417357 Compare August 15, 2026 05:20
@github-actions github-actions Bot removed lang:js PR changes/introduces Javascript/Typescript code lang:python PR changes/introduces Python code labels Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Feature a new feature lang:go PR changes/introduces Go code lang:rust PR changes/introduces Rust code size:XL PR is extra large

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bbednarski9