-
Notifications
You must be signed in to change notification settings - Fork 6
feat: replace HyperDX with configurable OTLP tracing #480
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
JuaniRios
wants to merge
1
commit into
master
Choose a base branch
from
juan/solver-victoriatraces
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| # Base solver tracing | ||
|
|
||
| The solver records round results, order diagnostics, events, and exceptions as | ||
| OpenTelemetry spans. VictoriaTraces stores these spans. This change does not | ||
| forward Docker stdout to VictoriaLogs or create alert rules. | ||
|
|
||
| ## Rollout | ||
|
|
||
| 1. Apply the `rain.devops` tailnet policy granting `tag:base-node` access to | ||
| `tag:rain-infra` on TCP 10428. VictoriaTraces already runs on the Rain | ||
| observability node. Keep this port private to the tailnet. | ||
| 2. Build and publish a solver image containing this change through the existing | ||
| release process. | ||
| 3. On `base-node`, update the environment used to create `base-solver`: | ||
|
|
||
| ```dotenv | ||
| OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://rain-management-observability.taile5cf8a.ts.net:10428/insert/opentelemetry/v1/traces | ||
| TRACER_SERVICE_NAME=base-bot | ||
| ``` | ||
|
|
||
| Remove the previous telemetry API key from the runtime environment. Preserve | ||
| the signer credentials, `CONFIG=./config.yml`, and the config bind mount from | ||
| `/root/solver/config.yml` to `/rain-solver/config.yml`. | ||
| 4. Recreate the container with the new image and environment through its existing | ||
| deployment process. A restart alone does not update Docker environment values. | ||
| 5. Verify DNS resolution and TCP 10428 reachability from inside the container, | ||
| not just from the host. Check Docker logs for exporter errors. | ||
| 6. Open [Rain Grafana](https://rain-management-observability.taile5cf8a.ts.net), | ||
| choose Explore and the `victoriatraces` Jaeger datasource, then search for | ||
| service `base-bot` over the last 15 minutes. Confirm fresh round and order | ||
| spans arrive, including attributes, events, exceptions, and child spans. | ||
|
|
||
| After verification, remove the obsolete telemetry ingestion secrets from GitHub | ||
| and the deployment secret store. GitHub previews now print spans in the Actions | ||
| run logs, linked from the preview deployment. | ||
|
|
||
| ## Endpoint configuration | ||
|
|
||
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` is the full trace ingestion URL and takes | ||
| precedence over `OTEL_EXPORTER_OTLP_ENDPOINT`. The generic setting is a base URL; | ||
| the SDK appends `/v1/traces`. The exporter sends gzip-compressed OTLP HTTP JSON. | ||
| Standard OTLP header environment settings remain available for other receivers. | ||
| No API key is needed for the private VictoriaTraces endpoint. | ||
|
|
||
| If both endpoint settings are absent, spans print to the console. For a temporary | ||
| fallback, remove both settings and recreate the container. Inspect its Docker | ||
| logs until trace ingestion is restored. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| import { createServer } from "node:http"; | ||
| import { gunzipSync } from "node:zlib"; | ||
| import { context, trace, propagation, SpanStatusCode } from "@opentelemetry/api"; | ||
| import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; | ||
| import { PreAssembledSpan, RainSolverLogger } from "."; | ||
|
|
||
| describe("OTLP trace export", () => { | ||
| beforeEach(() => { | ||
| trace.disable(); | ||
| context.disable(); | ||
| propagation.disable(); | ||
| vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", ""); | ||
| vi.stubEnv("OTEL_EXPORTER_OTLP_TRACES_ENDPOINT", ""); | ||
| vi.stubEnv("OTEL_EXPORTER_OTLP_HEADERS", ""); | ||
| vi.stubEnv("OTEL_EXPORTER_OTLP_TRACES_HEADERS", ""); | ||
| vi.stubEnv("TRACER_SERVICE_NAME", "base-bot"); | ||
| }); | ||
|
|
||
| afterEach(() => { | ||
| vi.unstubAllEnvs(); | ||
| trace.disable(); | ||
| context.disable(); | ||
| propagation.disable(); | ||
| }); | ||
|
|
||
| it.each(["trace-specific", "generic"])( | ||
| "exports queued diagnostics on shutdown using the %s endpoint", | ||
| async (setting) => { | ||
| const requests: { url?: string; encoding?: string; body: any }[] = []; | ||
| const server = createServer(async (req, res) => { | ||
| const chunks: Buffer[] = []; | ||
| for await (const chunk of req) chunks.push(Buffer.from(chunk)); | ||
| requests.push({ | ||
| url: req.url, | ||
| encoding: req.headers["content-encoding"], | ||
| body: JSON.parse(gunzipSync(Buffer.concat(chunks)).toString()), | ||
| }); | ||
| res.writeHead(200, { "Content-Type": "application/json" }); | ||
| res.end("{}"); | ||
| }); | ||
| await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve)); | ||
| const address = server.address(); | ||
| if (!address || typeof address === "string") throw new Error("Missing server port"); | ||
| const base = `http://127.0.0.1:${address.port}/insert/opentelemetry`; | ||
| if (setting === "trace-specific") { | ||
| vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://127.0.0.1:1/unused"); | ||
| vi.stubEnv("OTEL_EXPORTER_OTLP_TRACES_ENDPOINT", `${base}/v1/traces`); | ||
| } else { | ||
| vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", base); | ||
| } | ||
| const logger = new RainSolverLogger(); | ||
| try { | ||
| const child = new PreAssembledSpan("order") | ||
| .setAttr("order.id", "test-order") | ||
| .addEvent("quote", { duration: 12 }) | ||
| .recordException("quote failed") | ||
| .setStatus({ code: SpanStatusCode.ERROR, message: "quote failed" }) | ||
| .end(); | ||
| logger.exportPreAssembledSpan(new PreAssembledSpan("round").addChild(child).end()); | ||
| await logger.shutdown(); | ||
| expect(requests).toHaveLength(1); | ||
| expect(requests[0].url).toBe("/insert/opentelemetry/v1/traces"); | ||
| expect(requests[0].encoding).toBe("gzip"); | ||
| const resource = requests[0].body.resourceSpans[0]; | ||
| expect(resource.resource.attributes).toContainEqual({ | ||
| key: "service.name", | ||
| value: { stringValue: "base-bot" }, | ||
| }); | ||
| const spans = resource.scopeSpans[0].spans; | ||
| const order = spans.find((span: any) => span.name === "order"); | ||
| const round = spans.find((span: any) => span.name === "round"); | ||
| expect(order.traceId).toBe(round.traceId); | ||
| expect(order.parentSpanId).toBe(round.spanId); | ||
| expect(order.attributes).toContainEqual({ | ||
| key: "order.id", | ||
| value: { stringValue: "test-order" }, | ||
| }); | ||
| expect(order.events.map((event: any) => event.name)).toEqual([ | ||
| "quote", | ||
| "exception", | ||
| ]); | ||
| expect(order.status.code).toBe(SpanStatusCode.ERROR); | ||
| } finally { | ||
| await logger.shutdown(); | ||
| await new Promise<void>((resolve, reject) => | ||
| server.close((error) => (error ? reject(error) : resolve())), | ||
| ); | ||
| } | ||
| }, | ||
| ); | ||
| }); |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔒 Security & Privacy | 🛡️ Analyzed with Security Review | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: rainlanguage/rain.solver
Length of output: 21588
🌐 Web query:
OpenTelemetry JavaScript exporter-trace-otlp-http 0.49.1 OTEL_EXPORTER_OTLP_HEADERS OTEL_EXPORTER_OTLP_TRACES_HEADERS HTTP endpoint source💡 Result:
In OpenTelemetry, header configuration for the OTLP exporter is managed via standardized environment variables [1][2][3]. The variables OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_TRACES_HEADERS are used to specify key-value pairs for HTTP or gRPC requests [1][2]. Key details regarding these variables include: 1. Precedence: Signal-specific environment variables (e.g., OTEL_EXPORTER_OTLP_TRACES_HEADERS) take precedence over the generic, global variable (OTEL_EXPORTER_OTLP_HEADERS) [4]. 2. Format: Both variables expect a list of key-value pairs formatted as a W3C Baggage-compliant string (e.g., key1=value1,key2=value2) [2][3][4]. Semi-colon delimited metadata is not supported [2][3]. 3. JavaScript Implementation Context: For the
@opentelemetry/exporter-trace-otlp-httppackage (version 0.49.1), custom headers can also be configured programmatically via the headers option in the collectorOptions object when instantiating the OTLPTraceExporter [5][6][7]. Note that there have been historical discussions and issues within the OpenTelemetry JavaScript SDK regarding the consistent adherence of all exporters to these standardized environment variables [8]. Users should ensure their specific SDK version and exporter implementation align with the latest OpenTelemetry configuration specifications [1][3].Citations:
Sensitive Data Exposure
Reachability: Internal
Exploitability: Moderate
CWE: CWE-319 — Cleartext Transmission of Sensitive Information
Do not send configured OTLP headers over an unencrypted endpoint.
The OTLP exporter reads standard header environment variables, which can contain authorization tokens or API keys. Reject non-HTTPS endpoints when either header setting is nonempty. Keep the documented credential-free VictoriaTraces HTTP endpoint usable.
🤖 Prompt for AI Agents