Feature/vector log retention - #3
Merged
Merged
Conversation
Add automated log retention, rotation, and archival system for Vector logs to address SOC2/ISO27001 compliance requirements. ## Problem Vector logging (introduced in 0ec3a7f) captures all container logs but lacked: - Log rotation (files grow indefinitely) - Retention policy (no automated cleanup) - Archival capabilities (manual clearing loses history) - Long-term storage (no S3/GCS integration) This creates compliance gaps for SOC2/ISO27001 which require: - Defined retention policies - Secure archival of audit trails - Prevention of log data loss ## Solution ### 1. Daily Log Rotation Vector now writes to date-stamped files instead of single growing files: - platform-2026-01-05.json (not platform.json) - agents-2026-01-05.json (not agents.json) ### 2. Automated Archival Service Backend APScheduler runs nightly (default: 3 AM UTC) to: - Find log files older than retention period (default: 90 days) - Compress with gzip (achieves ~90% size reduction) - Verify integrity with SHA256 checksums - Upload to S3-compatible storage (optional) - Delete originals after successful archive ### 3. Admin API Endpoints - GET /api/logs/stats - Log file statistics and sizes - GET /api/logs/retention - Current retention configuration - PUT /api/logs/retention - Update retention (runtime) - POST /api/logs/archive - Manually trigger archival - GET /api/logs/health - Service health check ### 4. S3-Compatible Storage Optional integration with: - AWS S3 - MinIO (self-hosted) - Cloudflare R2 - Any S3-compatible service Gracefully disabled by default - works without S3 configuration. ## Configuration Add to .env (all optional, have sensible defaults): ```bash LOG_RETENTION_DAYS=90 # Days to keep logs LOG_ARCHIVE_ENABLED=true # Enable automated archival LOG_CLEANUP_HOUR=3 # Hour (UTC) to run nightly job LOG_S3_ENABLED=false # Upload archives to S3 LOG_S3_BUCKET= # S3 bucket name LOG_S3_ACCESS_KEY= # S3 access key LOG_S3_SECRET_KEY= # S3 secret key LOG_S3_ENDPOINT= # Custom endpoint (for MinIO, R2) LOG_S3_REGION=us-east-1 # AWS region ``` ## Files Changed New: - src/backend/services/log_archive_service.py (313 lines) - src/backend/services/s3_storage.py (177 lines) - src/backend/routers/logs.py (177 lines) - tests/test_log_archive.py (629 lines, 27 tests) Modified: - config/vector.yaml - Date-based file paths - docker-compose.yml - Env vars + trinity-archives volume - docker/backend/Dockerfile - Added boto3 dependency - src/backend/main.py - Router + scheduler integration - tests/requirements-test.txt - Test dependencies - docs/memory/changelog.md - Detailed changelog entry - docs/memory/feature-flows/vector-logging.md - Retention docs ## Testing All 27 tests passing: - 5 authentication tests - 7 configuration tests - 3 compression/integrity tests - 3 manual archive tests - 2 S3 integration tests - 2 integration workflow tests - 2 validation tests - 3 error handling tests ## Impact - No breaking changes to existing Vector setup - Disk space stabilizes after retention period - Compliance-ready audit trail with secure archival - Production-ready with comprehensive test coverage ## Compliance Addresses SOC2 CC7.2 (System Monitoring) and ISO27001 A.12.4.1 (Event Logging) requirements for log retention and archival. 🤖 Generated with Claude Sonnet 4.5 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add uplot npm package to fix import error in SparklineChart.vue component. This dependency is required for the telemetry sparkline charts feature. Error fixed: - Failed to resolve import "uplot" from "src/components/SparklineChart.vue" Changes: - src/frontend/package-lock.json - Added uplot package and dependencies - tests/test_log_archive.py - Whitespace cleanup
Architectural change following feedback to maintain full data sovereignty. All logs and archives now stay within operator-controlled infrastructure. Changes: - Created pluggable ArchiveStorage interface with LocalArchiveStorage - Removed external S3/boto3 dependency entirely - Fixed scheduler restart bug in retention update endpoint - Updated docs with sovereign backup strategies (NAS, rsync, volumes) - All 19 API tests passing Breaking: S3 configuration no longer supported (by design) Non-breaking: Archives still stored in trinity-archives volume Files: + src/backend/services/archive_storage.py (new storage abstraction) - src/backend/services/s3_storage.py (deleted) M docker-compose.yml (removed S3 vars, added LOG_ARCHIVE_PATH) M docker/backend/Dockerfile (removed boto3) M src/backend/routers/logs.py (fixed responses + scheduler) M src/backend/services/log_archive_service.py (uses interface) M tests/requirements-test.txt (removed boto3) M tests/test_log_archive.py (removed S3 tests)
vybe
added a commit
that referenced
this pull request
Jan 7, 2026
- Add Phase 13-14 requirements for scalability & process orchestration - Document simplified role model (Executor/Monitor/Informed) - Add human approval steps concept for business processes - Enhance demo-analyst and demo-researcher templates - Improve create-demo-agent-fleet command - Fix replay mode activity & context simulation in network.js - Merge conflict resolution for PR #3 and #4 changelog entries 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
pavshulin
pushed a commit
that referenced
this pull request
Mar 31, 2026
- requirements.md: Added SLACK-FILES requirement (§15.1b-iii) - feature-flows/slack-file-sharing.md: Full flow document - feature-flows.md: Index updated with new flow - task_execution_service.py: Move start_time before try block (review item #3) - message_router.py: Sanitize session_id in upload path (review item #4) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
vybe
added a commit
that referenced
this pull request
Mar 31, 2026
…container (#222) * feat: Bidirectional file sharing — Slack to agent (inbound) (#222) Enable Slack users to upload files that agents can process. Images are embedded as base64 data URIs for Claude vision. Text files (CSV, JSON, TXT, etc.) are copied into per-session container directories via Docker put_archive API. Security: - Filename sanitization (path traversal prevention, hidden file rejection) - Per-user file upload rate limiting (5 files/min) - Size caps: 5MB per image, 10MB per file, 10MB total inline images - Max 10 files per message - Unsupported formats rejected with user message (PDF, archives, video, audio) - Per-session upload dirs cleaned up after execution (all exit paths) Architecture: - FileAttachment model + files field on NormalizedMessage (channel-agnostic) - adapter.download_file() — each channel implements its own download auth - files:read OAuth scope added for Slack workspace installs - container_put_archive() async wrapper in docker_utils - Task execution logging: timeout, HTTP status, elapsed time Tests: 36 unit tests covering sanitization, type routing, size limits, extraction, rate limiting, session directories Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: Add requirements, feature flow for Slack file sharing (#222) - requirements.md: Added SLACK-FILES requirement (§15.1b-iii) - feature-flows/slack-file-sharing.md: Full flow document - feature-flows.md: Index updated with new flow - task_execution_service.py: Move start_time before try block (review item #3) - message_router.py: Sanitize session_id in upload path (review item #4) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Pavlo <pash@pashs-MBP.home> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: vybe <me@evyborov.com>
This was referenced Apr 4, 2026
This was referenced Apr 20, 2026
Closed
vybe
added a commit
that referenced
this pull request
Apr 21, 2026
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Apr 22, 2026
Closed
7 tasks
3 tasks
This was referenced Apr 27, 2026
Closed
trinity-ability
pushed a commit
that referenced
this pull request
Aug 21, 2026
…e surface that never rendered (ent#429) Slices 3+4 of ent#364. Most of the rendering shipped in #2300; auditing the acceptance criteria against the code left three things genuinely undone, one of them a promise the feature has been making falsely since it landed. **AC #2 — nothing homeless.** `workspace_session_id` was READ by the ask projection and written by nothing, so `chat_id` was always null and an ask raised by a scheduled run belonged nowhere. It is now resolved at RAISE time (`client_portal.service.ensure_thread_for_ask`), reusing the client's existing thread with that agent and opening one only if they have never chatted — the same `_resolve_session_id(..., None)` a first client turn takes, so an ask lands IN the conversation rather than beside it. Render-time resolution would not be an attachment: it is a guess repeated per view, with nothing durable to audit. **And it is platform-written, which nothing enforced.** The projection's docstring already promised "platform-written context only" while the clamp happily passed an agent-authored `workspace_session_id` straight through — `chat_id` is where the Workspace SENDS a reader, so an agent could choose that destination. Stripped unconditionally now, before the real one is written, and stripped whether or not an addressee resolves. Three ordering details, each load-bearing: * the addressee is resolved BEFORE the context block, because the thread id is written INTO context and doing it after would add bytes the size cap had already signed off on; * the context dict is REBUILT, not popped — `out = dict(req)` is a shallow copy, so popping would reach into the caller's own dict and the clamp's contract is that it never mutates the request; * the thread id survives the oversize-context marker, or an agent-controlled oversize context would be the one way to produce a homeless ask. Fail-SOFT, deliberately the opposite direction to `_validated_addressee` beside it: that fails closed because it is an authorization decision, this one only decides where a link points, and losing the whole question over a missing link is the worse trade. **AC #1 — the inline-in-chat surface has never rendered.** `PortalConversation` passes `props.agent`, the agent OBJECT, to `asksForAgent`, which compares it against the string `agent_name`. It matched nothing, and the wrapper is `v-if="agentAsks.length"`, so the third of ent#364's three surfaces silently did not exist. Verified with a throwaway probe before believing it: asksForAgent('scout') -> 1 ask asksForAgent({name: 'scout'}) -> 0 asks Pinned against the SOURCE, because this repo's vitest runs in `node` with no component-mount harness — crude, but it is the difference between catching this class again and not. **AC #3 — expiry now says WHEN.** The row said "This expired before it was answered" with no time, and the #1142 sweep DELETES terminal rows, so between lapsing and being swept this is the only evidence the question was ever asked. An hour ago and last March are different situations. `expiredLabel` lives in `portalUtils` with `now` injected, per this file's rule that a sentence composed inside a component is a sentence no test can reach; it degrades to the bare wording on a missing, garbled or future timestamp, because a wrong time is worse than no time — the reader would act on it. Plus `askThreadLink`: "Open the conversation", shown only when the ask names a thread the reader is not already in. Additive — it never hides an ask from the thread being read. Narrowing the inline surface to only its own thread is the open question, not this. All Workspace work in the OSS repo, per the standing rule. Backend slice: 687 passed, 1 skipped. Frontend: 52 files / 1097 tests, tokens OK, build OK. 6 of the 9 new backend cases fail without this change; the other 3 are regression guards for properties it could have broken. Related to ent#429. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
trinity-ability
pushed a commit
that referenced
this pull request
Aug 21, 2026
…e surface that never rendered (ent#429) Slices 3+4 of ent#364. Most of the rendering shipped in #2300; auditing the acceptance criteria against the code left three things genuinely undone, one of them a promise the feature has been making falsely since it landed. **AC #2 — nothing homeless.** `workspace_session_id` was READ by the ask projection and written by nothing, so `chat_id` was always null and an ask raised by a scheduled run belonged nowhere. It is now resolved at RAISE time (`client_portal.service.ensure_thread_for_ask`), reusing the client's existing thread with that agent and opening one only if they have never chatted — the same `_resolve_session_id(..., None)` a first client turn takes, so an ask lands IN the conversation rather than beside it. Render-time resolution would not be an attachment: it is a guess repeated per view, with nothing durable to audit. **And it is platform-written, which nothing enforced.** The projection's docstring already promised "platform-written context only" while the clamp happily passed an agent-authored `workspace_session_id` straight through — `chat_id` is where the Workspace SENDS a reader, so an agent could choose that destination. Stripped unconditionally now, before the real one is written, and stripped whether or not an addressee resolves. Three ordering details, each load-bearing: * the addressee is resolved BEFORE the context block, because the thread id is written INTO context and doing it after would add bytes the size cap had already signed off on; * the context dict is REBUILT, not popped — `out = dict(req)` is a shallow copy, so popping would reach into the caller's own dict and the clamp's contract is that it never mutates the request; * the thread id survives the oversize-context marker, or an agent-controlled oversize context would be the one way to produce a homeless ask. Fail-SOFT, deliberately the opposite direction to `_validated_addressee` beside it: that fails closed because it is an authorization decision, this one only decides where a link points, and losing the whole question over a missing link is the worse trade. **AC #1 — the inline-in-chat surface has never rendered.** `PortalConversation` passes `props.agent`, the agent OBJECT, to `asksForAgent`, which compares it against the string `agent_name`. It matched nothing, and the wrapper is `v-if="agentAsks.length"`, so the third of ent#364's three surfaces silently did not exist. Verified with a throwaway probe before believing it: asksForAgent('scout') -> 1 ask asksForAgent({name: 'scout'}) -> 0 asks Pinned against the SOURCE, because this repo's vitest runs in `node` with no component-mount harness — crude, but it is the difference between catching this class again and not. **AC #3 — expiry now says WHEN.** The row said "This expired before it was answered" with no time, and the #1142 sweep DELETES terminal rows, so between lapsing and being swept this is the only evidence the question was ever asked. An hour ago and last March are different situations. `expiredLabel` lives in `portalUtils` with `now` injected, per this file's rule that a sentence composed inside a component is a sentence no test can reach; it degrades to the bare wording on a missing, garbled or future timestamp, because a wrong time is worse than no time — the reader would act on it. Plus `askThreadLink`: "Open the conversation", shown only when the ask names a thread the reader is not already in. Additive — it never hides an ask from the thread being read. Narrowing the inline surface to only its own thread is the open question, not this. All Workspace work in the OSS repo, per the standing rule. Backend slice: 687 passed, 1 skipped. Frontend: 52 files / 1097 tests, tokens OK, build OK. 6 of the 9 new backend cases fail without this change; the other 3 are regression guards for properties it could have broken. Related to ent#429. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 tasks
This was referenced Aug 24, 2026
6 tasks
This was referenced Aug 28, 2026
This was referenced Aug 31, 2026
6 tasks
5 tasks
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.