✨ Sigma file: reference documents, GC, search quality, and codebase refactoring - #33
Merged
Conversation
…lize imports (Phase 1)
….2 + P1.3) - src/shared/http.py: create_client, head_url, download_file with retry/backoff - src/shared/utils/registry_utils.py: unified build_registry_entry() - sigma_ref_downloader: use shared http and registry builder (-173 lines) - sigma_ref_processor: use shared http and registry builder (-112 lines) - Tests: 28 tests for http utils, 8 tests for registry builder, 20 regression tests
…ule (P1.1) - SigmaValidator.validate() returns SigmaRule instead of dict[str, Any] - SigmaRule.name property aliases title for Sigma YAML compatibility - ChatService._uploaded_rule: dict -> SigmaRule | None - RAGPipeline accepts SigmaRule in explain_rule, analyze_coverage, etc. - Merged level/status validation from dead validate_sigma_rule() - Removed validate_sigma_rule(), documents.models ValidationError/Result - Normalized name/title duality in validator - Updated all consumer tests
… (R1.1 + R1.2)
- download_sigma_references(db, output_dir, mode='scan'|'registry')
- download_references() delegates to mode='scan'
- process_sigma_refs() delegates to mode='registry' (-332 lines)
- {url_hash}{ext} naming used consistently in both modes
- Removed _sanitize_filename() (registry path naming mismatch)
- Contract tests: same URL → same filename in both modes
…NSW config P1.4 — Unify _scan_all_github + _scan_all_spec into _scan_all(prepare_fn), unify _write_entries + _write_spec_entries with batch_upsert_fn param, extract _collect_repo_files() for shared rglob+selected_dirs logic P2.5 — Add NULL_UUID constant in src/shared/constants.py, migrate 6 files away from hardcoded null UUID string Q1.1 — Add ScalarQuantization INT8 to all 3 collection creation paths (collections.py, storage.py, qdrant.py), controlled via enable_quantization Q1.2 — Bump embed_batch_size 8->64, num_workers 0->4 in ingestion.py Q2.3 — Add per-collection HNSW config via collection_hnsw_config(): sigma_rules in-RAM ef_construct=200, sigma_docs/sigma_spec on-disk R1.4 follow-up — delete_unreferenced_entries() + DocGCWorker integration (delete_head_verified_orphans + unreferenced cleanup in GC cycle) ruff clean, 146+ relevant tests green
…(P2.3) - Nouveau module src/shared/session.py — SessionStore thread-safe - ChatService délègue rule/tool_state à SessionStore - _handle_explain/_handle_search/_handle_coverage et leurs streams acceptent (message, prompt_id, sid) - Routes API lisent X-Session-ID header avec fallback '_default' - P2.1: _chunk_rule() éclatée en 13 méthodes spécialisées - P2.6: FILETYPE_TO_EXT dans identify_file_type.py, utilisé dans sigma_ref_downloader.py - Tests mis à jour pour nouvelles signatures
- initdb.sql: strftime('%Y-%m-%dT%H:%M:%fZ', current_timestamp)
(DuckDB 1.5+ ne tolère plus deux string literals)
- doc_ops.py: cutoff calculé en Python pour éviter strftime à 3 args
- duckdb.py: ajout rule_references à _EXPECTED_TABLES
- test_duckdb.py: assert tables_missing 10→11, clean no-op passe
- test_feedback.py: C:\absolute\path.db → /tmp/absolute/path.db (Linux)
- test_datadir.py: count 14→15 (specification dir ajouté)
- Nouveau FILETYPE_TO_SUBDIR dans identify_file_type.py (markdown/, html/, pdf/, plain_text/, office/) - sigma_ref_downloader.py: _sigmaref_write_path / _sigmaref_resolve_path avec fallback flat pour backward compat - indexer.py: résolution sigmaref → subdir puis flat - gc_worker.py: _file_exists_locally vérifie subdir + flat - Fix bug: compute_sha256_bytes() au lieu de compute_sha256_str() sur contenu binaire
- _gc_orphaned_sigmaref_files() : scanne sigmaref/ (subdirs + flat) pour les fichiers dont le SHA256 stem n'est pas dans doc_registry et les déplace vers .trash/ - _is_orphan_candidate() : vérifie format 64-char hex + appartenance - Supprime les sous-répertoires vides après nettoyage - Intégré dans DocGCWorker.process() avec compteur dans le log - 13 tests unitaires (format, flat, subdir, trash, edge cases)
- ALPHA_BY_COLLECTION globale avec valeurs tunées : sigma_rules=0.5, sigma_docs=0.7, sigma_spec=0.3 - SearchEngine accepte alpha_by_collection en override - alpha global devient fallback (default 0.3) - Rétrocompatible : tous les appelants existants inchangés
Q2.2: - rrf_k configurable (default 60) dans SearchEngine - rrf_weights par collection (weighted RRF) - col_idx aligné sur collections pour lookup weight Q3.1: - on_disk=True ajouté aux 3 chemins de création collection (qdrant.py, storage.py, collections.py) pour sparse vectors
- P2.6: Deduplicate FILETYPE_TO_EXT/SUBDIR → FILETYPE_INFO + helpers - P2.2: Share RAGPipeline singleton in translate endpoint - P2.1: Extract _assemble_chunks() from _chunk_rule() - R2.2: Add trash cleanup to DocGCWorker (max_age_days=7)
- Add golden set loader/save (JSON) - Implement retrieval metrics (recall@k, precision@k, MRR, AP, context_precision/recall) - Add SearchEvaluator runner with per-query and aggregated results - Add exact vs approximate search comparison baseline - 64 unit tests for eval modules
…odules - sigma_ref_paths.py: path resolution helpers (subdir_for, sigmaref_write_path, sigmaref_resolve_path) - sigma_ref_url.py: URL type detection (detect_url_type, is_reference_url, resolve_ext) - sigma_ref_registry.py: registry management via DB (load_registry, save_registry, load_error_registry, maybe_record_error) - Refactored sigma_ref_downloader.py to use the new modules - Fixed detect_url_type to properly map extensions to file types - All 51 tests passing, ruff and mypy clean
…odules - sigma_ref_paths.py: path resolution helpers (subdir_for, sigmaref_write_path, sigmaref_resolve_path) - sigma_ref_url.py: URL type detection (detect_url_type, is_reference_url, resolve_ext) - sigma_ref_registry.py: registry management via DB (load_registry, save_registry, load_error_registry, maybe_record_error) - Refactored sigma_ref_downloader.py to use the new modules - Fixed detect_url_type to properly map extensions to file types - All 51 tests passing, ruff and mypy clean
- Add get_pooled_client() with httpx.HTTPTransport for connection reuse - Pool keyed by (timeout, follow_redirects) for correct isolation - Limits: 100 max connections, 20 keepalive, 30s expiry - Update head_url() and download_file() to use pooled clients - Add close_all_pooled_clients() for clean shutdown - Update all tests to mock get_pooled_client instead of create_client - Add 4 tests for pool behavior (dedup, isolation, cleanup)
- Add index_after_upload bool param to index_sigma_rules() - When True, skip delete+full-reindex for incremental indexing - Add index_after_upload query param to /api/v1/chat/upload endpoint - Upload response includes indexed/chunks count when index_after_upload=True - Enables immediate Qdrant indexing after file upload without full reindex
- Update qdrant-client dependency to >=1.18.0 - Remove on_disk from SparseVectorParams (removed in 1.18) - Remove rescore/oversampling from ScalarQuantization (removed in 1.18) - Rename full_scan_threshold_kb to full_scan_threshold in HnswConfigDiff - Sort dependencies alphabetically in pyproject.toml - Update direct dependencies to latest versions
- Pass repo_key via task params instead of scanning all repos - Add GITHUB_DISCOVERY worker trigger to select-dirs endpoint - Fix parameter naming conflict (request -> request_body)
…erence files - Add _garbage_collect_github/local/spec methods to GenericDiscoveryWorker - Add _cleanup_rule_references to delete downloaded reference files - Fix _process_github to GC all repos with active selections + scanned repo - Add delete_doc_registry_by_url_hashes and get_doc_registry_url_hashes_by_repo to DatabaseService - Add get_rule_id_by_url_hash and get_rule_reference_paths for reference cleanup - Fix sigmaref_resolve_path and resolve_rule_path fallbacks to use subdir-aware paths - Add 5 P0 tests for garbage collection scenarios - Add mocks for new DB methods in conftest
- fix: use huggingface_hub.constants.HF_HUB_OFFLINE instead of os.environ (env var is cached at import, mutations had no effect) - fix: normalize empty HF_TOKEN to None to prevent invalid Bearer header - fix: bypass offline mode for online API calls (search, model info, download) - fix: restrict reference downloads to markdown only - fix: remove catch-all text/plain → markdown mapping in URL type detection - fix: make list_gguf_files/get_model_info async (were blocking event loop) - fix: frontend now reads error body instead of discarding it on HTTP errors - fix: backend returns 400 with actual error message instead of generic 500 - refactor: remove dead os.environ manipulation in ingestion.py - docs: add AirGap / Offline Mode section to README
- fix: move stale file cleanup in download_binary() to after successful download + extraction, so a failed download no longer deletes the existing binary - fix: add missing @patch('start_download') to two test methods (test_admin_download and test_orchestration_download) that were triggering the real download pipeline with default config and deleting the binary
…t_models - fix: list_models(task=None) no longer filters by feature-extraction, allowing LLM model search to find GGUF repos without pipeline_tag - fix: add DownloadError handling to /llm/search endpoint (same as /embeddings/search) to return meaningful error messages
…wnload in LLM download - add missing os import to models_llm.py - wrap hf_hub_download with save/restore of hc.HF_HUB_OFFLINE to allow online access even when app is in AirGap mode - normalize empty HF_TOKEN to None to avoid invalid Bearer header
…ure started flags - Wrap _start_qd() and _start_llm() in separate try/except so one failure doesn't prevent the other from running - Remove _qdrant_started_by_us / _llamacpp_started_by_us = True that was set before raising ServiceStartError on health check timeout
…lumn
- Add source_type TEXT column to git_selected_dirs table
- Auto-migration via ALTER TABLE ADD COLUMN IF NOT EXISTS
- Backward-compat loading from old DB files (missing source_type)
- Thread source_type from routers ('github' / 'spec') to save_selected_dirs()
- Filter get_repos_with_selected_dirs(source_type=) in discovery worker
- Update schema_version to 20260704
…s, and sigma_docs provenance
Add html, yaml, pdf, and plain_text to SUPPORTED_REFERENCE_DOC_TYPES so the downloader processes them instead of skipping. Add yaml entry to FILETYPE_INFO for path resolution.
…a_docs on references: filter Update router prompt to describe sigma_docs as containing external reference documents linked to rules. Auto-include sigma_docs in search when a 'references:' filter is present in the query.
… threshold Add _log_download_summary helper to log download results and warn when the failure rate exceeds 5%. Apply to both scan and registry modes.
Add three tests covering orphan deletion, noop when all refs are referenced, and scoping to sigmaref org only (skipping local/github).
…n param The second definition shadowed the first, causing a TypeError when spec discovery called _write_entries with batch_upsert_fn keyword.
…injection, nested lock, deprecated typing - BUG 1 (critical): move assignment before if/else in discovery_worker.py - BUG 2 (high): add in block in storage.py - BUG 3 (high): parameterize SQL query with instead of f-string in gc_worker.py - BUG 8: remove redundant inner (RLock but unnecessary) - BUG 9: replace deprecated with ; with - Bonus: migrate to
… delete, reorder sidebar nav
Remove html, yaml, pdf, plain_text from SUPPORTED_REFERENCE_DOC_TYPES since only markdown and sigma_rule have dedicated transforms.
…e types Keep only markdown and sigma in FILETYPE_INFO; remove pdf, plain_text, html, yaml subdirs. Add sigma_rule entry so files land under sigma/ subdirectory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Résumé
Cette PR regroupe le travail sur la branche
sigma-file— gestion des documents de référence Sigma, nettoyage (GC), qualité de recherche, et refactoring du codebase.Nouveautés principales
📄 Documents de référence Sigma
references:sigmaref/markdown/,sigmaref/sigma/)rule_references(M:N) entre règles et références🧹 Garbage Collection
DocGCWorker: nettoyage des entrées orphelines, fichiers fantômes, corbeilleDiscoveryWorker: GC inline après scan GitHub/local🔍 Qualité de recherche
referenceskeyword dans Qdrant pour filtragesigma_docssur les requêtes avec filtrereferences:format_result_by_collectionappliqué sur les résultats API et chat stream♻️ Refactoring
sigma_ref_downloader.pyen modules spécialisés (sigma_ref_paths,sigma_ref_url,sigma_ref_registry)http.py,crypto_utils.py,url_utils.py,registry_utils.pySessionStore✅ Tests
test_gc_worker.py,test_discovery_workers.py,test_sigma_ref_paths.py,test_sigma_ref_url.py,test_sigma_ref_registry.py,test_sparse_encoder.py,test_http.py,test_registry_utils.py, eval suite (test_exact_search,test_golden_set,test_metrics,test_runner)🔧 Infrastructure
source_typenullable-safesigma_docs