[Perf] PDF·DOCX 50·100문서 전체 인덱싱 E2E 부하 검증 - #144
Conversation
|
Warning Review limit reached
Next review available in: 52 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughPDF·DOCX 문서를 병렬 업로드하고 전체 인덱싱을 수행하는 E2E 부하 Benchmark를 추가했습니다. 50·100문서 Profile의 처리량, 지연시간, 데이터 정합성을 검증하고 JSON 결과를 기록합니다. ChangesPDF·DOCX 인덱싱 Benchmark
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Benchmark as DocumentIndexingE2ELoadBenchmark
participant Worker
participant Storage as PostgreSQL and MinIO
participant Embedding as BAAI/bge-m3
Benchmark->>Worker: PDF·DOCX Multipart 업로드
Worker->>Storage: 객체와 인덱싱 상태 저장
Worker->>Embedding: 임베딩 생성 요청
Embedding-->>Worker: Vector 반환
Worker->>Storage: Chunk·Embedding 저장
Storage-->>Benchmark: 인덱싱 완료 상태 반환
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
src/test/java/com/opensource/docgrid/e2e/DocumentIndexingE2ELoadBenchmark.java (1)
232-232: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winWorker 최대 동시성 계약이 세 곳에 흩어져 있습니다. 테스트는 동시성을
2로 고정 주입하고 같은 값을 다시 단정하며, 설계 문서는 제품 설정을 사용한다고 기술합니다. 값을 바꾸면 세 곳이 동시에 어긋납니다.
src/test/java/com/opensource/docgrid/e2e/DocumentIndexingE2ELoadBenchmark.java#L232-L232: 상수 하나를 도입해 line 144의 주입값과 이 단정이 같은 정의를 참조하게 하십시오.docs/design/gimin-#143-pdf-docx-indexing-e2e-load-benchmark.md#L59-L61: 고정 동시성2를 사용한다는 사실과 그 이유를 명시하거나, 코드를 제품 설정 사용으로 바꾸십시오.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/test/java/com/opensource/docgrid/e2e/DocumentIndexingE2ELoadBenchmark.java` at line 232, Centralize the worker concurrency value in DocumentIndexingE2ELoadBenchmark by introducing one constant and reusing it for both the line 144 injection and the getMaxConcurrency() assertion at lines 232-232. Update docs/design/gimin-#143-pdf-docx-indexing-e2e-load-benchmark.md lines 59-61 to explicitly document the fixed concurrency of 2 and its rationale, or change the benchmark implementation and documentation together to use the product setting instead.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/design/gimin-`#143-pdf-docx-indexing-e2e-load-benchmark.md:
- Around line 59-61: Update the benchmark documentation’s worker-concurrency
description to match DocumentIndexingE2ELoadBenchmark’s forced
indexing.worker.max-concurrency value of 2, and state that this fixed value is
intentional with its rationale. Do not describe the benchmark as using the
product-configured maximum concurrency.
In
`@src/test/java/com/opensource/docgrid/e2e/DocumentIndexingE2ELoadBenchmark.java`:
- Around line 396-402: Update awaitIndexedAndIdle to pass a deferred
Supplier<String> to awaitCondition so jobSnapshot(uploads) is evaluated when the
wait fails rather than before waiting; add or use an awaitCondition overload
accepting Supplier<String> while preserving existing string-call compatibility
and avoiding snapshot queries on successful waits.
---
Nitpick comments:
In
`@src/test/java/com/opensource/docgrid/e2e/DocumentIndexingE2ELoadBenchmark.java`:
- Line 232: Centralize the worker concurrency value in
DocumentIndexingE2ELoadBenchmark by introducing one constant and reusing it for
both the line 144 injection and the getMaxConcurrency() assertion at lines
232-232. Update docs/design/gimin-#143-pdf-docx-indexing-e2e-load-benchmark.md
lines 59-61 to explicitly document the fixed concurrency of 2 and its rationale,
or change the benchmark implementation and documentation together to use the
product setting instead.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 20dc53d7-8755-4bff-beb3-a68913801fc5
📒 Files selected for processing (7)
build.gradledocs/design/gimin-#143-pdf-docx-indexing-e2e-load-benchmark.mddocs/test-results/gimin-#143-pdf-docx-indexing-e2e-load-benchmark.mdsrc/test/java/com/opensource/docgrid/e2e/DocumentIndexingE2ELoadBenchmark.javasrc/test/java/com/opensource/docgrid/e2e/DocumentIndexingE2ELoadStatistics.javasrc/test/java/com/opensource/docgrid/e2e/DocumentIndexingE2ELoadStatisticsTest.javasrc/test/java/com/opensource/docgrid/e2e/LocalE2eDocumentFactory.java
개요
실제 Text Layer PDF와 OOXML DOCX를 HTTP Multipart로 업로드하고 PostgreSQL 17, MinIO, BAAI/bge-m3와 pgvector를 통과하는 전체 인덱싱 부하 Benchmark를 추가했습니다.
변경 사항
실측 결과
검증
범위 밖
Closes #143
Summary by CodeRabbit
새로운 기능
문서화
테스트