[Feat] Flyway 마이그레이션 + embedding_models seed - #42
Conversation
- V32: embeddings.vector, search_queries.query_vector TEXT → vector(1024) 변환 및 HNSW 인덱스 추가 - R__seed_bge_m3_embedding_model: mock 모델 seed를 실제 BAAI/bge-m3(HUGGINGFACE) seed로 교체 - R__seed_test_fixtures: 벡터 검색 동작 확인용 개발 더미 데이터 추가 (문서 2개, 청크 4개, 임베딩 4개) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- VectorType: pgvector의 vector(1024) SQL 타입을 float[]로 매핑하는 커스텀 Hibernate UserType - Embedding.vector, SearchQuery.queryVector: String → float[] + @type(VectorType.class)로 교체 - postgresql 의존성을 runtimeOnly → implementation으로 변경 (PGobject 컴파일 타임 사용) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
📝 WalkthroughWalkthroughpgvector 기반으로 벡터 컬럼과 Java Changes벡터 검색 DB 인프라
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
src/main/resources/db/migration/V32__convert_vector_columns_to_pgvector.sql (1)
10-10: 🩺 Stability & Availability | 🔵 Trivial대용량 테이블 인덱스 생성 시 쓰기 잠금(Write Lock)을 고려하세요.
테이블의 데이터가 많을 경우
CREATE INDEX는 인덱스 생성이 완료될 때까지 테이블의 쓰기(Insert/Update/Delete) 작업을 차단합니다. 만약 데이터 규모가 큰 운영 환경을 가정한다면 가동 시간 확보를 위해CONCURRENTLY옵션 사용을 고려해볼 수 있습니다. (단, Flyway에서CONCURRENTLY를 사용하려면 해당 스크립트가 트랜잭션 외부에서 실행되도록 별도로 분리 설정되어야 합니다.)As per coding guidelines, do not make assumptions silently; state assumptions, surface uncertainty, and present multiple interpretations when applicable.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/resources/db/migration/V32__convert_vector_columns_to_pgvector.sql` at line 10, Update the HNSW index creation statement to use concurrent creation for the large embeddings table, and configure this migration to run outside Flyway’s transaction as required by CREATE INDEX CONCURRENTLY. Keep the existing index definition and vector_cosine_ops operator class unchanged.Source: Coding guidelines
src/main/java/com/opensource/docgrid/global/common/type/VectorType.java (1)
43-48: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win문자열 파싱 예외 처리를 강화하세요.
데이터베이스의 벡터 문자열 포맷이 훼손되었거나 빈 텍스트일 경우, 현재 로직은
StringIndexOutOfBoundsException혹은NumberFormatException런타임 에러를 발생시킬 수 있습니다. 시스템 안정성을 위해 방어적 검증 로직을 추가하고 명시적인SQLException으로 래핑하는 것을 권장합니다.As per path instructions, SOLID 원칙, 스프링 어노테이션, 의존성 주입 패턴, 예외 처리에 중점을 둔다.
🛠️ Proposed fix to improve exception handling
- String[] parts = value.substring(1, value.length() - 1).split(","); - float[] result = new float[parts.length]; - for (int i = 0; i < parts.length; i++) { - result[i] = Float.parseFloat(parts[i].trim()); - } - return result; + if (value.length() < 2 || !value.startsWith("[") || !value.endsWith("]")) { + throw new SQLException("Invalid vector string format: " + value); + } + String inner = value.substring(1, value.length() - 1).trim(); + if (inner.isEmpty()) { + return new float[0]; + } + String[] parts = inner.split(","); + float[] result = new float[parts.length]; + try { + for (int i = 0; i < parts.length; i++) { + result[i] = Float.parseFloat(parts[i].trim()); + } + } catch (NumberFormatException e) { + throw new SQLException("Failed to parse vector elements", e); + } + return result;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/java/com/opensource/docgrid/global/common/type/VectorType.java` around lines 43 - 48, Update the string parsing logic in VectorType to validate null/empty input and the expected bracketed vector format before substring and numeric conversion. Catch StringIndexOutOfBoundsException and NumberFormatException, then wrap parsing failures in an explicit SQLException while preserving successful parsing behavior.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/design/kangcheolung-`#41-vector-search-db-infrastructure.md:
- Around line 16-17: Update the embeddings.vector migration to preserve existing
vectors by converting the column in place with ALTER COLUMN ... TYPE
vector(1024) USING ..., or add a temporary column, backfill it, then enforce NOT
NULL before replacing the original. Remove the DROP COLUMN and direct NOT NULL
ADD COLUMN sequence so existing rows are retained and the migration succeeds
without a default.
In `@src/main/resources/db/migration/V32__convert_vector_columns_to_pgvector.sql`:
- Around line 3-4: Replace the destructive DROP COLUMN/ADD COLUMN sequence in
migration V32 with an ALTER COLUMN TYPE conversion using a USING expression,
matching the existing search_queries migration pattern, so embeddings.vector
data is preserved while becoming vector(1024). Ensure the conversion handles the
column’s current type explicitly and retains the NOT NULL constraint.
---
Nitpick comments:
In `@src/main/java/com/opensource/docgrid/global/common/type/VectorType.java`:
- Around line 43-48: Update the string parsing logic in VectorType to validate
null/empty input and the expected bracketed vector format before substring and
numeric conversion. Catch StringIndexOutOfBoundsException and
NumberFormatException, then wrap parsing failures in an explicit SQLException
while preserving successful parsing behavior.
In `@src/main/resources/db/migration/V32__convert_vector_columns_to_pgvector.sql`:
- Line 10: Update the HNSW index creation statement to use concurrent creation
for the large embeddings table, and configure this migration to run outside
Flyway’s transaction as required by CREATE INDEX CONCURRENTLY. Keep the existing
index definition and vector_cosine_ops operator class unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: a6c4963e-7fb1-4b05-8208-966c7b2ff3c8
📒 Files selected for processing (9)
build.gradledocs/design/kangcheolung-#41-vector-search-db-infrastructure.mdsrc/main/java/com/opensource/docgrid/domain/embedding/entity/Embedding.javasrc/main/java/com/opensource/docgrid/domain/search/entity/SearchQuery.javasrc/main/java/com/opensource/docgrid/global/common/type/VectorType.javasrc/main/resources/db/migration/V32__convert_vector_columns_to_pgvector.sqlsrc/main/resources/db/seed/R__seed_bge_m3_embedding_model.sqlsrc/main/resources/db/seed/R__seed_mock_embedding_model.sqlsrc/main/resources/db/seed/R__seed_test_fixtures.sql
💤 Files with no reviewable changes (1)
- src/main/resources/db/seed/R__seed_mock_embedding_model.sql
| ALTER TABLE embeddings DROP COLUMN vector; | ||
| ALTER TABLE embeddings ADD COLUMN vector vector(1024) NOT NULL; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find relevant files and inspect the doc plus migration references.
printf '\n== FILE LIST ==\n'
git ls-files | rg 'docs/design/kangcheolung-#41-vector-search-db-infrastructure\.md|V32|embeddings|vector'
printf '\n== DOC CONTEXT ==\n'
doc='docs/design/kangcheolung-#41-vector-search-db-infrastructure.md'
if [ -f "$doc" ]; then
nl -ba "$doc" | sed -n '1,80p'
fi
printf '\n== SEARCH FOR V32 / EMBEDDINGS / VECTOR ==\n'
rg -n --hidden -S 'V32|embeddings|vector\(1024\)|ALTER TABLE embeddings|DROP COLUMN vector|ADD COLUMN vector' .Repository: DocGrid/backend
Length of output: 429
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# If a V32 migration exists, print its path and relevant lines around vector changes.
rg -n --hidden -S 'CREATE TABLE embeddings|ALTER TABLE embeddings|vector\(1024\)|DROP COLUMN vector|ADD COLUMN vector|USING .*vector' . \
| sed -n '1,200p'Repository: DocGrid/backend
Length of output: 3592
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Narrow search for migration filenames and any vector-related schema definitions.
fd -a -t f 'V32|vector|embedding|embeddings' . | sed -n '1,200p'Repository: DocGrid/backend
Length of output: 1063
🏁 Script executed:
#!/bin/bash
set -euo pipefail
for f in src/main/resources/db/migration/V16__create_embeddings.sql \
src/main/resources/db/migration/V32__convert_vector_columns_to_pgvector.sql \
src/main/java/com/opensource/docgrid/domain/embedding/entity/Embedding.java \
src/main/java/com/opensource/docgrid/domain/search/entity/SearchQuery.java
do
printf '\n===== %s =====\n' "$f"
if [ -f "$f" ]; then
sed -n '1,220p' "$f"
else
echo "missing"
fi
doneRepository: DocGrid/backend
Length of output: 11505
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -S 'insert into embeddings|into embeddings|save\(.*Embedding|new Embedding|`@Entity`\s*.*Embedding|embedding' \
src/main/resources src/main/java | sed -n '1,240p'Repository: DocGrid/backend
Length of output: 34404
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== MIGRATION REFERENCES ==\n'
rg -n -S 'embeddings' src/main/resources/db/migration src/main/resources/db/seed | sed -n '1,240p'
printf '\n== ENTITY/REPOSITORY REFERENCES ==\n'
rg -n -S 'EmbeddingRepository|EmbeddingService|saveAll?\(|save\(.*Embedding|new Embedding|Embeddings' src/main/java | sed -n '1,240p'Repository: DocGrid/backend
Length of output: 3016
🏁 Script executed:
#!/bin/bash
set -euo pipefail
for f in src/main/resources/db/migration/V16__create_embeddings.sql \
src/main/resources/db/seed/R__seed_test_fixtures.sql \
src/main/resources/db/seed/R__seed_bge_m3_embedding_model.sql
do
printf '\n===== %s =====\n' "$f"
if [ -f "$f" ]; then
python3 - <<'PY' "$f"
import sys, pathlib
p = pathlib.Path(sys.argv[1])
for i, line in enumerate(p.read_text().splitlines(), 1):
if i <= 200:
print(f"{i:4d}: {line}")
PY
else
echo "missing"
fi
doneRepository: DocGrid/backend
Length of output: 9978
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '1,220p' src/main/resources/db/migration/V16__create_embeddings.sqlRepository: DocGrid/backend
Length of output: 1621
V32__convert_vector_columns_to_pgvector.sql는 drop/add 대신 변환으로 바꾸세요.
embeddings.vector를 삭제했다가 vector(1024) NOT NULL로 다시 추가하면 기존 행의 벡터가 유실되고, 기본값 없는 NOT NULL 추가도 실패합니다. ALTER COLUMN ... TYPE vector(1024) USING ...로 변환하거나 새 컬럼 추가 → 백필 → NOT NULL 적용 순서로 처리해야 합니다.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/design/kangcheolung-`#41-vector-search-db-infrastructure.md around lines
16 - 17, Update the embeddings.vector migration to preserve existing vectors by
converting the column in place with ALTER COLUMN ... TYPE vector(1024) USING
..., or add a temporary column, backfill it, then enforce NOT NULL before
replacing the original. Remove the DROP COLUMN and direct NOT NULL ADD COLUMN
sequence so existing rows are retained and the migration succeeds without a
default.
| ALTER TABLE embeddings DROP COLUMN vector; | ||
| ALTER TABLE embeddings ADD COLUMN vector vector(1024) NOT NULL; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win
실 데이터 유실 위험이 있는 DROP COLUMN 대신 타입 캐스팅을 사용하세요.
개발 환경에만 적용된다는 가정이 있으나, 해당 Flyway 마이그레이션 스크립트가 운영 및 스테이징 환경에 배포될 경우 embeddings 테이블의 기존 벡터 데이터가 영구적으로 삭제됩니다. search_queries 테이블(7행)과 동일하게 USING 절을 통한 타입 변환을 권장합니다.
As per coding guidelines, do not make assumptions silently; state assumptions, surface uncertainty, and present multiple interpretations when applicable.
🛡️ Proposed fix to preserve data
-ALTER TABLE embeddings DROP COLUMN vector;
-ALTER TABLE embeddings ADD COLUMN vector vector(1024) NOT NULL;
+ALTER TABLE embeddings ALTER COLUMN vector TYPE vector(1024) USING vector::vector;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ALTER TABLE embeddings DROP COLUMN vector; | |
| ALTER TABLE embeddings ADD COLUMN vector vector(1024) NOT NULL; | |
| ALTER TABLE embeddings ALTER COLUMN vector TYPE vector(1024) USING vector::vector; |
🧰 Tools
🪛 Squawk (2.59.0)
[warning] 3-3: Dropping a column may break existing clients.
(ban-drop-column)
[warning] 4-4: Adding a new column that is NOT NULL and has no default value to an existing table effectively makes it required. Make the field nullable or add a non-VOLATILE DEFAULT
(adding-required-field)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/main/resources/db/migration/V32__convert_vector_columns_to_pgvector.sql`
around lines 3 - 4, Replace the destructive DROP COLUMN/ADD COLUMN sequence in
migration V32 with an ALTER COLUMN TYPE conversion using a USING expression,
matching the existing search_queries migration pattern, so embeddings.vector
data is preserved while becoming vector(1024). Ensure the conversion handles the
column’s current type explicitly and retains the NOT NULL constraint.
Source: Coding guidelines
🔍 작업 내용
✨ 상세 설명
Flyway V32 — vector 컬럼 변환 및 HNSW 인덱스
embeddings.vector,search_queries.query_vectorTEXT → vector(1024) 타입 변환db/seed 정비
R__seed_bge_m3_embedding_model.sql로 교체 (BAAI/bge-m3, 1024차원,model_version 1.0)
VectorType — 커스텀 Hibernate UserType
global/common/type/VectorType.java: pgvector의vector(1024)(Types#OTHER)를float[]로 매핑PGobject만으로 구현, 스키마 검증 통과 확인엔티티 수정
Embedding.vector,SearchQuery.queryVector:String→float[]+@Type(VectorType.class)build.gradle: postgresql 의존성runtimeOnly→implementation(PGobject 컴파일 타임 사용)🛠️추후 리팩토링 및 고도화 계획
📸 스크린샷 (선택)
💬 리뷰 요구사항
VectorType외부 라이브러리 없이 직접 구현한 방식이 적절한지R__seed 파일의 ON CONFLICT 처리 누락된 케이스 없는지Summary by CodeRabbit
새 기능
vector(1024)형식으로 저장하고 조회할 수 있습니다.문서