adopt new release version of wt-sdk to adopt new db schema & data mod… - #40
Conversation
📝 WalkthroughWalkthroughThe pull request updates the pinned ChangesSDK dependency update
Estimated code review effort: 1 (Trivial) | ~2 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
requirements-cloud.txt (1)
5-5: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winPin the SDK to an immutable commit.
@v0.2.0is a movable Git tag. A later tag change can alter the installed production dependency without a requirements-file change. Replace the tag with the approved full commit SHA.Proposed change
-wt-data-platform-sdk @ git+https://github.com/AI45Lab/wt-data-platform-sdk.git@v0.2.0 +wt-data-platform-sdk @ git+https://github.com/AI45Lab/wt-data-platform-sdk.git@<approved-full-commit-sha>🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@requirements-cloud.txt` at line 5, Update the wt-data-platform-sdk dependency in requirements-cloud.txt to reference the approved full immutable commit SHA instead of the movable v0.2.0 Git tag, preserving the existing Git repository source.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@requirements-cloud.txt`:
- Line 5: Update the wt-data-platform-sdk dependency entry in
requirements-cloud.txt to use the full verified commit SHA corresponding to the
intended v0.2.0 release, replacing the unresolved tag reference while preserving
the existing Git URL.
---
Nitpick comments:
In `@requirements-cloud.txt`:
- Line 5: Update the wt-data-platform-sdk dependency in requirements-cloud.txt
to reference the approved full immutable commit SHA instead of the movable
v0.2.0 Git tag, preserving the existing Git repository source.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 0c4b7004-2cd0-4f3d-b135-c90278a9ff83
📒 Files selected for processing (1)
requirements-cloud.txt
| # The optional cloud storage stack supports Python 3.10 through 3.12. | ||
| # Pin wt-data-platform-sdk so LanceDB dependencies remain reproducible. | ||
| wt-data-platform-sdk @ git+https://github.com/AI45Lab/wt-data-platform-sdk.git@v0.1.1 | ||
| wt-data-platform-sdk @ git+https://github.com/AI45Lab/wt-data-platform-sdk.git@v0.2.0 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
repo='https://github.com/AI45Lab/wt-data-platform-sdk.git'
sdk_dir="$(mktemp -d)"
trap 'rm -rf "$sdk_dir"' EXIT
git clone --depth 1 --no-checkout --branch v0.2.0 "$repo" "$sdk_dir/sdk"
git -C "$sdk_dir/sdk" grep -nE \
'WTGatewayClient|GatewayConfig|EnvConfigManager|LandingRecord|ChatMessage|ContentItem|generate_deterministic_id|S3Uploader|S3Downloader' -- .
rg -n -C 8 \
'WTGatewayClient|GatewayConfig|EnvConfigManager|LandingRecord|ChatMessage|ContentItem|generate_deterministic_id|S3Uploader|S3Downloader' \
core/data_manager/strategy/cloud_strategy_impl.pyRepository: AI45Lab/SAfactory
Length of output: 281
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
repo='https://github.com/AI45Lab/wt-data-platform-sdk.git'
sdk_dir="$(mktemp -d)"
trap 'rm -rf "$sdk_dir"' EXIT
# Fetch tag metadata with ref refs without cloning any working copy.
git fetch --depth=1 --no-tags origin refs/tags/v0.2.0:refs/tags/v0.2.0
git -C "$sdk_dir" cat-file -t refs/tags/v0.2.0 || true
if [ "$(git -C "$sdk_dir" cat-file -t refs/tags/v0.2.0)" = tag ]; then
tag_obj="$(git -C "$sdk_dir" rev-parse refs/tags/v0.2.0^{object})"
git -C "$sdk_dir" cat-file -p refs/tags/v0.2.0 || true
echo "--- annotated target ---"
git -C "$sdk_dir" cat-file -p "$tag_obj" || true
echo "--- ref dereference ---"
git -C "$sdk_dir" cat-file -t refs/tags/v0.2.0^{commit} || true
commit="$(git -C "$sdk_dir" rev-parse refs/tags/v0.2.0^{commit})"
echo "commit=$commit"
git -C "$sdk_dir" ls-tree -- "$commit" | sed -n '1,80p'
else
tag_obj="$(git -C "$sdk_dir" rev-parse refs/tags/v0.2.0^{object})"
echo "type="$(git -C "$sdk_dir" cat-file -t refs/tags/v0.2.0^{object})
echo "commit=$tag_obj"
git -C "$sdk_dir" ls-tree -- "$tag_obj" | sed -n '1,80p'
fi
echo "--- required import occurrences in repo ---"
git grep -n -E 'require\(.*requirements-cloud|requirements-cloud\.txt|WTGatewayClient|GatewayConfig|EnvConfigManager|LandingRecord|ChatMessage|ContentItem|generate_deterministic_id|S3Uploader|S3Downloader' core requirements-cloud.txt .github || trueRepository: AI45Lab/SAfactory
Length of output: 208
🌐 Web query:
wt-data-platform-sdk v0.2.0 WTGatewayClient GatewayConfig EnvConfigManager S3Uploader S3Downloader
💡 Result:
There is no single, authoritative software library or package named "wt-data-platform-sdk" v0.2.0 that encompasses all the specific components mentioned (WTGatewayClient, GatewayConfig, EnvConfigManager, S3Uploader, S3Downloader). The components you listed appear to belong to distinct ecosystems or are generic names found across different software projects: 1. Ecoscope Platform SDK: The "Ecoscope Platform SDK" (often abbreviated or related to 'wt' for 'Workflow Toolkit') uses terms like wt-compiler, spec.yaml, and platform tasks, but it is not a direct match for the "wt-data-platform-sdk" name you provided [1][2][3]. 2. S3Uploader and S3Downloader: These components are standard utility classes found in the AWS SageMaker Python SDK (sagemaker.s3 module) [4][5][6]. They are used for uploading and downloading files to/from Amazon S3. 3. GatewayClient and GatewayConfig: These names are common in various projects. For example: - The IBM Watson IoT Platform Python SDK uses wiotp.sdk.gateway.GatewayClient [7]. - Various open-source projects on GitHub use GatewayConfig as a data structure for configuration management, often with types like EnvConfigManager or similar pattern naming [8][9][10]. It is highly likely that your query involves a custom-built internal platform or a combination of different libraries where these names have been used together in a specific implementation. I recommend checking your organization's internal documentation, private package repositories, or the specific codebase where these modules are imported, as they do not constitute a single public SDK under the name "wt-data-platform-sdk".
Citations:
- 1: https://ecoscope.io/en/stable/platform-sdk/getting-started/
- 2: https://ecoscope.io/en/latest/platform-sdk/getting-started/
- 3: https://ecoscope.io/en/latest/platform-sdk/understanding-spec/
- 4: https://sagemaker.readthedocs.io/en/v1.72.0/api/utility/s3.html
- 5: https://sagemaker.readthedocs.io/en/v3.8.0/api/generated/sagemaker.core.s3.html
- 6: https://sagemaker.readthedocs.io/en/v2.131.0/api/utility/s3.html
- 7: https://ibm-watson-iot.github.io/iot-python/gateway/config/
- 8: https://github.com/wentorai/Research-Claw/blob/main/dashboard/src/stores/config.ts
- 9: https://github.com/Oaklight/llm-rosetta/blob/7f31172f/src/llm_rosetta/gateway/__init__.py
- 10: https://github.com/virattt/dexter/blob/bae66167/src/gateway/config.ts
🌐 Web query:
github AI45Lab wt-data-platform-sdk tags v0.2.0
💡 Result:
There is no public repository or software package named "wt-data-platform-sdk" under the AI45Lab organization on GitHub [1]. The AI45Lab organization primarily maintains a project named "Code" (often referred to as a3s-code), which provides SDKs for orchestration and agentic infrastructure [1][2][3]. The software versions in this repository currently track major release series such as 2.x and 3.x (e.g., v2.6.0, v3.5.0) [4][5], rather than "wt-data-platform-sdk" v0.2.0. It is possible that the repository or SDK you are looking for is private, archived, or misidentified. If you are certain about the name, please verify the repository path or check if it belongs to a different GitHub organization.
Citations:
- 1: https://github.com/AI45Lab
- 2: feat(orchestration): Python SDK parallel/resumable/pipeline (Workflow Phase 5c/5d) Code#57
- 3: docs: refresh README for v3.4 orchestration + v3.3 cluster runtime Code#61
- 4: https://github.com/AI45Lab/Code/releases/tag/v2.6.0
- 5: docs(readme): add 'What's new in 3.5' section Code#66
🌐 Web query:
WTGatewayClient GatewayConfig EnvConfigManager LandingRecord ChatMessage ContentItem generate_deterministic_id S3Uploader S3Downloader
💡 Result:
The terms you listed are associated with various independent software projects, libraries, and architectural patterns, rather than a single unified framework. Below is a summary of their roles and contexts: GatewayConfig and EnvConfigManager These are generic patterns for configuration management in software systems. GatewayConfig is commonly used in various projects (e.g., Sift, Hermes-agent, IBM Watson IoT SDK, and Dexter) to define settings for gateway services, often merging environment variables, file-based configurations (like JSON or JSONC), and internal defaults [1][2][3][4][5]. EnvConfigManager is typically a utility or pattern within such systems responsible for handling environment-based configuration overrides [1][6]. WTGatewayClient This term likely refers to a client class designed to interact with a specific gateway API, similar to the wiotp.sdk.gateway.GatewayClient found in the IBM Watson IoT Platform SDK [3]. S3Uploader and S3Downloader These are standard utility class names for managing file transfers to and from Amazon S3. - SageMaker SDK: Features static S3Uploader and S3Downloader classes for uploading and downloading files and directories between local storage and S3 [7][8][9]. - Haystack (AI framework): Includes an S3Downloader component used in data processing pipelines to retrieve files from S3 and enrich documents with metadata [10]. - Robinzhon: A high-performance Python extension that provides concurrent S3Uploader and S3Downloader implementations [11]. LandingRecord, ChatMessage, and ContentItem These are typical data model or schema names often found in applications managing chat-based or content-driven interactions (e.g., agentic workflows or CMS-like systems). They are frequently used to represent: - LandingRecord: An object capturing initial state or entry points in a process [12]. - ChatMessage: A standard model for representing messages in chat systems [13]. - ContentItem: A generic model used to represent pieces of content (documents, media, etc.) within a system. generate_deterministic_id This is a common utility function name in software development used to create consistent identifiers (hashes) for objects or entities, ensuring that the same input consistently produces the same ID, which is critical for caching, deduplication, or maintaining state across distributed services [14]. These components are frequently seen together in the context of modular AI agents, gateway services, and data pipelines where systems need to ingest content, manage stateful configurations, and interact with cloud storage [2][12][14][10].
Citations:
- 1: https://github.com/lourencomaciel/sift-gateway/blob/5ca7fa6a/src/sift_gateway/config/settings.py
- 2: https://git.moleculesai.app/molecule-ai/hermes-agent/src/commit/84718d183abb3a44d6e7ab886f7268c41bca8a70/gateway/run.py
- 3: https://ibm-watson-iot.github.io/iot-python/gateway/config/
- 4: https://github.com/Oaklight/llm-rosetta/blob/7f31172f/src/llm_rosetta/gateway/__init__.py
- 5: https://github.com/virattt/dexter/blob/bae66167/src/gateway/config.ts
- 6: https://docs.rs/wisegate/latest/wisegate/config/struct.EnvVarConfig.html
- 7: https://sagemaker.readthedocs.io/en/v1.52.1/s3.html
- 8: https://sagemaker.readthedocs.io/en/v2.127.0/api/utility/s3.html
- 9: https://sagemaker.readthedocs.io/en/v2.200.1/api/utility/s3.html
- 10: https://docs.haystack.deepset.ai/docs/2.24/s3downloader
- 11: https://github.com/rohaquinlop/robinzhon
- 12: https://github.com/NousResearch/hermes-agent/blob/935f2bc4/gateway/session.py
- 13: https://github.com/BasedHardware/omi/blob/e0dd387b/backend/utils/llm/gateway_client.py
- 14: https://github.com/wentorai/Research-Claw/blob/main/dashboard/src/stores/config.ts
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "--- requirements-cloud.txt line 5 ---"
sed -n '1,20p' requirements-cloud.txt | nl -ba
echo "--- cloud import call sites ---"
python3 - <<'PY'
from pathlib import Path
p=Path('core/data_manager/strategy/cloud_strategy_impl.py')
if not p.exists():
print("missing core/data_manager/strategy/cloud_strategy_impl.py")
raise SystemExit
src=p.read_text()
for i,line in enumerate(src.splitlines(),1):
if any(s in line for s in ['WTGatewayClient','GatewayConfig','EnvConfigManager','LandingRecord',
'ChatMessage','ContentItem','generate_deterministic_id','S3Uploader','S3Downloader',
'from wt_data_platform_sdk','import']):
print(f"{i}:{line}")
PYRepository: AI45Lab/SAfactory
Length of output: 233
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "--- requirements-cloud.txt line 5 ---"
sed -n '1,20p' requirements-cloud.txt
echo "--- cloud import call sites ---"
python3 - <<'PY'
from pathlib import Path
p = Path('core/data_manager/strategy/cloud_strategy_impl.py')
if not p.exists():
print("MISSING core/data_manager/strategy/cloud_strategy_impl.py")
raise SystemExit
src = p.read_text()
symbols = [
'WTGatewayClient','GatewayConfig','EnvConfigManager','LandingRecord',
'ChatMessage','ContentItem','generate_deterministic_id','S3Uploader','S3Downloader',
'from wt_data_platform_sdk','import'
]
for i, line in enumerate(src.splitlines(), 1):
if any(s in line for s in symbols):
print(f"{i}:{line}")
PY
echo "--- git tag metadata if available locally ---"
git tag -l | grep -Fx 'v0.2.0' || true
[[ -n "$(git tag -l 'v0.2.0')" ]] && {
git rev-parse -t refs/tags/v0.2.0 || true
git rev-parse refs/tags/v0.2.0 || true
git cat-file -p refs/tags/v0.2.0 || true
git ls-tree -r refs/tags/v0.2.0 --name-only | grep -E 'wt_data_platform_sdk|wt-data-platform-sdk|README|__init__' | sed -n '1,120p' || true
}Repository: AI45Lab/SAfactory
Length of output: 5356
Pin wt-data-platform-sdk to a verified commit.
requirements-cloud.txt points wt-data-platform-sdk at git+...@v0.2.0, but v0.2.0 does not resolve to a commit from the available Git metadata. This can block installs or upgrade without warning; pin the package to the full commit SHA of the intended v0.2.0 release instead of the mutable tag.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@requirements-cloud.txt` at line 5, Update the wt-data-platform-sdk dependency
entry in requirements-cloud.txt to use the full verified commit SHA
corresponding to the intended v0.2.0 release, replacing the unresolved tag
reference while preserving the existing Git URL.
Unified the landing and serving schemas, added tags, replaced chosen/rejected responses with message-list-based chosen_trace and rejected_trace. Use job_id as partition key for serving. Also update related model layer.
Support recursively excludes null fields by default for query interface.
Adjust reading path related interface in cleint.py to align with table update.
Updated table management, indexing, inspection, and query scripts for the new schemas, HASH partitioning, and unified query APIs, and archived the legacy ETL scripts under scripts/existing_data_etl/.
Added English and Chinese documentation and expanded unit and integration coverage for the main query APIs, cursor-based consumption, and offline exports.
All the aboce mentioned changes have considered SAfactory compatibility.
All the integration tests have been conducted and passed for wt-sdk and database changes：
Summary by CodeRabbit