Repository navigation
feat(drive): describe the GroveDB structure as code - #4845
Conversation
Nothing showed the whole GroveDB layout Drive builds: the book draws single subtrees and the root tree comment in drive/mod.rs had fallen behind (no ContractGroups). This adds a description of every level, from the 18 root trees down to the leaves, declared in Rust from the real key constants and kept honest by tests. - `drive::structure` holds the node model, a builder, a lint, a conformance walker and the JSON export types. Each area declares its part in a `structure.rs` beside its `paths.rs`. The module is compiled for tests and under the new `structure` feature only, so the node never builds it. - A node carries its key (fixed bytes and the constant they come from, or a template such as "identity id, 32 bytes"), the element kinds that can sit there, the first protocol version it exists in, whether it is created with its parent, what an item holds, reference and recursion targets, its source file and book chapter. - The conformance walker reads a real GroveDB layer by layer and reports every element no node describes, every kind mismatch, every node outside its protocol versions and every missing node. Tests run it on the initial state structure of every protocol version and on populated fixtures (identities, contracts with documents and history, tokens, group actions, address balances). A coverage gate lists the nodes no fixture reaches yet; the list can only shrink. - `packages/rs-drive/grovedb-structure.json` is the serialized description the GroveDB structure viewer reads by git ref. A test fails when it is stale; `UPDATE_GROVEDB_STRUCTURE=1 cargo test -p drive --lib structure::tests` rewrites it. - The JSON also records the exact Merk binary tree of layers whose keys are all fixed, rebuilt by replaying the proof of a full range query (`grovedb-merk` as a dev-dependency). The root tree comment now shows ContractGroups below Versions, as the recorded shape does. `initialization::genesis_core_height` becomes `pub(crate)` so the Misc description can use `GENESIS_CORE_HEIGHT_KEY`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
PR HygieneState: waiting-bots · commit
Self-review is an author attestation that you have read the diff: This check passes when the policy is satisfied; the repository decides whether merging requires it. |
|
Warning Review limit reachedNext included review available in 25 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Repository: dashpay/platform/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (22)
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository: dashpay/platform/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (49)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughChangesThe PR adds a feature-gated GroveDB structure model for Drive. It describes storage trees, validates declarations and live elements, replays fixed-key Merk layers, exports structure metadata, and tests conformance across protocol versions and populated fixtures. Drive structure model
Priority: ➖ Normal Estimated code review effort: 4 (Complex) | ~60 minutes Change: Feature Sequence Diagram(s)sequenceDiagram
participant StructureTests
participant Drive
participant GroveDB
participant StructureNode
participant ConformanceReport
StructureTests->>Drive: build structure fixtures
StructureTests->>StructureNode: obtain drive_structure()
StructureTests->>GroveDB: create and inspect storage
StructureTests->>ConformanceReport: run check_conformance()
ConformanceReport->>StructureNode: resolve keys and validate elements
ConformanceReport-->>StructureTests: return violations and visited nodes
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 75.51% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 98 functions across 46 files. (2 skipped: 2 unsupported.) ✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
🕓 Queued for automated review — 1st in line, estimated start in ~20 min (commit e035450)
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## v4.2-dev #4845 +/- ##
=============================================
- Coverage 88.06% 76.13% -11.93%
=============================================
Files 2980 3006 +26
Lines 389888 440822 +50934
=============================================
- Hits 343348 335621 -7727
- Misses 46540 105201 +58661
🚀 New features to boost your workflow:
|
… values Two ways a valid state failed the structure conformance check. An epoch's storage fee item is created with the epoch tree and deleted when the epoch is paid out, while the tree stays. The description could only say "created with its parent", so a paid epoch reported the item as missing. Presence gains a third state, `UntilDeleted`, which the walker does not require. The proposers tree and the processing fee item, already lazy, now say that payout deletes them too. Below a contested index a 32 byte key is a contender's identity id at the last level and an index value at the levels before it. The walker chose a template by key length and kind alone, read such a value as a contender, and then reported the poll's own keys below it as undescribed. Where several templates accept a key and list the element's kind, the walker now tries each and keeps the one whose description fits what is below the element. The contender's description no longer documents the limitation. Fixtures for both: an epoch while it runs and after payout, and two DPNS contests, one with a 32 byte label. The contest fixture fails without the lookahead with the violations from the review. They also reach the epoch start fields, the proposers and the whole active poll layout, which leave the UNVERIFIED list; the poll layout written from reading the insert code turned out right. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
rustfmt gives up on a whole method chain when one string in it does not fit the line, so the description files were left as written, with chains on one line. Long descriptions now use line continuations and one long identifier became a constant, which lets rustfmt lay the chains out. No text changes: grovedb-structure.json is the same. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Both findings from the review at 1. Paid epochs. 2. 32 byte contested index values. Key length cannot tell an index value from a contender one level up, and I had documented that in the node instead of fixing it. Where several templates accept a key and list the element's kind, the walker now tries each and keeps the one whose description fits what is below the element: a contender has The two fixtures also reach the epoch start fields, the proposers and the whole active poll layout, so those leave the e035450 is formatting only: rustfmt had been skipping the description files because of over-long string literals. |
…anlist-suspension-b6a28a Describes the contract's other tree (the version item, the banlist and the suspension list) in the GroveDB structure that #4845 introduced, with a fixture that reaches the moderation lists, and regenerates grovedb-structure.json. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Issue being fixed or feature implemented
Nothing shows the whole GroveDB layout Drive builds. The book draws single subtrees, and the root tree comment in
drive/mod.rshad fallen behind (noContractGroups). There was no machine readable description at all, so nothing could draw the structure or tell what a pull request adds to it.This is the first of three parts:
packages/rs-drive/grovedb-structure.json.What was done?
drive::structure(packages/rs-drive/src/structure): the node model, a builder, a lint, a conformance walker and the export types. Each area declares its part in astructure.rsbeside itspaths.rs, from the real key constants (RootTree::Tokens as u8,TOKEN_BALANCES_KEY,IdentityRootStructure::IdentityTreeKeys as u8), so key bytes cannot drift. 215 nodes cover all 18 root trees down to the leaves.recursetargets (index levels repeat to any depth), its source file and its book chapter.structurefeature only (structure = ["server"], not indefault). drive-abci does not enable it, so the node never builds it and nothing on the block execution path can depend on it.ElementKind::ofmapsgrovedb::Elementwith an exhaustive match and no wildcard, so a GroveDB upgrade that adds an element variant fails to compile until the description knows about it.check_conformancewalks a real GroveDB layer by layer and reports every element no node describes, every kind mismatch, every node outside its protocol versions and every missing node that should exist with its parent.grovedb-structure.jsonalso records the exact Merk binary tree of every layer whose keys are all fixed (12 layers, the root included). GroveDB does not expose Merk links, but a proof does: the layer is proved with a full range query and the merk proof operations are replayed (grovedb-merkadded as a dev-dependency of rs-drive, already in the lockfile). Shapes carry their origin (genesis@14), since the shape depends on insertion order and an upgraded chain can differ.drive/mod.rsnow showsContractGroupsbelowVersions, as the recorded shape does.initialization::genesis_core_heightbecomespub(crate)so the Misc description can useGENESIS_CORE_HEIGHT_KEY.A change that adds a root tree, a subtree key or a level now fails rs-drive tests until it is described, and describing it changes the JSON, which is what the follow-up workflow keys on.
How Has This Been Tested?
cargo test -p drive --lib structure::(15 tests):sourcefile exists and names the constant a key claims to come from, no two templates of a layer can claim the same element;should_match_initial_structure_for_every_protocol_version: for protocol versions 1 to 14, builds the initial state structure and requires zero violations. This pins everysinceto whatcreate_initial_state_structurev0 to v4 and the version gates in the withdrawal structure really build;UNVERIFIEDwith the reason. The test also fails when a listed node is reached, so the list can only shrink. Still listed: token distributions, contract bound identity keys, key budgets, epoch fields written by block execution, withdrawal and asset lock contents, shielded pool contents, contested votes and contract groups;UPDATE_GROVEDB_STRUCTURE=1 cargo test -p drive --lib structure::testsrewrites it) and a test that the recorded root shape has DataContractDocuments on top with Identities and Balances below.Also:
cargo clippy -p drive --all-targetsclean,cargo check -p drive --features structure,--no-default-features --features serverand--no-default-features --features verify.Breaking Changes
None. No consensus code changes; the new module is not compiled into the node.
Checklist:
For repository code-owners and collaborators only
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Tests