Skip to content

feat(cli): Add cli storage subcommands per ADR 0016-v2#850

Merged
gtema merged 1 commit into
mainfrom
cli-raft
Jun 26, 2026
Merged

feat(cli): Add cli storage subcommands per ADR 0016-v2#850
gtema merged 1 commit into
mainfrom
cli-raft

Conversation

@gtema

@gtema gtema commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Implement backup, restore, rotate-dek, confirm-rotate-dek, and metrics
subcommands for keystone-manage storage, completing CLI coverage for all
Raft cluster operations defined in ADR 0016-v2.

  • Add Backup/Restore RPCs to ClusterAdminService (proto + server
    handlers): server-streaming Backup chunks the encrypted snapshot at
    256 KiB; client-streaming Restore accumulates chunks and calls
    install_full_snapshot
  • Expose FjallStateMachine::snapshot_dir() and decode_backup_blob() for
    use by the restore handler (validates AES-256-GCM envelope, returns
    OpenRaft Snapshot ready for install_full_snapshot)
  • Wire sm: Arc into ClusterAdminServiceImpl so the
    backup handler can call build_snapshot() and read the resulting
    encrypted file
  • New CLI commands: backup (--output), restore (--snapshot), rotate-dek
    (--emergency flag for dual-control path), confirm-rotate-dek
    (--rotation-id), metrics (leader / voters / raw OpenRaft metrics)
    • Update doc/src/raft_storage.md: add CLI reference table, Cluster
      Metrics runbook section, and fix all flag inconsistencies
      (--node-addr/--leader-addr -> --addr; positional args for
      init/join/promote)

Assisted-By: Claude Sonnet 4.6 noreply@anthropic.com
Signed-Off-By: Artem Goncharov artem.goncharov@gmail.com

Implement backup, restore, rotate-dek, confirm-rotate-dek, and metrics
subcommands for keystone-manage storage, completing CLI coverage for all
Raft cluster operations defined in ADR 0016-v2.

- Add Backup/Restore RPCs to ClusterAdminService (proto + server
  handlers): server-streaming Backup chunks the encrypted snapshot at
  256 KiB; client-streaming Restore accumulates chunks and calls
  install_full_snapshot
- Expose FjallStateMachine::snapshot_dir() and decode_backup_blob() for
  use by the restore handler (validates AES-256-GCM envelope, returns
  OpenRaft Snapshot ready for install_full_snapshot)
- Wire sm: Arc<StateMachineStore> into ClusterAdminServiceImpl so the
  backup handler can call build_snapshot() and read the resulting
  encrypted file
- New CLI commands: backup (--output), restore (--snapshot), rotate-dek
  (--emergency flag for dual-control path), confirm-rotate-dek
  (--rotation-id), metrics (leader / voters / raw OpenRaft metrics)
  - Update doc/src/raft_storage.md: add CLI reference table, Cluster
  Metrics runbook section, and fix all flag inconsistencies
  (--node-addr/--leader-addr -> --addr; positional args for
  init/join/promote)

Assisted-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-Off-By: Artem Goncharov <artem.goncharov@gmail.com>
@github-actions

Copy link
Copy Markdown

🦢 Load Test Results

Goose Attack Report

Plan Overview

Action Started Stopped Elapsed Users
Increasing 26-06-26 07:50:13 26-06-26 07:50:15 00:00:02 0 → 4
Maintaining 26-06-26 07:50:15 26-06-26 07:50:45 00:00:30 4
Decreasing 26-06-26 07:50:45 26-06-26 07:50:45 00:00:00 0 ← 4

Request Metrics

Method Name # Requests # Fails Average (ms) Min (ms) Max (ms) RPS Failures/s
GET 6960 0 16.75 11 33 232.00 0.00
Aggregated 6960 0 16.75 11 33 232.00 0.00

Response Time Metrics

Method Name 50%ile (ms) 60%ile (ms) 70%ile (ms) 80%ile (ms) 90%ile (ms) 95%ile (ms) 99%ile (ms) 100%ile (ms)
GET 14 17 21 21 22 23 25 33
Aggregated 14 17 21 21 22 23 25 33

Status Code Metrics

Method Name Status Codes
GET 6,960 [200]
Aggregated 6,960 [200]

Transaction Metrics

Transaction # Times Run # Fails Average (ms) Min (ms) Max (ms) RPS Failures/s
ListUsers
0.0 0 0 0.00 0 0 0.00 0.00
0.1 4246 0 13.66 11 25 141.53 0.00
ValidateToken
1.0 0 0 0.00 0 0 0.00 0.00
1.1 2714 0 21.68 17 33 90.47 0.00
Aggregated 6960 0 16.75 11 33 232.00 0.00

Scenario Metrics

Transaction # Users # Times Run Average (ms) Min (ms) Max (ms) Scenarios/s Iterations
ListUsers 2 4244 13.66 11 25 141.47 2122.00
ValidateToken 2 2712 21.68 18 33 90.40 1356.00
Aggregated 4 6956 16.79 11 33 231.87 3478.00

View full report

@github-actions

Copy link
Copy Markdown

🐰 Bencher Report

Branchcli-raft
Testbedubuntu-latest

🚨 2 Alerts

BenchmarkMeasure
Units
ViewBenchmark Result
(Result Δ%)
Upper Boundary
(Limit %)
Payload_encryption/unpack/set_cmdLatency
nanoseconds (ns)
📈 plot
🚷 threshold
🚨 alert (🔔)
311.83 ns
(+20.79%)Baseline: 258.16 ns
311.53 ns
(100.10%)

Raft_1Node_Latency/read/1nodeLatency
microseconds (µs)
📈 plot
🚷 threshold
🚨 alert (🔔)
1.39 µs
(+124.92%)Baseline: 0.62 µs
1.10 µs
(126.76%)

Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)
Upper Boundary
nanoseconds (ns)
(Limit %)
Command_Serde/apply/remove📈 view plot
🚷 view threshold
190,470.00 ns
(-8.62%)Baseline: 208,443.53 ns
993,156.76 ns
(19.18%)
Command_Serde/apply/set📈 view plot
🚷 view threshold
281,690.00 ns
(+44.33%)Baseline: 195,175.89 ns
856,762.52 ns
(32.88%)
Command_Serde/pack/delete📈 view plot
🚷 view threshold
121.63 ns
(+1.71%)Baseline: 119.59 ns
144.46 ns
(84.20%)
Command_Serde/pack/delete_index📈 view plot
🚷 view threshold
109.54 ns
(-1.08%)Baseline: 110.74 ns
134.95 ns
(81.17%)
Command_Serde/pack/set📈 view plot
🚷 view threshold
195.47 ns
(+1.27%)Baseline: 193.02 ns
238.27 ns
(82.04%)
Command_Serde/pack/set_index📈 view plot
🚷 view threshold
109.41 ns
(-1.35%)Baseline: 110.91 ns
134.94 ns
(81.08%)
Command_Serde/unpack/delete📈 view plot
🚷 view threshold
187.97 ns
(-1.46%)Baseline: 190.75 ns
234.04 ns
(80.31%)
Command_Serde/unpack/delete_index📈 view plot
🚷 view threshold
162.34 ns
(-1.35%)Baseline: 164.56 ns
204.04 ns
(79.56%)
Command_Serde/unpack/set📈 view plot
🚷 view threshold
274.46 ns
(+11.30%)Baseline: 246.59 ns
298.27 ns
(92.02%)
Command_Serde/unpack/set_index📈 view plot
🚷 view threshold
157.92 ns
(-2.97%)Baseline: 162.76 ns
200.20 ns
(78.88%)
Payload_encryption/pack/remove_cmd📈 view plot
🚷 view threshold
112.59 ns
(-5.41%)Baseline: 119.03 ns
153.34 ns
(73.43%)
Payload_encryption/pack/set_cmd📈 view plot
🚷 view threshold
228.87 ns
(+6.54%)Baseline: 214.83 ns
285.42 ns
(80.19%)
Payload_encryption/unpack/remove_cmd📈 view plot
🚷 view threshold
205.18 ns
(+2.13%)Baseline: 200.89 ns
246.86 ns
(83.12%)
Payload_encryption/unpack/set_cmd📈 view plot
🚷 view threshold
🚨 view alert (🔔)
311.83 ns
(+20.79%)Baseline: 258.16 ns
311.53 ns
(100.10%)

Raft_1Node_Latency/prefix/1node📈 view plot
🚷 view threshold
2,007,500.00 ns
(-41.30%)Baseline: 3,420,077.81 ns
7,026,405.79 ns
(28.57%)
Raft_1Node_Latency/read/1node📈 view plot
🚷 view threshold
🚨 view alert (🔔)
1,391.60 ns
(+124.92%)Baseline: 618.72 ns
1,097.79 ns
(126.76%)

Raft_1Node_Latency/remove/1node📈 view plot
🚷 view threshold
567,040.00 ns
(+13.88%)Baseline: 497,929.06 ns
2,187,439.70 ns
(25.92%)
Raft_1Node_Latency/write/1node📈 view plot
🚷 view threshold
596,200.00 ns
(+29.91%)Baseline: 458,917.03 ns
1,589,881.60 ns
(37.50%)
build_snapshot/default📈 view plot
🚷 view threshold
102,970.00 ns
(+5.89%)Baseline: 97,237.88 ns
172,671.19 ns
(59.63%)
fernet token/project📈 view plot
🚷 view threshold
1,423.70 ns
(+2.31%)Baseline: 1,391.60 ns
1,634.44 ns
(87.11%)
get_data_keyspace📈 view plot
🚷 view threshold
0.31 ns
(-2.42%)Baseline: 0.32 ns
0.38 ns
(81.60%)
get_db📈 view plot
🚷 view threshold
0.31 ns
(-2.73%)Baseline: 0.32 ns
0.38 ns
(81.36%)
get_fernet_token_timestamp/project📈 view plot
🚷 view threshold
172.78 ns
(+19.76%)Baseline: 144.27 ns
176.00 ns
(98.17%)
get_keyspace📈 view plot
🚷 view threshold
4.38 ns
(-7.94%)Baseline: 4.76 ns
8.80 ns
(49.79%)
🐰 View full continuous benchmarking report in Bencher

@gtema
gtema merged commit e44eb9f into main Jun 26, 2026
43 of 45 checks passed
@gtema
gtema deleted the cli-raft branch July 1, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant