Skip to content

SWIP 050 - Sequential Transformation Scheme 1 - #100

Open
lat-murmeldjur wants to merge 5 commits into
ethersphere:masterfrom
lat-murmeldjur:swip_050_sts
Open

lat-murmeldjur wants to merge 5 commits into
ethersphere:masterfrom
lat-murmeldjur:swip_050_sts

Conversation

@lat-murmeldjur

Copy link
Copy Markdown

Sequential Transformation Scheme 1, or STS-1, extends Swarm redistribution from chunk-only sampling to a sequenced chunk-and-stamp proof. It keeps the existing running weighted truth-selection process, but an entry receives selection weight only after its selected stamp witnesses and STS-1 chunk-binding proofs pass.

@lat-murmeldjur lat-murmeldjur changed the title SWIP 050 Sequential Transformation Scheme 1 SWIP 050 - Sequential Transformation Scheme 1 Jul 6, 2026
@significance

Copy link
Copy Markdown
Member

I think there is an opportunity to simplify the protocol by separating density from pre-knowledge, as they are distinct security properties.

The density signal only needs to come from postage stamps. Since every valid stamp is cryptographically bound to exactly one chunk,

$$ \forall s \in S,\qquad s \mapsto c, $$

a valid stamp proof already implies the existence of its corresponding chunk. Therefore, if the required stamp density is satisfied,

$$ |S_{\mathrm{valid}}| \ge D, $$

then the existence of at least (D) qualifying chunks follows immediately. A separate chunk-density metric provides no additional information and is therefore redundant.

The chunk commitment serves a different purpose entirely. Its role is not to measure density, but to prove prior possession of the reserve. Let

$$ R_c = \mathrm{MerkleRoot}\left({T(c_i)}\right), $$

where (T) is a deterministic transformation (e.g. repeated Keccak hashing). For each sampled stamp (s_i), the participant proves

$$ s_i \rightarrow c_i, \qquad T(c_i) \in R_c. $$

This establishes that the challenged chunk belonged to the participant's committed reserve before the challenge was known, preventing adaptive fetching after witness selection.

The protocol therefore decomposes cleanly into two independent proofs:

$$ \boxed{\mathrm{Stamp\ Proofs}\Longrightarrow\mathrm{Density}} $$

$$ \boxed{\mathrm{Chunk\ Commitment}\Longrightarrow\mathrm{PreKnowledge}} $$

The chunk commitment is therefore a participant-specific commitment, not a density signal or consensus object.

As a consequence, the second source of randomness and the additional commit/reveal phase appear unnecessary. A single commitment followed by a single unpredictable challenge is sufficient:

$$ \boxed{ \mathrm{Commit} \longrightarrow \mathrm{Random\ Challenge} \longrightarrow \mathrm{Prove} } $$

rather than

$$ \mathrm{Commit} \longrightarrow \mathrm{Random} \longrightarrow \mathrm{Commit} \longrightarrow \mathrm{Random} \longrightarrow \mathrm{Prove}. $$

This preserves the anti-adaptive-storage property while significantly simplifying the protocol. It also cleanly separates the economic weighting mechanism (derived entirely from stamps) from the cryptographic proof of prior reserve possession (derived entirely from the participant's chunk commitment).

@significance

Copy link
Copy Markdown
Member

btw I also think that there may be better way to approach sampling in general - we need to not prove every segment and also could perhaps find a more efficient means of doing so

i also note the current cardinality of the sample is unworkably low and needs to be addressed and also that the consensus should be arrived at by an approach utilising the methodology outlined in my RS Fuzzy Sampling paper

@0xCardiE

Copy link
Copy Markdown
Collaborator

Is the phase structure settled?
@significance raised that density comes from stamps alone, that the chunk commitment only proves prior possession, and that the second commit/reveal round is therefore unnecessary. That hasn't been answered.

We implemented the six phase version and it works end to end, but if it collapses to Commit → Random → Prove a fair bit of the plumbing moves. Worth settling before people build against it.

@0xCardiE

Copy link
Copy Markdown
Collaborator

sampleMaxValueForDepth isn't defined, and the depth argument looks wrong
The function isn't defined anywhere in the SWIP, but the bigger issue is that it takes depth at all.

Today estimateSize compares the sample against a fixed sampleMaxValue. That fixed ceiling is what makes claiming a larger depth cost something. A node's reserve is capacity bounded so its 16th transformed value doesn't really move with depth, while stakeDensity grows as stake * 2^(depth - height).

If the ceiling scales with depth then overreporting by k bits gains 2^k on base density, L gains 2^k as well, so L/x is unchanged and the density coefficient doesn't react. Overreporting becomes free.

The Coefficient safety argument only works through underreporting. Is there an analysis for the other direction? We made the ceiling depth independent.

@0xCardiE

Copy link
Copy Markdown
Collaborator

Both coefficients are capped at 2^32, not 2x
MAX_COEFFICIENT_Q64 = (1 << 32) * Q64. With bucketDepth 16 and a depth 32 batch there are 65536 slots per bucket, so three witnesses at index 0 give an average index ratio of 1/65536, a benefit ratio of 32768 and a 32x coefficient. Both coefficients multiply.

The tables stop at 2x and the text says worse utilization isn't penalised below 1x, which reads like the useful range is 1x to 2x.

As written, the cheapest way to multiply your weight by 32 is to buy a deep batch, stamp only index 0 of each bucket and never fill it. That rewards wasting capacity, which is the opposite of the stated motivation. It also happens to be exactly what a self stamper produces, since they choose their own indexes.

We capped both at 2x. Was 2^32 deliberate?

@0xCardiE

Copy link
Copy Markdown
Collaborator

bucketDepth < claimedDepth rejects every batch we have
bucketDepth is fixed at creation and is 16 for essentially every mainnet batch. claimedDepth is already above 16 and goes up as the network grows. So this check rejects every real batch at any realistic depth and no proof could ever pass.

The neighbourhood binding it seems to be reaching for is already there twice: getPostageBucket(index) == addressToBucket(chunkAddress, bucketDepth) binds the stamp to the chunk, and inProximity(chunkAddress, firstAnchor, claimedDepth) binds the chunk to the neighbourhood.

We dropped it. Was it guarding something else?

@0xCardiE

Copy link
Copy Markdown
Collaborator

pendingCompletion is unbounded and can stop claim working permanently
finalizeIncompleteCommitments walks the whole array in one claim transaction. Entries are added on every stage one commit and removed only by a successful proof or a successful claim.

STS-1 has five stages to clear, so any round where nothing claims carries its entire commit set forward. After enough of those the first successful claim runs out of gas, which means no claim can succeed, which means the array never drains. It doesn't need an attacker, a quiet stretch on a small network gets there.

SWIP-51 Option B already caps a round at MAX_COMMITS, and its finalize freezes non-finishers when the next round's first commit lands. So an unfinished participant is always resolved within one round, from an array that's bounded by construction.

We dropped pendingCompletion and made that path treat "revealed but never proved" the same as "never revealed". Does that lose anything the carry-over list was for?

@lat-murmeldjur

lat-murmeldjur commented Sep 21, 2026

Copy link
Copy Markdown
Author

Is the phase structure settled? @significance raised that density comes from stamps alone, that the chunk commitment only proves prior possession, and that the second commit/reveal round is therefore unnecessary. That hasn't been answered.

We implemented the six phase version and it works end to end, but if it collapses to Commit → Random → Prove a fair bit of the plumbing moves. Worth settling before people build against it.

The phase structure is settled. A different phase structure would form a different proposal different SWIP etc.

density comes from stamps alone

yes however without the binding to chunks this density would only represent holding stamps, not storing chunks. The stage 1 chunk stage 2 stamp commitment is properly motivated, the second commit reveal is absolutely necessary.

Commit → Random → Prove

as stated above that would be a different proposal, perhaps @significance will create that, but for the purposes of this proposal the 6 stage setup will remain.

@lat-murmeldjur

Copy link
Copy Markdown
Author

sampleMaxValueForDepth isn't defined, and the depth argument looks wrong The function isn't defined anywhere in the SWIP, but the bigger issue is that it takes depth at all.

Today estimateSize compares the sample against a fixed sampleMaxValue. That fixed ceiling is what makes claiming a larger depth cost something. A node's reserve is capacity bounded so its 16th transformed value doesn't really move with depth, while stakeDensity grows as stake * 2^(depth - height).

If the ceiling scales with depth then overreporting by k bits gains 2^k on base density, L gains 2^k as well, so L/x is unchanged and the density coefficient doesn't react. Overreporting becomes free.

The Coefficient safety argument only works through underreporting. Is there an analysis for the other direction? We made the ceiling depth independent.

Yes. The big picture is, the appendix code suggestion is not reliable in the sense that some mistakes have not been spotted by me in the first generated version, this one of them. (in any case the text part of the proposal would be the more reliable)

In this case, the sampleMaxValue would remain unchanged, sampleMaxValueForDepth is in fact not necessary. estimateSize etc can be basically unchanged.

Okay so with coefficient safety, overreporting by +1 basically halves the usable entries, so overreporting is mostly only inhibited by the sampleMaxValue - it becomes less probable that overreporting still allows to sample low enough values if the number of usable entries is less than a minimal reserve.

@lat-murmeldjur

lat-murmeldjur commented Sep 21, 2026

Copy link
Copy Markdown
Author

Both coefficients are capped at 2^32, not 2x MAX_COEFFICIENT_Q64 = (1 << 32) * Q64. With bucketDepth 16 and a depth 32 batch there are 65536 slots per bucket, so three witnesses at index 0 give an average index ratio of 1/65536, a benefit ratio of 32768 and a 32x coefficient. Both coefficients multiply.

The tables stop at 2x and the text says worse utilization isn't penalised below 1x, which reads like the useful range is 1x to 2x.

We capped both at 2x. Was 2^32 deliberate?

Density and Utilization coefficients should be capped at 32x each, while Reported Depth based coefficient should be capped at 2^16.
2x is not the limit the table is not showing every value just an illustration of a few of the early values up until 2x

As written, the cheapest way to multiply your weight by 32 is to buy a deep batch, stamp only index 0 of each bucket and never fill it. That rewards wasting capacity, which is the opposite of the stated motivation. It also happens to be exactly what a self stamper produces, since they choose their own indexes.

The actual aim is to reward wasting utilization here. A self-stamper using every index is cheaper, than buying magnitudes more indexes and only being able to use low indexes - and still needing 2.5 million of those to be able to pass stamp density.

Wasting utilization is something the real network would exhibit more frequently, while economically rational attackers would use all indexes to minimize cost. Only using index 0 and having 128 indexes in a bucket would mean only having a handful of usable indexes ( 64 buckets for depth 10 for example - 64 usable index 0 stamps). This would require (2.5million / 64) individual batches with that size to be able to provide stamp density for depth 10 - making this extremely costly to exploit.

@lat-murmeldjur

lat-murmeldjur commented Sep 21, 2026

Copy link
Copy Markdown
Author

bucketDepth < claimedDepth rejects every batch we have bucketDepth is fixed at creation and is 16 for essentially every mainnet batch. claimedDepth is already above 16 and goes up as the network grows. So this check rejects every real batch at any realistic depth and no proof could ever pass.

The neighbourhood binding it seems to be reaching for is already there twice: getPostageBucket(index) == addressToBucket(chunkAddress, bucketDepth) binds the stamp to the chunk, and inProximity(chunkAddress, firstAnchor, claimedDepth) binds the chunk to the neighbourhood.

We dropped it. Was it guarding something else?

It's actually an important check that should be kept but it confused me as well re-reviewing it.
What this says if the reported depth is higher than bucket depth (reported depth 17 and above) we reject that, because, with bucket depth 16 and reported depth 17 only half of the bucket indexes should be valid, but it's not possible to distinguish which, so for now this is a limitation.

@lat-murmeldjur

Copy link
Copy Markdown
Author

pendingCompletion is unbounded and can stop claim working permanently finalizeIncompleteCommitments walks the whole array in one claim transaction. Entries are added on every stage one commit and removed only by a successful proof or a successful claim.

STS-1 has five stages to clear, so any round where nothing claims carries its entire commit set forward. After enough of those the first successful claim runs out of gas, which means no claim can succeed, which means the array never drains. It doesn't need an attacker, a quiet stretch on a small network gets there.

SWIP-51 Option B already caps a round at MAX_COMMITS, and its finalize freezes non-finishers when the next round's first commit lands. So an unfinished participant is always resolved within one round, from an array that's bounded by construction.

We dropped pendingCompletion and made that path treat "revealed but never proved" the same as "never revealed". Does that lose anything the carry-over list was for?

Another error, sorry once again.
What really should handle this after an unfinalized round (that has some commits not removed by proof, and with no claim to clear up), is the first commit in the next round. So when the first upcoming commit does this cleanup it cant accumulate more than 1 rounds commits.

@0xCardiE

Copy link
Copy Markdown
Collaborator

Trying to determine do we expect Redistribution to keep any funds, and the only reason Redistribution would need to hold tokens at all is the pull payout.

Was pull deliberate? claim is already a pull, someone has to send that transaction for anything
to happen. So the appendix adds a second collection step: someone calls claim, then each node
calls withdraw separately.

I would do it the other way in the implementation. PostageStamp gains
withdrawShares(beneficiaries, weights) and transfers directly. Redistribution decides the split,
PostageStamp moves the money, no funds ever sit in Redistribution.

The one thing pull buys is that a recipient reverting on receipt can't block the others. BZZ has
no transfer hooks or blocklist so that seemed acceptable, and the current contract already pushes.

Anything I'm missing?

@0xCardiE

Copy link
Copy Markdown
Collaborator

You mentioned "Reported Depth based coefficient should be capped at 2^16". I can't find it in the SWIP, only stampDensityCoefficient and utilizationCoefficient.

A 2^16 cap next to two 32x caps is a much wider range than the current coefficient safety argument covers.

Also worth knowing: Redistribution doesn't fit under EIP-170 with STS-1 inlined, at any optimizer setting. I had to move the witness verification and the Q64.64 maths into a deployed library to get under the limit. A third coefficient would add to that.

@0xCardiE

Copy link
Copy Markdown
Collaborator

Two things I wanted to check:

Is a maximum reported depth an intended constraint of this proposal, or is there a path past it
later? Worth stating either way so client and batch tooling know.

Also the check is against the full reported depth rather than depth minus height. A node using
height reports a depth larger than its neighbourhood responsibility, so it reaches the ceiling
earlier than a height-0 node covering the same neighbourhood?

@0xCardiE

Copy link
Copy Markdown
Collaborator

STS-1 drops chunk sample witnesses, so nothing checks chunkSampleHash against anything. A node
that can pass the stamp proofs, which needs real batches and real chunks but says nothing about
the chunk sample, can put an arbitrary value there. If its weight wins the truth draw then every
honest node in the neighbourhood disagrees and gets frozen while it takes the pot. Today that
isn't possible, the winner has to open three chunk witnesses against winner.hash.

The Rationale says it stays because it coordinates the chunk side convention among honest nodes.
That's a benefit, but being freezable on a value nobody checks is a cost, and Security
considerations doesn't mention it.

Keeping one chunk witness would close it. So would dropping chunkSampleHash from the truth tuple
and letting stampSampleHash carry the Schelling point alone.

@0xCardiE

Copy link
Copy Markdown
Collaborator

claim has no msg.sender check, same as before, but the payoff changed. The winner used to take
the whole pot so it had an obvious reason to pay the gas. Now it splits N ways, so the caller
pays full gas for about a quarter of it at redundancy 4, an eighth at 8, or nothing at all if
they aren't a beneficiary.

If nobody calls, currentClaimRound doesn't advance, nothing is paid and the pot rolls forward.
Nobody is frozen for it, but everyone who proved loses that round.

Probably fine with a healthy pot, but every client needs a policy or they all wait for each
other. Is it meant to be any matching participant, first prover, lowest overlay?

@0xCardiE

Copy link
Copy Markdown
Collaborator

chunkTransformRoot is fixed in stage one, but the stamp indexes are only derived once the stamp
anchor exists. If you build the tree in discovery order the root changes between those two points
and every membership proof fails. Leaves have to be in ascending order. I hit this in
implementation and every client will.

One sentence covering the tree construction would do: leaves ascending, parent is keccak of the
sorted pair, odd node promoted.

Separately, the text says the root is over the "complete" list of the reserve but nothing checks
completeness. Fine in practice since you can't predict what's useful, but "complete" probably
shouldn't be normative when it isn't enforced.

@0xCardiE

Copy link
Copy Markdown
Collaborator

The proof seed mixes block.prevrandao at the block of the first valid stamp reveal. Whoever
reveals first picks that block, so they can look at the resulting seed, see which positions would
be opened, and not send the transaction, retrying later in the nineteen block window.

The sample is already committed so this doesn't let anyone invent one. It does let a node with a
partly supported sample steer away from its weak positions, which is what that section says it
prevents.

Accumulating prevrandao across all stamp reveals, the way updateRandomness already does for the
round seed, would remove it.

@lat-murmeldjur

Copy link
Copy Markdown
Author

Trying to determine do we expect Redistribution to keep any funds, and the only reason Redistribution would need to hold tokens at all is the pull payout.

Was pull deliberate? claim is already a pull, someone has to send that transaction for anything to happen. So the appendix adds a second collection step: someone calls claim, then each node calls withdraw separately.

I would do it the other way in the implementation. PostageStamp gains withdrawShares(beneficiaries, weights) and transfers directly. Redistribution decides the split, PostageStamp moves the money, no funds ever sit in Redistribution.

The one thing pull buys is that a recipient reverting on receipt can't block the others. BZZ has no transfer hooks or blocklist so that seemed acceptable, and the current contract already pushes.

Anything I'm missing?

Yes actually the separate collection step is unnecessary - with claim we would ideally send "withdraw" transaction to all participants that reported the selected truth - proportionate to stake size.

Indeed redistribution does not need to hold funds.

@lat-murmeldjur

Copy link
Copy Markdown
Author

You mentioned "Reported Depth based coefficient should be capped at 2^16". I can't find it in the SWIP, only stampDensityCoefficient and utilizationCoefficient.

A 2^16 cap next to two 32x caps is a much wider range than the current coefficient safety argument covers.

Also worth knowing: Redistribution doesn't fit under EIP-170 with STS-1 inlined, at any optimizer setting. I had to move the witness verification and the Q64.64 maths into a deployed library to get under the limit. A third coefficient would add to that.

Reported depth based coefficient is basically unchanged compared to the currently deployed contracts. The reason for the 2^16 is because of the bucket depth - with current bucket depth 16 the maximum reported storage depth we can accept is also 16 (where stamp density can be strictly checked still). Therefore 2^"reported depth" can maximum be 2^16.

Contract size fitting - good to know. Like Q64.64 is one option, but I'm not sure if any other alternatives would help bringing that down.

@lat-murmeldjur

lat-murmeldjur commented Sep 22, 2026

Copy link
Copy Markdown
Author

Two things I wanted to check:

Is a maximum reported depth an intended constraint of this proposal, or is there a path past it later? Worth stating either way so client and batch tooling know.

Also the check is against the full reported depth rather than depth minus height. A node using height reports a depth larger than its neighbourhood responsibility, so it reaches the ceiling earlier than a height-0 node covering the same neighbourhood?

Yes, the maximum reported storage depth is 16 - because of the current network wide bucket depths make stamps unambigous until this depth.

Good question about height. For any purposes we are not interested whether a node is storing another neighborhoods content - it should report the storage depth of a single neighborhood, and the "height" should only allow it to participate in the sybling neighborhood with one less Proximity Order to the anchor than the storage depth would require.

So PO(overlay, anchor) >= reported depth - height

Edit: the reported depth based coefficient is also effected by height, in fact it should remain as it is now, 2^(reported depth-height)

@lat-murmeldjur

lat-murmeldjur commented Sep 22, 2026

Copy link
Copy Markdown
Author

STS-1 drops chunk sample witnesses, so nothing checks chunkSampleHash against anything. A node that can pass the stamp proofs, which needs real batches and real chunks but says nothing about the chunk sample, can put an arbitrary value there. If its weight wins the truth draw then every honest node in the neighbourhood disagrees and gets frozen while it takes the pot. Today that isn't possible, the winner has to open three chunk witnesses against winner.hash.

The Rationale says it stays because it coordinates the chunk side convention among honest nodes. That's a benefit, but being freezable on a value nobody checks is a cost, and Security considerations doesn't mention it.

Keeping one chunk witness would close it. So would dropping chunkSampleHash from the truth tuple and letting stampSampleHash carry the Schelling point alone.

No we don't want or need chunk witness. The aim of having the chunk sample at all is only to motivate following stamp rewrites.
If a stamp was already used, and the uploader reuses it for a different chunk, a node that has the old version of the stamp plus old chunk would be able to produce the same stamp sample as other nodes. However, because of this chunk sample being part of the schelling point, it now risks that using the old version of a chunk instead of the newest makes it fall off the schelling point of the honest nodes. Intentionally putting a random hash there is also leading to falling off the schelling point, and that is fine, because the aim is that the only safe strategy is trying to stay on the schelling point, anything else increases the risk of being frozen by other honest nodes that are polling their stakes by staying on the same schelling point.

@lat-murmeldjur

Copy link
Copy Markdown
Author

claim has no msg.sender check, same as before, but the payoff changed. The winner used to take the whole pot so it had an obvious reason to pay the gas. Now it splits N ways, so the caller pays full gas for about a quarter of it at redundancy 4, an eighth at 8, or nothing at all if they aren't a beneficiary.

If nobody calls, currentClaimRound doesn't advance, nothing is paid and the pot rolls forward. Nobody is frozen for it, but everyone who proved loses that round.

Probably fine with a healthy pot, but every client needs a policy or they all wait for each other. Is it meant to be any matching participant, first prover, lowest overlay?

Good question, this is kind of undesigned - who should call the claim from all of the beneficiaries. For now we can probably leave it as it is - since everybody who is winning is motivated to make sure this is done, we can introduce a "convention view function" that tells a bee node whether it should call the claim - but still not restricting to that one participant only. Convention can be anything that is cheap to calculate, last node who proved from the winners sounds like one option.

@lat-murmeldjur

Copy link
Copy Markdown
Author

chunkTransformRoot is fixed in stage one, but the stamp indexes are only derived once the stamp anchor exists. If you build the tree in discovery order the root changes between those two points and every membership proof fails. Leaves have to be in ascending order. I hit this in implementation and every client will.

One sentence covering the tree construction would do: leaves ascending, parent is keccak of the sorted pair, odd node promoted.

Separately, the text says the root is over the "complete" list of the reserve but nothing checks completeness. Fine in practice since you can't predict what's useful, but "complete" probably shouldn't be normative when it isn't enforced.

The order of chunks behind chunkTransformRoot is not strictly requiring any ordering (and in fact sorting can create more work).
The bee node can create the chunkTransformRoot in a random-discovery-order of chunks in the reserve that have a valid usable stamp by swip49 standards. While this chunkTransformRoot is committed, the file itself can be kept in memory/disc until the round is over.

Later the stamp sampling (also using the same range of stamps valid and usable by swip49 standards) gets the 16 lowest transformed stamp addresses by randomness 2. Now to find the matching entries in the chunktransform file, one only needs to create their transformed chunk addresses of the corresponding chunks of these 16 stamps using the randomness 1 and find the resulting transformed chunk addresses in the file for which the roottransformedhash was committed - by just checking where that transformed address is in the file, iterating through the 32 byte segments.

Also this can be left for optimization by sorting, or by recording the list of stamps in the same order when doing the initial chunk transformation iteration, and finding their corresponding transformed chunk address in stage 2 based on the index of the stamp in this array.

@lat-murmeldjur

Copy link
Copy Markdown
Author

Separately, the text says the root is over the "complete" list of the reserve but nothing checks completeness. Fine in practice since you can't predict what's useful, but "complete" probably shouldn't be normative when it isn't enforced.

Yes, good insight. Completeness is motivated by higher coefficients coming from higher density samples, and the aforementioned not being able to predict what will become useful.

@lat-murmeldjur

Copy link
Copy Markdown
Author

The proof seed mixes block.prevrandao at the block of the first valid stamp reveal. Whoever reveals first picks that block, so they can look at the resulting seed, see which positions would be opened, and not send the transaction, retrying later in the nineteen block window.

The sample is already committed so this doesn't let anyone invent one. It does let a node with a partly supported sample steer away from its weak positions, which is what that section says it prevents.

Accumulating prevrandao across all stamp reveals, the way updateRandomness already does for the round seed, would remove it.

The reason why the first reveal is the source of the randomness - because that can be short-circuited by anyone who reveals early, removing any more entropy.

However always using a new randomness on a new reveal would allow as many rerandomizations as many reveals can be done - so one could have a number of commits and only reveal until a benefitting seed is created and accept the freeze for the non revealing identities. This would allow much more manipulation.

@0xCardiE

Copy link
Copy Markdown
Collaborator

STS-1 drops chunk sample witnesses, so nothing checks chunkSampleHash against anything. A node that can pass the stamp proofs, which needs real batches and real chunks but says nothing about the chunk sample, can put an arbitrary value there. If its weight wins the truth draw then every honest node in the neighbourhood disagrees and gets frozen while it takes the pot. Today that isn't possible, the winner has to open three chunk witnesses against winner.hash.
The Rationale says it stays because it coordinates the chunk side convention among honest nodes. That's a benefit, but being freezable on a value nobody checks is a cost, and Security considerations doesn't mention it.
Keeping one chunk witness would close it. So would dropping chunkSampleHash from the truth tuple and letting stampSampleHash carry the Schelling point alone.

No we don't want or need chunk witness. The aim of having the chunk sample at all is only to motivate following stamp rewrites. If a stamp was already used, and the uploader reuses it for a different chunk, a node that has the old version of the stamp plus old chunk would be able to produce the same stamp sample as other nodes. However, because of this chunk sample being part of the schelling point, it now risks that using the old version of a chunk instead of the newest makes it fall off the schelling point of the honest nodes. Intentionally putting a random hash there is also leading to falling off the schelling point, and that is fine, because the aim is that the only safe strategy is trying to stay on the schelling point, anything else increases the risk of being frozen by other honest nodes that are polling their stakes by staying on the same schelling point.

That makes sense, thanks.

One thing that might be worth a line in Security considerations. With chunk witnesses a majority
attacker still had to commit to a provable sample, so honest nodes with the same reserve agreed
with it and weren't frozen. Now a majority attacker can commit garbage and guarantee everyone
else falls off. It needs majority weight in the neighbourhood, which already enables other
griefing, so it's a narrowing of an existing assumption rather than a new attack. But the
assumption did change.

@0xCardiE

Copy link
Copy Markdown
Collaborator

Two things I wanted to check:
Is a maximum reported depth an intended constraint of this proposal, or is there a path past it later? Worth stating either way so client and batch tooling know.
Also the check is against the full reported depth rather than depth minus height. A node using height reports a depth larger than its neighbourhood responsibility, so it reaches the ceiling earlier than a height-0 node covering the same neighbourhood?

Yes, the maximum reported storage depth is 16 - because of the current network wide bucket depths make stamps unambigous until this depth.

Good question about height. For any purposes we are not interested whether a node is storing another neighborhoods content - it should report the storage depth of a single neighborhood, and the "height" should only allow it to participate in the sybling neighborhood with one less Proximity Order to the anchor than the storage depth would require.

So PO(overlay, anchor) >= reported depth - height

Edit: the reported depth based coefficient is also effected by height, in fact it should remain as it is now, 2^(reported depth-height)

Worth stating it in the proposal body rather than only in this thread, since client and batch tooling both need to know
the ceiling exists and why.

@lat-murmeldjur

lat-murmeldjur commented Sep 22, 2026

Copy link
Copy Markdown
Author

STS-1 drops chunk sample witnesses, so nothing checks chunkSampleHash against anything. A node that can pass the stamp proofs, which needs real batches and real chunks but says nothing about the chunk sample, can put an arbitrary value there. If its weight wins the truth draw then every honest node in the neighbourhood disagrees and gets frozen while it takes the pot. Today that isn't possible, the winner has to open three chunk witnesses against winner.hash.
The Rationale says it stays because it coordinates the chunk side convention among honest nodes. That's a benefit, but being freezable on a value nobody checks is a cost, and Security considerations doesn't mention it.
Keeping one chunk witness would close it. So would dropping chunkSampleHash from the truth tuple and letting stampSampleHash carry the Schelling point alone.

No we don't want or need chunk witness. The aim of having the chunk sample at all is only to motivate following stamp rewrites. If a stamp was already used, and the uploader reuses it for a different chunk, a node that has the old version of the stamp plus old chunk would be able to produce the same stamp sample as other nodes. However, because of this chunk sample being part of the schelling point, it now risks that using the old version of a chunk instead of the newest makes it fall off the schelling point of the honest nodes. Intentionally putting a random hash there is also leading to falling off the schelling point, and that is fine, because the aim is that the only safe strategy is trying to stay on the schelling point, anything else increases the risk of being frozen by other honest nodes that are polling their stakes by staying on the same schelling point.

That makes sense, thanks.

One thing that might be worth a line in Security considerations. With chunk witnesses a majority attacker still had to commit to a provable sample, so honest nodes with the same reserve agreed with it and weren't frozen. Now a majority attacker can commit garbage and guarantee everyone else falls off. It needs majority weight in the neighbourhood, which already enables other griefing, so it's a narrowing of an existing assumption rather than a new attack. But the assumption did change.

Yes that's a valid observation. The reason why we don't want to keep chunk witnesses, is because they are expensive (more inclusion proofs) and even if we would require them, an attacker that really wants to step off the schelling point and disagree could still create a different chunk sample anyway with little difficulty. Swip39 combined with fixed stakes should make it more difficult to achieve the majority weight.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants