Skip to content

Prove the JustDummies generator invariants with a property-based suite#306

Merged
Reefact merged 7 commits into
mainfrom
claude/property-based-testing-dummies-vefjvh
Jul 26, 2026
Merged

Prove the JustDummies generator invariants with a property-based suite#306
Reefact merged 7 commits into
mainfrom
claude/property-based-testing-dummies-vefjvh

Conversation

@Reefact

@Reefact Reefact commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

JustDummies claims that every value it builds satisfies every constraint declared on it, for every legal combination of constraint arguments — a universal quantification. The example suite proved that claim by sampling: it pinned the constraint arguments (Between(10, 20), WithLength(12)) and varied only the drawn values, instantiating the claim at a handful of hand-picked points. This adds JustDummies.PropertyTests, which generates the constraint arguments too and quantifies over them, and moves the invariants that belong there out of the example suite.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Refactoring
  • Analyzer / diagnostic change
  • Tests
  • Documentation
  • Build / CI / tooling

Changes

  • Added JustDummies.PropertyTests (FsCheck + xUnit v3), nested under the tests solution folder and added to the .NET Framework 4.7.2 floor job in ci.yml.
  • 210 properties across twelve families: interval algebra for every integer width and the continuous types, lattice constraints (MultipleOf, WithScale, WithGranularity — whose anchor differs per type), string shape, collections, composition, explicit pools, temporal types with the offset dimension, seeded determinism, URIs, and the net8-only generators.
  • Regular-expression round trip: patterns are generated recursively over the supported subset — nested alternation, named and non-capturing groups, negated classes, ranges, lazy and bounded quantifiers, \x/\u escapes — and every value drawn from one is checked against the real .NET engine as oracle. Sampling the generator shows 100 distinct patterns per run averaging ~82 characters, so this fuzzes the home-grown engine rather than re-pinning a corpus.
  • Shared PropertyTestSupport.cs: ordered bound pairs, edge-biased numeric generators (FsCheck's own cluster near zero, so the domain ends would almost never be drawn), and per-case multi-draw helpers.
  • Pruned JustDummies.UnitTests from 535 to 519 tests, removing only what a property strictly subsumes. Each pruned class carries a note saying which half of its contract it keeps.
  • Two draws assembled a 64-bit word by or-ing shifted fields (CS0675); the fields are disjoint, so they are summed instead.

Testing

  • dotnet build FirstClassErrors.sln
  • dotnet test FirstClassErrors.sln
  • Analyzer tests pass (FirstClassErrors.Analyzers.UnitTests)

Additionally run locally, since a plain dotnet test does not cover them:

  • dotnet build JustDummies.PropertyTests -c Release -f net472 -p:EnableNet472Floor=true — succeeds, as does the same for JustDummies.UnitTests.
  • Negative control: a deliberately false property was added, confirmed to fail, and removed — the harness detects a broken property rather than passing vacuously.

Whole solution green, no warnings. JustDummies.UnitTests 519, JustDummies.PropertyTests 210.

Documentation

  • Public API / error documentation updated
  • README / doc/ updated
  • French translation (doc/handwritten/for-users/README.fr.md) updated if user-facing behavior changed
  • No documentation change required

No public API changed and no user-facing behavior changed, so the user-facing README and its French translation are untouched. New maintainer documentation:

  • doc/handwritten/for-maintainers/WritingJustDummiesTests.{en,fr}.md — where a new test belongs and how to write it, reduced to one question: does the assertion have an input space? Indexed in the maintainer README.
  • Pointers from CONTRIBUTING.md, CLAUDE.md and AGENTS.md, so the rule is found before a test is written rather than after.

Architecture decisions

  • No architectural decision in this pull request
  • New decision recorded — ADR-0040, Split the JustDummies test bed between an example suite and a property suite
  • Supersedes an existing ADR — successor proposed, status not flipped: ADR-____
  • ⚠️ Conflicts with an existing ADR — flagged for the maintainer: ADR-____

Drafted as Proposed; the maintainer then explicitly instructed acceptance, applied in bee8c2c (status flipped in both language versions and in the index, Date unchanged).

Two notes for the reviewer:

  • Adding a *.PropertyTests project is not what the ADR records — neither existing one has an ADR. What it records is the boundary, and the fact that tests are actively moved out of the example suite.
  • ADR-0040 leaves one follow-up: the July audit flagged that ADR-0025 cites "a property test" against the real regex engine while what existed was a fixed-seed oracle test. That claim is now literally true, but ADR-0025 is Accepted and is left untouched — only the maintainer amends or supersedes it.

Reviewer notes — deliberate limits

  • The pruning is conservative: 3 files of 26. The signed/unsigned suites interleave a sampled invariant with a conflict-message assertion inside the same method, so removing the loop would take the assertion with it. Splitting them is a rewrite, not a prune, and was not done unprompted.
  • The dated regression for Dummies: AnyDecimal never generates the upper half of a range #206 was kept, although the new property covers the same ground over all intervals. The property proves the class of defect; the regression pins the coordinates where it actually occurred. ADR-0040 records this as the rule.
  • Some properties were deliberately omitted rather than guessed: OneOf combined with an interval (the conflict surface there is value-dependent), and strict bounds on decimal near MaxValue (the 1E-28 exclusion nudge vanishes in rounding there — documented behavior, not a defect).

Related issues

Relates to #206 — the decimal interval defect whose class is now quantified rather than pinned to one interval.


Generated by Claude Code

claude added 7 commits July 26, 2026 16:25
The example-based suite pins hand-picked constraint arguments -- Between(10,
20), WithLength(12) -- and can only prove a generator right for those. The
sampled loops it wraps them in are property tests in spirit, but the constraint
space itself stays fixed, so a bound that overflows for one interval in a
million is never drawn. Issue #206 was exactly that: a decimal interval whose
draws never left the lower half, found by hand and pinned as a seeded
regression.

JustDummies.PropertyTests inverts the split. FsCheck generates the CONSTRAINTS
-- the bounds, lengths, counts and seeds a caller declares -- while JustDummies
generates the value that must satisfy them, and the property asserts the
invariant over the whole space. A failure shrinks to its minimal
counter-example instead of surfacing as an opaque sample. Drawing the
constraints from FsCheck also breaks a circularity, since the suite no longer
uses the component under test to decide what to test it with.

The project mirrors its siblings: JustDummies only, so the library's standalone
boundary stays visible in its own test bed, and the net472 floor leg so the
invariants are proven against the netstandard2.0 asset .NET Framework consumers
actually load.

This commit carries the scaffold, the shared constraint generators, and the
Int32 interval family as the exemplar the remaining families follow.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019v9nNWTjPcLdv3K54adsWr
JustDummies claims that every value it builds satisfies every constraint
declared on it, for every legal combination of constraint arguments -- a
universal quantification. The example suite proves that claim by sampling: it
pins the constraint arguments and varies only the drawn values, so it
instantiates the claim at a handful of hand-picked points. Issue #206 lived in
the dimension that leaves unexplored, the relation between an arbitrary bound
and the produced value.

Adding a sibling *.PropertyTests project is established practice here and
needed no record on its own -- neither existing one has an ADR. What is new and
lasting is the BOUNDARY, and the fact that tests are actively moved out of the
example suite: properties own invariants quantified over generated constraint
arguments, examples own message content, argument validation, structural
conventions and dated regressions. That rule is what a future maintainer will
question when the same bound appears tested in two places, and it survives any
reorganization of the files themselves.

Drafted as Proposed and indexed; only the maintainer accepts an ADR.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019v9nNWTjPcLdv3K54adsWr
Four families following the Int32 exemplar, each quantifying over generated
constraint arguments rather than hand-picked ones.

Scalars cover the integer widths other than Int32, with the invariant set
spread so every one is proven on at least one signed and one unsigned width.
Continuous covers Double, Single and Decimal, including the both-halves
reachability that issue #206 violated -- now asserted over arbitrary intervals
instead of the single [0, 100] the seeded regression pins. Lattice covers
MultipleOf, WithScale and WithGranularity, whose per-type anchor (zero,
DateTime.MinValue, DateTimeOffset.MinValue) is the classic place to get the
grid wrong. String covers the length, affix, charset and casing algebra,
including the value-dependent legality ADR-0035 describes: WithLength(n) with
StartingWith(prefix) is valid exactly when n is at least the prefix length, and
the property branches on that relation rather than assuming either outcome.

87 properties pass on net10.0; the project also compiles on the net472 floor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019v9nNWTjPcLdv3K54adsWr
Completes the property suite at 210 properties across twelve families.

The regular-expression round trip is the centrepiece: patterns are generated
recursively over the supported subset -- nested alternation, named and
non-capturing groups, negated classes, ranges, lazy and bounded quantifiers,
\x and \u escapes -- and every value drawn from one is checked against the real
.NET engine as oracle. Sampling the generator shows 100 distinct patterns per
run averaging ~82 characters, so this fuzzes the home-grown engine rather than
re-pinning a corpus. ADR-0025's safety net is now literally a property test,
which is what the July audit asked for.

Seeds are quantified rather than hand-picked: the example suite pins 12345, 777
and 31415, which says nothing about seeds in general. Collections quantify the
count algebra including the eager cardinality conflict, where a distinct count
exceeding what the element generator advertises must be refused at declaration
time. Composition covers As, OrNull, Combine and the explicit pools, including
that ElementOf materializes a lazy source exactly once. Temporal covers the
offset dimension over arbitrary whole-minute offsets within the legal range.
Modern types cover the net8-only generators and are excluded from the net472
leg by the project file.

Two draws assembled a 64-bit word by or-ing shifted fields, which the compiler
reads as or-ing a sign-extended operand (CS0675). The fields are disjoint, so
they are summed instead: same word, no warning.

210 properties pass on net10.0 with no warnings; the suite also compiles on the
net472 floor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019v9nNWTjPcLdv3K54adsWr
Sixteen tests asserted an invariant that holds for every legal constraint
argument while pinning exactly one: Between containment and pinning, bound
strictness and inclusiveness, OneOf and Except membership, recipe immutability,
crossed-argument rejection, non-finite argument rejection, and seed agreement
pinned to 12345, 777 and 31415. Each is now quantified over generated arguments
in JustDummies.PropertyTests, which proves strictly more, so keeping the pinned
version would assert a special case of a property one file away (ADR-0040).

Removed only what a property strictly subsumes. Everything whose subject is a
specific named case stays: conflict messages, which name both offending
constraints in direction-aware wording; null and empty argument guards; the
named domain extremes; that a bounded range is actually reached, which is
statistical rather than universal; and the seeded regression for issue #206,
which pins the interval where the defect occurred and is not retired by the
property covering the class.

Nothing was removed from the signed, unsigned or remaining suites: their
methods interleave a sampled invariant with a conflict-message assertion, so
removing the loop would take the assertion with it. Each pruned class now
carries a note saying which half of its contract it keeps, so the absence reads
as a decision rather than an oversight.

JustDummies.UnitTests 535 -> 519; JustDummies.PropertyTests adds 210.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019v9nNWTjPcLdv3K54adsWr
ADR-0040 records WHY the test bed is split; a contributor adding a constraint
or fixing a defect needs the HOW. The guide reduces the boundary to one
question -- does the assertion have an input space? -- and applies it: bounds,
lengths, counts, seeds and patterns are input spaces and belong to a property;
message wording, null arguments, named domain extremes, structural conventions
and dated regressions are not, and stay examples.

It also carries the two disciplines that keep such a suite honest and that no
ADR would hold: pin a seed for anything statistical, since "both halves are
reached" is probabilistic and flakes without one; and break each property on
purpose once to confirm it can go red, since a property returning true for the
wrong reason passes silently and proves nothing.

Indexed in the maintainer README and pointed to from CONTRIBUTING, CLAUDE and
AGENTS, so it is found before a test is written rather than after -- the same
reason the solution-nesting reminder was added to those files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019v9nNWTjPcLdv3K54adsWr
ADR-0040 ("Split the JustDummies test bed between an example suite and a
property suite") records the boundary the property suite and the pruning of the
example suite implement. The maintainer accepts the decision; the Date stays
2026-07-26, the day it took effect.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019v9nNWTjPcLdv3K54adsWr
@Reefact
Reefact merged commit 9fc624a into main Jul 26, 2026
16 checks passed
@Reefact
Reefact deleted the claude/property-based-testing-dummies-vefjvh branch July 26, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants