Skip to content

LWDiD precursor: canonical LW paper reviews + REGISTRY entry + Prop 99 / Walmart dataset loaders#685

Merged
igerber merged 3 commits into
mainfrom
LWDiD
Jul 13, 2026
Merged

LWDiD precursor: canonical LW paper reviews + REGISTRY entry + Prop 99 / Walmart dataset loaders#685
igerber merged 3 commits into
mainfrom
LWDiD

Conversation

@igerber

@igerber igerber commented Jul 13, 2026

Copy link
Copy Markdown
Owner

Summary

  • Maintainer-side precursor to feat: add LWDiD estimator (Lee & Wooldridge 2025, 2026) #588 (LWDiD estimator), per the plan agreed there: canonical methodology notes + REGISTRY entry + dataset loaders, so the estimator branch can rebase onto a main that already carries its reference spec and data infrastructure.
  • Canonical paper reviews for both Lee-Wooldridge papers against their current SSRN revisions (docs/methodology/papers/lee-wooldridge-{2025,2026}-review.md): the June 8, 2026 revision of the transformation paper (SSRN 4516518) and the February 3, 2026 revision of the small-sample inference paper (SSRN 5325686).
  • Maintainer-authored ## LWDiD REGISTRY.md section (assumptions NAC/CPTC/OVLC + CNAS/CPTS/OVLS + CHT, transformations, RA/IPW/IPWRA, Algorithm 1 multiplier bootstrap, exact-t small-N inference layer, WATT(r) + composite aggregation, edge cases, replication targets) following the LP-DiD precedent of landing the registry entry with the paper review, ahead of the implementation.
  • load_prop99() + load_walmart() dataset loaders: download the MIT-licensed ancillary datasets of the authors' Stata lwdid package (SSC, Boston College host) - the Prop 99 smoking panel (39 states, single 1989-treated unit) and the Walmart entry county panel (1,277 counties, staggered 1986-1999, 391 never-treated; the current paper revision's sample). Binary download helper with pinned SHA-256 verification on every byte-load (host is HTTP-only), stale-cache re-download, post-download structural validation against source invariants, loud UserWarning + df.attrs["source"] marker on synthetic fallback, seeded local-RNG fallback constructors.
  • Docs surfaces per house convention: docs/api/datasets.rst, index.rst autosummary + tracked _autosummary stubs, llms.txt / llms-full.txt catalogs, docs/references.rst Lee-Wooldridge subsection, CHANGELOG entry for the loaders, TODO row for aligning the four legacy loaders with the loud-fallback pattern.

Methodology references (required if estimator / math changes)

Validation

  • Tests added/updated: tests/test_datasets.py (+12 tests, 48 passing): per-dataset fallback structure/treatment/values/size, loud-fallback warning + attrs marker, checksum-mismatch rejection + stale-cache re-download, per-invariant source-validator rejection tests against a valid-shaped fixture, clear_cache .dta behavior, load_dataset dispatch, integration fits (DifferenceInDifferences, CallawaySantAnna).
  • Backtest / simulation / notebook evidence (if applicable): real-download smoke verified both files against pinned checksums and all value-level source invariants (single 1989 cohort + 38 donors; 1,277 counties, cohorts 1986-1999, 391 never-treated).

Security / privacy

  • Confirm no secrets/PII in this PR: Yes. Downloads are checksum-pinned (SHA-256) because the SSC host is HTTP-only; datasets are MIT-licensed public research data.

Generated with Claude Code

@github-actions

Copy link
Copy Markdown

Overall Assessment

⚠️ Needs changes — two P1 methodology-documentation issues are unmitigated.

Executive Summary

  • No estimator, variance, or inference code is changed in this PR.
  • The new load_prop99() / load_walmart() loader source URLs match the SSC/RePEc lwdid sample data entries, and checksum pinning is appropriate for HTTP-only data. (ideas.repec.org)
  • P1: The new methodology files cite SSRN revision dates/page counts that do not match the public SSRN records I could access.
  • P1: The LWDiD registry hard-codes balanced-panel WATT event-study weights while also claiming unbalanced-panel support.
  • I could not run tests: this container is missing pytest and even core runtime deps such as numpy.

Methodology

P1 — Cited LW paper revisions are not traceable to public SSRN metadata

Location: docs/methodology/REGISTRY.md:L2138-L2140, docs/methodology/papers/lee-wooldridge-2025-review.md:L4-L5, docs/methodology/papers/lee-wooldridge-2026-review.md:L4-L5, docs/references.rst:L281-L285

Impact: The registry and paper reviews claim the transformation paper was reviewed as a June 8, 2026, 61-page SSRN revision and the small-N paper as a February 3, 2026 revision. The public SSRN pages currently show SSRN 4516518 as 52 pages, posted July 27, 2023, last revised December 25, 2025, date written December 24, 2025; and SSRN 5325686 as 36 pages, last revised January 3, 2026, date written January 3, 2026. That makes the new methodology contract non-reproducible against the cited public source. (papers.ssrn.com)

Concrete fix: Reconcile all LW citations/review headers/registry dates with the public SSRN metadata, or add a stable archived source for the newer PDFs and explicitly document the SSRN-metadata discrepancy.

P1 — WATT aggregation spec loses the unbalanced-panel weighting nuance

Location: docs/methodology/REGISTRY.md:L2169-L2170, conflicting nuance in docs/methodology/papers/lee-wooldridge-2025-review.md:L195

Impact: The registry specifies WATT(r) = sum_g (N_g / N_{G_r}) ATT(g, g+r), while the paper review itself notes that under unbalanced panels the operative weight is the number of treated units in cohort g contributing at event time r, not the full cohort size. Because the registry also lists unbalanced-panel support, this can lead the forthcoming implementation to use balanced-panel shorthand where contributor-count weights are required.

Concrete fix: Change the registry aggregation bullet and checklist to define omega_{g,r} using contributing treated units for event time r, with N_g / N_{G_r} documented only as the balanced-panel simplification.

Code Quality

No P0/P1 findings. The new binary download helper verifies cached and freshly downloaded bytes before use, re-downloads stale cache entries, and avoids caching checksum-mismatched fresh downloads.

Performance

No findings. The added code paths are dataset-loading paths and do not affect estimator hot loops.

Maintainability

No P1 findings. The dataset dispatch and public exports are consistently wired through diff_diff/__init__.py, list_datasets(), and load_dataset().

Tech Debt

P3 — Legacy loader fallback alignment is tracked

Location: TODO.md:L46-L52

Impact: The four older loaders still have quieter synthetic fallback behavior than the new loaders, but the PR adds a TODO entry tracking this.

Concrete fix: No approval-blocking action required; this is properly tracked deferred work.

Security

No findings. The new data downloads are HTTP, but every cached/fresh byte load is pinned by SHA-256 before parsing; the RePEc entry also confirms the sample data file URLs and MIT license. (ideas.repec.org)

Documentation / Tests

P3 — Test execution not verified in this review environment

Location: tests/test_datasets.py:L245-L541

Impact: The added tests cover fallback construction, loader dispatch, checksum mismatch, stale-cache replacement, validators, and integration examples, but I could not execute them here because required tooling/dependencies are unavailable.

Concrete fix: Run pytest -q tests/test_datasets.py in CI or a dev environment with the project dependencies installed.

Path to Approval

  1. Update the LWDiD paper-review headers, REGISTRY.md, and references.rst so the cited revision dates/page counts match the public SSRN records, or add a stable archived citation for the newer PDFs and document why SSRN metadata differs.
  2. Update the registry WATT aggregation requirement to use event-time contributor counts under unbalanced panels, with cohort-size weights only as the balanced-panel special case.

@igerber

igerber commented Jul 13, 2026

Copy link
Copy Markdown
Owner Author

Both P1s addressed in 823a756.

P1 (SSRN metadata): I re-verified the live SSRN pages today (2026-07-13):

  • 4516518: "61 Pages ● Posted: 27 Jul 2023 ● Last revised: 8 Jun 2026", Date Written: April 26, 2026 - exactly as cited in the review docs.
  • 5325686: "36 Pages ● Posted: 27 Jun 2025 ● Last revised: 13 Jun 2026". One subtlety: SSRN's author-entered "Date Written" field says January 03, 2026, while the delivered PDF's cover page is dated February 3, 2026 - our docs cite the cover date of the artifact actually reviewed.

The metadata quoted in the review (52 pages / last revised Dec 25, 2025 / Jan 3, 2026) matches the superseded December 2025 / January 2026 revisions - consistent with a stale cache or mirror of the SSRN pages. Per the suggested fallback fix, provenance is now self-documenting regardless: both paper-review headers pin the exact reviewed artifact (PDF SHA-256 + cover-page date) alongside a live-verified SSRN metadata block with an explicit stale-cache note; DOIs added to REGISTRY.md and references.rst.

P1 (WATT weights): Agreed - good catch, and it was already recorded as a nuance in our own review doc. The REGISTRY aggregation bullet and requirements checklist now define omega_{g,r} by treated units of cohort g contributing at event time r (the LW 2025 Appendix E.1 operative definition, required under unbalanced panels), with N_g / N_{G_r} documented as the balanced-panel simplification only.

P3 (test execution): expected - tests run under the label-gated CI; the 48 tests in tests/test_datasets.py pass locally in both backends' pure-Python mode.

@igerber

igerber commented Jul 13, 2026

Copy link
Copy Markdown
Owner Author

/ai-review

@github-actions

Copy link
Copy Markdown

🔁 AI review rerun (requested by @igerber)

Head SHA: 823a756f2b7cca4c753041ca460fde235b566969


Overall Assessment

Looks good — no unmitigated P0/P1 findings.

Executive Summary

  • No estimator, weighting, variance, SE, or inference code is changed in this PR.
  • The prior WATT registry issue is addressed: REGISTRY.md now requires contributor-count event-time weights for unbalanced panels.
  • The prior SSRN metadata issue is materially addressed with reviewed-PDF SHA pins and explicit live-metadata notes in the paper-review headers.
  • New dataset loaders avoid parsing untrusted bytes by SHA-256 checking before read_stata, validate source invariants, and mark synthetic fallback frames.
  • Remaining issues are non-blocking: cleanup of generated Playwright artifacts, one shorthand wording cleanup, and test execution not verified here because pytest is missing.

Methodology

P3 — Balanced-panel WATT shorthand remains in secondary docs

Location: docs/methodology/papers/lee-wooldridge-2025-review.md:L87-L91, docs/methodology/papers/lee-wooldridge-2025-review.md:L142, docs/references.rst:L281-L283

Impact: The authoritative registry now correctly uses contributing treated units for omega_{g,r} under unbalanced panels at docs/methodology/REGISTRY.md:L2171-L2173, so this is not an approval blocker. But secondary documentation still says “cohort-size-weighted” without immediately carrying the unbalanced-panel caveat, which could reintroduce confusion.

Concrete fix: Rephrase these secondary mentions to “contributing-treated-unit weighted, simplifying to cohort-size weights in balanced panels.”

Code Quality

No P0/P1 findings. The binary cache helper verifies cached and fresh .dta bytes before parsing, avoids caching checksum-mismatched downloads, and re-downloads stale cache entries.

Performance

No findings. The new paths are dataset-loading and validation paths, not estimator hot loops.

Maintainability

P2 — Generated Playwright capture artifacts are committed

Location: .playwright-mcp/console-2026-07-13T12-56-12-336Z.log, .playwright-mcp/console-2026-07-13T12-57-11-432Z.log, .playwright-mcp/page-2026-07-13T12-56-12-580Z.yml, .playwright-mcp/page-2026-07-13T12-56-30-704Z.yml, .playwright-mcp/page-2026-07-13T12-57-13-335Z.yml

Impact: These are browser-session artifacts, not library/docs source. They add noisy, timestamped external-page captures and one binary-ish log file to the repository.

Concrete fix: Remove .playwright-mcp/* from the PR and add .playwright-mcp/ to .gitignore if this tooling is used locally.

Tech Debt

P3 — Legacy loader fallback alignment is tracked

Location: TODO.md:L46-L52

Impact: The older CSV-backed loaders still have quieter fallback behavior than the new loaders, but this PR tracks the limitation explicitly.

Concrete fix: No approval-blocking action required.

Security

No findings. The new HTTP downloads are checksum-pinned before parsing, and the committed textual Playwright logs did not show secrets in the inspected output. Removing the generated artifacts is still recommended under Maintainability.

Documentation / Tests

P3 — Test execution not verified in this environment

Location: tests/test_datasets.py:L240-L647

Impact: Added tests cover fallback structures, source validators, checksum mismatch behavior, stale-cache replacement, cache clearing, dispatch, and estimator integration examples. I could not execute them here because pytest is not installed.

Concrete fix: Run python -m pytest -q tests/test_datasets.py in CI or a dev environment with project test dependencies installed.

@github-actions

Copy link
Copy Markdown

🔁 AI review rerun (requested by @igerber)

Head SHA: 3dcd9e19455358bc796af10fec63f263909b9465


Overall Assessment

Looks good — no unmitigated P0/P1 findings.

Executive Summary

  • No estimator, weighting, variance/SE, inference, or default estimator behavior is changed.
  • The prior WATT weighting concern is resolved: the registry now requires contributing-treated-unit weights, with cohort-size weights only as the balanced-panel simplification.
  • The prior Playwright artifact concern is resolved: artifacts are no longer in the changed-file list, and .playwright-mcp/ is ignored.
  • New dataset loaders checksum-verify .dta bytes before parsing, validate source invariants, and loudly mark synthetic fallbacks.
  • I could not run tests here because pytest and numpy are not installed.

Methodology

No P0/P1 findings.

The LWDiD registry is documentation-only and explicitly marks implementation requirements as unchecked until the estimator lands. The relevant WATT text is now methodologically sound at docs/methodology/REGISTRY.md:L2171-L2173, with matching secondary wording at docs/methodology/papers/lee-wooldridge-2025-review.md:L87-L91 and docs/references.rst:L281-L287.

Code Quality

No findings.

The new binary download helper verifies cached and fresh bytes before pd.read_stata, rejects checksum mismatches, and re-downloads stale cache entries: diff_diff/datasets.py:L64-L108. Public loader dispatch and exports are propagated through diff_diff/datasets.py:L1175-L1225 and diff_diff/__init__.py:L67-L77.

Performance

No findings.

The changed runtime paths are dataset loading, cache I/O, validation, and synthetic fallback generation, not estimator hot paths.

Maintainability

No findings.

The prior generated-artifact issue is addressed by ignoring .playwright-mcp/ at .gitignore:L77-L78; no .playwright-mcp/* files are present in the changed-file set.

Tech Debt

P3 — Legacy loader fallback alignment is tracked

Location: TODO.md:L46-L52

Impact: The four older dataset loaders still have quieter fallback behavior than load_prop99() / load_walmart(), but this is explicitly tracked as actionable backlog.

Concrete fix: No approval-blocking action required.

Security

No findings.

The HTTP-only .dta sources are protected by pinned SHA-256 verification before parsing, and checksum-mismatched fresh downloads are not cached: diff_diff/datasets.py:L64-L108.

Documentation / Tests

P3 — Test execution not verified in this environment

Location: tests/test_datasets.py:L245-L647

Impact: The added tests cover fallback shape/treatment behavior, source validators, checksum mismatch rejection, stale-cache replacement, cache clearing, dataset dispatch, and estimator integration examples, but I could not execute them here.

Concrete fix: Run python -m pytest -q tests/test_datasets.py in CI or a dev environment with test dependencies installed.

igerber and others added 2 commits July 13, 2026 11:53
…on rubric)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FZK3FD9jxWGxBrPDw5APSg
…D REGISTRY entry + references (precursor to #588)

Loaders download the MIT-licensed ancillary datasets of the authors' Stata
lwdid package (SSC): pinned SHA-256 verification of every byte-load (HTTP-only
host), stale-cache re-download, structural validation against source
invariants, loud UserWarning + df.attrs['source'] marker on synthetic
fallback, seeded local-RNG fallback constructors. REGISTRY.md gains the
maintainer-authored LWDiD section (E.1 contributing-unit WATT weights;
provenance pinned via reviewed-PDF SHA-256 + live-verified SSRN metadata).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FZK3FD9jxWGxBrPDw5APSg
@github-actions

Copy link
Copy Markdown

🔁 AI review rerun (requested by @igerber)

Head SHA: 4692e72274b40c6aa364fcbcb1f338940b30aaf7


Overall Assessment

Looks good — no unmitigated P0/P1 findings.

Executive Summary

  • No estimator, weighting, variance/SE, inference, identification, or default estimator behavior is changed.
  • Affected method surface is documentation/infrastructure for future LWDiD, plus two dataset loaders.
  • Prior re-review concerns remain resolved: WATT aggregation now uses contributing-treated-unit weights, and Playwright MCP artifacts are ignored.
  • The new binary loaders checksum-verify cached and downloaded bytes before parsing and loudly mark synthetic fallbacks.
  • I could not run tests here because pytest/numpy are not installed.

Methodology

No findings.

The PR adds the LWDiD registry entry and paper reviews but no estimator implementation. The registry’s WATT wording at docs/methodology/REGISTRY.md:L2171-L2173 matches the documented contributing-treated-unit interpretation, with cohort-size weights limited to the balanced-panel simplification. The public RePEc entry also supports the selected Stata lwdid package and lists lw_smoking.dta / lw_walmart.dta as sample data files. (ideas.repec.org)

Code Quality

No findings.

The loader plumbing is complete: public exports at diff_diff/__init__.py:L67-L77 and diff_diff/__init__.py:L537-L544, dispatch at diff_diff/datasets.py:L1212-L1225, and validation/fallback handling at diff_diff/datasets.py:L877-L914 and diff_diff/datasets.py:L1048-L1088.

Performance

No findings.

The added paths are dataset download/cache/validation and small synthetic fallback generation, not estimator hot paths.

Maintainability

No findings.

The source validators are explicit and covered by focused tests for duplicate rows, non-constant cohorts, missing cohorts, and wrong never-treated counts at tests/test_datasets.py:L366-L468.

Tech Debt

P3 — Legacy loader fallback alignment is tracked

Location: TODO.md:L48-L53

Impact: Older dataset loaders still have quieter fallback behavior than the new load_prop99() / load_walmart() pattern, but this is explicitly tracked as actionable backlog.

Concrete fix: No approval-blocking action required.

Security

No findings.

HTTP-only binary downloads are guarded by pinned SHA-256 checks before cache write or Stata parsing at diff_diff/datasets.py:L64-L108; checksum-mismatched fresh bytes are rejected by the helper and not cached.

Documentation / Tests

P3 — Test execution not verified in this environment

Location: tests/test_datasets.py:L245-L647

Impact: Added tests cover fallback structures, source validators, checksum mismatch rejection, stale-cache replacement, cache clearing, dispatch, and estimator-smoke paths, but I could not execute them locally because pytest and numpy are missing.

Concrete fix: Run python -m pytest -q tests/test_datasets.py in CI or a dev environment with the project test dependencies installed.

@igerber igerber added the ready-for-ci Triggers CI test workflows label Jul 13, 2026
…oc-snippet mock loaders with prop99/walmart

The doc-snippets harness replaces sys.modules['diff_diff.datasets'] with a
mock module; the new loaders' snippets need mock counterparts there. The
wrapped Returns bullet in load_prop99 broke the -W docs build (unexpected
indentation).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FZK3FD9jxWGxBrPDw5APSg
@github-actions

Copy link
Copy Markdown

🔁 AI review rerun (requested by @igerber)

Head SHA: 8587b356ac3dd36cb0b422e56bb07817c85635fb


Overall Assessment
Looks good — no unmitigated P0/P1 findings.

Executive Summary

  • Re-review focus items remain resolved: WATT aggregation is documented as contributing-treated-unit weighted, and Playwright MCP artifacts are ignored.
  • No estimator, weighting, variance/SE, inference, identification, or default estimator behavior changes are introduced.
  • Affected method surface is documentation/specification for future LWDiD, plus load_prop99() / load_walmart() dataset infrastructure.
  • The binary loaders checksum-verify bytes before parsing, validate source invariants, and loudly mark synthetic fallbacks.
  • I could not execute tests in this environment because pytest and numpy are not installed.

Methodology
No findings.

Affected method: LWDiD documentation/spec only. The Registry entry at docs/methodology/REGISTRY.md:L2136-L2220 is consistent with the added paper-review files and records the earlier WATT concern correctly at docs/methodology/REGISTRY.md:L2171-L2173.

Code Quality
No findings.

Loader export and dispatch paths are wired through diff_diff/__init__.py:L67-L77, diff_diff/__init__.py:L537-L544, and diff_diff/datasets.py:L1211-L1224.

Performance
No findings.

The added work is download/cache/validation and small synthetic fallback generation, not estimator hot-path code.

Maintainability
No findings.

Source validators and fallback tests are explicit at tests/test_datasets.py:L245-L515.

Tech Debt

P3 — Legacy loader fallback alignment is tracked

Location: TODO.md:L48-L53

Impact: Older dataset loaders still have quieter fallback behavior than the new load_prop99() / load_walmart() pattern, but the PR tracks that as actionable backlog.

Concrete fix: No approval-blocking action required.

Security
No findings.

HTTP-only .dta downloads are guarded by pinned SHA-256 checks before cache write or Stata parsing at diff_diff/datasets.py:L64-L108; .mcp.json remains ignored and .playwright-mcp/ is newly ignored at .gitignore:L77-L81.

Documentation/Tests

P3 — Test execution not verified in this environment

Location: tests/test_datasets.py:L245-L647

Impact: Added tests cover fallback structure, source validators, checksum mismatch rejection, stale-cache replacement, cache clearing, dispatch, and estimator smoke paths, but I could not run them here because pytest and numpy are missing.

Concrete fix: Run python -m pytest -q tests/test_datasets.py tests/test_doc_snippets.py in CI or a dev environment with project dependencies installed.

@igerber igerber merged commit 94c979e into main Jul 13, 2026
30 checks passed
@igerber igerber deleted the LWDiD branch July 13, 2026 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-ci Triggers CI test workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant