Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 86 additions & 13 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,44 @@
# Working in this fork

`hotdata-dev/liquid-cache` is a fork of `datafusion-contrib/liquid-cache`.
`main` is upstream's `0033b15` with our patches on top — upstream's history is
fully contained, so `git merge-base main <upstream commit>` resolves.
`main` contains upstream's history in full plus our patches, so
`git merge-base main <any upstream commit>` resolves.

Nothing here states where upstream currently is, or where we are: both move.
Ask git instead.

```
git fetch https://github.com/datafusion-contrib/liquid-cache main
git log --oneline FETCH_HEAD..main # ours that upstream does not have
git log --oneline main..FETCH_HEAD # upstream's that we do not have
```

`AGENTS.md` and `README.md` are upstream's and describe the project itself.
This file is ours and describes only what differs here. Nothing in this repo
should edit an upstream-owned file to record a fork convention: that conflicts
on every sync. Add a file upstream does not have instead.

## Remotes

**Check before using a remote name. They vary by clone and they are not what
you would guess** — in at least one working copy `origin` is *upstream* and the
fork is a second remote named `fork`, which is the reverse of the usual
arrangement.

```
git remote -v
```

Commands below name repositories by URL rather than by remote, so they are
correct in any clone. Do the same when writing instructions for anyone else: a
bare `origin/main` is ambiguous here and has already been misread as the
opposite of what it meant.

## Branches

- Work off `main`. Fetch first — a local `main` goes stale with no signal, and
branching off a stale one silently drops everything merged since. The remote
is `origin` in a fresh clone; if you cloned upstream and added this fork as a
second remote, use that name instead.
- Work off `main`, and fetch before branching — a local `main` goes stale with
no signal, and branching off a stale one silently drops everything merged
since.
- **Upstream PRs branch from upstream, not from `main`**, and are named
`upstream/<topic>`. A branch cut from `main` carries our whole patch stack
into the PR diff.
Expand All @@ -26,8 +50,9 @@ on every sync. Add a file upstream does not have instead.

Before pushing, this must list only the commits you wrote — anything else is
a fork patch that would land in the upstream diff. Re-fetch upstream on the
line above it: `FETCH_HEAD` holds whatever the last fetch wrote, so after a
`git fetch origin` it is *our* `main` and the check hides every fork patch.
line above it: `FETCH_HEAD` holds whatever the last fetch wrote, so after
fetching any other remote it is no longer upstream and the check hides every
fork patch.

```
git fetch https://github.com/datafusion-contrib/liquid-cache main
Expand All @@ -46,16 +71,64 @@ on every sync. Add a file upstream does not have instead.
have (`DiskResidue`, `reclaim_orphaned_disk`, `settle`) and are not
upstreamable at all.

## Syncing from upstream

Merge upstream into a branch off `main` and raise a PR; do not use GitHub's
"Sync fork" button, which offers to discard our commits when the merge is not
a fast-forward.

Fetch *this fork* first and branch from what came back, not from local `main`
— the upstream fetch below never refreshes `main`, so a stale one would make
the sync PR revert fork commits merged since. Use each `FETCH_HEAD`
immediately: it holds only the last fetch.

```
git fetch https://github.com/hotdata-dev/liquid-cache main
git checkout -b sync/upstream-<date> FETCH_HEAD

git fetch https://github.com/datafusion-contrib/liquid-cache main
git merge FETCH_HEAD
```

**Merge this PR, do not squash it.** A squash gives the result a single parent,
so upstream's history never enters `main`'s ancestry: the merge-base does not
move, the same upstream commits stay missing, and nothing reports it.

Verify afterwards. The merge happens on GitHub, so local `main` does not have
it and must not be what you check; and `FETCH_HEAD` holds only the last fetch,
so capture upstream before fetching the fork over it:

```
git fetch https://github.com/datafusion-contrib/liquid-cache main
upstream=$(git rev-parse FETCH_HEAD)
git fetch https://github.com/hotdata-dev/liquid-cache main
git merge-base --is-ancestor "$upstream" FETCH_HEAD && echo ok
```

A change we contributed upstream comes back as their squash of it. The content
matches but the commit does not, so the merge conflicts where both sides
touched the same lines — typically a module list that each side appended to.
Keep ours *for the returned change*, and keep any other upstream edit in the
same hunk: upstream may have appended something of its own next to it, and
taking the whole hunk from our side drops that silently. Read the hunk rather
than resolving by rule.

Sync promptly rather than letting such a conflict wait: alone it is obvious,
bundled with real upstream work later it is not.

## Building and testing

- **`cargo +1.96.0`.** The dependency tree needs 1.95+ (`vortex-*`, `sysinfo`)
and DataFusion 55 needs 1.94. A bare `cargo` on an older default fails
resolution with a wall of `requires rustc 1.9x` lines.
- **Pin the toolchain: `cargo +1.96.0 ...`.** There is no
`rust-toolchain.toml`, so a bare `cargo` uses whatever default is installed,
and an older one fails resolution with a wall of `requires rustc 1.9x` lines
naming `vortex-*` and `sysinfo`. Those lines state the minimum each crate
wants; use a toolchain at least that new.
- **Shuttle tests need a filter**: `cargo +1.96.0 test -p liquid-cache
--features shuttle --lib shuttle_`. The feature swaps `crate::sync` to
shuttle primitives for the whole test build, so running it unfiltered fails
~49 unrelated tests with "Are you accessing a Shuttle primitive outside of a
Shuttle test?". That is by design, not a regression.
every test that touches a lock outside a shuttle runner, with "Are you
accessing a Shuttle primitive outside of a Shuttle test?". That is by design,
not a regression.
- **`dev-tools` needs `dev/dev-tools/assets/tailwind.css`**, which CI generates
and the repo does not carry. To run its tests locally, create a placeholder
and delete it before committing. Do not habitually pass
Expand Down
Loading