Skip to content

add a README covering usage, theory, architecture and limits - #8

Open
rmarable wants to merge 3 commits into
vixie:masterfrom
rmarable:pgfiler-documentation
Open

rmarable wants to merge 3 commits into
vixie:masterfrom
rmarable:pgfiler-documentation

Conversation

@rmarable

Copy link
Copy Markdown

The repository had no documentation. This adds one file.

  • The CLI surface: the usage text verbatim, since there is no --help;
    the five operations with the SQL each one builds; and every option
    with its actual default.
  • The theory: addressing a row by key is addressing a file by name, and
    why single-key whole-value access is the pattern that scales.
  • The architecture: the two data paths through get() and put(), the
    mmap-or-spool decision, and the temp-file-and-rename output.
  • The limits: the size ceiling, the mmap snapshot hazard, what select
    does and does not preserve when it replaces a file, and the rest.
  • The build dependency: libpq, with the package name for macOS, Debian
    and RHEL, since pg_config is on none of them by default.

Verified against a throwaway PostgreSQL 18.6 cluster, 36 checks:

  • The documented examples, with cmp proving the awk round trip is
    byte-identical, and the same for the tar pipe.
  • Every operation and its failure mode: missing key, multiple rows, NULL
    value, duplicate insert, replace and append against no row, append
    over NULL.
  • integer, inet and uuid key columns.
  • Both -M forms: to_timestamp(N) for a regular file, 'now'::TIMESTAMP
    for a pipe.
  • File behaviour: an existing target keeps its mode, a fresh retrieval
    of a 755 executable lands at 644, no trailing newline into a file, and
    a failed select leaves the target byte-identical with no temp file
    left behind.

The second commit is what that testing changed. The first draft said the
write direction of -b was unchecked. In fact the server refuses a write
to a bytea column without -b as a type mismatch, so only one of the four
combinations is silent: a write to a text column with -b, which stores
the hex form at twice the file's length plus two.

Two claims that had rested on the documented cast rules are now observed.
-b into a text column stores hex, and a text column rejects a zero byte
with invalid byte sequence for encoding "UTF8": 0x00.

The third commit is wording only.

Not covered: the roughly 1 GB size ceiling, and the /proc behaviour,
since the testing ran on macOS.

🤖 Generated with Claude Code

rmarable and others added 3 commits September 21, 2026 19:38
The repository had no documentation.  This adds one file: the CLI
surface, with the usage text reproduced verbatim since there is no
--help; the five operations and the SQL each one builds; the theory of
addressing rows as files; the two data paths through get() and put();
and a limits section stating the hazards a caller can hit.

The build section names the libpq dependency and the package that
carries it on macOS, Debian and RHEL.  pg_config is on none of those by
default, and its absence is the first thing a reader hits.

Verified on macOS against Homebrew's libpq: the tree compiles clean
under the Makefile's -Werror set, and the usage block diffs identically
against the program's own output.  The two claims that need a server are
not verified and rest on the documented cast rules -- that -b on a text
column stores the hex representation, and that a text column rejects a
zero byte.

Co-Authored-By: Claude <noreply@anthropic.com>
The README said the write direction was unchecked, which understated
what happens.  A write to a bytea column without -b is refused by the
server as a type mismatch, so three of the four combinations of -b
against a text or bytea column are caught: that one, a select from a
bytea column without -b, which pgfiler refuses itself, and a select from
a text column with -b, which returns the same bytes.  Only a write to a
text column with -b is silent, and it stores the hex form at twice the
file's length plus two.

Verified against a throwaway PostgreSQL 18.6 cluster, which also settled
the two claims the first draft could only derive from the cast rules:
-b into a text column stores hex, and a text column rejects a zero byte
with "invalid byte sequence for encoding UTF8: 0x00".

36 checks in all, covering the documented examples, every operation and
its failure mode, integer/inet/uuid keys, the two -M forms, and the file
behaviours: mode preservation, the missing exec bit on a fresh
retrieval, no trailing newline into a file, and an intact target after a
failed select.

Co-Authored-By: Claude <noreply@anthropic.com>

@vixie vixie left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the program is a work in progress, and the proposed readme is about to be out of date. i'll leave this pr open for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants