Skip to content

MINOR: Close output stream when ParquetFileWriter construction fails on encryption validation#3663

Open
anxkhn wants to merge 1 commit into
apache:masterfrom
anxkhn:patch-5
Open

MINOR: Close output stream when ParquetFileWriter construction fails on encryption validation#3663
anxkhn wants to merge 1 commit into
apache:masterfrom
anxkhn:patch-5

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

The shared private ParquetFileWriter constructor opens the output stream early
(OutputFile.create / createOrOverwrite), and only afterwards validates that
every column named in the encryption properties actually exists in the file
schema. When a configured encrypted column is missing, it throws
ParquetCryptoRuntimeException.

That validation runs after the stream is opened and is not guarded, so on the
throw the half-constructed writer is never returned to the caller. Its close()
is the only place that closes the stream, and it can never run, so the open
stream (and its underlying file/DFS handle and buffers) leaks. A service that
repeatedly constructs writers against a misconfigured encryption schema
accumulates leaked handles until exhaustion.

This is the same acquire-then-throw hazard the sibling ParquetFileReader
constructor already guards against by closing its stream before rethrowing, so
the writer should behave consistently.

What changes are included in this PR?

  • In ParquetFileWriter, wrap the encryption-setup and encrypted-column-in-schema
    validation block (which runs after this.out is assigned) in a
    try { ... } catch (Exception e) { out.close(); throw e; }. On any exception
    the already-opened stream is closed before the exception propagates, mirroring
    the existing guard in the ParquetFileReader constructor. The early return for
    the no-encryption case is left outside the guard (nothing to clean up there).
  • Add a regression test, testConstructorClosesStreamWhenEncryptedColumnMissing,
    that constructs a writer with an encrypted column absent from the schema,
    asserts the expected ParquetCryptoRuntimeException, and asserts the stream was
    closed. It uses a small RecordingOutputFile whose PositionOutputStream
    flips an AtomicBoolean on close(), so it needs no disk I/O.

Two files change: ParquetFileWriter.java (the guard) and
TestParquetFileWriter.java (the test plus helper).

Are these changes tested?

Yes.

  • The new test is red without the source fix (the stream is left open) and green
    with it. It runs parameterized (vectored true/false), 2/2 passing.
  • Regression sweep passes: TestParquetFileWriter (38), TestParquetWriter
    (19), TestEncryptionOptions (1), 0 failures / 0 errors.
  • Spotless is clean on both changed files.

Commands used:

mvn -q -pl parquet-hadoop -am -DskipTests -Dspotless.check.skip=true install
mvn -pl parquet-hadoop -Dtest=TestParquetFileWriter#testConstructorClosesStreamWhenEncryptedColumnMissing test
mvn -pl parquet-hadoop -Dtest=TestParquetFileWriter,TestParquetWriter,TestEncryptionOptions test

Are there any user-facing changes?

No. Behavior on the success path is unchanged; the only difference is that a
failed writer construction (invalid encrypted-column configuration) now releases
the output stream it opened instead of leaking it. No public API changes.

…on encryption validation

The shared ParquetFileWriter constructor opens the output stream early (via
OutputFile.create/createOrOverwrite), then validates that every encrypted
column named in the encryption properties exists in the file schema. When a
configured encrypted column is absent, it throws ParquetCryptoRuntimeException.
Because this validation runs after the stream is opened and is not guarded, the
half-constructed writer is never returned to the caller, so its close() (the
only place that closes the stream) can never run and the open stream leaks. A
service that repeatedly builds writers with a misconfigured encryption schema
accumulates leaked handles until exhaustion.

Wrap the encryption setup in a try/catch that closes the already-opened stream
before rethrowing, mirroring the existing guard in the ParquetFileReader
constructor. Add a regression test that constructs a writer with an encrypted
column missing from the schema and asserts the stream is closed on failure.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant