feat: workflows etcd backend - #2
Merged
ibrarahmad merged 1 commit intoDec 4, 2024
Merged
Conversation
jason-lynch
force-pushed
the
jason-lynch/storage-operations
branch
from
November 22, 2024 00:03
2f916e6 to
a1a0373
Compare
jason-lynch
force-pushed
the
jason-lynch/workflows-backend
branch
from
November 22, 2024 00:03
5b0197b to
8712b8b
Compare
This commit adds a custom Etcd backend for the go-workflows library. This implementation should be compliant and functional, but see the note in etcd_test.go for an issue I ran into with the E2E tests provided by the library. The other TODO in this commit is that one of the go-workflows packages that I needed is marked as internal. The current state is that there is an edge case where we won't be able to properly bubble up an error if a sub workflow gets created twice. As far as I can tell, this shouldn't happen unless there's a bug in the library, so I consider it a non-blocking issue. See the go-workflows documentation for details on how this will be used: https://github.com/cschleiden/go-workflows
jason-lynch
force-pushed
the
jason-lynch/storage-operations
branch
from
November 22, 2024 17:29
a1a0373 to
6d7cec9
Compare
jason-lynch
force-pushed
the
jason-lynch/workflows-backend
branch
from
November 22, 2024 17:29
8712b8b to
ab22d06
Compare
AntTheLimey
added a commit
that referenced
this pull request
Jul 20, 2026
A ColdFront data node needs the coldfront / pg_duckdb stack loaded at server start, which nothing configured: pg_duckdb and coldfront must be in shared_preload_libraries, and pg_duckdb needs three duckdb.* GUCs (extension_directory pointing at the package path, allow_unsigned_extensions=true, and autoinstall_known_extensions=false — true would silently pull the unpatched upstream iceberg and 409 under concurrent cold writes). Detect the coldfront requirement once in Spec.NodeInstances() (any service of type lakekeeper), carry it as InstanceSpec.ColdFrontEnabled through Clone(), and consume it in the Patroni config generator. The append + duckdb GUCs are applied as the last step of parameters(), after the user postgresql_conf merge, so they win over both the defaults and any override while preserving the existing libraries — shared_preload is appended (with dedup, keeping reconciles stable), never replaced, so spock is retained. The generator is the single choke point feeding both the DCS and postgresql.conf parameters, and NodeInstances() is regenerated on every create/update/restore, so adding a lakekeeper service to an existing database recomputes the flag and the shared_preload change is applied by the standard update path (PatroniConfig.Update -> restartIfNeeded), which restarts the instance to load the new libraries. Resolves findings #6a and #6b (the latter subsumes #2 — the duckdb extension directory is a GUC, not a $PGDATA copy).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit adds a custom Etcd backend for the go-workflows library. This implementation should be compliant and functional, but see the note in etcd_test.go for an issue I ran into with the E2E tests provided by the library.
The other TODO in this commit is that one of the go-workflows packages that I needed is marked as internal. The current state is that there is an edge case where we won't be able to properly bubble up an error if a sub workflow gets created twice. As far as I can tell, this shouldn't happen unless there's a bug in the library, so I consider it a non-blocking issue.
See the go-workflows documentation for details on how this will be used: https://github.com/cschleiden/go-workflows