Skip to content

replace go-dsqueue with cascadeq - #1194

Draft
gammazero wants to merge 3 commits into
mainfrom
replace-go-dsqueue
Draft

replace go-dsqueue with cascadeq#1194
gammazero wants to merge 3 commits into
mainfrom
replace-go-dsqueue

Conversation

@gammazero

@gammazero gammazero commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Replace the provide queue use of go-dsqueue with cascadeq.

The intended benefits are:

  • Overall efficiency (faster, less memory overhead)
  • Reliable performance regardless of datastore in use

The efficiency of go-dsqueue depends on the ability of the datastore in use to support ordered query responses without reading all queued items into memory to provide that ordering of queued item(s). With go-dsqueue is it also necessary to store additional ordering information with each queued item, taking up additional space.

Using cascadeq eliminates the need for a datastore implementation to efficiently return ordered query results as these are always stored in queue order when persisted to disk. Storage is also more efficient as there is no need to store additional ordering information with each item.

A potential downside to this PR is the case where is user wants all data written to S3, so an S3-backed datastore is used. In this case, cascadeq would not allow the user to write all data to S3, without some additional work.

Related PRs:

@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 47.05882% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.83%. Comparing base (f20139a) to head (5b06ee7).

Files with missing lines Patch % Lines
provider/reprovider.go 53.33% 4 Missing and 3 partials ⚠️
provider/noop.go 0.00% 2 Missing ⚠️

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1194      +/-   ##
==========================================
- Coverage   63.85%   63.83%   -0.03%     
==========================================
  Files         269      269              
  Lines       27144    27151       +7     
==========================================
- Hits        17334    17333       -1     
- Misses       8097     8102       +5     
- Partials     1713     1716       +3     
Files with missing lines Coverage Δ
provider/provider.go 90.16% <ø> (ø)
provider/noop.go 0.00% <0.00%> (ø)
provider/reprovider.go 58.27% <53.33%> (+0.67%) ⬆️

... and 15 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Replace the provide queue use of go-dsqueue with cascadeq.

The efficiency of go-dsqueue depends on the ability of the datastore in use to support ordered query responses without reading all queued items into memory to provide that ordering of queued item(s). With go-dsqueue is it also necessary to store additional ordering information with each queued item, taking up additional space.

Using cascadeq eliminates the need for a datastore implementation to efficiently return ordered query results as these are always stored in queue order when persisted to disk. Storage is also more efficient as there is no need to store additional ordering information with each item.
@gammazero
gammazero force-pushed the replace-go-dsqueue branch from 6427b70 to 1e7ee08 Compare August 4, 2026 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant