We're now manually writing a lot of queueing and retrying/jobs logic into events.
Queuing: https://github.com/PostHog/posthog-plugin-replicator/blob/974854579a7a51f815f9d3601e9c5b0f075200d8/index.ts#L19-L27
Retrying: https://github.com/PostHog/s3-export-plugin/blob/91e4813e84237e88142107c00929dc6b147963b7/index.ts#L37-L41
This is super powerful stuff, but our export plugins could be better. Something like:
export async function exportEvents (events, { global, config }) {
try {
await fetch(`https://${config.host}/e`, {
method: 'POST',
body: JSON.stringify(events),
headers: { 'Content-Type': 'application/json' },
})
} catch (error) {
throw new RetryError() // ask to retry
}
}
It would just be an abstraction over existing tools, and it would take care of 1) retry queues, 2) batching, 3) async processing .
You literally write the data upload call and we take care of the rest.
Plus, this will unlock one really strong feature: "start from the beginning". More on that later. :D
Open questions: I'm not sure what's the best way to configure this (e.g. some queues can take 1MB of buffer, some want 100kb, some go just one by one)
We're now manually writing a lot of queueing and retrying/jobs logic into events.
Queuing: https://github.com/PostHog/posthog-plugin-replicator/blob/974854579a7a51f815f9d3601e9c5b0f075200d8/index.ts#L19-L27
Retrying: https://github.com/PostHog/s3-export-plugin/blob/91e4813e84237e88142107c00929dc6b147963b7/index.ts#L37-L41
This is super powerful stuff, but our export plugins could be better. Something like:
It would just be an abstraction over existing tools, and it would take care of 1) retry queues, 2) batching, 3) async processing .
You literally write the data upload call and we take care of the rest.
Plus, this will unlock one really strong feature: "start from the beginning". More on that later. :D
Open questions: I'm not sure what's the best way to configure this (e.g. some queues can take 1MB of buffer, some want 100kb, some go just one by one)