We now have many export plugins. However what if you don't want to only export all new events, but also events from the past? We have them all in the database, and if we enable the option to write plugins like PostHog/plugin-server#404 (function exportEvents(events) { fetch(...) or throw new RetryError() } ), we could easily support exporting all events.
This requires a few things:
- Some globally managed cursor logic
- Some way to keep track of which events were exported on the fly and which weren't.
- Some code to start/pause/track ongoing "from the beginning" exports.
Few thoughts on the above:
- The "cursor logic" refers to just storing a timestamp and ID of an event somewhere in the database, and proclaiming that everything before this
order by timestamp, uuid "cursor" is fully exported. This could be stored in the plugin storage for example.
- On a run of the export script, we fetch events from the database that are
>= cursor, run the plugin on a batch of them them, and then save the updated cursor for the next run.
- This script must also use the
$plugins_succeeded and $plugins_failed properties to know if it can skip an event or not. (we should add another property for onEvent plugins!)
I would imagine this system should give plenty of feedback to the user. Possibly in the form of some long progress bar and "2 hours remaining" in the plugins page.
We now have many export plugins. However what if you don't want to only export all new events, but also events from the past? We have them all in the database, and if we enable the option to write plugins like PostHog/plugin-server#404 (
function exportEvents(events) { fetch(...) or throw new RetryError() }), we could easily support exporting all events.This requires a few things:
Few thoughts on the above:
order by timestamp, uuid"cursor" is fully exported. This could be stored in the pluginstoragefor example.>= cursor, run the plugin on a batch of them them, and then save the updated cursor for the next run.$plugins_succeededand$plugins_failedproperties to know if it can skip an event or not. (we should add another property foronEventplugins!)I would imagine this system should give plenty of feedback to the user. Possibly in the form of some long progress bar and "2 hours remaining" in the plugins page.