Skip to content
This repository was archived by the owner on Nov 4, 2021. It is now read-only.
This repository was archived by the owner on Nov 4, 2021. It is now read-only.

Create exportEvents function in plugins that abstracts away all the queueing, batching, retrying logic #404

Description

@mariusandra

We're now manually writing a lot of queueing and retrying/jobs logic into events.

Queuing: https://github.com/PostHog/posthog-plugin-replicator/blob/974854579a7a51f815f9d3601e9c5b0f075200d8/index.ts#L19-L27

Retrying: https://github.com/PostHog/s3-export-plugin/blob/91e4813e84237e88142107c00929dc6b147963b7/index.ts#L37-L41

This is super powerful stuff, but our export plugins could be better. Something like:

export async function exportEvents (events, { global, config }) {
    try {
        await fetch(`https://${config.host}/e`, {
            method: 'POST',
            body: JSON.stringify(events),
            headers: { 'Content-Type': 'application/json' },
        })
    } catch (error) {
         throw new RetryError() // ask to retry
    }
}

It would just be an abstraction over existing tools, and it would take care of 1) retry queues, 2) batching, 3) async processing .

You literally write the data upload call and we take care of the rest.

Plus, this will unlock one really strong feature: "start from the beginning". More on that later. :D

Open questions: I'm not sure what's the best way to configure this (e.g. some queues can take 1MB of buffer, some want 100kb, some go just one by one)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions