Add a benchmarking system

[Twixes]

It'd be super useful to know what are the limits of the plugins server and how is performance affected by various changes we make.

[mariusandra]

After PR #41 lands, we will have a basic benchmarking system under tests/worker.test.ts, which tests the throughput of the Worker Threads + Plugin VM system. It omits celery for the time being.

Looking at the results below, the bottleneck will probably be in the redis/celery pipe, not in the plugin server.

Running these three blocks of code:

        {
            testName: 'simple',
            events: 10000,
            testCode: `
                function processEvent (event, meta) {
                    event.properties = { "somewhere": "over the rainbow" };
                    return event
                }
            `,
        },
        {
            testName: 'for200k',
            events: 10000,
            testCode: `
                function processEvent (event, meta) {
                    let j = 0; for(let i = 0; i < 200000; i++) { j = i };
                    event.properties = { "somewhere": "over the rainbow" };
                    return event
                }
            `,
        },
        {
            testName: 'timeout100ms',
            events: 2000,
            testCode: `
                async function processEvent (event, meta) {
                    await new Promise(resolve => __jestSetTimeout(() => resolve(), 100))
                    event.properties = { "somewhere": "over the rainbow" };
                    return event             
                }
            `,
        },

the system is able to process this many events:

    // 2.4GHz 8-core/16-thread i9 (16" 2019)
    ┌─────────┬────────────────┬───────────┬─────────┬─────────┬─────────┬─────────┬──────────┬──────────┐
    │ (index) │    testName    │ coreCount │ 1 cores │ 2 cores │ 4 cores │ 8 cores │ 12 cores │ 16 cores │
    ├─────────┼────────────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────┼──────────┤
    │    0    │    'simple'    │    16     │  26162  │  45892  │  55995  │  53125  │  44863   │  43446   │
    │    1    │   'for200k'    │    16     │  11780  │  21345  │  36296  │  37861  │  42382   │  34826   │
    │    2    │ 'timeout100ms' │    16     │  1867   │  3619   │  6040   │  9108   │  14077   │  14185   │
    └─────────┴────────────────┴───────────┴─────────┴─────────┴─────────┴─────────┴──────────┴──────────┘

    // M1 Air with native nodejs (4 performance + 4 slow cores)... 8 cores, but pushing to 16 threads for fun and profit
    ┌─────────┬────────────────┬───────────┬─────────┬─────────┬─────────┬─────────┬──────────┬──────────┐
    │ (index) │    testName    │ coreCount │ 1 cores │ 2 cores │ 4 cores │ 8 cores │ 12 cores │ 16 cores │
    ├─────────┼────────────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────┼──────────┤
    │    0    │    'simple'    │     8     │  45929  │  79412  │  93268  │  82478  │  70884   │  67301   │
    │    1    │   'for200k'    │     8     │  12720  │  23627  │  38912  │  44066  │  46039   │  42644   │
    │    2    │ 'timeout100ms' │     8     │  1811   │  3566   │  6006   │  9118   │  15051   │  15038   │
    └─────────┴────────────────┴───────────┴─────────┴─────────┴─────────┴─────────┴──────────┴──────────┘

These tests were run with TASKS_PER_WORKER=100 (per-thread async concurrency). Increasing this could increase the timeout100ms test's results, though there might be tradeoffs with making over 100 http requests in parallel for example. I'm not sure what's a good production value for this. To improve this and give plugins the ability to make one batched request instead of 100 separate requests, there's event batching: #39.

Also, for simplicity the benchmarks are now part of the jest tests and just output console.tables, without doing any assertions. This could be refactored into its own system, though I prefer to keeps things simple at least this sprint.

[mariusandra]

Benchmarks are now there. Still to do:

• E2E benchmarks for Celery and Kafka E2E benchmarks for celery and Kafka #56
• Extracting benchmarks from the other tests Extract benchmarks from tests #55

Let's follow that up in the other issues.