@sentry/node memory leaks on node 10.13.0

[abraxxas]

Package + Version

• @sentry/browser
• @sentry/node

Version:

4.3.4

Description

I tried integrating sentry into ah next.js project. I tried it using this template https://github.com/sheerun/next.js/tree/with-sentry-fix/examples/with-sentry and discovered what seems to be a memory-leak in sentry. If you check out this project and add memwatch

if (!dev) {
  memwatch.on("leak", info => {
    console.log("memwatch::leak");
    console.error(info);
  });

  memwatch.on("stats", stats => {
    console.log("memwatch::stats");
    console.error(Util.inspect(stats, true, null));
  });
}

and bombard the server with requests, i requested the following ressources:

    "/",
    "/_next/static/r1zovjaZ1TujaA0hJEp91/pages/_app.js",
    "/_next/static/r1zovjaZ1TujaA0hJEp91/pages/_error.js",
    "/_next/static/r1zovjaZ1TujaA0hJEp91/pages/index.js",
    "/_next/static/runtime/main-1eaa6d1d0c8e7d048efd.js",
    "/_next/static/chunks/commons.b34d260fee0c4a698139.js",
    "/_next/static/runtime/webpack-42652fa8b82c329c0559.js"

With this the memory-usage grows endlessly for me. As soon as i remove the request and errorHandler from server.js the memory-leak stops. So it seems to be connected to those 2

[kamilogorek]

@abraxxas can you help me reproduce this issue? How exactly are you triggering this memleak?
I tried your description with no luck, it just produces stats events, never leak one.

[abraxxas]

@kamilogorek Thanks for your response. What i did was the following. I checked out this example https://github.com/sheerun/next.js/tree/with-sentry-fix/examples/with-sentry and added memwatch to the server.js

if (!dev) {
  memwatch.on("leak", info => {
    console.log("memwatch::leak");
    console.error(info);
  });

  memwatch.on("stats", stats => {
    console.log("memwatch::stats");
    console.error(Util.inspect(stats, true, null));
  });
}

then i ran the example using node 10.x (with 8.x i observed no memory issues) and requested the following ressources using our gatling testsuite:

    "/",
    "/_next/static/r1zovjaZ1TujaA0hJEp91/pages/_app.js",
    "/_next/static/r1zovjaZ1TujaA0hJEp91/pages/_error.js",
    "/_next/static/r1zovjaZ1TujaA0hJEp91/pages/index.js",
    "/_next/static/runtime/main-1eaa6d1d0c8e7d048efd.js",
    "/_next/static/chunks/commons.b34d260fee0c4a698139.js",
    "/_next/static/runtime/webpack-42652fa8b82c329c0559.js"

(note the hashes might change for you)

but you should be able to achieve the same results with this approach very simple https://www.simonholywell.com/post/2015/06/parallel-benchmark-many-urls-with-apachebench/

after a few thousand requests our memory usage was at almost 1GB and never reduced even when idling. As soon as i remove the request and errorHandler from server.js the memory-leak stops. So it seems to be connected to those 2. Maybe you either had too few requests or used node 8.x?

[kamilogorek]

@abraxxas confirmed. Node 10 + ~300req for each resource does the job. Will investigate further. Thanks!

[abraxxas]

@kamilogorek intersting, but if you look on the heapsize in your reproduction you will see that it stays around 20mb without the handlers but increases rapidly with them. I think the memory-leak warning without the handlers happens just because due to the requests there is some constsnt memory increase. There is still a vast difference in memory usage between the version with sentry and without.

…

On Thu, 29 Nov 2018, 12:45 Kamil Ogórek ***@***.*** wrote: @abraxxas <https://github.com/abraxxas> I successfully reproduced it, however, it appears that server still leaks request objects on its own, even without Sentry handlers. https://streamable.com/bad9j The growth rate is a tad larger, as we do attach domain and our own scope object to the request, but it'd be utilized alongside the request by GC. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1762 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIbrNlgPjPd5Jra1aahR-Dthf7XvbCexks5uz8jjgaJpZM4YvOA2> .

[kamilogorek]

@abraxxas ignore my previous comment (the one I removed), it's totally on our side :)

It appears to be the issue in Node's core itself. Please refer to this comment #1762 (comment)

[tpbowden]

@kamilogorek Have you had chance to look into this yet? It is causing a huge memory leak for us. After looking at the heap, this line looks like it could be the root of problem:

sentry-javascript/packages/node/src/handlers.ts

Line 233 in c27e1e3

scope.addEventProcessor(async (event: SentryEvent) => parseRequest(event, req, options)),

The inspector showed thousands of entries into the eventProcessors list

I don't have any context around how things are architected, but we have noticed that requests are not correctly scoped and are giving the wrong metadata (see #1773) so it appears everything is being managed in global state and not cleaned up when a request ends

[kamilogorek]

@abraxxas @tpbowden there's an issue with leaking domain module in Node's core itself. We'll keep monitoring it and try to come up with a temporary solution before it's fixed in the core. Related issue: nodejs/node#23862