Large project, JavaScript heap out of memory

[uuksu]

Operating system

Debian 12

Eleventy

3.1.2

Describe the bug

Let's start by saying that I'm doing some unholy with Eleventy and maybe pushing it to the limits where it should not be pushed. But still, I'm suspecting that there might be some actual bug somewhere.

I have a project with around ~190k .liquid files. These files are built externally from a data source, by custom tool, made by me. Largest of the file is around 300 Kb, but most of the are under 50 Kb. Individual pages are content empty files, but they have a frontmatter with all the needed data. All the pages point to one common layout (and that uses default layout) and uses the data in the frontmatter. There is also a computed data in the layout that creates the title of the page from the pages frontmatter data.

I've tried to build the project simply by running

ELEVENTY_ENV=production npx @11ty/eleventy"

Eleventy does something for a long time (maybe 20-30 minutes) and then crashes. This leads to heap out of memory error mentioned in the title

<--- Last few GCs --->

[32711:0x456f1000]  1869082 ms: Scavenge 16180.2 (16513.7) -> 16180.2 (16513.7) MB, pooled: 0 MB, 240.00 / 0.00 ms  (average mu = 0.992, current mu = 0.990) allocation failure; 
[32711:0x456f1000]  1869317 ms: Scavenge 16180.2 (16513.7) -> 16180.2 (16513.7) MB, pooled: 0 MB, 235.14 / 0.00 ms  (average mu = 0.992, current mu = 0.990) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Scavenger: semi-space copy Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xe1603a node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
 2: 0x11e3540 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 3: 0x11e3817 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0x14110f5  [node]
 5: 0x14a13f4 heap::base::SlotCallbackResult v8::internal::Scavenger::ScavengeObject<v8::internal::FullHeapObjectSlot>(v8::internal::FullHeapObjectSlot, v8::internal::Tagged<v8::internal::HeapObject>) [node]
 6: 0x14a1a5f void v8::internal::BodyDescriptorBase::IteratePointers<v8::internal::ScavengeVisitor>(v8::internal::Tagged<v8::internal::HeapObject>, int, int, v8::internal::ScavengeVisitor*) [node]
 7: 0x14ad40e v8::internal::Scavenger::Process(v8::JobDelegate*) [node]
 8: 0x14ae1ef v8::internal::ScavengerCollector::JobTask::ProcessItems(v8::JobDelegate*, v8::internal::Scavenger*) [node]
 9: 0x14ae35a v8::internal::ScavengerCollector::JobTask::Run(v8::JobDelegate*) [node]
10: 0x21ce548 v8::platform::DefaultJobWorker::Run() [node]
11: 0xffba04  [node]
12: 0x7fafc60821f5  [/lib/x86_64-linux-gnu/libc.so.6]
13: 0x7fafc61028dc  [/lib/x86_64-linux-gnu/libc.so.6]

I tried first solution I found and increased the max memory size of V8's old memory section. By increasing this to 20 G, the Eleventy is able to do little bit more by writing couple of files (actually outside of the directory that contains the 190k files) but still ends up in the same error. That's a lot of memory so I'm thinking there must be leak somewhere.

When running with a Debug Mode (and with increased memory), last lines before the error are these

  Eleventy:ComputedData Computed data order of execution: [] +0ms
  Eleventy:Template Second round of computed data for './src/pages/thread/99998-1/99998.liquid' +0ms
  Eleventy:ComputedData Computed data order of execution: [] +0ms
  Eleventy:Template Second round of computed data for './src/pages/thread/99999-1/99999.liquid' +0ms
  Eleventy:ComputedData Computed data order of execution: [] +0ms
  Eleventy:Template Second round of computed data for './src/pages/thread/99997-1/99997.liquid' +0ms
  Eleventy:ComputedData Computed data order of execution: [] +0ms
  Eleventy:TemplateWriter Template map created. +0ms
  Eleventy:Transforms Running '@11ty/eleventy-bundle' transform on './src/pages/users.liquid': './_site/pages/users/index.html' +0ms
  Eleventy:Bundle Processing '/pages/users/' +0ms
  Eleventy:Transforms Running '@11ty/eleventy/html-transformer' transform on './src/pages/users.liquid': './_site/pages/users/index.html' +570ms
  Eleventy:Transforms Running '@11ty/eleventy-bundle/delayed' transform on './src/pages/users.liquid': './_site/pages/users/index.html' +41ms
  Eleventy:Logger Writing ./_site/pages/users/index.html from ./src/pages/users.liquid +0ms
  Eleventy:Template ./_site/pages/users/index.html written. +11m

I can provide more information if needed and even share the project privately if necessary.

Reproduction steps

1. Run the build

Expected behavior

Eleventy should build the static website.

Reproduction URL

No response

Screenshots

No response

[Ryuno-Ki]

Eleventy has documented how to CPU profile a build (as this was a bottleneck in the past).

Node itself has a section on memory profiling.

In #2470 --heap-prof was used.

Can you try to run those diagnostics and share results? (Possibly after redacting sensitive information).

Another idea would be to strip down the Eleventy-config by replacing filters and extensions with noops to narrow down the culprit.

[uuksu]

Thanks for the great ideas!

I'm having a hard time to get either CPU or heap dump. None of the --cpu-prof or --heap--prof fail to actually write any files, maybe this is because of Node crashing?

I managed to attach debugger to Eleventy process by using --inspect-brk and connecting from the Chrome. Here I should also be able to dump different memory profiles. Problem here too is that Chrome is unable to create the profile after the crash and just gets stuck generating them (I waited for a like 8 hours for that to happen).

While monitoring the memory usage gradually rising, I noticed that after Eleventy is processing computed data the memory usage is somewhere around 10-11 GB. Just before the crash, when Eleventy starts writing the actual pages, I noticed that the memory usage starts steeply increasing and then the actual crash occurs.

I'm trying to to get heap snapshot just before the crash by pausing the execution and taking the snapshot. Even this method has a problem that for some reason the Eleventy process dies before the snapshotting finishes.

[zachleat]

Sorry, but we do need access to code to diagnose!

We do also have a large-build benchmark here that maybe you could modify to reproduce the issue? https://github.com/11ty/eleventy-benchmark

I would also note that 11ty.js is by and large the most efficient template syntax

[uuksu]

I understand, one would say it's almost impossible to diagnose this without the code :)

Like I said at the start, I can share the project, but prefer to do it in private. The project has nothing secret or confidential, but I don't feel like sharing publicly a code I've not intended to be released in public. Is there a prefered way I could make it available to your team?

That's good to know about the11ty.js but would not this mean in practice that I would need to convert even my Liquid layout to JavaScript? That is something I really would not like to do, because of flexibility of working with Liquid.

[groenroos]

We've got the same problem with our large project, also making heavy use of CMS data sources to generate pages via pagination (31 endpoints, each with n pages of data, each with large payloads of complex JSON). All the fetches are done via eleventy-fetch, and a fresh build creates 19 megabytes of files in .cache.

Some of this data is then preprocessed/reformatted within its JS data file; I imagine JSDOM is particularly memory-heavy.

It's harder for me to debug though, because it doesn't appear to fail every time, and the only failures I have observed have been inside GitHub Actions runners on ubuntu-latest (16GB RAM, 4-core CPU), and never on my localhost (64GB RAM, M4 Max).

The fetches all represent pages that used to be local to Eleventy, and we never ran into any memory issues in the past, so I'd lean towards assuming the culprit is all of the data being held and processed in memory during the data processing step, prior to writing templates.

Thus, I wonder if it would be feasible and beneficial to write the processed data to disk for Eleventy to consume, thus not needing to hold all that data in memory during processing?