Ideas on How to Debug SIGKILL (OOM) While Building

[jeffdc]

Background

Running next build works OK locally. It fails however when I run it in a Docker container as part of my production build process. It dies via a SIGKILL during the static assets generation:

...
info  - Compiled successfully
info  - Collecting page data...
info  - Generating static pages (0/739)
info  - Generating static pages (184/739)
info  - Generating static pages (369/739)
info  - Generating static pages (554/739)
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with signal "SIGKILL".
ERROR: Service 'gallformers' failed to build : The command '/bin/sh -c NODE_OPTIONS="--max_old_space_size=1024 --report-on-fatalerror" npx prisma generate && yarn add --dev typescript @types/node && yarn build' returned a non-zero code: 1
make: *** [build] Error 1

Through some experimentation (increasing mem allocated to Docker) I can get it to complete. However it requires 8GB! to complete. Currently I have 739 static pages. The number of pages is only going to grow and it surely will not be long before even 8GB is not enough. I also imagine that it will eventually fail locally as well.

Annoying wildcard... Sometimes, but not always, if I run the build again it will succeed.

Things That I Have Already Tried

• Decreasing the parallelization of webpack via the experimental.cpus flag in my next.js.config file
• Capping the max_old_space of V8 via NODE_OPTIONS

I shelled into the container itself and tried just the yarn build command and it fails in the same way. Running dmesg in the container confirms that it is an OOM error:

...
[91697.831706] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/63aeeae1424c6cc8fd9499fde93a87affeaec63638db2e734666d8286642cfdc,task=node,pid=51138,uid=0
[91697.834716] Out of memory: Killed process 51138 (node) total-vm:526444kB, anon-rss:54948kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:7344kB oom_score_adj:0
...

Help Needed

I need some guidance on how to debug this issue. I feel like it must be an issue with how I have the project structured and that something is causing a huge increase in mem usage that is out of line with how many static pages that I have. I have never used next.js (or React for that matter) before, so I have no doubt that this is an error I have made. I have already spent many hours on this with no progress.

The project is open-source and can be found here: https://github.com/jeffdc/gallformers

[jeffdc]

I figured out my problem (leaky DB connections) but still do not know how to really debug a next.js build appropriately. Given that code that I wrote is executing (getStaticProps/Paths) at build time it would be super useful to have some docs/tooling to support debugging this process. It is all very opaque right now.

[jeffdc]

What I had to do was to first check the system logs in the container where it was built to figure out what was causing the SIGKILL, in my case an OOM (probably the same for you). Then I had to think about likely sources of memory leaks or large memory usage. I added some debug output (console.log) to my code and figured out that my DB connection was getting created over and over again (which was really a separate issue as seen here: prisma/orm#5007 (comment)). I hacked in some code to prevent excessive creations of DB connections and my SIGKILL problem went away.

Sorry I can not offer anything any better. As you can see, no one ever responded to my original query.

[nandorojo]

Got it, well I do appreciate you getting back to me. It seems like there's some sort of memory leak in my Babel config, but while I'm very experienced in React I really have no idea how to debug Babel.

I noticed that in my case, the second my build script gets to my Babel config file, it sits slowly for about 7 minutes. 1/10 times I get this error, and other times it works for the same code, indicating there must be some issue beyond a typical slow build. Unfortunately, I have no clue how to debug this.

How did you go from this yarn script to viewing system logs?

[jeffdc]

I am not using vercel so I can not speak to how you would tackle this on that platform, but for me it was all in a docker container so I had to shell in to the container after the build failed and inspect the system logs that way.

[jeffdc]

Presumably you can build fine locally? If so, then it is almost certainly the resource constraints of the vercel build env (memory I would guess) that causes it to die. I doubt you can get much info out of a vercel build container. If so, maybe try and replicate the issue locally by running the build in a more resource constrained way (docker container/VM)?

[nandorojo]

I actually have the same issue on my local computer with building (maxed out MacBook Pro.) I notice when running yarn next dev locally, it does work, but it gets hung up when loading in my custom Babel config as well.

Thanks for your advice here. I'll try some of it out. I'm a beginner when I go outside of React and into figuring out what's going on in my machine with something like this.

[walidvb]

I'm having the same issue atm. Any update on this topic? Any way to reproduce the build environment locally?

44:6  Warning: React Hook React.useEffect has a missing dependency: 'initialTheme'. Either include it or remove the dependency array.  react-hooks/exhaustive-deps
--
11:47:09.031 |  
11:47:09.031 | info  - Need to disable some ESLint rules? Learn more here: https://nextjs.org/docs/basic-features/eslint#disabling-rules
11:47:09.041 | info  - Creating an optimized production build...
11:47:10.436 |  
11:47:10.437 | warn - You have enabled the JIT engine which is currently in preview.
11:47:10.437 | warn - Preview features are not covered by semver, may introduce breaking changes, and can change at any time.
11:48:01.705 | error Command failed with signal "SIGKILL".
11:48:01.705 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
11:48:01.726 | Error: Command "yarn run build" exited with 1

[walidvb]

I was able to fix this build by disabling swcMinify in next-config.js

[SPBabbar]

Browserslist: caniuse-lite is outdated. Please run:

16:47:58.698 | npx browserslist@latest --update-db
16:47:58.698 |  
16:47:58.698 | Why you should do it regularly:
16:47:58.698 | https://github.com/browserslist/browserslist#browsers-data-updating
16:47:59.259 | Browserslist: caniuse-lite is outdated. Please run:
16:47:59.259 | npx browserslist@latest --update-db
16:47:59.259 |  
16:47:59.259 | Why you should do it regularly:
16:47:59.260 | https://github.com/browserslist/browserslist#browsers-data-updating
16:49:43.541 | Error: Command "next build" exited with SIGKILL

It has started coming since February 3, 2022 all of a sudden!

I have no avenues to go through! I could only think of the possibility of a bloated package, but i have not used any heavy package as well!

[franklin-vidal]

In my case, using wsl2, removing node_modules and reinstall solve this issue.

[Bikash9609]

I face this every time I try to build when the .next folder is already present. So I remove it and node_modules as well as the lock file

• rm -rf node_modules .next yarn.lock
• yarn && next build

[coderbahrom]

I have the same issue with building my React project

Module level directives cause errors when bundled, 'use client' was ignored.
Module level directives cause errors when bundled, 'use client' was ignored.
Module level directives cause errors when bundled, 'use client' was ignored.
Module level directives cause errors when bundled, 'use client' was ignored.
Module level directives cause errors when bundled, 'use client' was ignored.
Module level directives cause errors when bundled, 'use client' was ignored.
Module level directives cause errors when bundled, 'use client' was ignored.
Module level directives cause errors when bundled, 'use client' was ignored.
transforming...
✓ 6 modules transformed.
rendering chunks...
Module level directives cause errors when bundled, 'use client' was ignored.
Module level directives cause errors when bundled, 'use client' was ignored.
Module level directives cause errors when bundled, 'use client' was ignored.
✓ 3665 modules transformed.
rendering chunks...
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with signal "SIGKILL".
The command '/bin/sh -c yarn build --mode ${ENV}' returned a non-zero code: 1
make: *** [Makefile:52: build-image] Error 1
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

[prashantchothani]

Facing tremendous issues with Vercel using Nextjs regarding SIGKILL. My local machine builds in 17 seconds but on Vercel it just does not build and fails with SIGKILL error

[Bikash9609]

The deployment headache with NextJs was so much that i reverted to CRA and whatever static pages I needed I used svelte.

…

On Thu, 2 Nov, 2023, 3:04 pm Prashant Chothani, ***@***.***> wrote: Facing tremendous issues with Vercel using Nextjs regarding SIGKILL. My local machine builds in 17 seconds but on Vercel it just does not build and fails with SIGKILL error — Reply to this email directly, view it on GitHub <#19198 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMETNIHZVEYVLKF4CPZF7I3YCNSJNAVCNFSM4TWN4BUKU5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TONBVGQ2TAMA> . You are receiving this because you commented.Message ID: <vercel/next. ***@***.***>

[jokull]

Here's my log

[08:39:14.575] Running "pnpm run build"
[08:39:15.065] 
[08:39:15.065] > @trip/next@0.1.0 build /vercel/path0/apps/next
[08:39:15.065] > next build
[08:39:15.065] 
[08:39:15.768]    ▲ Next.js 14.1.0
[08:39:15.768] 
[08:39:15.847]    Creating an optimized production build ...
[08:40:52.715] Error: Command "pnpm run build" exited with SIGKILL
[08:40:53.787] ▲ Build system report
[08:40:53.787] ▲ To always completely log this report, add VERCEL_BUILD_SYSTEM_REPORT=1 as an Environment Variable to your project.
[08:40:53.787] • At least one "Out of Memory" ("OOM") event was detected during the build.
[08:40:53.787]   • This occurs when processes or applications running during the build completely fill up the available memory (RAM) in the build container. When this happens, the build container terminates one of the processes during the build with a SIGKILL signal.

No idea how to debug this. It always works if I redeploy without build cache.

[danoph]

I have the same issue. Just started a couple weeks ago.

[jokull]

Fixed for me in the canary - as of today

[spiritbuilder]

Please can you be clearer

[manoranjaninfoviz]

[nextapp builder 4/4] RUN yarn build:
3.457 yarn run v1.22.22
3.627 $ set NODE_OPTIONS=--max-old-space-size=4096 && next build
4.917 info - Loaded env from /app/.env.local
6.558 Attention: Next.js now collects completely anonymous telemetry regarding usage.
6.558 This information is used to shape Next.js' roadmap and prioritize features.
6.558 You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL:
6.558 https://nextjs.org/telemetry
6.558
6.964 info - Skipping validation of types
6.964 info - Skipping linting
8.677 info - Creating an optimized production build...
74.08 error - No Sentry auth token configured. Source maps will not be uploaded.
74.08 You can find information on how to generate a Sentry auth token here: https://docs.sentry.io/api/auth/
74.08 After generating a Sentry auth token, set it via the SENTRY_AUTH_TOKEN environment variable during the build.
74.08
177.0 (node:28) [DEP_WEBPACK_MODULE_ID] DeprecationWarning: Module.id: Use new ChunkGraph API
177.0 (Use node --trace-deprecation ... to show where the warning was created)
259.6 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
259.6 error Command failed with signal "SIGKILL".

failed to solve: process "/bin/sh -c yarn build" did not complete successfully: exit code: 1
I am facing this error when i am build it on docker

[zhyd1997]

Enable build woker and worker threads in the next config file may work in the scenario:

experimental: {
  // --- TODO: Remove the 2 options if possible ---
  webpackBuildWorker: true,
  workerThreads: true,
  // ----------------------------------------------
},

[codercatdev]

Seeing the same after adding Sentry. Here is an example of a working build log and a failed one.

https://gist.github.com/codercatdev/c55e9e7116e382e2310c6ca228d628d3

[baroxdev]

I've faced this issues for a week. Finally I've just figure out the reason is because we missed the lock file.

I've fixed by delete node_modules and run npm install --legacy-peer-deps.

[xabierlameiro]

Six years on, the question in the title still hasn't been answered, and you said as much yourself back in January 2021. Here's an attempt, because the thing that makes this hard is visible in your very first post and nobody picked it up.

Your dmesg line:

Out of memory: Killed process 51138 (node) total-vm:526444kB, anon-rss:54948kB

The kernel killed a node process holding 55 MB. That's not the process that ate your memory. The OOM killer picks by score across the whole cgroup, and in a next build there are several node processes: the parent that compiles, and the static-generation workers. Pressure from the siblings gets one of them killed, and which one takes the bullet tells you almost nothing about which one grew. Everyone in this thread, me included until I started measuring it, reads the corpse instead of the tree.

Two consequences worth stating plainly.

A SIGKILL is not a V8 heap error. If you got JavaScript heap out of memory, that's V8 hitting its own ceiling and it prints a stack. Exit 137 with nothing in the logs is the kernel, and V8 never got a say. They're different failures and they take different fixes, which is why the thread has people reporting success with contradictory workarounds.

That also means raising --max-old-space-size can make it worse. The flag tells V8 it may grow before collecting hard. Under a cgroup limit, a lazier GC means higher RSS, and the kernel kills you sooner. On a V8 heap error it's the right knob; on a cgroup kill it's pointed the wrong way.

Then there's your "sometimes it passes, sometimes it doesn't". I measured peak memory of the same build three times today, same machine, nothing changed between runs: 216 MB, 334 MB, 450 MB. A single build is not a measurement of anything. If your ceiling sits anywhere near that spread, the build is a coin flip, and it will look like whatever you changed last is what fixed it. That's most of what's in this thread: clearing node_modules, dropping swcMinify, regenerating the lockfile. Some of those genuinely change the memory profile. From one run each you can't tell which.

So, concretely: sample the whole process tree over the build, not one number at the end, and keep the parent and the workers apart. If the workers climb page after page, the useful figure is MB per page and your 739 pages are the multiplier. If it's a single peak while it compiles, page count is irrelevant and no amount of experimental.cpus will help. Whatever you change, measure it three times per arm, alternating, before believing it.

I package that as next-leak:

npx next-leak build .

It samples the tree, reports the parent peak and the worker curve separately, and gives MB per page. Node 22+, no output: "standalone" needed on the build path. Worth saying: it can't help with a kill that lands faster than its sampling interval, and on webpack builds it won't name what's retained, only show the curves.

For your original case, leaky DB connections in getStaticProps is exactly the shape that shows up as MB per page climbing on the workers while the parent stays flat. That would have pointed at it in one run.