Are chunk hashes not deterministic based on BUILD_ID?

[CaptainN]

Summary

I have a site that we have been running with 2 horizontally scaled pods, which uses generateBuildId to return the git hash of the current commit for reliable scaling. I thought that meant the chunk hashes would be deterministically generated - that is, they'd be the same between pods (which are built at startup in the runtime environment, since static generation occurs at build time, and the necessary services are not available in the build environment).

This worked for a long time, but we just had a site meltdown because the chunk hash names didn't match, and when balancing occured, it was causing errors in loading. The solution was to scale back down to one instance for the time being. I did confirm by inspecting the build artifacts that .next/BUILD_ID matched between the pods, but the chunk names which were erroring did not.

Is BUILD_ID not intended to be used this way?

Additional information

I'm currently on [email protected]

Operating System:
  Platform: darwin (dev machine - we run node:20-alpine in production)
  Arch: arm64
  Version: Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000
Binaries:
  Node: 20.11.1
  npm: 10.2.4 (this project uses npm)
  Yarn: 1.22.22
  pnpm: 8.15.4
Relevant Packages:
  next: 14.1.4
  eslint-config-next: 14.1.3
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.4.2
Next.js Config:
  output: N/A

Example

No response

[CaptainN]

I just did a quick test, and can confirm, the chunk names are not the same between builds. All I did is build once locally, delete the .next folder, then build again. The results are different chunk names:

# First Build:
❯ cat BUILD_ID
3cdb0cdf731951b8f03f4d8c16c600c5ead0d6eb

# in:
/.next/static/chunks/app/about/methodology/[slug]

❯ ls
page-fc743ef5cdb60c1d.js

# Second Build:
❯ cat BUILD_ID
3cdb0cdf731951b8f03f4d8c16c600c5ead0d6eb

# in:
/.next/static/chunks/app/about/methodology/[slug]

❯ ls
page-8acd4de2256ef601.js

Many of the build artifacts do have the same names - it seems to be these app chunks that don't get the same (deterministic) hashes.

[CaptainN]

Is this expected behavior, or is there a bug?

[CaptainN]

(Clearly a bug.)

[CaptainN]

bump

[Djay77]

Hey,

We have exactly the same problem here !
Multiples pods who build at startup and generate different chunks => some JS are in 404, then the page fall in error

What is the real utility of the Next.js config generateBuildId ... ?

[Djay77]

We have to override the webpack config in next.config file

webpack: (
        config,
        { buildId }
    ) => {
        config.output.filename = config.output.filename.replace('[chunkhash]', buildId);

        return config
    },

We have now the same name generated for our files BUT the content change

[CaptainN]

I tried to create an issue, but it requires a bunch of work since you have to create a reproduction repo. This is easy to understand, and should be in a unit test somewhere.

[austin-raey]

I'd like to concur on this issue,

The codebase I'm working with is quite complex so there is likely a lot of moving parts causing this discrepancy, however I would like to note:

I have a pretty simple layout file that is shared for a few pages, seen with this code:

// layout-component.tsx
import Script from 'next/script';

import Footer from './footer';
import Header from './header';
import Main from './main';

export default function DefaultLayout({
  children,
}: Readonly<{
  children?: React.ReactNode;
}>) {
  return (
    <>
      {process.env.NODE_ENV === 'production' && <Script src="/analytics.js" />}
      <Header />
      <Main>{children}</Main>
      <Footer />
    </>
  );
}

With that code in that particular file, and a generateBuildId config present to always generate the same Build ID, I will consistently produce page-e1ea5a6b17a0777a.js in .next/static/chunks/app/.... I have replicated this running build many, many times.

However, simply removing the <Script> tag now makes all pages dependent on this have inconsistent filenames in .next/static/chunks/app/..., all else held equal:

// layout-component.tsx, modified to remove <Script />.
import Footer from './footer';
import Header from './header';
import Main from './main';

export default function DefaultLayout({
  children,
}: Readonly<{
  children?: React.ReactNode;
}>) {
  return (
    <>
      <Header />
      <Main>{children}</Main>
      <Footer />
    </>
  );
}

Now, the page.js file will cycle between about 3 or 4 different filenames: page-090c4b84bc1c1c00.js, page-c2924b123f71afcb.js, page-a711e6f388147d72.js. It becomes semi-deterministic?

Completely confounded as to why such a change would cause this issue, especially with a Build ID. Being that we are building to a couple different load balancers, this is a very irritating issue.

I had previously thought this non-deterministic issue was potentially caused by more complex node_module dependencies, but with this I have no idea.

[owenallenaz]

For what it's worth, what we do is docker build which runs the next build and then we re-use the image in two different Kube pods using replicas: 2. So then it's the exact same content, rather than needing to run build twice (which would just be a performance slog in CI anyways).

[austin-raey]

Yep, this is what we're looking at right now. Our operation had gotten by with 2 nodes, so we only needed 2 separate builds. This has generally worked o.k. in the past.

The solution going forward is we're going to just do as you say -- build once, and in our case, export the resulting Docker Image to both nodes. This ensures content is always exactly the same regardless of Build IDs or whatever content hashes come up. I'm sure Kubernetes is geared for this exact use case but I'll have to review this in future 😵‍💫

Honestly, this also seems better practice anyways. 😂 I could imagine scaling to 5 or 10 nodes, and then having to run 5 or 10 separate builds. That doesn't seem optimal lol.

Nonetheless, an interesting issue this is.

[CaptainN]

The problem with this is compile and generate are both done in the build context. In our deployment, we don't even have access to the services that would create the generate artifacts, since build and runtime are in two entirely different bits of infrastructure. But also, you aren't supposed to produce varying build artifacts - that's a docker no-no.

There is a kind of away around it, and it does actually address the bug described in this thread. You can do "compile" in the build environment, which creates all the chunks, and therefor, can't vary the chunk names in the runtime. This is still a bug, and it should still be fixed, because who knows what else it's doing. Then you just have to do generate in the runtime environment at startup. This also speeds up pod startup dramatically.

You still might end up with variable static generation artifacts, but you can work around that by using a custom redis backing store or something.

What would be even better is some straightforward way to just disable static generation at startup of all kinds, and let it fill the ISR cache at runtime. Maybe one day.

[JohnRPB]

This is a bit confounding to me as well. I'm using a consistent generateBuildId() output so my generated webpack files should all have the same names, but a few don't and cause key requests to fail with a 404 error. This means that caching pages across builds doesn't work; you have to delete the page entries in the cache on deployment. Not the behavior I was expecting from generateBuildId() :-/

[CaptainN]

I guess someone is going to have to make a reproduction repro, and file a ticket, even though there's more than enough information on this thread to see the problem...

@JohnRPB That used to work as expected. Now we get variable chunk names. There are workarounds available (described above), but none of that should be necessary.

[michael-meetkai]

hey @CaptainN @JohnRPB have u found a solution ? having the same issue where the server chunks are different between my local env and CI/CD

[CaptainN]

My solution is to split build - compile in GHA, which locks the artifacts, and then do generation in startup. This works because the variable output happens in the compile stage. This relies on side effects being stable, which is not super amazing. It'd be nicer if the next.js team would fix the core issue.

[mhmmdglc]

is there any update ?

[samcarton]

Adding my 2c to this - This is a blocker especially given source maps are either all or nothing. If you want a public build without source maps, and a separate source-mapped build for external monitoring (sentry/datadog etc), the chunk names are going to be different. In this related discussion #7579 (comment) there are only half-solutions which involve stripping out sourcemaps from the deployed build which seems to cause 404s in some browsers.

[mhmmdglc]

I think Vercel is intentionally not solving this so that we use their platform :D

[CaptainN]

I don't think it's intentional, but at this point I would not recommend running next.js in production outside of vercel. It's clearly not well tested or maintained for that scenario.

(OpenNext exists, but it shouldn't have to.)

[junerd]

/** @type {import('next').NextConfig} */
const nextConfig = {
  output: 'export',
  productionBrowserSourceMaps: true,
  generateBuildId: () => 'static-build-id',
  webpack: (config) => {
    // Next.js output is not stable when there's no code change and this is a work around
    config.output.filename = config.output.filename.replace('[chunkhash]', '[contenthash]');
    return config;
  },
  devIndicators: false,
};

This worked for me I'm on "next": "^15",

[junerd]

// this is madness, but it's working with next 15.2.4
function removeRandomness(el) {
  const prefix = `[\\"$\\",\\"$1\\",\\"`;
  const suffix = `\\"`;
  const prefixIndexes = [];
  let index = -1;
  do {
    if (typeof index === 'number') {
      prefixIndexes.push(index);
    }
    index = el.innerHTML.indexOf(prefix, index + 1);
  } while (index !== -1);
  prefixIndexes.shift();
  prefixIndexes.forEach((prefixIndex) => {
    const startIndex = prefixIndex + prefix.length;
    const endIndex = el.innerHTML.indexOf(suffix, startIndex);
    if (endIndex - startIndex < 2) {
      return;
    }
    const randomness = el.innerHTML.substring(startIndex, endIndex);
    const original = el.innerHTML.substring(prefixIndex, endIndex + suffix.length);
    el.innerHTML = el.innerHTML.replace(randomness, 'stable-id');

    const modified = el.innerHTML.substring(
      prefixIndex,
      el.innerHTML.indexOf(suffix, startIndex) + suffix.length
    );
    console.log(`👉 Replaced ${original} with ${modified} to stabilize build output`);
  });
}

I tracked it down to an inline script element, the code above works to stabilize the generated html for me now but yeah no idea why the the random string is added, couldn't tell by quickly looking at nextjs repo

[barroudjo]

Where do you put this code @junerd ?

On my end I mostly solved this issue with this workaround:

const webpackConfig = (config, params) => {
    // config.optimization.realContentHash = true;
    if (config.name === 'client') {
      config.output.chunkFilename = `static/js/[name]-${params.buildId}.chunk.js`;
      config.output.filename = `static/js/[name]-${params.buildId}.js`;
      const index = /** @type {import('webpack').WebpackPluginInstance[]}  */ (
        config.plugins
      ).findIndex((plugin) => plugin instanceof NextMiniCssExtractPlugin);
      if (index !== -1) {
        const oldOptions = config.plugins[index].options;
        // eslint-disable-next-line @typescript-eslint/ban-ts-comment
        // @ts-ignore
        config.plugins[index] = new NextMiniCssExtractPlugin({
          ...oldOptions,
          filename: `static/css/[name]-${params.buildId}.css`,
          chunkFilename: `static/css/[name]-${params.buildId}.css`,
        });
        console.log(
          '!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Updated MiniCssExtractPlugin options',
        );
      }
    }
    return config;
  }

[junerd]

I think that might be a different issue since the chunk (js or css) files themselves are stable for me but I did have to alter the next.js.config to

/** @type {import('next').NextConfig} */
const nextConfig = {
  output: 'export',
  productionBrowserSourceMaps: true,
  generateBuildId: () => 'static-build-id',
  webpack: (config) => {
    // Next.js output is not stable when there's no code change and this is a work around
    config.output.filename = config.output.filename.replace('[chunkhash]', '[contenthash]');
    return config;
  },
  devIndicators: false,
};

The removeRandomness function is in a custom post build script which parses the generated index.html and modifies that script element

[barroudjo]

Thanks ! What a mess, you'd think having a deterministic build would be a priority for the Next team...

[cmdcolin]

for what it's worth, I managed to get @junerd's code to output stable ID like this: months later edit: removed. It didn't work. I will leave the details in a <details> below but know that it did not work as far as i could tell, there was still randomness in the hashes

I asked AI to generate a script to run the replaceRandomness function with jsdom

// replaceRandomness function credit @junerd
// https://github.com/vercel/next.js/discussions/65856#discussioncomment-12667717
function removeRandomness(el) {
  const prefix = `[\\"$\\",\\"$1\\",\\"`
  const suffix = `\\"`
  const prefixIndexes = []
  let index = -1
  do {
    if (typeof index === 'number') {
      prefixIndexes.push(index)
    }
    index = el.innerHTML.indexOf(prefix, index + 1)
  } while (index !== -1)
  prefixIndexes.shift()
  prefixIndexes.forEach(prefixIndex => {
    const startIndex = prefixIndex + prefix.length
    const endIndex = el.innerHTML.indexOf(suffix, startIndex)
    if (endIndex - startIndex < 2) {
      return
    }
    const randomness = el.innerHTML.substring(startIndex, endIndex)
    const original = el.innerHTML.substring(
      prefixIndex,
      endIndex + suffix.length,
    )
    el.innerHTML = el.innerHTML.replace(randomness, 'stable-id')

    const modified = el.innerHTML.substring(
      prefixIndex,
      el.innerHTML.indexOf(suffix, startIndex) + suffix.length,
    )
    // console.log(
    //   `👉 Replaced ${original} with ${modified} to stabilize build output`,
    // )
  })
}

/**
 * Example script to demonstrate how to use the stabilizeHtml.js file
 * This script processes an HTML file and removes randomness from it
 *
 * Usage: node processHtmlExample.js <input-html-file> [output-html-file]
 */

// Get command line arguments
const args = process.argv.slice(2)

if (args.length < 1) {
  console.error(
    'Usage: node processHtmlExample.js <input-html-file> [output-html-file]',
  )
  process.exit(1)
}

const inputFilePath = args[0]
const outputFilePath =
  args.length > 1 ? args[1] : inputFilePath.replace('.html', '.stable.html')

// Process the HTML file
// console.log(`Processing HTML file: ${inputFilePath}`)
// console.log(`Output will be saved to: ${outputFilePath}`)

// Import required modules
const fs = require('fs')
const path = require('path')
const { JSDOM } = require('jsdom')

/**
 * Processes an HTML file to remove randomness
 * @param {string} inputFilePath - Path to the input HTML file
 * @param {string} outputFilePath - Path to save the processed HTML file (optional)
 * @returns {Promise<string>} - The processed HTML content
 */
async function processHtmlFile(inputFilePath, outputFilePath) {
  try {
    // Read the HTML file
    const htmlContent = fs.readFileSync(inputFilePath, 'utf8')

    // Parse the HTML using JSDOM
    const dom = new JSDOM(htmlContent)
    const document = dom.window.document

    // Find all elements in the document
    const allElements = document.querySelectorAll('*')

    // Apply removeRandomness to each element
    allElements.forEach(el => {
      // Create a wrapper with innerHTML property for compatibility with the function
      const wrapper = {
        innerHTML: el.outerHTML,
      }

      // Apply the removeRandomness function
      removeRandomness(wrapper)

      // If the innerHTML was changed, update the element
      if (wrapper.innerHTML !== el.outerHTML) {
        // Replace the element with the modified version
        const tempDiv = document.createElement('div')
        tempDiv.innerHTML = wrapper.innerHTML
        el.parentNode.replaceChild(tempDiv.firstChild, el)
      }
    })

    // Get the processed HTML
    const processedHtml = dom.serialize()

    // Save to output file if specified
    if (outputFilePath) {
      fs.writeFileSync(outputFilePath, processedHtml, 'utf8')
      console.log(`Processed HTML saved to ${outputFilePath}`)
    }

    return processedHtml
  } catch (error) {
    console.error('Error processing HTML file:', error)
    throw error
  }
}

processHtmlFile(inputFilePath, outputFilePath)
  .then(() => {
    console.log('HTML processing completed successfully')
  })
  .catch(error => {
    console.error('Failed to process HTML:', error)
    process.exit(1)
  })

then I run this over all files in the "out" directory (output of nextjs export:static build) with fd-find+gnu parallel

 "postbuild": "fd html out| parallel \"node createReproducibleHtml.js {} {}\"",

now, when I rebuild the site, I can use rclone to upload to AWS S3 and it only uploads the couple files that were updated.... (note: I use rclone sync because it is able to do checksums for file upload, aws s3 sync command only does file-last-modified which doesn't work since of course all file times are modified after build! rclone command with -c which is checksum: rclone sync -c -v out mysite:bucketname/folder/)

[dmitrc]

@leerob, the issue described in this thread is a serious blocker to scalable self-hosting.
Given the team's focus on deployment options and level playing field, is this something that can be looked at?

[junerd]

Combination of #65856 (comment) and #65856 (reply in thread) made it stable for me. The second one is pretty hacky though I have to say

[CaptainN]

The recent RFP was about stablizing the alternative build internals - the build apis that Vercel uses, for it's deployment strategy. The idea there is that OpenNext partners can use that, instead of relying on next-server.

The issue is that OpenNext wouldn't use the undocumented internal APIs that Vercel uses, because they were undocumented, and potentially unstable. But now they can.

The key takeaway is that FEWER key players will be using the standalone next-server, not more. I would not count on patches to next-server, not even for key problems like the one described here, or the various memory and even CPU cycle leaks. They are unlikely to get patched.

[dmitrc]

I am not sure how one relates to other. These are still the build outputs, whether they are declared in the manifest for the new adapters API, or not. It will still not solve the non-determinism of these builds when running multiple times for different stages etc, and result in the same category of issues with 404s from CDN and more. Not to mention we don't have a crystal ball re: the end usage of deployment solutions, and manual Docker/Kubernetes will always remain desirable for a non-negligible subset of users.

We shouldn't have to resort to these hacks that @junerd in describing, when generateBuildId's whole purpose is to handle cases like these.

[CaptainN]

To be clear - this is a bug. It's not a feature. The way to verify it is easy (use create next app to create a new app, then build it multiple times and note how the hash names change with each build, perhaps with a small subset of possible outputs.) It's just a question of whether the bug will be prioritized and fixed.

[sidoruk-sv]

Sorry, but wow!

Year 2025

The popular js/react "framework" still can not produce deterministic builds, so only hacks and post-build scripting save 😿

[GabenGar]

Works on vercel.

[sidoruk-sv]

@GabenGar does not work on my computer

[CNsixgod]

Does this happen in turbo build Mode?

[sidoruk-sv]

no, only Webpack variant is affected (so it's hash generation is based not on a content of the files but also includes timestamp)

rechecked myself and I see the amount of changed files is less for turbopack, but still because the folder name (which contains files like _buildManifest.js and _ssgManifest.js) are changing every build, we have a lot of "affected files". Even though _buildManifest.js and _ssgManifest.js are identical in both folders

[sidoruk-sv]

My concern here is not about reproducible builds across different machines (though that would be nice to have in 2025).

The real issue for me is that we deploy new releases to devices over very slow and unreliable internet connections. To save bandwidth, we try to upload only changed files between builds. But since the Next.js build output is not deterministic, file hashes often change even when the content hasn’t, which forces us to re-upload almost the entire out folder (we have output: 'export') unnecessarily.