[Bug]: Product updates with many variants and images memory usage/problem

[mikkel-lindstrom]

Package.json file

{
  "name": "my-medusa-performance-store",
  "version": "0.0.1",
  "description": "A starter for Medusa projects.",
  "author": "Medusa (https://medusajs.com)",
  "license": "MIT",
  "keywords": [
    "sqlite",
    "postgres",
    "typescript",
    "ecommerce",
    "headless",
    "medusa"
  ],
  "scripts": {
    "build": "medusa build",
    "seed": "medusa exec ./src/scripts/seed.ts",
    "start": "medusa start",
    "dev": "medusa develop",
    "test:integration:http": "TEST_TYPE=integration:http NODE_OPTIONS=--experimental-vm-modules jest --silent=false --runInBand --forceExit",
    "test:integration:modules": "TEST_TYPE=integration:modules NODE_OPTIONS=--experimental-vm-modules jest --silent=false --runInBand --forceExit",
    "test:unit": "TEST_TYPE=unit NODE_OPTIONS=--experimental-vm-modules jest --silent --runInBand --forceExit"
  },
  "dependencies": {
    "@medusajs/admin-sdk": "2.10.3",
    "@medusajs/cli": "2.10.3",
    "@medusajs/framework": "2.10.3",
    "@medusajs/medusa": "2.10.3",
    "@mikro-orm/core": "6.4.3",
    "@mikro-orm/knex": "6.4.3",
    "@mikro-orm/migrations": "6.4.3",
    "@mikro-orm/postgresql": "6.4.3",
    "awilix": "^8.0.1",
    "pg": "^8.13.0"
  },
  "devDependencies": {
    "@medusajs/test-utils": "2.10.3",
    "@mikro-orm/cli": "6.4.3",
    "@swc/core": "1.5.7",
    "@swc/jest": "^0.2.36",
    "@types/jest": "^29.5.13",
    "@types/node": "^20.0.0",
    "@types/react": "^18.3.2",
    "@types/react-dom": "^18.2.25",
    "jest": "^29.7.0",
    "prop-types": "^15.8.1",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "ts-node": "^10.9.2",
    "typescript": "^5.6.2",
    "vite": "^5.2.11",
    "yalc": "^1.0.0-pre.53"
  },
  "engines": {
    "node": ">=20"
  },
  "packageManager": "yarn@1.22.22+sha512.a6b2f7906b721bba3d67d4aff083df04dad64c399707841b7acf00f6b133b7ac24255f2652fa22ae3534329dc6180534e98d17432037ff6fd140556e2bb3137e"
}

Node.js version

v20.x

Database and its version

PostgreSQL 16.6

Operating system name and version

Windows/Linux

Browser name

No response

What happended?

When updating products with large quantity of images the memory usage explodes and often causes heap memory issues. Using UpdateProductsWorkflow.

OBS: it doesn't seem to be isolated to version 2.10.3 as i also reproduced on 2.8.3.

Expected behavior

Product updates shouldn't be memory intensive when updating default columns field for the object regarding the amount of variants or images

Actual behavior

When updating products with large quantity of images and variants the memory usage explodes and often causes heap memory issues. Using UpdateProductsWorkflow.
If a product has 1 variant and 100 images its fine, but if the product has 100 variants and 100 images something weird is going on and the memory easily consumes 6gb memory in seconds often causes the heap to run out.

Link to reproduction repo

https://github.com/mikkel-lindstrom/medusajs-2.10.3-product-performance

[HectaSquareUK]

Same issue observed when adding 180+ variants , slow, and when images added, it crashes

[anubus298]

same here!

[anubus298]

i noticed that reading/updating/creating product with many variants and images cause the same issue.

[mikkel-lindstrom]

i noticed that reading/updating/creating product with many variants and images cause the same issue.

I have noticed the same, if you overwrite the middleware and select the fields you want it helps alot when navigating in medusa
Heres what ours look like (Some fields like collection.id and collection_id is used in different widgets in default medusa, so so make sure all function still work i keep them)

export default defineMiddlewares({
routes: [
{
matcher: '/admin/products/:id',
method: 'GET',
middlewares: [
validateAndTransformQuery(createFindParams(), {
defaults: [
'id',
'title',
'description',
'subtitle',
'discountable',
'handle',
'weight',
'length',
'height',
'width',
'hs_code',
'origin_country',
'mid_code',
'material',
'status',
'metadata',
'collection.id',
'collection.title',
'collection_id',
'tags.id',
'tags.value',
'sales_channels.id',
'sales_channels.name',
'images.id',
'images.url',
'images.rank',
'type_id',
'type.value',
'options.id',
'options.title',
'options.product_id',
'options.values.id',
'options.values.value'
]
})
]
},

[mikkel-lindstrom]

@olivermrbl Are you aware of these performance issues?

[NicolasGorga]

Hello @mikkel-lindstrom thank you for the reproduction repo. I could reproduce this on both 2.10.3 and the latest 2.11.3. I think this is also related to another reported issue, and the culprit, off the top of my head, seems to be too many related entities being loaded in memory when expanding relations. I will let you know once I have an update :)

[mikkel-lindstrom]

Yes i also experience the same. But it seems to be more of a problem when its images that are a large quantity, i dont know if its something to do with how they are fetched/stored.
Another problem is that when it then loads/updates these big products it uses all the cpu, so instead of just being slow it forces alot of timeouts for requests i have not looked at the source code but i guess its because it parrallelize the request to fetch all the related entities