Issues with npm datasource updates hitting memory limits on Mend-hosted infrastructure

[jamietanna]

How are you running Renovate?

A Mend.io-hosted app

Which platform you running Renovate on?

GitHub.com

Which version of Renovate are you using?

42.95.2

Please tell us more about your question or problem

We've recently seen an increase in projects using pnpm and yarn hitting memory issues when installing dependencies on a project

• pnpm Renovate stopped working on the open-source software fast-check #40541
• yarn Job fails with kernel-out-of-memory #40352
• yarn: Frequent intermittent kernel out of memory errors (with `npm` datasource) #38880
• pnpm: Renovate started failing with "kernel-out-of-memory" #40940
• pnpm: Increase memory - job fails with kernel-out-of-memory #40935
• pnpm: Renovate-job started to fail with a kernel-out-of-memory Exception #40943
• pnpm: Renovate job fails with kernel-out-of-memory #41020
• pnpm: Increased resources for OSS organization gemini-hlsw #41685

We're in the process of investigating this further, and will share updates in this Discussion.

Related upstream issue: pnpm/pnpm#8441

Logs (if relevant)

No response

[jamietanna]

One attempt at fixing using pnpm configuration: #40541 (reply in thread)

[jamietanna]

#40957 is a suggested change (which we can then use on Mend-hosted)

[ssbarnea]

Is this the magic conversion trick for funnelling more free users into paying customers? -- clearly would not be able to squeeze much from unfunded open source project using it. /sarcasm

Now here is an funny aspect that might have contributed to the OOM issue: the recent new dependency added to the project was renovate npm package itself, which:

• is the only way to install renovate-config-validate (linting config)
• has a huge list of dependencies
• has a huge list of outdated and compromised dependencies
• is updated extremely often (not a problem by itself but if you only need linting config... it becomes one)

I wonder if we can spot a pattern among those affected by this bug.

Considering the number of users reporting this issue very recently, is almost for sure related to increase of memory needs from newer releases. Still, the increase might prove to not be critical and an increase of node memory of 10-20% might address it but sadly I did not see any signs of this being tried on mend.io services.

I seen the need to increase this in the past with tools like webpack.

[jraoult]

Hi @jamietanna, we've been hitting this consistently in our repo for 4 days in a row (therefore blocking any dependency upgrades). Is there an official workaround at the moment?

Happy for you to use our repo if you need a reproduction case.

[jamietanna]

Short answer, no, sorry! We're working on the CLI changes that would allow specifying these environment variables (but a few other things have also popped up needing work on them)

[jamietanna]

A couple of tests I've done so far with (against a few repos who've reported issues):

{
    "toolSettings": {
      "nodeMaxMemory": 1024
    }
  }

seem to be going positively - I'll do a little bit more testing, then we'll have this set across all repos

[Methuselah96]

We had to increase nodeMaxMemory to 1.5GB to fix the out-of-memory errors

[CHC383]

If I understand correctly, toolSettings.nodeMaxMemory is to limit the memory taken by Node, and thus avoid the kernel-out-of-memory issue right?

Unfortunately, setting it to 1024 is not a stable workaround. I still get kernel-out-of-memory from time to time (limit doesn't work or components other than pnpm still take too much memory?). And sometimes even if the run didn't fail with kernel-out-of-memory, pnpm failed with FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory.

Is it really due to pnpm being memory consuming, or is there memory leak somewhere?

Environments:

• Node: v24.14.1
• pnpm: v10.33.0

[Methuselah96]

Never mind, we're still experiencing issues. We've tried setting nodeMaxMemory to 1.0GB, 1.5GB, 2.0GB, and 2.5GB and they all result in a kernel-out-of-memory error.

[jamietanna]

From the debugging I've done, it seems to appear to line up with pnpm / yarn on very large repos - I think if we can get a bug report together for pnpm, they may be able to advise better

[CHC383]

Here is an existing issue on pnpm: pnpm/pnpm#8441

[CHC383]

@jamietanna As mentioned in pnpm/pnpm#8441 (comment), this situation is getting worse on pnpm v11. I wonder if Renovate could expose a way to set PNPM_MAX_WORKERS or PNPM_WORKERS (code). Reducing the number of workers might lower pnpm’s memory usage.

[jamietanna]

Note that I did try that setting a while back (#40541 (reply in thread)) and it didn't seem to change much

If you can share a repository / Job ID for a repository (if you don't want to share repo name publicly) I can set the config for you on your repo, to see if it makes any difference right now

[CHC383]

Here is the job id: d8c69a06-9ac6-4af0-a35d-b965e2825d1b, thanks

[jamietanna]

Actually, re-reviewing Renovate's code, I think we can't actually specify PNPM_MAX_WORKERS as something that goes to the child without explicitly passing it, as it's not marked as basicEnvVars (https://github.com/renovatebot/renovate/blob/HEAD/lib/util/exec/env.ts) or in global:safeEnv to allow setting it in repo config 🤔

[PatrykWalach]

I've pinned pnpm to 10.23.0, this is last version before concurrency was changed from 1-16 to 16-64 https://github.com/pnpm/pnpm/pull/10215/changes

[CHC383]

@PatrykWalach Thanks for the findings. Combining networkConcurrency: 16 on pnpm-workspace.yaml (pnpm v11.0.8) and toolSettings. nodeMaxMemory : 1024 at least unblock Renovate from creating the PR, here are some of my cases just for reference:

• pnpm v10 starts to have kernel OOM from some point
• pnpm v10 + nodeMaxMemory 1024: PR was able to be created occasionally (kernel OOM for the rest), but package-lock.yaml failed to be generated due to heap OOM
• pnpm v11 + nodeMaxMemory 1024: Always kernel OOM

Just based on a few runs, didn't do exhausted tests

• pnpm v11 + networkConcurrency 16 + nodeMaxMemory 1024: PR created, package-lock.yaml failed
• pnpm v11 + networkConcurrency 16: kernel OOM
• pnpm v11 + networkConcurrency 12: kernel OOM

---

Update: Unfortunately pnpm v11 + networkConcurrency 16 + nodeMaxMemory 1024 still fails most of the time except for the one partial success above

[jamietanna]

OK, we can now specify PNPM_MAX_WORKERS and PNPM_WORKERS across Mend-hosted repos - anyone want to test it out?

It'll need some rebasing of open PRs to check that it definitely does work, so would prefer not to test it without the repo owner's consent!

[CHC383]

My repo doesn't have an open PR as the PR creation job keeps failing, maybe you could try to adjust the pnpm workers setup to see if you could create a Renovate PR successfully.

I have pnpm networkConcurrency set to 16 and toolSettings.nodeMaxMemory set to 1024. Please let me know if you want me to remove them to eliminate these factors.

Job id: 019e1c21-92b4-71d1-aad3-d1841bceb020

[jano-duvo]

We experience same issue with either silent OOM crash (no lockfile update) or kernel-out-of-memory failure, I am going to experiment with dedupePeers: true next, as that should simplify the dependency graph.

maxNodeMem=2.5gb
PNPM_MAX_WORKERS=1

Job id: 0fbf631f-e48b-4018-811a-5feba8a4e96f

[jamietanna]

Sorry folks - bit of a busy week so haven't been able to test this out for folks. However, it's now possible to use env in your repo config, so you can test this out as you'd like to see if this works for you 🤞🏽

(We're deciding whether we would set value(s) for everyone, by default)

[CHC383]

Took a closer look on the pnpm latest (v11) source code, I don't think setting either PNPM_WORKERS or PNPM_MAX_WORKER would help:

• The only place uses these two environment variables is: calcMaxWorkers
  • PNPM_MAX_WORKERS: As the name suggests, set the max workers
  • PNPM_WORKERS: The name is confusing, it actually means the idle CPUs you want to keep. That's why the calculation is available cpus - PNPM_WORKERS - 1 (likely reserved for pnpm main process)
• The only place uses calcMaxWorkers is calculating network concurrency, however, the minimum is 16, so max workers <= 5 won't have any impact, and max workers > 5 would increase the network concurrency.
• I have tried to explicitly set opts.networkConcurrency to 16 which is the minimum we can have by setting max works in the current pnpm implementation, and it doesn't work as my comment above.

---

As of writing this, I think we could try lowering network concurrency value, like 8 or 4, afterall before 10.23.0 it was just number of cores.

https://github.com/pnpm/pnpm/pull/10215/changes

I will do some experiments later.

[CHC383]

I have tried lowering the pnpm opt.networkConcurrency from 8 to 4 to 2 and to 1, unfortunately Mend hosted job still got kernel OOM even with concurrency set to 1.

Another thing I noticed is that all the failed jobs in this round of testing ends at running corepack use pnpm@xxx based on the last line of logs. Previously some of them failed at "pnpm install xxx". Not sure if this is due to latest log not being captured, or kernel OOM actually happened at the corepack step. For the latter case it might mean something else is going on.

@jamietanna For testing and experimenting, I wonder if it makes sense for you to clone Renovate CLI repo itself, use preset "group:allNonMajor" to group updates, wait for a few days to pile up some updates, and run on the free Mend hosted app plan.

[esetnik]

We're still stuck on this issue. None of the proposed solutions are working.

[jsa4000]

Me too, I am also getting OOM 99.9% of cases.

[CHC383]

With the memory fix/optimization in pnpm v11.3, the situation seems to be improved. Besides, pnpm is moving to Rust implementation, the memory pressure might be further down with pnpm v12.

---

P.S. Node v24.16.0

[haines]

Unfortunately I can't report the same. We have upgraded to pnpm v11.3.0, set trustLockfile: true to disable the supply chain integrity checks, set toolSettings.nodeMaxMemory to 1536, and Renovate still OOMs 😢

[haines]

Here is a reproduction: https://github.com/haines/pnpm-memory

Without toolSettings.nodeMaxMemory, the job fails with kernel-out-of-memory.

With toolSettings.nodeMaxMemory set to 1536, the job still fails with kernel-out-of-memory.

With toolSettings.nodeMaxMemory set to 1024, the job completes and creates a pull request, but pnpm install ran out of memory so there is an artifact update problem reported:

<--- Last few GCs --->

[1013:0x23bcf000]   112428 ms: Scavenge 1015.7 (1022.8) -> 1013.0 (1022.8) MB, pooled: 0 MB, 5.39 / 0.00 ms  (average mu = 0.317, current mu = 0.283) allocation failure; 
[1013:0x23bcf000]   114162 ms: Mark-Compact (reduce) 1015.7 (1022.8) -> 1008.4 (1012.8) MB, pooled: 0 MB, 873.22 / 0.01 ms  (+ 821.5 ms in 42 steps since start of marking, biggest step 23.7 ms, walltime since start of marking 1734 ms) (average mu = 0.308,
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x744ae8 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/opt/containerbase/tools/node/24.16.0/bin/node]
 2: 0xc1a4b0  [/opt/containerbase/tools/node/24.16.0/bin/node]
 3: 0xc1a59f  [/opt/containerbase/tools/node/24.16.0/bin/node]
 4: 0xebdda5  [/opt/containerbase/tools/node/24.16.0/bin/node]
 5: 0xebddd2  [/opt/containerbase/tools/node/24.16.0/bin/node]
 6: 0xebe0ca  [/opt/containerbase/tools/node/24.16.0/bin/node]
 7: 0xecedca  [/opt/containerbase/tools/node/24.16.0/bin/node]
 8: 0xed3170  [/opt/containerbase/tools/node/24.16.0/bin/node]
 9: 0x19655d1  [/opt/containerbase/tools/node/24.16.0/bin/node]
/usr/local/bin/node: line 18:  1013 Aborted                 /opt/containerbase/tools/node/24.16.0/bin/node "$@"

I've linked the repro on the pnpm issue: pnpm/pnpm#8441 (comment)

[Methuselah96]

This was not happening to me for the past couple months, but it's now happening to me on every run after upgrading from pnpm 11.3.0 to 11.5.0.

[Methuselah96]

Things seem to be working for me again on pnpm 11.5.2 so far

[kopach]

Seems like an issue with PNPM itself. As stated here, downgrading to PNPM 11.1.2 might solve the issue (as temporary workaround)

[kopach]

Setting the PNPM_CONFIG_TRUST_LOCKFILE=true env variable would solve the issue. Here is a PR to the Renovate repo which would allow this #43776

Context:
PNPM 11.1.3+ introduces an important security update that now checks every dependency in the lockfile on every install against security policies.
It is generally safe to disable this security mechanism for renovate runs if you have other checks on PR which will do the same security check when running pnpm install

[haines]

As noted here, that setting was insufficient to solve the issue for us. It's probably worth doing, but it's not a complete fix.

[kopach]

I see, yes, you're right; that might not help then

[Mathias-S]

@jamietanna What is the status of this issue? What should we be doing? Renovate has more or less ceased working for us now, we need trigger the jobs manually and even then it only works once in a blue moon:

The comment and reproduction by haines is exactly what we are experiencing: #40942 (comment)

This was happening with both pnpm 10 and 11, but it's worse in pnpm 11. We set trustLockfile: true, but it makes no difference.

toolSettings.nodeMaxMemory set to 1024 causes javascript heap OOM errors in pnpm.

toolSettings.nodeMaxMemory set to 1536 or more causes kernel OOM errors.

[esetnik]

Same issue. Basically can't use renovate anymore 😢