Temp workaround (that works for me!) for SSL certificate renewal bug

[EDIflyer]

I know there are already lots of issues on this topic - I've tried to link to most of them below. I've just had to renew 16 sites on one server (running the latest v.2.10.2) and thought I'd go through the process that seemed to work reliably for me in case it helps others (with thanks to posters in other issues where I've gleaned this info from!).

Given how it works I suspect the issue is the the requests to the ACME endpoint not being allowed through when force SSL is enabled (as mentioned in some bug reports) and I'm hopeful @jc21 can merge in #2038 that seems to be an option (but is unfortunately now based off an older base).

Symptom

SSL certificates do not automatically renew and you receive a warning email from LetsEncrypt about an upcoming expiring certificate (typically I seem to get them when <20 days left to go). Attempts to manually review end up just showing an 'Internal server error'

Workaround
Part 1 - clear any certbot.lock files
I've found there is sometime an error caused by a a duplicate instance of CertBot running. You can check whether there are .certbot.lock files in your system:

find / -type f -name ".certbot.lock"

If there are, you can remove them:

find / -type f -name ".certbot.lock" -exec rm {} \;

(from https://community.letsencrypt.org/t/solved-another-instance-of-certbot-is-already-running/44690/2)

Part 2 - turn off Force SSL and then renew

After clearing any certbot lock, I then went through site by site and 1) disabled Force SSL on the proxy host page then 2) requested certificate renewal on the SSL page and then 3) re-enabled SSL and all sub-options back on the proxy host page.

As I say it takes a while and is frustrating but I found it worked reliably and they're all now renewed for the next 3 months. If you don't switch off Force SSL then you just end up with an internal error.

Related issues on this topic (in the hope that once this issue is resolved these can all be closed)
#1771 #1816 #1856 #2048 #2251 #2258 #2267 #210 #2418 #2499 #2593 #2642 #2713 #2860

[ririko5834]

It really works thanks. Certs were before normally renewed, found out that it stopped working when I updated to the latest version. When it will be fixed?

[EDIflyer]

Agree, @ririko5834 - it used to work fine for me but seemed to stop a few version ago. Hopefully the PR can be merged in to the latest codebase....

[pd5rm]

Just did this, thanks for the workaround writeup.

[jhalak1984]

Hmm, for me, no such files were found, hence, didn't work for me

[EDIflyer]

@jc21 any word on when a fix might be coming for this SSL cert renewal issue? That's me having to manually renew another 10 sites this evening 😔

[JohnnyLAmpAz]

Hmm, for me, no such files were found, hence, didn't work for me

Neither I found any lock file but the trick disabling force SSL its the important part! Try anyway

[EDIflyer]

Hmm, for me, no such files were found, hence, didn't work for me

@jhalak1984 did you try part 2? The main bit seems to be force SSL not allowing an ACME exclusion, the first bit is just to ensure no conflicting certbot instances running.

[EDIflyer]

Thanks to the work from @the1ts in #2038 and the comments from @Whoopsadaisy re regex on that PR #2038 (comment) I've created a new PR #3121 that combines their comments to stop /.well-known/acme-challenge requests from being redirected to https.

The new PR has been build (you can access it in a docker compose file by commenting out your current image and using image: 'jc21/nginx-proxy-manager:github-pr-3121' instead). The only change I made was to the one force-ssl-.conf file, but it is based off the current develop branch (2.10.4 as of today) so will include any other changes on there.

I've tried it on two servers that I run - on the first I was now enable to renew OK just by clicking 'renew now' on the SSL page (something that previously errored out). On the other one I initially still got the internal error but when I ran the first bit of the code in my OP above I found 3 certbot instances running so once I cleared them it seemed to renew OK. Out of interest I've only renewed one certificate on that server to see if the rest renew OK automatically. In both cases everything still seems to redirect to https OK and the regex seems to check out OK - (https://regex101.com/r/H58N25/1)

If you're happy to do so then please test it out - it is showing as OK to merge so if this merges hopefully it'll be accepted by @jc21 😃

PS - I checked back 10-15 min later and it seems that all the other certs have autorenewed too so that saved me quite a bit of work switching force SSL off/on for each one!

[jhalak1984]

Awesome!! Works without a hitch now. Thank you!!!

[Panoramiac]

Thanks to the work from @the1ts in #2038 and the comments from @Whoopsadaisy re regex on that PR #2038 (comment) I've created a new PR #3121 that combines their comments to stop /.well-known/acme-challenge requests from being redirected to https.

The new PR has been build (you can access it in a docker compose file by commenting out your current image and using image: 'jc21/nginx-proxy-manager:github-pr-3121' instead). The only change I made was to the one force-ssl-.conf file, but it is based off the current develop branch (2.10.4 as of today) so will include any other changes on there.

I've tried it on two servers that I run - on the first I was now enable to renew OK just by clicking 'renew now' on the SSL page (something that previously errored out). On the other one I initially still got the internal error but when I ran the first bit of the code in my OP above I found 3 certbot instances running so once I cleared them it seemed to renew OK. Out of interest I've only renewed one certificate on that server to see if the rest renew OK automatically. In both cases everything still seems to redirect to https OK and the regex seems to check out OK - (https://regex101.com/r/H58N25/1)

If you're happy to do so then please test it out - it is showing as OK to merge so if this merges hopefully it'll be accepted by @jc21 😃

PS - I checked back 10-15 min later and it seems that all the other certs have autorenewed too so that saved me quite a bit of work switching force SSL off/on for each one!

So what do I need to do to get this working with Nginx Proxy Manager Addon running on HomeAssisant? I got yesterday the mails that my certs will expire soon. I do not know what went wrong, but I can not renew them and currently, the certs are also not accepted anymore by the Android App (I guess I messed something up by adding my subdomains to the DuckDns Addon).

[EDIflyer]

@Panoramiac sorry I'm running it on VPS with Docker/Portainer so can easily specify a different image to use - I'm not 100% sure re Home Assistant how to do that (I do run HA but only on my NAS on my home network) - might be worth asking in the HA forums if someone knows how to specify a different image to use?

[mtojay]

hmm for me it does not work. received a mail about expiring certs. went and try to renew. didnt work. looked up on the internet and found this workaround. but for me it does not work. there are not certbot.lock files in my docker container. and disabling ssh also does not do anything. tried rebooting the container, the vps and all steps here. its still always: "internal error". not quite sure where to go from here.

[EDIflyer]

@mtojay did you try the different version in the PR I submitted? I've been running it since I created it and all auto-renewals going through fine for me.

[mtojay]

thx for your answer @EDIflyer, but yeah i tried taht. i pulled the docker image with your PR, but i still get "Internal Error". After recreating the container with the new image i tried looking for locked Certbots again, but i dont have any locked certbot instances. I dont know what or if am doing wrong, but i cant get new ceritificates no matter how often i try what had been suggested here in this thread.

I probably have an unrelated issue. If my certs expire in t he coming days i will retry again.

[EDIflyer]

Ah OK, sorry to hear that @mtojay. At least you've ruled out locked Certbots. Is there anything more in the NPM logs that you can see when you try to renew and get the error? The issue at #1816 goes through some of what others found. Sorry I can't help more!

[fhazal]

this workaround is not working for me too, i change the image to image: 'jc21/nginx-proxy-manager:github-pr-3121' still can't get it work, i follow the instruction and delete and reinstall NPM still can't create SSL cert or renew the cert, please help.

[EDIflyer]

Hmm weird, I'm still using that one and it works OK. Does it pull the image down OK?

[fhazal]

Hmm weird, I'm still using that one and it works OK. Does it pull the image down OK?

yup it did pull the image without any error.

[EDIflyer]

OK - I presume port 80 is open on your firewall to allow the certbot requests to get through to the acme endpoint? I'm afraid I don't have many other ideas!

[broetchenrackete36]

PR #3121 worked great for me. Finally my certs are renewed automagically again, thx :)

[peterge1998]

I guess I experience the same problem, the certs aren't renewing in my instance too. docker logs gives this:

[11/10/2023] [7:55:11 AM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --work-dir "/tmp/letsencrypt-lib" --logs-dir "/tmp/letsencrypt-log" --preferred-challenges "dns,http" --disable-hook-validation  
Another instance of Certbot is already running.

    at ChildProcess.exithandler (node:child_process:402:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1100:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)

I guess a fix will be released in near future?

[EDIflyer]

@peterge1998 it's been like this for months so sadly I'm not sure a fix is imminent. Did you try the code posted above re duplicate certbot instances or running the PR version I created? Worth a shot but no guarantees it'll help I'm afraid!

[peterge1998]

@peterge1998 it's been like this for months so sadly I'm not sure a fix is imminent. Did you try the code posted above re duplicate certbot instances or running the PR version I created? Worth a shot but no guarantees it'll help I'm afraid!

How can I run the PR version with docker?

[EDIflyer]

How can I run the PR version with docker?
If you replace the normal image with image: 'jc21/nginx-proxy-manager:github-pr-3121' in your docker compose (or tweak appropriately for docker run) command then that should do the trick.

[apoorv569]

@apoorv569 have you tried the version in my PR #3121 ? If turning off SSL worked for you that should hopefully fix it for renewals.

Sorry for the late response.

Fix as in I have to apply a PR to my install?

[EDIflyer]

@apoorv569 as in running the Docker container generated by the PR instead of latest

[apoorv569]

@apoorv569 as in running the Docker container generated by the PR instead of latest

Like I have to build the docker container my self using commits from the PR?

Why is your PR not merged yet? almost 1.5 years now.. what's stopping the dev from merging it?

BTW can you in short tell me why this happens?

[EDIflyer]

@apoorv569 no if you cjeck in the PR they build a container based on the PR, so I your docker compose you just point to that instead. Let me know if you can't find it and I'll look it out.

I'm not sure why they've not merged it yet, I think it would close lots of the open issues!

[EDIflyer]

Here it is:

nginxproxymanager/nginx-proxy-manager-dev:pr-3121

[apoorv569]

Here it is:

nginxproxymanager/nginx-proxy-manager-dev:pr-3121

Thanks. All my certs are up to date ATM but next time they fail to auto update I'll try out your build, which should be in March I think as its valid for 3 months IIRC.

[fcar12]

What worked for me was to temporary add a forward rule on the router to enable port 80 traffic to the website. Because apparently there were problems with the accessibility of the website from Let's Encrypt. After the refresh I turned off the forward rule to port 80 again.

The most simple and efficient solution until this is fixed with the PR! :)

[apoorv569]

Here it is:

nginxproxymanager/nginx-proxy-manager-dev:pr-3121

Hey, my certificates expires today and it still fails to update them. So should I continue with this version only? or do you a newer one as well?

[EDIflyer]

Hi @apoorv569 i updated this a few months ago and it's still working well for me. I'll look at syncing the updates to the latest version when I've got a chance to test it'll still work for me. Even better @jc21 hopefully might merge in the PR 🤞

[apoorv569]

Hi @apoorv569 i updated this a few months ago and it's still working well for me. I'll look at syncing the updates to the latest version when I've got a chance to test it'll still work for me. Even better @jc21 hopefully might merge in the PR 🤞

Oh cool. BTW if update to your version will I be able to go back to the official/stable tag later when it finally gets merged?

[EDIflyer]

AFAIK yes, my changes are tiny (you'll see them in the PR) and don't do anything database-wise, although it's always advised to back it up first!

[apoorv569]

AFAIK yes, my changes are tiny (you'll see them in the PR) and don't do anything database-wise, although it's always advised to back it up first!

I see. All I want to do is update my certs nothing else.

[EDIflyer]

Yep, it should do the trick - just trying to follow best practice and recommend backing up 🙂

[apoorv569]

Yep, it should do the trick - just trying to follow best practice and recommend backing up 🙂

Hey after updating to your version, I started getting this error,

app-1  | [3/9/2025] [1:57:04 PM] [Migrate  ] › ℹ  info      Current database version: 20240427161436
app-1  | [3/9/2025] [1:57:04 PM] [Global   ] › ✖  error     The migration directory is corrupt, the following files are missing: 20240427161436_stream_ssl.js Error: The migration directory is corrupt, the following files are missing: 20240427161436_stream_ssl.js
app-1  |     at validateMigrationList (/app/node_modules/knex/lib/migrations/migrate/Migrator.js:566:11)
app-1  |     at Migrator.latest (/app/node_modules/knex/lib/migrations/migrate/Migrator.js:69:7)
app-1  |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Thankfully, I was able to revert back to the image jc21/nginx-proxy-manager:latest and it worked without errors. I guess I'll update it with the hack "turning off force ssl" for now.. maybe next time the certs expires the PR would be merged by then 🤞

[EDIflyer]

@apoorv569 ah OK - were you coming from the latest public current version? I wonder if something has changed in the DB and this would have effectively been a downgrade. It looks like there have been 42 changes to the latest version of dev:

Looks like some streams DB stuff in it:

I'll sync to the latest and see how that goes

[apoorv569]

@apoorv569 ah OK - were you coming from the latest public current version? I wonder if something has changed in the DB and this would have effectively been a downgrade. It looks like there have been 42 changes to the latest version of dev:

Looks like some streams DB stuff in it:

I'll sync to the latest and see how that goes

I use the :latest so whatever is the "latest" I guess.

[EDIflyer]

OK cool - I've synchronised the fork now, just waiting for the updated Docker image to be built.

[EDIflyer]

@apoorv569 FYI that's the docker image for 3121 updated now.

[apoorv569]

@apoorv569 FYI that's the docker image for 3121 updated now.

I see. Appreciate it, but all my certs are updated now, sorry I had to update them some of the services refuses to work without a good cert.

I can try to check the DB issue though if you want.

[EDIflyer]

@apoorv569 don't worry - I've just pulled the new Docker image for one of my sites and it seemed to update to 2.12.3 fine and I managed to manually renew a cert without a prob (and without turning off SSL!) so hopefully is all good.

[MahmoudAlyuDeen]

Certificates expired today. Time to try traefik then 🤷‍♂️

[EDIflyer]

@MahmoudAlyuDeen or feel free to try the latest version from my 3121 PR and see if that works for you 😃

[bengivre]

@MahmoudAlyuDeen or feel free to try the latest version from my 3121 PR and see if that works for you 😃

In my case, I still have to disable the "Force SSL" on my Proxy Hosts, if I want to renew manually from the UI . I did not try auto with the new version .

That would be pretty good if the process of auto-renew & from the manual "Renew now" from the admin panel" could do that on it's own.

1- disable Force SSL if enable
2- renew SSL
3- re-activate Force SSL

That would be it.

Thanks !

[EDIflyer]

Hmm, interesting @bengivre - manual renewals as well as automatic work OK for me with the PR version.

[bengivre]

Hmm, interesting @bengivre - manual renewals as well as automatic work OK for me with the PR version.

My bad, I did not use the PR version. I just upgrade to latest . That must be the problem. Sorry about that.
I'll wait they merge your PR :)

[EDIflyer]

Ahh OK that makes sense (and at least confirms my PR is still required 😂) - basically all my PR does is bypass the force SSL just for the Acme certificate check. The PR is pretty up to date with the latest version so feel free to try it out (just backup the DB first just in case)

[bengivre]

Thanks for this. Hopefully we finally get something that work as expected ;)

[virtualdj]

Did you try to ping the maintainer to try to merge this PR?

[EDIflyer]

Did you try to ping the maintainer to try to merge this PR?

Yep, multiple times over the months (year?) 😥