[Kamal 2] Are ENV variables set in .env available in .kamal/secrets?

[jessevdp]

After initializing a new project with Kamal 2.0.0.rc3, I have a setup like the following:

# config/deploy.yml

# ...

registry:
  username: my-username
  password:
    - KAMAL_REGISTRY_PASSWORD

# .kamal/secrets
KAMAL_REGISTRY_PASSWORD=$KAMAL_REGISTRY_PASSWORD

# .env
KAMAL_REGISTRY_PASSWORD="<REDACTED>"

Running kamal registry login results in the following error:

$ kamal registry login
  INFO [078e4b73] Running docker login -u [REDACTED] -p [REDACTED] as me@localhost
 DEBUG [078e4b73] Command: docker login -u [REDACTED] -p [REDACTED]
 DEBUG [078e4b73]       flag needs an argument: 'p' in -p
 ERROR (SSHKit::Command::Failed): docker exit status: 256
docker stdout: Nothing written
docker stderr: flag needs an argument: 'p' in -p

---

This leads me to believe that $KAMAL_REGISTRY_PASSWORD is never filled. Do we have access to those ENV variables in .kamal/secrets?

[nickhammond]

@jessevdp There are some WIP docs for 2.0, here are a few helpful pages:

.env to Kamal secrets: https://github.com/basecamp/kamal-site/blob/kamal-2/docs/upgrading/secrets-changes.md
Kamal Secrets: https://github.com/basecamp/kamal-site/blob/kamal-2/docs/commands/secrets.md

[jessevdp]

Hey!

Thanks for taking the time to write back!

I found the same WIP documentation. It mentions that you can use values from ENV inside of the .kamal/secrets file. And that .kamal/secrets is in-and-of-itself loaded through dotenv.

I guess that this means that you can only load variables from your shell ENV. Not from dotenvs .env file.

But I wasn't completely sure...

Re-reading through those docs a bit more I suppose the section on environment variables in deploy.yml solidifies my assumption even more. The .env file is NEVER loaded in Kamal 2.

---

I want to move to 1password stuff through the kamal secrets CLI. I just hadn't gotten around to it. Perhaps I should just bite that bullet.

[tillcarlos]

For me it works to just call it with dotenv kamal deploy

[jessevdp]

Yeah fair!

That adds to my conclusion:

Kamal 2 does not load the .env file. You can only access ENV variables from your shell.

You can of course add another tool into the chain; as you suggest by prefixing with the dotenv CLI, or by using a tool like https://direnv.net/, etc.

Thanks!

[nickhammond]

Kamal just loads .kamal/secrets, .kamal/secrets.production, .kamal/secrets-common instead of .env. You can interpolate or pull from your actual ENV, it's up to you.

The upgrade guide is also out which goes over this https://kamal-deploy.org/docs/upgrading/overview/

[laptopmutia]

@nickhammond I still cannot make it works, can you help me where to put this .env files? did we need to add a gem to make it works? did I need to execute some command to make it works?

the doc is not really helping me at all

I have my old .env that works on kamal 1 that I put on my rails project roots folder
and I have set my .kamal/secrets

and it still not working

[buncis]

@jessevdp I don't what to be that guy who nitpicking

but could you just mark @tillcarlos answer as the answer, because its more straight forward, and practically solve the question.

so people that googling the answer could just find the solution,

by that we could lessen the confusions and avoiding bad approach like putting the pass to .secrets or using cat commands to copy values in the env files

[charnould]

Hi,
Kind of same problem here; I don't get how environnement variables and/or secrets work.

With Kamal 1.x, everything was working:

• simple .env file with my secrets
• in deploy.yml:

env:
   secret:
    - API_KEY

With Kamal 2.x:

• in .kamal/secrets:
  API_KEY=$API_KEY

• in .env:
  API_KEY=my-super-secret-key

• in deploy.yml:

<% require "dotenv"; Dotenv.load(".env") %>
env:
   secret:
      - <%= ENV["API_KEY"] %>

... and when I run kamal config: CRASH

 ERROR (Psych::SyntaxError): (<unknown>): found character that cannot start any token while scanning for the next token at line 51 column 7

(There is nothing weird at line 51: just another secret: - <%= ENV["PASSWORDS"] %>)

Any clue?
Docs are not that clear for me :-(
Thanks!

[jessevdp]

I don’t think you want to use ERB interpolation under env.secret. Just put the name of the ENV variables there. Does that work?

<% require "dotenv"; Dotenv.load(".env") %>
env:
   secret:
      - API_KEY

[charnould]

I don’t think you want to use ERB interpolation under env.secret.
Just put the name of the ENV variables there. Does that work?

<% require "dotenv"; Dotenv.load(".env") %>
env:
   secret:
      - API_KEY

Thanks, I've made some progress: I've got a new kind of error :-)

.env:
A list of env var.

deploy.yml:

<% require "dotenv"; Dotenv.load(".env") %>
env:
  secret:
    - OPENAI_API_KEY
    - REGISTRY_TOKEN
    - AUTH_SECRET
    - PASSWORDS
    - TELEMETRY

.kamal/secret:

OPENAI_API_KEY=$OPENAI_API_KEY
ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY
REGISTRY_TOKEN=$REGISTRY_TOKEN
AUTH_SECRET=$AUTH_SECRET
PASSWORDS=$PASSWORDS
TELEMETRY=$TELEMETRY
ACCOUNT_SID=$ACCOUNT_SID
AUTH_TOKEN=$AUTH_TOKEN

...and when I run:

• dotenvx get PASSWORDS.... my passwords print correctly!
• kamal registry login... it succeeds!
• dotenvx run -- kamal upgrade (or just kamal upgrade, it outputs:

ERROR (SSHKit::Command::Failed): Exception while executing on host 138.201.246.225: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy

Hard to upgrade!
Anu other clue?
Thanks

[jessevdp]

This is a new kind of issue indeed.

Completely unrelated to the ENV variables.
BTW, does the setup you have now work without the “dotenv load” ERB tag in your deploy.yaml? Since you’re running the dotenvx command you shouldn’t need that line.

---

I’m actually running into a very similar issue. In my case it seems to have something to do with a docker healthcheck. My container isn’t reporting .State.Health or something like that.

Perhaps best to look for a discussion around health checks, or start a new one if there isn’t one already there.

Could you link that discussion here for me? Many thanks!

[cpburm]

ERROR (SSHKit::Command::Failed): Exception while executing on host 138.201.246.225: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy

I had the same issue after fixing my initial failed deployment due to KAMAL_REGISTRY_PASSWORD not being set. I fixed it by running kamal restore to wipe the failed first deployment from the server and then running kamal setup again

[nickhammond]

@jessevdp Are you using ENV at all in your deploy recipes and do you use destinations?

[jessevdp]

Not yet, I’m not upgrading or anything I’m migrating a pretty old Rails app from Capistrano to Kamal. So for now, things that might need to be an ENV variable, or different between stages is just not setup yet.

[antarr]

This is the solution I went with. Nothing else seemed to work other than setting them in ~/.zshrc but that conflicted with my development env.

.kamal/secret

APPSIGNAL_PUSH_API_KEY=$(cat .env.production | grep APPSIGNAL_PUSH_API_KEY | cut -d '=' -f 2)
ELASTIC_APM_SERVER_URL=$(cat .env.production | grep ELASTIC_APM_SERVER_URL | cut -d '=' -f 2)
ELASTICSEARCH_URL=$(cat .env.production | grep ELASTICSEARCH_URL | cut -d '=' -f 2)
KAMAL_REGISTRY_PASSWORD=$(cat .env.production | grep KAMAL_REGISTRY_PASSWORD | cut -d '=' -f 2)
LEGISCAN_API_KEY=$(cat .env.production | grep LEGISCAN_API_KEY | cut -d '=' -f 2)
LEGISCAN_DATABASE_URL=$(cat .env.production | grep LEGISCAN_DATABASE_URL | cut -d '=' -f 2)
RAILS_MASTER_KEY=$(cat config/credentials/production.key)
VOTER_DATA_URL=$(cat .env.production | grep VOTER_DATA_URL | cut -d '=' -f 2)
S3_ACCESS_KEY_ID=$(cat .env.production | grep AWS_ACCESS_KEY_ID | cut -d '=' -f 2)
S3_SECRET_ACCESS=$(cat .env.production | grep AWS_SECRET_ACCESS_KEY | cut -d '=' -f 2)

[jessevdp]

@antarr you need the -f flag in the dotenv CLI since you're using a non-standard .env file.

dotenv -f ".env.production" kamal deploy

https://github.com/bkeepers/dotenv?tab=readme-ov-file#cli

[tillcarlos]

Yeah, I'd rather use dotenv kamal deploy (or with env.production - but then you could also scope the variables in secrets.production:

RAILS_MASTER_KEY=$PRODUCTION_MASTER_KEY

Just an example. Another idea could be to use https://direnv.net/ - then you would read it from the directory and it loads the ENVs directly into every call.

[jessevdp]

Scoping the ENV variables names is a good idea yeah!

[antarr]

is there something that prevent you to prepend your kamal command with dotenv?

Just didn't work

[tillcarlos]

@antarr - would be awesome if you could provide a bit context.

Some ideas how to debug:

Compare these different outputs:

rails runner ' ENV.each { |key, value| puts "#{key}=#{value}" }'
dotenv rails runner ' ENV.each { |key, value| puts "#{key}=#{value}" }'

I installed dotenv in my rubygems, not in the Gemfile. That might also be a hint.

And I just checked my .env file - looks like this, so I don't need to prepend that -f production file.

➜  cat .env                                                                                                                                                  
PORT=3000

# for .kamal/secrets
KAMAL_REGISTRY_PASSWORD=.....

# for .kamal/secrets-production
PRODUCTION_RAILS_MASTER_KEY=....
PRODUCTION_LITESTREAM_REPLICA_URL=s3://....roduction-replica
PRODUCTION_LITESTREAM_ACCESS_KEY_ID=....
PRODUCTION_LITESTREAM_SECRET_ACCESS_KEY=....

[paicha]

I prefer using Rails credentials to store secrets. To facilitate this, I created a custom Rails task for reading credentials:

# lib/tasks/credentials.rake
namespace :credentials do
  desc "Read a specific credential"
  task read: :environment do
    key_path = ENV['KEY'].to_s.split(',').map(&:to_sym)
    value = Rails.application.credentials.dig(*key_path)
    puts value
  end
end

This task is then utilized in the Kamal secrets file:

# .kamal/secrets

# Read secrets based on the specific environment
KAMAL_REGISTRY_PASSWORD=$(RAILS_ENV=development KEY=kamal,registry_password bin/rails credentials:read)
MYSQL_ROOT_PASSWORD=$(RAILS_ENV=production KEY=database,password bin/rails credentials:read)

# From bin/rails credentials:edit --environment=production
RAILS_MASTER_KEY=$(cat config/credentials/production.key)

This approach offers several advantages:

1. The MYSQL_ROOT_PASSWORD only appears in the accessories configuration.
2. The value for MYSQL_ROOT_PASSWORD is read from Rails.application.credentials, which is the same source used by config/database.yml.
3. It eliminates the need for additional .env files, centralizing all sensitive information in the encrypted credentials.yml.enc.

[tillcarlos]

Interesting solution. If it works for you - cool!

The only caveat for me is that usually you want to share basic development credentials (not anything being able to access production). But here you store your production credentials in the development file. If at one point the developers share the development.key you expose production.

[paicha]

The credentials are encrypted, and only I possess the production.key, so only I can decrypt the credentials for the production environment (and only I can deploy to the production server). Other developers have their own master.key to edit the credentials for their development environments.

[ACPK]

In Kamal 2, the following line causes confusion, as it implies that the key is being loaded from the .env file. This is due to the text referencing "SECRET_FROM_ENV".

# .kamal/secrets

SECRET_FROM_ENV=$SECRET_FROM_ENV

[wdiechmann]

I'm not sure I'm "at the right desk" but anyways here goes:

I found .env -> .kamal/secrets -> deploy.yml to be a very potent way to handle variables but since like yesterday (after taking a hint from dependabot to bump rails & friends to the very shiny new, I find myself faced with issues that taste like the ones mentioned in this thread - kind'a

My deploys have started to fail - and I don't really know where to start debug 😢

#18 [build 6/6] RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile
#18 1.124 bin/rails aborted!
#18 1.125 KeyError: key not found: "MS_AD_ID" (KeyError)
#18 1.125 /rails/config/initializers/devise.rb:337:in `fetch'
#18 1.125 /rails/config/initializers/devise.rb:337:in `block in <main>'
#18 1.125 /rails/config/initializers/devise.rb:14:in `<main>'
#18 1.125 /rails/config/environment.rb:15:in `<main>'
#18 1.125 Tasks: TOP => assets:precompile => environment
#18 1.125 (See full trace by running task with --trace)
#18 ERROR: process "/bin/sh -c SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile" did not complete successfully: exit code: 1
------
 > [build 6/6] RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile:
1.124 bin/rails aborted!
1.125 KeyError: key not found: "MS_AD_ID" (KeyError)
1.125 /rails/config/initializers/devise.rb:337:in `fetch'
1.125 /rails/config/initializers/devise.rb:337:in `block in <main>'
1.125 /rails/config/initializers/devise.rb:14:in `<main>'
1.125 /rails/config/environment.rb:15:in `<main>'
1.125 Tasks: TOP => assets:precompile => environment
1.125 (See full trace by running task with --trace)
------
Dockerfile:50
--------------------
  48 |     
  49 |     # Precompiling assets for production without requiring secret RAILS_MASTER_KEY
  50 | >>> RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile
  51 |     
  52 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile" did not complete successfully: exit code: 1

grep MS_AD_ID .env tells me that MS_AD_ID=9cf--8<--0d and likewise does grep MS_AD_ID .kamal/secrets inform me that MS_AD_ID=$(cat .env | grep MS_AD_ID | cut -d'=' -f2) and finally grep MS_AD_ID config/deploy.yml offers me - MS_AD_ID

the config/deploy.yml actually looks more like this:

service: app
image: meondocker/app
servers:
  web:
    - 2.2.2.2
proxy: 
  ssl: true
  host: app.mortimer.pro
registry:
  username: meondocker
  password:
    - KAMAL_REGISTRY_PASSWORD
builder:
  remote: ssh://[email protected]
  arch: arm64
env:
  secret:
    - SECRET_KEY_BASE
    - RAILS_MASTER_KEY
    - MS_AD_ID
    - SMTP_HOST
  clear:
    SOLID_QUEUE_IN_PUMA: true
    PORT: 3000
    RAILS_ENV: production
    PDF_HOST: 4.4.4.4
aliases:
  console: app exec --interactive --reuse "bin/rails console"
  shell: app exec --interactive --reuse "bash"
  logs: app logs -f
  dbc: app exec --interactive --reuse "bin/rails dbconsole"
volumes:
  - "storage:/rails/storage"

In config/initializers/devise.rb I apply one of the ENV's like this

  APP_ID = ENV.fetch("MS_AD_ID")

I suspect the missing ENV's are due to Kamal not loading the ENV "early on" but that's just guesswork...

[jessevdp]

It looks like you're missing an ENV variable at build time.

Kamal doesn't inject your .env nor kamal/secrets when running docker build. And for good reason! You don't want to put secrets in at the build step.

What's likely happening is that some Rails plugin, during the Rails boot process (in an initializer of some sort) is testing for that ENV variable and failing when it's not there.

This is usually a good idea. You don't want to start your app when you know you're missing an essential setting.

Except for the fact that all initializers run on rails assets:precompile.

I faced similar problems with some of the code that I owned. One of my very own initializers was doing something like raise "missing a thing" unless ENV["A_THING"].present?.

I ended up hooking into the already existing SECRET_KEY_BASE_DUMMY ENV variable which is set right there in the Dockerfile to detect this situation.

Hope this helps!

(This is a rails & built-time thing, not a Kamal thing.)

[wdiechmann]

@jessevdp thx for noting but with R8 and 'no-build' - and your point (This is a rails & built-time thing, not a Kamal thing.) - where does that leave me? If all is well in development^* then I reasoned that it had to be during the 'docker build' step (which I believe is a Kamal thing)

Solution:
I did my best to escape the config/credentials.yml.enc - but decided that battle would have to wait to another day so I begged my way back to the encrypted altar 😎

APP_ID = "#{Rails.application.credentials.dig(:microsoft, :ad_id)}"

I hope, some day, I'll find a way to avoid the credentials alltogether

Again @jessevdp - thx for pointing the catalyzing finger forcing me on my way 🫶

^* I don't do assets:precompile - only to drive my point

SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile
Rebuilding...
Done in 760ms.

[wdiechmann]

well - you are right, of cause 😄

I woke one morning with an urge to get rid of credentials - it has been an itch since day zero to me

Debugging my commits backwards I have learned that the moment I rid the initializers of credentials variables my deploy troubles started - so, yeah, you're right -

I like the .env -> .kamal/secrets -> config/deploy.yml so much that I (wrongly) thought I could rely on that construct all over the place but as the rest of the Internet apparently knows - that will not fly until the framework is fully loaded (or at least it will not afford me ENV's in my initializers

That - I think is/was the problem 😃

[jessevdp]

One though I have here... I'm not sure putting the value in credentials solves anything...

You're not putting your RAILS_MASTER_KEY in your docker build step right?

If you don't have the master key... you can't decrypt the credentials. I'm pretty sure that raises an error of sorts. Which crashes the docker build.

The SECRET_KEY_BASE ENV variable is the best example here. Usually the secret key base is stored in encrypted credentials.

Rails included that SECRET_KEY_BASE_DUMMY variable to get around this issue: don't rely on encrypted credentials during assets:precompile.

[wdiechmann]

nope - and I have all files tasting like secrecy added to .gitignore (.kamal/*, master.key, credentials*, more)

(it works btw - using APP_ID = "#{Rails.application.credentials.dig(:microsoft, :ad_id)}" in config/initializers/devise.rb)

but in line with the 12-factor rites IMHO .env is where "configuration" should live - no matter if this configuration pertains to setting up gems (and their initialization) or usage of the app in general with that one exception that all configuration decided by the user should "live" in a CRUD resource like /settings or like construct

From the list of issues here I sense not being the only one to 'fight' this part of the framework - I reckon Rails is in effect paying some kind of technical debt here, showing some signs of design fatigue probing various ways to solving what is not a trivial issue.

[JohnSmall]

Obviously we can't put secrets into Dockerfile, so that when the docker build process gets to assets:precompile if there's anything in an initializer that expects to see an environment variable then the process will fail. There's no point in adding rails master key into secrets because these secrets are loaded into the app before it runs, not into the docker build process.

I ran into this problem, and Claude came up with a good suggestion, use BuildKit's secret mounting instead. But that requires a change to the Kamal build step.

For me the only way out seems to be to remove that step from the Dockerfile and then to compile the assets and check in to git before running Kamal. Very very non-ideal but it does work.

[tillcarlos]

Hey John - what do you have in the initializer that fails without certain ENVs? I think, best practice is to disable features if the ENV it not there. Then it'll be easier to test as well.

Only the master key was required before, and that's skipped by using

RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile (as described above by @jessevdp )

[token-ed]

Hello!

I am having a similar issue.
I am deploying an application on a Hetzner server. When it's all deployed, I navigate to login and NextAuth breaks with

[next-auth][error][NO_SECRET] 
https://next-auth.js.org/errors#no_secret Please define a `secret` in production. MissingSecret [MissingSecretError]: Please define a `secret` in production.
    at t.assertConfig (/src/.next/server/chunks/4020.js:6:115040)
    at AuthHandler (/src/.next/server/chunks/4020.js:6:108189)
    at async getServerSession (/src/.next/server/chunks/4020.js:30:21263)
    at async LoginPage (/src/.next/server/app/(landing-page)/sign-in/page.js:1:7008) {
  code: 'NO_SECRET'
}

I am not sure what is the issue as I am setting NEXTAUTH_SECRET, like this:

.kamal/secrets

NEXTAUTH_SECRET=$NEXTAUTH_SECRET

deploy.yml

env:
  secret:
    - NEXTAUTH_SECRET

In the host machine I can see the variable if I run printenv | grep NEXTAUTH_SECRET

Not really sure if it's a mapping problem between the container and the host machine or what is going on.
Maybe I am not configuring the environment variables correctly?

[jessevdp]

You can also check the values kamal has for secrets: I think you can run kamal secrets print.

Also: the secrets are uploaded and stored in a plaintext file on the server somewhere under ~/.kamal/apps/…. You can always SSH into the server and check!

---

Where is this ENV variable being exported from? Is it generally available in your shell? (Is it available withecho $NEXTAUTH_SECRET?) Or loaded from a .env file? (Kamal won’t automatically load .env files for you.)

It’s better to put these kinds of secrets in a password manager of sorts, of course.

You can try to put the value into a file on disk. You can load secrets from a file in .kamal/secrets by just FOO=$(cat some/file.secret).

[token-ed]

Hi @jessevdp!
Yes, I checked ~/.kamal/apps/…, the variables were all defined there but completely unassigned to any value.
And no, the variable was not available in the shell.

Then you said "(Kamal won’t automatically load .env files for you.)" and you shed some light on me. I was being a bit silly as I was not making the values available to Kamal. So I stored them in the GitHub Action secrets bucket so that .kamal/secrets load the values from there, which are then stored in ~/.kamal/apps/…
Everything is working now, thank you very much!

But I think I prefer to have full control of these variables and use a persistent volume in host machine and use something like you mentioned FOO=$(cat some/file.secret). It feels safer? Not sure, what do you think?

[jessevdp]

Happy to help!

Are you running kamal deploy from your local machine or from a GitHub action or something like that?

In case of the latter, I would use https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions.

When deploying manually from my local machine I would use a secret store like 1password (through kamal secrets extract) so the secrets are NOT stored as plain text on my computer.

https://kamal-deploy.org/docs/commands/secrets/

[token-ed]

Yes sir, using kamal deploy from a GitHub action, that's why Kamal couldn't find the secrets because they were not defined in the GitHub Action environment. Setting the secrets in the Github action secrets bucket works.
And you answered my question about "having full control of the variables in host machine" (that would be if I would run Kamal locally), although I could've phrased it better. Thanks again!

[JohnSmall]

You can pass secrets into the build process It's documented here,Kamal builders

Example,

In deploy.yml

builder:
  arch: amd64
  # # Pass arguments and secrets to the Docker build process
  # args:
  #   RUBY_VERSION: 3.3.6
  secrets:
    - RAILS_MASTER_KEY

In your Dockerfile

RUN --mount=type=secret,id=RAILS_MASTER_KEY,env=RAILS_MASTER_KEY \
  echo "Rails Masterkey: ${RAILS_MASTER_KEY}"

But keep in mind that the environment variable is only available for that one command. After that's finished it's gone

[tillcarlos]

Can't really give a good example, but it doesn't feel right to pass in credentials and then push the Dockerfile around. Another point to worry about. I might be wrong, would like to know what the other people think here.