self-host storage got relation "buckets" does not exist thought schema and tables already exists

[customkandura]

• I confirm this is a bug with Supabase, not my own application.
• I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

• self-host
• docker-compose
• image: supabase/storage-api:v0.40.4

env:

      ANON_KEY: ${ANON_KEY}
      SERVICE_KEY: ${SERVICE_ROLE_KEY}
      POSTGREST_URL: http://rest:3000
      PGRST_JWT_SECRET: ${JWT_SECRET}
      DATABASE_URL: postgres://postgres:[email protected]:5434/myDb
      FILE_SIZE_LIMIT: 52428800
      STORAGE_BACKEND: file
      FILE_STORAGE_BACKEND_PATH: /var/lib/storage
      TENANT_ID: stub
      # TODO: https://github.com/supabase/storage-api/issues/55
      REGION: stub
      GLOBAL_S3_BUCKET: dev-sss
      ENABLE_IMAGE_TRANSFORMATION: "true"
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_DEFAULT_REGION: ap-southeast-1
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      DB_DRIVER: postgres
      DB_HOST: localhost
      DB_NAME: lionparcel
      DB_PORT: "5434"
      DB_SSL: disable
      # GLOBAL_S3_BUCKET: dev-sss
      GLOBAL_S3_FORCE_PATH_STYLE: "true"
      PGOPTIONS: -c search_path=storage,public

the error I got:

originalError: error: select "id", "name", "public", "owner", "created_at", "updated_at", "file_size_limit", "allowed_mime_types" from "buckets" - relation "buckets" does not exist

when I tried to run my docker-compose was no error, but after on Supabase Studio I got those errors,
i though my schema and tables already there, is that something wrong I missed on my env or do I need to fix my table/schema ??

To Reproduce

Steps to reproduce the behavior, please provide code snippets or a repository:

1. please use the same version storage on docker-compose
2. then run
3. check the container log

my docker-compose.yaml

version: "3"
services:
  storage:
    container_name: LocalCloudStorage
    image: supabase/storage-api:v0.40.4
    depends_on:
      rest:
        condition: service_started
    healthcheck:
      test:
        [
          "CMD",
          "wget",
          "--no-verbose",
          "--tries=1",
          "--spider",
          "http://localhost:5000/status"
        ]
      timeout: 5s
      interval: 5s
      retries: 3
    restart: unless-stopped
    environment:
      ANON_KEY: ${ANON_KEY}
      SERVICE_KEY: ${SERVICE_ROLE_KEY}
      POSTGREST_URL: http://rest:3000
      PGRST_JWT_SECRET: ${JWT_SECRET}
      DATABASE_URL: postgres://postgres:password@localhost:5434/myDb
      FILE_SIZE_LIMIT: 52428800
      STORAGE_BACKEND: file
      FILE_STORAGE_BACKEND_PATH: /var/lib/storage
      TENANT_ID: stub
      # TODO: https://github.com/supabase/storage-api/issues/55
      REGION: stub
      GLOBAL_S3_BUCKET: dev-sss
      ENABLE_IMAGE_TRANSFORMATION: "true"
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_DEFAULT_REGION: ap-southeast-1
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      DB_DRIVER: postgres
      DB_HOST: localhost
      DB_NAME: lionparcel
      DB_PORT: "5434"
      DB_SSL: disable
      # GLOBAL_S3_BUCKET: dev-sss
      GLOBAL_S3_FORCE_PATH_STYLE: "true"
      PGOPTIONS: -c search_path=storage,public
    volumes:
      - ./volumes/storage:/var/lib/storage:z

Expected behavior

i want to see my local storage and create new storage on local

Screenshots

System information

• OS: [Windows WSL2] WSL2 Linux Debian

[fenos]

Hello @customkandura could you please provide the records of your migrations table under the storage schema?

[glani]

Interestingly a new connection url to cloud hosted storage DB causes the same error. Looks like a balancer always points to public scheme instead of "storage".
Steps to reproduce are simple.
I created a project:
Spawned a storage service in docker (I need it because the configuration of cloud version is pretty weak)
connection via pgbouncer works smoothly. However when I switched to AWS balancer of postgres all queries started failing with the above mentioned issue.

on the 15th of January there will be no support anymore for pgbouncer and looks like the current solution is pretty raw.

[fenos]

@customkandura I have dug deeper into this issue and apparently the original error was caused by not having the initial migration as part of the service.

My recent PR #409 will now include the initial migration and the Storage service will now start without manually run the migrate command.

A release is yet to be done for this feature I'll keep you up to date.

---

@glani Storage requires the pooler to support either transaction mode or session where transaction mode is the preferred setting for optimum performance.

Could you please link the AWS pool service you are trying to use, i'll have a look at their docs. Also can you make sure the DATABASE_POOL_URL env is set with the correct user

[glani]

@fenos now it starts working when I specified DATABASE_POOL_URL. When a new docker image will be available or release ? I built a current master to check it. However, merged PR of a new migration system is very questionable (b0331a1). No way to deploy it as independent database with storage schema. Role "authenticator" is not checked for a creation.

[fenos]

Hello @glani the image is now available 0.44.1
Also not sure i understand what you by is not checked for creation?

If you want to disable role creation you would want to pass the DB_INSTALL_ROLES=false as an env.
This is useful if the roles are created externally from storage

[fenos]

Closing the issue as this is now supported

[fenos]

here is a docker-compose file example:
https://github.com/supabase/storage/blob/master/docker-compose.yml

[Coderwelsch]

Hello @glani the image is now available 0.44.1 Also not sure i understand what you by is not checked for creation?

If you want to disable role creation you would want to pass the DB_INSTALL_ROLES=false as an env. This is useful if the roles are created externally from storage

this solved also my issue with supabase/storage-api:v0.46.3 ! thanks a lot! had the problem that storage migrations failed with role "anon" already exists

[glani]

For self hosted storage api that still points to cloud db. I would recommend to mount the migration folder from the docker to empty dir. No idea why supabase guys decided to go for file uploads. From my point of view it is slightly irrelevant to the main product. Still the main product(postgres as a rest service) is amazing 👏

…

On Tue, Jan 23, 2024, 22:02 Joseph Ribbe ***@***.***> wrote: Hello @glani <https://github.com/glani> the image is now available 0.44.1 Also not sure i understand what you by is not checked for creation? If you want to disable role creation you would want to pass the DB_INSTALL_ROLES=false as an env. This is useful if the roles are created externally from storage this solved also my issue with supabase/storage-api:v0.46.3 ! thanks a lot! had the problem that storage migrations failed with role "anon" already exists — Reply to this email directly, view it on GitHub <#383 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBRSOOGZBFYROSXD34N6MDYQAQNRAVCNFSM6AAAAAA6ADVPAWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBWHEYTENJTGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>