docs: Doc improvement (#256)

* feat: update docs and self host deploy configs

* feat: use default ports for dev

* feat: revamp self host

* feat: use nginx resolver

* feat: cargo fmt

* feat: ci integration test with nginx routing

* feat: cargo fmt

* feat: make docker ci use deploy env rather than dev env

* feat: update api external url for deploy test

* fix: if sqlx is set to offline dont run casbin for tests

* fix: set api external url for tests

* fix: ignore sqlx test with cfg

* fix: sqlx tests

* fix: test use env

* feat: update the docs on optional components

.github/workflows/docker.yml

@@ -8,8 +8,11 @@ on:
     branches: [ main ]
 
 env:
-  SQLX_VERSION: 0.7.1
-  SQLX_FEATURES: "rustls,postgres"
+  LOCALHOST_URL: http://localhost
+  LOCALHOST_WS: ws://localhost/ws
+  LOCALHOST_GOTRUE: http://localhost/gotrue
+  DATABASE_URL: postgres://postgres:password@localhost:5432/postgres
+  SQLX_OFFLINE: true
 
 jobs:
   test:
@@ -24,8 +27,8 @@ jobs:
           workspaces: |
             AppFlowy-Cloud
 
-      - name: Copy and rename dev.env to .env
-        run: cp dev.env .env
+      - name: Copy and rename deploy.env to .env
+        run: cp deploy.env .env
 
       - name: Replace values in .env
         run: |
@@ -35,6 +38,10 @@ jobs:
           sed -i 's/GOTRUE_SMTP_PASS=.*/GOTRUE_SMTP_PASS=${{ secrets.GOTRUE_SMTP_PASS }}/' .env
           sed -i 's/GOTRUE_SMTP_ADMIN_EMAIL=.*/GOTRUE_SMTP_ADMIN_EMAIL=${{ secrets.GOTRUE_SMTP_ADMIN_EMAIL }}/' .env
           sed -i 's/GOTRUE_EXTERNAL_GOOGLE_ENABLED=.*/GOTRUE_EXTERNAL_GOOGLE_ENABLED=true/' .env
+          sed -i 's/GOTRUE_MAILER_AUTOCONFIRM=.*/GOTRUE_MAILER_AUTOCONFIRM=false/' .env
+          sed -i 's/API_EXTERNAL_URL=http:\/\/your-host/API_EXTERNAL_URL=http:\/\/localhost/' .env
+          # expose port for sqlx tests
+          sed -i '38s/$/\n    ports:\n      - 5432:5432/' docker-compose.yml
 
       - name: Disable appflowyinc images
         run: |
@@ -47,5 +54,4 @@ jobs:
 
       - name: Run tests
         run: |
-          cargo install sqlx-cli --version=${{ env.SQLX_VERSION }} --features ${{ env.SQLX_FEATURES }} --no-default-features --locked
-          RUST_LOG=debug cargo test
+          cargo test

Dockerfile

@@ -28,7 +28,6 @@ RUN apt-get update -y \
     && rm -rf /var/lib/apt/lists/*
 
 COPY --from=builder /app/target/release/appflowy_cloud /usr/local/bin/appflowy_cloud
-COPY --from=builder /app/configuration configuration
 ENV APP_ENVIRONMENT production
 ENV RUST_BACKTRACE 1
 CMD ["appflowy_cloud"]

README.md

@@ -17,7 +17,7 @@
 
 # AppFlowy Cloud
 
-AppFlowy Cloud is part of the AppFlowy ecosystem, offering secure user authentication, file storage, 
+AppFlowy Cloud is part of the AppFlowy ecosystem, offering secure user authentication, file storage,
 and real-time WebSocket communication for an efficient and collaborative user experience.
 
 ## Table of Contents
@@ -48,7 +48,7 @@ You'll need to install:
 ./build/run_local_server.sh
 ```
 
-This process will execute all the dependencies and start the AppFlowy-Cloud server. Alternatively, 
+This process will execute all the dependencies and start the AppFlowy-Cloud server. Alternatively,
 you have the option to run the AppFlowy-Cloud server independently
 
 ### Run the AppFlowy-Cloud
@@ -85,7 +85,7 @@ cargo test
 
 ## 🐞Debugging
 
-Effective debugging is essential for maintaining a healthy application. Here are some tools and commands to help you 
+Effective debugging is essential for maintaining a healthy application. Here are some tools and commands to help you
 troubleshoot issues in various components of the AppFlowy cloud server:
 
 ### Postgres
@@ -95,23 +95,23 @@ A web-based administration tool for PostgreSQL. Access it at [PgAdmin](http://lo
 - OR command line:
 ```bash
     export PGPASSWORD=password
-    psql --host=localhost --username=postgres --port=5433
+    psql --host=localhost --username=postgres --port=5432
 ```
 
 - Redis
 
 Redis offers a powerful command line interface for managing your Redis instance. Connect using the following command:
 
 ```bash
-    redis-cli -p 6380
+    redis-cli -p 6379
 ```
 
 ### Minio
 
 Minio provides a Web UI for easy management of your files and buckets. Access it at [Web UI](http://localhost:9001)
 
 ### Portainer
-For managing Docker containers, Portainer's Web UI is an excellent tool. Access it at Web UI to easily manage Docker 
+For managing Docker containers, Portainer's Web UI is an excellent tool. Access it at Web UI to easily manage Docker
 environments, including container deployment, networking, volume management, and more. Access it at [Web UI](http://localhost:9442)
 
 

admin_frontend/dev.env

@@ -1,3 +1,3 @@
-GOTRUE_URL=http://localhost:9998
-REDIS_URL=redis://localhost:6380
+GOTRUE_URL=http://localhost:9999
+REDIS_URL=redis://localhost:6379
 RUST_LOG=trace

build/init_gotrue.sh

@@ -7,9 +7,6 @@ git clone https://github.com/supabase/gotrue.git
 cp gotrue.env.docker gotrue/.env.docker
 cd gotrue
 
-# avoid port conflict with host postgres
-sed -i "s/'5432:5432'/'5433:5432'/" docker-compose-dev.yml
-
 make dev &
 
 while true; do

build/run_local_server.sh

@@ -6,7 +6,7 @@ cd "$(dirname "$0")/.."
 
 DB_USER="${POSTGRES_USER:=postgres}"
 DB_PASSWORD="${POSTGRES_PASSWORD:=password}"
-DB_PORT="${POSTGRES_PORT:=5433}"
+DB_PORT="${POSTGRES_PORT:=5432}"
 DB_HOST="${POSTGRES_HOST:=localhost}"
 
 # Stop and remove any existing containers to avoid conflicts
@@ -34,7 +34,7 @@ if [ $ATTEMPTS -eq $MAX_ATTEMPTS ]; then
   exit 1
 fi
 
-until curl localhost:9998/health; do
+until curl localhost:9999/health; do
   sleep 1
 done
 

deploy.env

@@ -1,3 +1,8 @@
+# GoTrue URL that the appflowy service will use to connect to gotrue
+# In docker environment, `gotrue` is the hostname of the gotrue service
+APPFLOWY_GOTRUE_BASE_URL=http://gotrue:9999
+APPFLOWY_DATABASE_URL=postgres://postgres:password@postgres:5432/postgres
+
 # This file is a template for docker compose deployment
 # Copy this file to .env and change the values as needed
 
@@ -26,9 +31,13 @@ GOTRUE_ADMIN_PASSWORD=password
 # change this to your own domain where you host the docker-compose or gotrue
 API_EXTERNAL_URL=http://your-host
 
+# In docker environment, `postgres` is the hostname of the postgres service
+# GoTrue connect to postgres using this url
+GOTRUE_DATABASE_URL=postgres://supabase_auth_admin:root@postgres:5432/postgres
+
 # Refer to this for details: https://github.com/AppFlowy-IO/AppFlowy-Cloud/blob/main/doc/AUTHENTICATION.md
 # Google OAuth2
-GOTRUE_EXTERNAL_GOOGLE_ENABLED=true
+GOTRUE_EXTERNAL_GOOGLE_ENABLED=false
 GOTRUE_EXTERNAL_GOOGLE_CLIENT_ID=
 GOTRUE_EXTERNAL_GOOGLE_SECRET=
 GOTRUE_EXTERNAL_GOOGLE_REDIRECT_URI=http://your-host/callback
@@ -59,9 +68,13 @@ PGADMIN_DEFAULT_PASSWORD=password
 # Portainer (username: admin)
 PORTAINER_PASSWORD=password1234
 
-# Grafana Dashboard
-GF_SECURITY_ADMIN_USER=admin
-GF_SECURITY_ADMIN_PASSWORD=password
-
 # Cloudflare tunnel token
 CLOUDFLARE_TUNNEL_TOKEN=
+
+# If you are using a different postgres database, change the following values
+# POSTGRES_USER=
+# POSTGRES_DB=
+# POSTGRES_PASSWORD=
+# POSTGRES_HOST=
+# GOTRUE_DATABASE_URL=postgres://supabase_auth_admin:$POSTGRES_PASSWORD@$POSTGRES_HOST:5432/$POSTGRES_DB
+# APPFLOWY_DATABASE_URL=postgres://POSTGRES_USER:$POSTGRES_PASSWORD@$POSTGRES_HOST:5432/$POSTGRES_DB

dev.env

@@ -1,3 +1,7 @@
+# gotrue URL that the appflowy service will use to connect to gotrue
+APPFLOWY_GOTRUE_BASE_URL=http://localhost:9999
+APPFLOWY_DATABASE_URL=postgres://postgres:password@localhost:5432/postgres
+
 # This file is used to set the environment variables for local development
 # Copy this file to .env and change the values as needed
 
@@ -26,29 +30,32 @@ GOTRUE_ADMIN_PASSWORD=password
 # If you're using an Nginx proxy as part of your setup, this host should be set to the domain managed by the proxy.
 # For instance, if you're running your application locally using 'docker compose up -d',
 # you can set this value to 'http://localhost'.
-API_EXTERNAL_URL=http://localhost:9998
+API_EXTERNAL_URL=http://localhost:9999
 
-# url to the postgres database
-DATABASE_URL=postgres://postgres:password@localhost:5433/postgres
+# url for sqlx
+DATABASE_URL=postgres://postgres:password@localhost:5432/postgres
 # uncomment this to enable build without database
 # .sqlx files must be pregenerated
-SQLX_OFFLINE=false
+# SQLX_OFFLINE=true
+
+# database url that gotrue will use
+GOTRUE_DATABASE_URL=postgres://supabase_auth_admin:root@postgres:5432/postgres
 
 # Google OAuth2
 GOTRUE_EXTERNAL_GOOGLE_ENABLED=true
 GOTRUE_EXTERNAL_GOOGLE_CLIENT_ID=
 GOTRUE_EXTERNAL_GOOGLE_SECRET=
-GOTRUE_EXTERNAL_GOOGLE_REDIRECT_URI=http://localhost:9998/callback
+GOTRUE_EXTERNAL_GOOGLE_REDIRECT_URI=http://localhost:9999/callback
 # GitHub OAuth2
 GOTRUE_EXTERNAL_GITHUB_ENABLED=false
 GOTRUE_EXTERNAL_GITHUB_CLIENT_ID=
 GOTRUE_EXTERNAL_GITHUB_SECRET=
-GOTRUE_EXTERNAL_GITHUB_REDIRECT_URI=http://localhost:9998/callback
+GOTRUE_EXTERNAL_GITHUB_REDIRECT_URI=http://localhost:9999/callback
 # Discord OAuth2
 GOTRUE_EXTERNAL_DISCORD_ENABLED=false
 GOTRUE_EXTERNAL_DISCORD_CLIENT_ID=
 GOTRUE_EXTERNAL_DISCORD_SECRET=
-GOTRUE_EXTERNAL_DISCORD_REDIRECT_URI=http://localhost:9998/callback
+GOTRUE_EXTERNAL_DISCORD_REDIRECT_URI=http://localhost:9999/callback
 # File Storage
 USE_MINIO=true
 # MINIO_URL=http://localhost:9000 # change this if you are using a different address for minio

doc/DEPLOYMENT.md

@@ -6,21 +6,26 @@
 - Minimum 2GB Ram (4GB Recommended)
 - Ports 80/443 available
 - Because AppFlowy-Cloud will have to be running persistently (or at least whenever users need access),
-we recommend deploying it on a cloud compute services as host server (if deploying it on a home server is not an option for you) such as
+it's probably a good idea to run it on a non-end user device. It is best if you already have a home server(check software requirements),
+but if you don't, you can also deploy it on a cloud compute services as host server such as
     - [Amazon EC2](https://aws.amazon.com/ec2/) or
     - [Azure Virtual Machines](https://azure.microsoft.com/en-gb/products/virtual-machines/)
 
-
 ## Software Requirements
-
-Ensure you have Docker Compose installed on your host server. Follow the official guidelines for a reliable setup:
+Ensure you have Docker Compose(v2) installed on your host server. Follow the official guidelines for a reliable setup:
 
 Docker Compose is included with Docker Engine:
 - **Docker Engine:** We suggest adhering to the instructions provided by Docker for [installing Docker Engine](https://docs.docker.com/engine/install/).
 
 For older versions of Docker Engine that do not include Docker Compose:
 - **Docker Compose:** Install it as per the [official documentation](https://docs.docker.com/compose/install/).
 
+Once you have it installed, you can check by running the command:
+```
+docker compose version
+# Docker Compose version 2.23.3
+```
+Note: `docker-compose` (with the hyphen) may not be supported. You are advise to use `docker compose`(without hyphen) instead.
 
 ## Steps
 
@@ -115,17 +120,7 @@ GOTRUE_EXTERNAL_DISCORD_SECRET=your-discord-secret
 GOTRUE_EXTERNAL_DISCORD_REDIRECT_URI=http://your-host/gotrue/callback
 ```
 
-### 3. Optional Services
-Some services in `docker-compose.yml` are optinal and can be commented out if you dont need them, the will be marked as `Optional`
-- `pgadmin` (Web UI configured easy view into deployed postgres database)
-- `portainer`/`portainer_init` (Web UI for providing some monitoring and ease of container management)
-- `tunnel` (cloud flare tunnel: provide secure way to connect appflowy to Cloudflare without a publicly routable IP address)
-- `admin_frontend` (admin portal to manage accounts and adding authentication method, recommended to keep)
-If you did not deploy an optional component, else the nginx server will not start properly
-1. comment out the corresponding `depends_on` in `nginx` service in file `docker-compose.yml`
-2. comment out the corresponding `location` section in file `nginx/nginx.conf`
-
-### 4. Running the services
+### 3. Running the services
 
 #### Start and run AppFlowy-Cloud
 - The following command will build and start the AppFlowy-Cloud.
@@ -138,18 +133,35 @@ docker compose up -d
 docker ps -a
 ```
 
+### 4. Optional Services
+There are optional services that are non essential in core functionalities of AppFlowy Cloud, there can be useful for administrative or debugging purpose.
+The files containing these services are in `docker-compose-extra.yml`.
+- `pgadmin` (Web UI configured easy view into deployed postgres database)
+- `portainer`/`portainer_init` (Web UI for providing some monitoring and ease of container management)
+- `tunnel` (cloud flare tunnel: provide secure way to connect appflowy to Cloudflare without a publicly routable IP address)
+- `admin_frontend` (admin portal to manage accounts and adding authentication method, recommended to keep)
+If you wish to deploy those, edit this file accordingly and do:
+```
+docker compose --file docker-compose-extra.yml up -d
+```
+You may ignore the orphan containers warning message from docker
+
+
 > When using the `docker compose up -d` command without specifying a tag, Docker Compose will pull the `latest`
 tag for the `appflowy_cloud` and `admin_frontend` images from Docker Hub by default. If you've set the `BACKEND_VERSION`
 environment variable, it will pull the specified version instead. If `BACKEND_VERSION` is not set, Docker Compose
 defaults to using the `latest` tag.
 
-- The metrics endpoint can also be used to verify that the AppFlowy-Cloud server is running. It should return a status of 200 OK.
-- This command should only be run in the host machine as port 8000 should not be exposed
+- Check that services are running correctly `docker ps -a`
+- If you find a particular service not working properly, you can inspect the logs:
 ```bash
-curl -v localhost:8000/metrics
+# Getting logs for a particular docker compose service
+# You can obtain name by `docker ps -a`
+docker logs <NAME>
+# e.g. docker logs appflowy-cloud-admin_frontend-1
 ```
 
-### 4. Reconfiguring and redeployment
+### 5. Reconfiguring and redeployment
 - It is very common to reconfigure and restart. To do so, simply edit the `.env` and do `docker compose up -d` again
 
 ## Ports

doc/EC2_SELF_HOST_GUIDE.md

@@ -179,17 +179,5 @@ docker system prune -af
 
 **Caution:** This command removes all unused Docker images, containers, volumes, and networks. Be sure to backup any important data before proceeding.
 
-#### 3. Modify Docker Compose Configuration
-Alternatively, you can selectively reduce Docker's disk usage:
-
-- **Disable the Tunnel Server**: Temporarily comment out or remove the tunnel server service in the `docker-compose.yml` file.
-
-  ![Commenting Out Tunnel Server](../assets/images/comment_out_tunnel.png)
-
-- **Disable Admin Frontend**: Similarly, you can disable the `admin_frontend` service in the `docker-compose.yml` file. This involves commenting out the relevant sections.
-
-  ![Commenting Out Admin Frontend](../assets/images/comment_out_admin_frontend.png)
-
-  Additionally, ensure that dependencies on the `admin_frontend` service are also commented out as needed.
-
-  ![Commenting Out Dependencies on Admin Frontend](../assets/images/comment_out_deps_on_admin_frontend.png)
+#### 3. Additional Docker Compose Services
+- You can add additional docker compose services, refer to [main guide](./DEPLOYMENT.md) for more info