fix: protect Crawl4AI nginx exposure

[botAGI]

Motivation

• Prevent unauthenticated access to the Crawl4AI REST API which allowed API discovery, submission of arbitrary crawl jobs and SSRF against internal/metadata endpoints.
• Ensure that when Authelia is enabled it cannot be silently bypassed by the Crawl4AI nginx vhost and that Crawl4AI host binding does not default to 0.0.0.0.

Description

• Require Authelia before enabling the Crawl4AI nginx blocks by updating the activation condition in lib/config.sh so the Crawl4AI markers are only stripped when both ENABLE_CRAWL4AI and ENABLE_AUTHELIA are true.
• Add Authelia auth_request /authelia-auth handling and an internal /authelia-auth endpoint to the Crawl4AI dedicated-port and mDNS nginx vhosts in templates/nginx.conf.template so requests to :11235 and agmind-crawl.local can be authenticated when enabled.
• Default the published host bind for the Crawl4AI port to loopback by adding CRAWL4AI_BIND_ADDR and changing the port mapping in templates/docker-compose.yml to ${CRAWL4AI_BIND_ADDR:-127.0.0.1}:${EXPOSE_CRAWL4AI_PORT:-11235}:11235 and add CRAWL4AI_BIND_ADDR=127.0.0.1 to templates/env.lan.template.
• Add a regression unit test tests/unit/test_crawl4ai_auth_exposure.sh that asserts the nginx activation condition, presence of auth_request and /authelia-auth, explicit /health handling, and the default loopback binding, and update golden fixtures/checksums accordingly.

Testing

• Ran bash tests/unit/test_crawl4ai_auth_exposure.sh which passed and validates the new activation condition, auth_request markers and default bind settings.
• Ran bash tests/unit/test_nginx_no_static_proxy_pass.sh which passed and confirmed nginx proxy patterns are unchanged.
• Verified sha256 checksums for updated golden fixtures with sha256sum -c checksums.sha256 which succeeded for the updated fixtures.
• Performed a static check bash -n lib/config.sh tests/unit/test_crawl4ai_auth_exposure.sh which returned without syntax errors.
• bash tests/unit/test_compose_security_invariants.sh was skipped due to missing python3 + PyYAML in the environment, and bash tests/golden/run.sh full_lan --update was skipped due to Docker CLI / Docker Compose not available in the environment.

---

Codex Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9c2e47fc6e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Don't silently disable Crawl4AI for agents installs

For DEPLOY_PROFILE=agents (and custom installs that select Crawl4AI but decline Authelia), templates/services/registry.yaml still implies ENABLE_CRAWL4AI=true without ENABLE_AUTHELIA=true, and the wizard still advertises agmind-crawl.local. With this new condition, generate_nginx_config deletes every Crawl4AI server block in that configuration, while compose only publishes nginx's 11235 port rather than the Crawl4AI container itself, so the selected service starts but has no advertised/reachable host route. Either force Authelia when Crawl4AI is enabled, or stop enabling/advertising Crawl4AI in non-Authelia profiles.

Useful? React with 👍 / 👎.