Save logs to S3 (ceph)

[mvngne]

Hello, everyone. I'm trying to deploy Apache Airflow with the Helm chart to our Kubernetes cluster. After some initial difficulties, I have it running now. However, there is one remaining issue: Logging. Currently, without any special settings, the logs aren't persistent. Since the pods are deleted by default after execution, I can't access the logs afterwards. We would like to save the logs to S3 (Ceph), but I can't seem to get that to work. There are some parts in the documentation on how to configure AWS S3, which is similar but slightly different, that I tried to use, but it just won't work. My goal is to use the deployment to add all dependencies so that the logs from all DAGs, which are synced from Git, are automatically saved in S3 and can be viewed later.

Is there someone who has configured logging to S3 in Airflow 3+ with Helm who can give me hints on how to configure it?

[mvngne]

Okay. Finaly found a solution. For anyone with the same issues

extraSecrets:
    ceph-s3-creds:
      stringData: |-
        AWS_ACCESS_KEY_ID: "{{ s3_access_key }}"
        AWS_SECRET_ACCESS_KEY: "{{ s3_access_secret }}"
        AWS_ENDPOINT_URL: "{{ s3_endpoint_url }}"

  extraEnv: |-
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: ceph-s3-creds
          key: AWS_ACCESS_KEY_ID
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: ceph-s3-creds
          key: AWS_SECRET_ACCESS_KEY
    - name: AWS_ENDPOINT_URL
      valueFrom:
        secretKeyRef:
          name: ceph-s3-creds
          key: AWS_ENDPOINT_URL

[revolman]

Hi, what do you specify in the config?

config:
  logging:
    remote_logging: 'True'
    remote_base_log_folder: s3://bucket-name
    remote_log_conn_id:  # <-- Anything here?
    encrypt_s3_logs: 'False'

[mvngne]

Hey,

config:
    logging:
        remote_logging: "True"
        remote_base_log_folder: "s3://apacheairflow/logs"
        remote_log_conn_id: "ceph_s3_conn"
        encrypt_s3_logs: "False"

[revolman]

Then you should define the AIRFLOW_CONN_CEPH_S3_CONN env variable.
Could you please share the values you use for this connection? Like signature_version and others.
I'm getting XAmzContentSHA256Mismatch and SignatureDoesNotMatch when the UI tries to fetch logs from S3.

[mvngne]

All I did to get S3 to work is what you see here. I provided the s3 endpoint, access key and secret key as env vars and the config.logging I just provided in my values.yml.

[revolman]

Got it, thanks anyway!