Add file-based authentication support for CEL and HTTP JSON inputs

[SpencerLN]

Proposed commit message

feat(filebeat): add file auth support for CEL and HTTP JSON inputs

• Add auth.file configuration for CEL and HTTP JSON inputs to load header values from disk with optional prefix and refresh interval.
• wrap client transports to inject the refreshed file-based credential on each request and guard against invalid configs

Note

The code is mostly duplicated between httpjson and CELinputs. I considered putting it in a shared location that each could reference, but I didn't see a clear pattern on how to do that. As it isn't much code I've implemented it in each input at the moment.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None. Existing configurations continue to work; the new file auth mechanism is opt-in.

Author's Checklist

• [ ]

How to test this PR locally

• Create a file (i.e. /etc/elastic/gh_token_test) containing an invalid secret
• Start a Filebeat instance with a configuration that tries to retrieve data from a server requiring authentication:

filebeat.inputs:
  - type: httpjson
    interval: 5s
    auth.file:
      path: "/etc/elastic/gh_token_test"
      prefix: "token "
      refresh_interval: 10s
    request:
      url: https://api.github.com/repos/{org}/{repo}/issues
      method: GET
      transforms:
        - set:
            target: url.params.per_page
            value: "1"
        - set:
            target: url.params.state
            value: open
        - set:
            target: header.Accept
            value: application/vnd.github+json

• Confirm that Filebeat logs show a 401 error
• Update the secret file to contain a valid secret, wait 10-15 seconds, confirm the input now outputs the expected data

Related issues

• Closes [Filebeat] [httpjson/cel] Support setting authentication headers based on file content #47506

Use cases

Filebeat reads short-lived (JWT, OAuth, etc.) tokens projected to disk by an external secret manager and refreshes them without restarting the input.

Note

GitHub Copilot was used to generate some code for this PR.

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @SpencerLN? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/filebeat/filebeat-input-cel.md
• docs/reference/filebeat/filebeat-input-httpjson.md

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[colleenmcginnis]

Starting with v9.0, there is no longer a new documentation set published with every minor release. Instead, the same page stays valid over time and shows version-related evolutions. You can read more in Write cumulative documentation.

Based on the lack of backport labels, I'm assuming you're targeting 9.3.0 for this change, but feel free to adjust my suggestion if my assumption is incorrect!

[colleenmcginnis]

Suggested change

	* File
	* {applies_to}`stack: ga 9.3.0` File

[colleenmcginnis]

Suggested change

	* file-based headers
	* file-based headers {applies_to}`stack: ga 9.3.0`

[colleenmcginnis]

Suggested change

	### `auth.file.enabled` [_auth_file_enabled]
	### `auth.file.enabled` [_auth_file_enabled]
	```{applies_to}
	stack: ga 9.3.0
	```

[colleenmcginnis]

Suggested change

	### `auth.file.path` [_auth_file_path]
	### `auth.file.path` [_auth_file_path]
	```{applies_to}
	stack: ga 9.3.0
	```

[colleenmcginnis]

Suggested change

	### `auth.file.header` [_auth_file_header]
	### `auth.file.header` [_auth_file_header]
	```{applies_to}
	stack: ga 9.3.0
	```

[colleenmcginnis]

Suggested change

	### `auth.file.enabled` [_auth_file_enabled_2]
	### `auth.file.enabled` [_auth_file_enabled_2]
	```{applies_to}
	stack: ga 9.3.0
	```

[colleenmcginnis]

Suggested change

	### `auth.file.path` [_auth_file_path_2]
	### `auth.file.path` [_auth_file_path_2]
	```{applies_to}
	stack: ga 9.3.0
	```

[colleenmcginnis]

Suggested change

	### `auth.file.header` [_auth_file_header_2]
	### `auth.file.header` [_auth_file_header_2]
	```{applies_to}
	stack: ga 9.3.0
	```

[colleenmcginnis]

Suggested change

	### `auth.file.prefix` [_auth_file_prefix_2]
	### `auth.file.prefix` [_auth_file_prefix_2]
	```{applies_to}
	stack: ga 9.3.0
	```

[colleenmcginnis]

Suggested change

	### `auth.file.refresh_interval` [_auth_file_refresh_interval_2]
	### `auth.file.refresh_interval` [_auth_file_refresh_interval_2]
	```{applies_to}
	stack: ga 9.3.0
	```

[SpencerLN]

Based on the lack of backport labels, I'm assuming you're targeting 9.3.0 for this change, but feel free to adjust my suggestion if my assumption is incorrect!

I'll defer to the maintaners on which release this should go into, as I'm not sure what the policy is on adding minor new functionality. From my perspective, I'd like it as soon as possible and 9.3.0 still looks to be quite far out 😁.

---

I've submitted 21356c6 to fix the linting errors that were newly introduced by this code, the rest of the errors appear to be pre-existing issues.