Support Kubernetes Secrets CSI Driver injecting method

[JorTurFer]

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

Hello

We use this tool in our applications to configure them but we have found a gap running on Kubernetes. Kubernetes supports setting secrets as environment variables but there is a component known as "Secrets CSI Driver" which is based on providing secrets as files but not in a yaml format or so.

This component generates a file per secret using the secret name as file name and secret content as file content. Although it's technically possible to provide a yaml/json file as secret content, usually secrets are stored one by one on cloud secret vaults and mounted one by one into the Kubernetes pods (the component handles it to add all the files needed)

Proposed Solution

I'd like to propose a new configuration "remote config" system based on reading multiple files, binding a single key from each file. For example, this directory

/mnt/secrets
->mySecrets.secretA # (content: ValueForA)
->mySecrets.secretB # (content: ValueForB)
->appSecret # (content: ValueForAppSecret)

Would be treated as:

mySecrets:
  secretA: ValueForA
  secretB: ValueForB
appSecret: ValueForAppSecret

Additionally, to not enforce strange local setups, this config source should be optional, not failing if the folder doesn't exist on the local machine.

Alternatives Considered

No response

Additional Information

Currently, we are achieving the same behaviour with this function:

func setValuesFromFiles(keyPerFilePath string) error {
	if keyPerFilePath == "" {
		return nil
	}
	files, err := os.ReadDir(keyPerFilePath)
	if err != nil {
		return err
	}
	for _, file := range files {
		if !file.IsDir() {
			content, err := os.ReadFile(path.Join(keyPerFilePath, file.Name())) //nolint: gosec // this is intended
			if err != nil {
				return err
			}
			// Replace __ with . to generate the hierarchy
			viper.Set(strings.ReplaceAll(file.Name(), "__", "."), string(content))
		}
	}
	return nil
}

where basically we pass a path and include all the files as keys with the content as value.

This is enough for us but it's error prone when we need to add the code to each project. That's why we would like to contribute to this awesome tool adding the support for Kubernetes Secret CSI driver with a native configuration support

[JorTurFer]

We are willing to develop the feature if you think that it's useful for the project ❤️

[sagikazarmark]

Hey @JorTurFer !

Thanks for opening this issue!

Honestly, I'm not entirely sure that adding this to the core would beneficial to a lot of people.

The core library is already heavy as it is. I'm trying to introduce more abstractions so that something like this is possible.

Ultimately, my goal would be to reduce the core lib to a small set of features and allow people to extend it and share their implementations in an awesome-viper repo or something.

In Viper 2 there is going to be a config provider implementation which is precisely where this would fit.

However, I think there is a way to implement an "extension" without having to add code to every application:

You could implement a special io.Reader and use MergeConfig.

The reader would read all files on first read under the hood, and re-encode them as YAML. Then it would simply need to return the YAML content upon Read calls.

You could easily add this to a repo and reuse it across application. I would be happy to add it to the Viper docs or create the aforementioned awesome-viper repo.

Once the config provider support lands in Viper, you can rewrite the implementation to use that interface (basically you will return a map[string]any)

WDYT?

[github-actions]

Issues with no activity for 30 days are marked stale and subject to being closed.

[JorTurFer]

Hi,
sorry because I missed the answer :(

Finally we have used viper.Set() to enforce the values from files as last stop in our pipeline

func setValuesFromFiles(keyPerFilePath, prefix string) error {
	if keyPerFilePath == "" {
		return nil
	}
	files, err := os.ReadDir(keyPerFilePath)
	if err != nil {
		return err
	}
	for _, file := range files {
		if !strings.HasPrefix(file.Name(), prefix) {
			continue
		}
		if !file.IsDir() {
			content, err := os.ReadFile(path.Join(keyPerFilePath, file.Name())) //nolint: gosec // this is intended
			if err != nil {
				return err
			}
			// Replace __ with . to generate the hierarchy
			key := file.Name()
			key = strings.TrimPrefix(key, fmt.Sprintf("%s_", prefix))
			viper.Set(strings.ReplaceAll(key, "__", "."), string(content))
		}
	}
	return nil
}

Just with this function we can extend viper with the behaviour that we need (just in case someone can leverage it too). I close the issue and thanks for this awesome software