Complex Service Configuration

[ghost]

Many of the modules the Cloud Game Development Toolkit provides require complex configuration through environment variables and config files. Currently, we set these configurables with a number of "hacks."

• Helix Authentication Service web admin endpoint password configuration
• Helix Core user_data
• Helix Swarm sso configuration
• Unreal Engine Horde ASP.NET variable overrides

The question is, should we continue to approach configuration through this piecemeal exposure, or should we establish a more consistent approach for settings like these?

[ghost]

Configuration Files

Many of our modules provide configuration files:

• Horde server.json and appsettings.json
• Helix Swarm config.php
• Helix Authentication Service config.toml or .env
• Jenkins casc jenkins.yaml

Ideally, end users could define these files in their expected format external to the Toolkit. This way, the development of modules in the Toolkit does not hinge on our ability to stay up to date with application configuration changes.

These files need to persist across container / application restarts. In the case of Helix Swarm and Helix Authentication Service the underlying data directory is still ephemeral. Horde caches a version of server.json - but redeployments could also effect these configurations.

There are also security considerations. These configuration files may contain sensitive information (OIDC provider details, service account credentials). End users should be made aware of the implications of storing these configuration files locally alongside Terraform. We should also provide a secure way for these files to be pulled from cloud storage (S3).

Nonetheless, the value of abstracting configuration away from infrastructure deployment is undeniable. I think the juice is worth the squeeze on this one.

[pasotee]

Speaking about my experience with your Horde module, configuring the environment variables would be a game changer. We've basically forked this repo just to add more of them and then added it as a module.

Having a key-value map exposed where we can add additional field would be amazing.
Regarding the security concern you've raised we were able to store them in AWS secret manager and reference them this way (I think you did something in the examples with the GitHub token).

[ghost]

Yeah we've done this elsewhere. Thanks for the feedback! Would you rather configure those values via TF module variables, or via a Server.json config file alongside the module?

[pasotee]

A Server.json would be easier to type because it would make array inputs a lot more readable than env variables like Horde__Perforce__1__Username but I am afraid it make might secret management more complicated (or maybe there are some fancy ways to reference something from the secret manager from the JSON?).

But ultimately even with the messier syntax for complex types, I think Terraform has more flexibility because it allows us to reference variables and other bits of the infrastructure (e.g.: the horde FQDN, secrets from AWS secret manager).

Maybe a JSON file as a base with terraform overrides would solve everyone's problems, but that feels like bit much work.
But if I had to pick one, I would go for terraform.

Also, I wanted to make a PR for exactly what you are mentioning here - additional env vars via key-value map in terraform, but I wasn't sure if this is the path you want to take.
If you feel this would be a good addition, I am would love to make a proposal.

[runreal-warman]

For Horde configuration, I think having server.json templatefile would be easier to maintain than defining each config property as TF variables or environment variables. The Horde server configuration is also split into:

• server.json => requires restart
• global.json + $project.json => hot reloads

I think standardizing on something similar to (#444) where we store config in S3 might be the best approach. For Horde server:

• initial configuration is created in S3 by TF with certain properties interpolated from TF
• if server.json is updated => create new ecs task
• if *.json is updated => save to bucket and sync to config folder

Ideally having a S3 object notify + lambda setup would be useful so we don't have to manage the configuration via TF. For Horde, being able to just modify global.json or create some new config files from any process and have it sync would be really nice. That could be useful for other parts of the kit through an opt-in flag.

[ghost]

Ansible Playbooks and SSM Run Documents

Some configurations can be done remotely against the services after startup. We adopt this approach for build agent installation and registration as part of the Horde module. We could, theoretically, leverage the same tools for configuring module applications. We are in the process of doing this with Perforce Helix Core configuration (#427).

In scenarios where the configuration is tightly coupled to the repeated deployment of the application I think this should be baked into a Terraform apply. Changes to the underlying configuration file should trigger a redeployment via Terraform - not be executed via some other tool. This makes roleback / versioned plans reflective of changes to infrastructure of the underlying service as well as configuration.

[GrzesiekO]

I'd propose to set a standard for Ansible Playbooks and how we should run them.
To run properly against Amazon Linux 2023 I have updated AWS managed document: AWS-ApplyAnsiblePlaybooks that is using an "ExternalVariables" string that is passed to the playbook when the Association is being done:
" ansible-playbook -i \"localhost,\" --check -c local -e \"ansible_python_interpreter=$${PYTHON_INTERPRETER}\" -e \"{{ExtraVariables}}\" \"{{Verbose}}\" \"$${PlaybookFile}\"",

I suggest to have the variables defined by a "map" variable and create for these "reusable" variables entries in SSM Parameter store.

Why?
Because if these are defined in Parameter Store in case we want to change them it is enough to change a single parameter and rerun the association rather to rerun entire terraform also for visibility - All vital configurations are auditable in Parameter Store rather than from terraform state file.