Skip to content

Latest commit

 

History

History
127 lines (78 loc) · 14 KB

design_decisions.md

File metadata and controls

127 lines (78 loc) · 14 KB

Design Decisions

High-level design considerations

A Seqera Platform deployment has two separate phases:

  1. Initial stand-up
    Infrastructure is provisioned, the first set of containers are downloaded, and Seqera Platform-specific configuration files are created.

  2. Upgrade & maintenance
    Includes downloading new containers and/or modifying Seqera Platform configuration files.
    Depending on the initial stand-up decisions, new infrastructure may also be provisioned (e.g. switching from the containerized database to an RDS instance).

Regardless of phase, a system administrator must consider two themes as they do their work:

  1. How to provide necessary values (sensitive and non-sensitive) in configuration files.
  2. Executing the commands necessary to install packages, pull containers, and run programs.

With these phases and themes in mind, the following was considered when designing the tool:

  1. Security
    Sensitive values like passwords should be stored in a secure store rather than being written into configuration files as plain text.

  2. Repeatability
    Provided the same input values, the outputs generated by the solution should be consistent.

  3. Operational resilience
    Individuals change roles / leave organizations and workstations can fail. The solution should not be dependent on a single vulnerable asset.

  4. Visibility
    The solution should ensure the entire end-to-end process can be determined by a reader with access to these project files.

  5. Flexibility
    Clients have a wide range of environment configuration and deployment needs. The solution should avoid implementating a one-size-fits-all approach wherever possible.

  6. Speed & efficiency
    Slow deployment cycles increase the likelihood that a user will deviate from established deployment processes. The solution should minimize execution time to ensure best practices are followed.

  7. Familiarity
    Users have a varying range of familiarity with cloud infrastructure and are not guaranteed to have any experience with Terraform. The solution should use familiar patterns wherever possible, even if this means deviating from Terraform community best practices.

General design decisions

In response to the outlined design considerations, the following design decisions were made:

  1. Structure project and name files for the benefit of non-Terraform experts
    This project does not follow standard Terraform file naming conventions nor implement extensive hierarchical modularization. Priority was given to make this as easy as possible for non-Terraform experts (a majority of our anticipated users). As a result, cascading inputs/outputs were minimized as much as possible and the .tf files were named in a way to model the DAG execution flow to facilitate troubleshooting.

  2. Store sensitive values in AWS Systems Manager Parameter Store (SSM).
    Use a world-class secret store to protect values at rest and only retrieve the values at runtime using a native integration supplied by the Micronaut framework (upon which Seqera Platform is built).

  3. Store sensitive values in omnibus starter entries.
    Each SSM-based configuration value must have its own unique prefix. To minimimize the operational burden, clients will create a small number of complex SSM entries ahead of tool invocation. The tool extracts these entries into their own specific prefixes. TODO_ADD_LINK_TO_TEMPLATE.

  4. Do not supply default values for Terraform variables.
    Terraform variables can be given a default value as they are defined. Despite this behaviour being considered a best pratice by the Terraform community, we respectfully disagree as this spreads values across two different files (variables.tf and terraform.tfvars), and makes the values of some variables implicit while others must be defined explicitly. The solution requires all values to be defined explicitly within the terraform.tfvars file.

  5. Favour templatefiles over in-resource definitions.
    Terraform prefers to have practitioners define content within a resource (example: IAM Policy) which it transforms to JSON at runtime. In our opinion, this is sub-optimal because (1) it makes resource definitions bigger and (2) it makes it harder to compare JSON objects in the AWS console to the source files in the Terraform project. As a result, this solution uses discrete JSON and YML template files instead.

  6. Regenerate assets on every terraform apply.
    To minimize the risk of drift, the solution will regenerate secrets and Seqera Platform configuration files every time. This ensures that any change made to the starting omnibus secret objects and/or terraform.tfvars file are guaranteed to be applied to the resulting Seqera Platform instance.

  7. Regenerate files via null_resource instead of local_file.
    This project uses local-exec provisioners with null_resources to generate files. Our preference would have been to use the more targeted local_file resource (example), but local_file resources don't regenerate consistently.

  8. Do not optimize Terraform statement management Terraform offers a wide variety of backend options. These implementations can get quite complex and are beyond the scope of this solution. We default to local state storage by default. Clients are free to implement a more robust state management solution without their own project effort.

  9. Minimize reliance on third-party tooling where possible Supply-chain attacks are growing in popularity and scope. The best way to minimize risk is to reduce reliance on third-party software packages and favour standard libraries or native tooling wherever possible. There is a balance to be struck, however. Using AWS-supported Terraform modules can significantly reduce the number of resources that the project needs to generate/maintain. As a result, we use third-party packages issued by trusted upstream sources and try to minimize reliance on less well-known Github projects (even if this means the resulting code is less elegant).

Notable design decisions

In addition to the general design decisions noted above, there are a few decisions which must be called out specifically due to their influence on the project.

  1. The solution should create infrastructure, conduct VM configuration, and generate application configuration.
    Seqera Platform needs all three of these to run successfully. Whereas a Kubernetes deployment is more cleanly split between Terraform (infrastructure) and Helm (application configuration), a VM-based Docker-Compose solution isn't as easy. To provide clients with a true end-to-end experience, we need to blur the lines of segregated responsibly.

  2. Conduct VM configuration via Ansible rather than Packer.
    Hashicorp advises against using provisioners except as a last resort, and has a specific section regarding configuration management via Packer. This project uses Ansible instead of Packer.

    While Packer is a great tool, its inclusion would add a tool dependency and introduce delays into the first-time provisioning process (waiting for the image to generate so it can be referenced by the AWS EC2 instance resource). Changes to system packages would force AMI regeneration and would not remove the need for Ansible since it is used to execute further application configuration steps on the instance.

  3. Use bespoke Bash commands via local-exec to copy and run commands on the remote VM.

    Earlier versions of this tool used native Terraform provisioners like file and remote-exec to cleanly copy local files to the remote instance and execute commands. These features rely on the establishment of a connection to the target instance.

    Our clients have consistently said that they want to run their Seqera Platform instance in a private subnet and minimize public access to their VPC. AWS's Instance Connect Endpoint service offers a way to connect to a private subnet-based machine directly, but relies upon the ability to specify ProxyCommand arguments to establish a tunnel.

    The Terraform connection object does not appear to support ProxyCommand arguments, which negates the viability of these provisioners for this common use case. The Terraform community is actively discussing adding enhancements for the SSM tunnel, but this is currently only available as an unofficial module (terraform-ssh-tunnel). As a result, despite the known risks and general preference, bespoke logic is used execute interaction with the remote host.

  4. Use AWS Instance Connect Endpoint as the default method to connect to a private VM.

    AWS's Instance Connect Endpoint is an elegant way to shield the Seqera Platform VM instance from direct public access while still allowing administrators to reach it via SSH when necessary (even across the public internet). The security gains heavily outweigh the additional complexity needed to implement the solution, and thus has been implemented as the default approach.

    EICE is not a perfect solution, however. There is a limit of one endpoint per VPC (which can only connect to a single subnet) and access must be limited if deploying into an existing VPC with heavily managed routing infrastructure (split-tunnels, Palo Alto firewalls, etc). In such cases, where a VPN connection is already present, the EICE solution can be discarded and the private IP of the VM be used to connect directly.

  5. A single NAT will serve private subnets if a new VPC with private subnets is created. The VMs in private subnets must be able to egress to the public internet to accomplish their operational duties. As a result, a newly-generated VPC must have NAT infrastructure in place to support this egress. To minimize ongoing operational charges to our clients, the VPC module is configured by default to use a single NAT instance to support all private subnets within the VPC. Should this decision not meet a client's needs, they can modify the VPC module directly.

  6. Support DNS flexibility.
    DNS has proven to be surprisingly diverse in our client sites thus far, and it is not guaranteed that the installer will have the authority or access required to automatically generate the A record needed to direct traffic to a target load balancer or VM instance. As a result, the installer has been built to:

    • Use an existing public Route53 hosted zone in the same account
    • Use an existing or create a new private hosted zone in Route53 (same account)
    • Or entirely forego DNS record generation and only use a local hosts file entry in the event that a record must be created in an external system.

  7. Platform instance must be contacted via DNS name Due to modified redirection logic introduced in 23.3, DNS names should be used to contact the Platform instance rather than the VM IP. This is possible due to the range of DNS options the installer supports.

  8. Public VMs will be assigned an Elastic IP This provides better stability (especially if using a DNS solution not accessible to the installer) for a minor ongoing cost. In the event that the VM hosting the Seqera Platform is stopped and restarted, it will keep the same public IP, thereby avoiding the need to update any external systems.

  9. Solution will not manage database backups.
    There is too much variability in client operational procedures to implement a one-size-fits-all solution for database backup. This is left to each client to implement in a manner appropriate to their business needs.

  10. Solution is HIGHLY opinionated re: your Data Studio implementation. The Data Studio feature made available in v24.1 requires nested subdomains to function. While the feature itself supports the data studio URL being several layers below your Tower domain, this results in more complex configurations of your DNS and related TLS certificates.

    As a result, this tool has a highly opinionated implementation of the Data Studio feature, which is not adjustable without directly modifying non-terraform.tfvars settings.

    1. The Subdomain is one layer below your Tower URL.
    2. The Subdomain is called connect. (example: https://connnect.mytower.com).