(PE-39789) Support custom environment

[davidmalloncares]

Summary

Allowing user to specify custom environment

Checklist

• 🟢 Spec tests.
• 🟢 Acceptance tests.

Changes include test coverage?

• Yes
• [] Not needed

Have you updated the documentation?

• Yes, I've updated the appropriate docs
• [] Not needed

[davidmalloncares]

@bastelfreak do you remember when you tested these changes did you manage to get a successful install running using a non-production environment name? I have tested and the install fails for me with the following error:

  Resources failed to apply for shy-complement.delivery.puppetlabs.net
    Node_group[PE Infrastructure Agent]: Unable to update node_group ''
    Node_group[PE Master]: Unable to update node_group ''
    Node_group[PE Database]: Unable to update node_group ''
    Node_group[PE Primary A]: change from 'absent' to 'present' failed: Could not set 'present' on ensure: Unable to create node_group 'PE Primary A' (file: /Users/david.mallon/peadm/large_ha_peadm/.modules/peadm/manifests/setup/node_manager.pp, line: 103)
    Node_group[PE HA Replica]: change from 'absent' to 'present' failed: Could not set 'present' on ensure: Unable to create node_group 'PE HA Replica' (file: /Users/david.mallon/peadm/large_ha_peadm/.modules/peadm/manifests/setup/node_manager.pp, line: 155)
    Node_group[PE Primary B]: change from 'absent' to 'present' failed: Could not set 'present' on ensure: Unable to create node_group 'PE Primary B' (file: /Users/david.mallon/peadm/large_ha_peadm/.modules/peadm/manifests/setup/node_manager.pp, line: 164)
  changed: 4, failed: 6, unchanged: -3 skipped: 6, noop: -3

[bastelfreak]

Yes this worked and a few customer use this patch on a regular basis. I will try to have a look in the next days. Maybe something got lost in the last rebases. peadm creates a huge amount of output. somewhere in there should be a stacktrace for the failed resources. There's a chance that those traces have more details.

[davidmalloncares]

Yes this worked and a few customer use this patch on a regular basis. I will try to have a look in the next days. Maybe something got lost in the last rebases. peadm creates a huge amount of output. somewhere in there should be a stacktrace for the failed resources. There's a chance that those traces have more details.

Np, could you give a link to the patch so I have the definitive changes to compare to what I am testing? Thanks

[bastelfreak]

Those are the steps for a manual installation + peadm convert + peadm upgrade, all with a demo controlrepo:

prepare fresh EL9 instance

dnf update -y --quiet; \
dnf install -y --quiet langpacks-de tar epel-release bash-completion vim jq git; \
dnf install -y --quiet htop ccze tree screen; \
dnf update -y --quiet

download & install PE

curl --silent --remote-header-name --location --remote-name 'https://pm.puppet.com/cgi-bin/download.cgi?dist=el&rel=9&arch=x86_64&ver=2021.7.9'
cat >~/pe.conf <<EOL
{
  "puppet_enterprise::profile::master::r10k_remote": "https://github.com/bastelfreak/testcontrolrepo.git",
  "puppet_enterprise::profile::master::code_manager_auto_configure": true,
  "console_admin_password": "TimIsTesting2024!",
  "puppet_enterprise::puppet_master_host": "%{trusted.certname}"
}
EOL
tar xf puppet-enterprise-*-x86_64.tar.gz
rm -f puppet-enterprise-*.tar.gz
cd puppet-enterprise-*-x86_64/
DISABLE_ANALYTICS=1 ./puppet-enterprise-installer -c ~/pe.conf -y
puppet agent -t; \
puppet agent -t; \
puppet agent -t; \
echo 'TimIsTesting2024!' | puppet access login --verbose --username admin --lifetime=4y; \
puppet code deploy peadm --wait; \
cert="$(puppet config print hostcert)"; \
cacert="$(puppet config print localcacert)"; \
key="$(puppet config print hostprivkey)"; \
uri="https://$(puppet config print server):4433/classifier-api/v1/update-classes?environment=peadm"; \
curl --cert "$cert" --cacert "$cacert" --key "$key" --request POST "$uri"

create new environment node group

source /etc/profile.d/puppet-agent.sh
# get id for `all environments` node group
auth_header="X-Authentication: $(puppet-access show)"
cacert="$(puppet config print localcacert)"
groups_uri="https://$(puppet config print server):4433/classifier-api/v1/groups"
id=$(curl --cacert "$cacert" --header "$auth_header" "$groups_uri" --silent | jq --raw-output '.[] | select(.name=="All Environments").id')
uid='085e2797-32f3-4920-9412-8e9decf4ef65'
new_group_uri="${groups_uri}/${uid}"

# create new environment node group
data="
{
  \"name\": \"peadm\",
  \"parent\": \"$id\",
  \"environment\": \"peadm\",
  \"environment_trumps\": true,
  \"description\": \"Test environment for PEADM upgrades\",
  \"classes\": {
    \"profiles::boltprojects\": {}
  },
  \"rule\": [
    \"or\",
    [
      \"=\",
      \"name\",
      \"$(hostname -f)\"
    ]
  ]
}
"
type_header='Content-Type: application/json'
curl --request PUT --cacert "$cacert" --header "$auth_header" --header "$type_header" "$new_group_uri" --data "$data"

run puppet two times

puppet agent -t
puppet agent -t

add new class the node group

data="
{
  \"name\": \"peadm\",
  \"parent\": \"$id\",
  \"environment\": \"peadm\",
  \"environment_trumps\": true,
  \"description\": \"Test environment for PEADM upgrades\",
  \"classes\": {
    \"profiles::cleanup\": {},
    \"profiles::boltprojects\": {}
  },
  \"rule\": [
    \"or\",
    [
      \"=\",
      \"name\",
      \"$(hostname -f)\"
    ]
  ]
}
"
curl --request PUT --cacert "$cacert" --header "$auth_header" --header "$type_header" "$new_group_uri" --data "$data"

run puppet agent

puppet agent -t
puppet agent -t
puppet agent -t

start peadm::convert and peadm::upgrade as systemd service

systemctl start peadmmig@profiles::convertandupgradeto2023.service
# watch logs
journalctl --unit peadmmig@profiles::convertandupgradeto2023.service --no-hostname --follow

The used PEADM branch: https://github.com/bastelfreak/puppetlabs-peadm/tree/issue-469. I will check for new installations later.

[bastelfreak]

I think the install plan fails because the puppet-enterprise installer doesn't create the environment from the deploy_environment parameter, before setting up nodegroups. That's basically the same issue as reported in #469 , just in the opposite direction. I think the easiest way is to leave out the parameter in the install plan and just implement it in the convert plan. The customers I support all do the code deployment + environment switch after PEADM finishes.

Or peadm::install needs to create the environment before starting the installer (e.g. mkdir -p /etc/puppetlabs/code/environments/$env). That works as well. I can post my examples later.

[davidmalloncares]

Thanks, I tried setting the node_group_environment for these two values to the pe.conf for the primary but the install failed with a different error:

pe_install::install::classification::pe_node_group_environment
puppet_enterprise::master::recover_configuration::pe_environment

unless I need the env created before the install as you suggest - although I can't use 'pe-puppet' as the owner and group before the install so not sure what values to use there.

[davidmalloncares]

Hi @bastelfreak I was testing today and the changes look good - all the updated plans are picking up the custom environment name form the primary etc although the plans that add noes to the infrastructure (add replica and add database) do not add the new nodes to the custom environment group but to the 'production' environment group - maybe that is as expected?

I was not not able to get custom environment to work on an install though so I may have to remove those bits form the PR

[davidmalloncares]

e.g. see the groups the replica and db I added are members of:

[bastelfreak]

Hi,
it's totally fine to remove the parameter from peadm::install plan, I just added it for convenience. The bug we hit was during upgrades, that that's properly fixed.

I would prefer it if we could merge/modify my original implementation from https://github.com/bastelfreak/puppetlabs-peadm/tree/issue-469. that properly attributes the commits. Also my implementation is split into two commits, because IMO this is a feature and a bugfix, and the commits have a proper description.

[davidmalloncares]

Yeah I'm ok with that (I had pushed my PR up just for testing really) - are you able to push that up as a PR with the param removed from the install plan? What do you think about the add_database and add_replica plans I tested above - would you expect the added nodes to land in the 'production' group rather than the custom 'peadm' one?