$PATH env dissapears which breaks facter plugin, when local connection is used

[kajinamit]

I don't know if this is the right place to report this, but I have no clear idea about the potential root cause I'm reporting this here first to ask for some thoughts.

I'm now trying to on-board some of the acceptance tests which has been based on puppet-litmus, but I've seen consistent failure when setting up mysql grants using puppetlabs-mysql ( I've tried 16.3.0 or 16.1.0 so far).

Looking into the failure further I've identified that the mysqld_version fact[1] is resolved to '' after the mariadb package is installed.

[1] https://github.com/puppetlabs/puppetlabs-mysql/blob/1e550701515c2c144719b4f769aa954a0cf2645f/lib/facter/mysqld_version.rb#L3

I see no problem with manual local run.

[root@localhost ~]# env PATH=$PATH:/usr/libexec mysqld --no-defaults -V 2>/dev/null
mysqld  Ver 10.5.27-MariaDB for Linux on x86_64 (MariaDB Server)
[root@localhost ~]# rpm -q mariadb
mariadb-10.5.27-1.el9.x86_64

I've injected some debug logs and eventually found that the PATH environment is not resolved correctly when the command is being executed by the factor plugin.

  Debug: Facter: Executing command: /bin/echo $PATH
  Debug: Facter: /opt/puppetlabs/bin:/home/vagrant/puppet-nova/.bundled_gems/bin:/sbin:/bin:/usr/sbin:/usr/bin
  Debug: Facter: Executing command: /bin/env PATH=$PATH:/usr/libexec mysqld --no-defaults 2 > /dev/null
  Debug: Facter: Command /bin/env PATH=$PATH:/usr/libexec mysqld --no-defaults 2 > /dev/null completed with the following stderr message: /bin/env: ':/usr/libexec': No such file or directory

Note that I'm using LocalConnection.

$ cat spec/acceptance/nodesets/centos-9-x86_64.yml 
HOSTS:
  localhost:
    roles:
      - master
    platform: el-9-x86_64
    hypervisor: none
CONFIG:
  default_module_install_opts:
    append_cmds: '--ignore-dependencies --force'
  trace_limit: 200
  type: foss

[kajinamit]

I'll dump list of gems I have in that env in case it helps.

Gems included by the bundle:
  * addressable (2.8.7)
  * ast (2.4.3)
  * base64 (0.3.0)
  * bcrypt_pbkdf (1.1.1)
  * beaker (7.0.1)
  * beaker-docker (3.0.0)
  * beaker-hiera (2.0.0)
  * beaker-hostgenerator (3.3.0)
  * beaker-rspec (9.0.0)
  * beaker-vagrant (2.0.0)
  * beaker_puppet_helpers (3.0.0)
  * benchmark (0.4.1)
  * bigdecimal (3.2.3)
  * builder (3.3.0)
  * colored2 (3.1.2)
  * concurrent-ruby (1.3.5)
  * crack (1.0.0)
  * cri (2.15.12)
  * deep_merge (1.2.2)
  * diff-lcs (1.6.2)
  * docker-api (2.4.0)
  * ed25519 (1.4.0)
  * erubi (1.13.1)
  * excon (1.3.0)
  * facter (4.6.1)
  * facterdb (3.10.0)
  * faraday (2.13.4)
  * faraday-follow_redirects (0.3.0)
  * faraday-net_http (3.4.1)
  * fast_gettext (2.4.0)
  * ffi (1.17.2)
  * forwardable (1.3.3)
  * getoptlong (0.2.1)
  * gettext (3.5.1)
  * gettext-setup (1.1.0)
  * gssapi (1.3.1)
  * gyoku (1.4.0)
  * hashdiff (1.2.1)
  * hocon (1.4.0)
  * httpclient (2.9.0)
  * in-parallel (1.0.1)
  * inifile (3.0.0)
  * jgrep (1.5.4)
  * json (2.14.0)
  * json-schema (6.0.0)
  * jwt (2.10.2)
  * little-plugger (1.1.4)
  * locale (2.1.4)
  * log4r (1.1.10)
  * logger (1.7.0)
  * logging (2.4.0)
  * metadata-json-lint (4.3.0)
  * minitar (0.12.1)
  * minitest (5.25.5)
  * mocha (2.7.1)
  * multi_json (1.17.0)
  * mutex_m (0.3.0)
  * net-http (0.6.0)
  * net-scp (4.1.0)
  * net-ssh (7.3.0)
  * net-telnet (0.2.0)
  * nori (2.7.1)
  * openfact (5.1.0)
  * openvox (8.23.1)
  * openvox-strings (5.0.0)
  * ostruct (0.6.3)
  * parallel (1.27.0)
  * parallel_tests (5.4.0)
  * parser (3.3.9.0)
  * pathspec (2.1.0)
  * prime (0.1.4)
  * prism (1.5.1)
  * public_suffix (6.0.2)
  * puppet (8.10.0)
  * puppet-lint (4.3.0)
  * puppet-lint-absolute_classname-check (4.0.0)
  * puppet-lint-anchor-check (2.0.0)
  * puppet-lint-file_ensure-check (2.0.0)
  * puppet-lint-leading_zero-check (2.0.0)
  * puppet-lint-lookup_in_parameter-check (2.0.0)
  * puppet-lint-manifest_whitespace-check (0.3.0)
  * puppet-lint-optional_default-check (2.0.0)
  * puppet-lint-param-docs (2.0.0)
  * puppet-lint-param-types (2.0.0)
  * puppet-lint-params_empty_string-check (2.0.0)
  * puppet-lint-resource_reference_syntax (2.0.0)
  * puppet-lint-stdlib_deprecations (0.0.3)
  * puppet-lint-strict_indent-check (3.0.0)
  * puppet-lint-topscope-variable-check (2.0.0)
  * puppet-lint-trailing_comma-check (2.0.0)
  * puppet-lint-unquoted_string-check (3.0.0)
  * puppet-lint-variable_contains_upcase (2.0.0)
  * puppet-lint-version_comparison-check (2.0.0)
  * puppet-modulebuilder (2.1.0)
  * puppet-openstack_spec_helper (20.0.0)
  * puppet-resource_api (1.9.0)
  * puppet-syntax (6.0.0)
  * puppet_fixtures (1.0.1)
  * puppet_forge (5.0.4)
  * puppetlabs_spec_helper (1.1.1)
  * r10k (4.1.0)
  * racc (1.8.1)
  * rainbow (3.1.1)
  * rake (13.3.0)
  * regexp_parser (2.11.3)
  * rexml (3.4.4)
  * rgen (0.10.2)
  * rspec (3.13.1)
  * rspec-core (3.13.5)
  * rspec-expectations (3.13.5)
  * rspec-github (2.4.0)
  * rspec-its (2.0.0)
  * rspec-mocks (3.13.5)
  * rspec-puppet (5.0.0)
  * rspec-puppet-facts (5.4.0)
  * rspec-puppet-utils (3.4.0)
  * rspec-support (3.13.6)
  * rsync (1.0.9)
  * rubocop (1.50.2)
  * rubocop-ast (1.46.0)
  * rubocop-capybara (2.21.0)
  * rubocop-rake (0.6.0)
  * rubocop-rspec (2.20.0)
  * ruby-augeas (0.6.0)
  * ruby-progressbar (1.13.0)
  * ruby2_keywords (0.0.5)
  * rubyntlm (0.6.5)
  * scanf (1.0.0)
  * semantic_puppet (1.1.1)
  * serverspec (2.43.0)
  * sfl (2.3)
  * singleton (0.3.0)
  * spdx-licenses (1.3.0)
  * specinfra (2.94.1)
  * stringify-hash (0.0.2)
  * syslog (0.3.0)
  * text (1.3.1)
  * thor (1.4.0)
  * unicode-display_width (2.6.0)
  * uri (1.0.3)
  * voxpupuli-acceptance (4.0.0)
  * voxpupuli-puppet-lint-plugins (5.0.0)
  * voxpupuli-test (11.1.0)
  * webmock (3.25.1)
  * winrm (2.3.9)
  * yard (0.9.36)

[kajinamit]

This problem is not seen when I run the manifest by executing the puppet command directly or even when the test is run via puppet-litmus. So far the problem with mysqld_version fact is the new thing I'm facing only in acceptance/beaker tests set up using voxpupuli-acceptance.

[kajinamit]

So I tested ssh connection to localhost by updating nodeset definition to

HOSTS:
  mynode:
    ip: 127.0.0.1
    roles:
      - master
    platform: el-9-x86_64
    hypervisor: none
CONFIG:
  default_module_install_opts:
    append_cmds: '--ignore-dependencies --force'
  trace_limit: 200
  type: foss

and this resolves the problem. However I'm still looking for a way to unblock local node connection because of our own CI infrastructure which can spawn fresh VMs per test run (hosted in opendev, consumed by OpenStack projects).

[kajinamit]

I wonder this could be a problem caused by that command executed in mysqld_version facter plugin and puppetlabs/puppetlabs-mysql#1687 resolve the issue.