More "Failed to create temporary directory."

[ghost]

I've been struggling with the converge issue where Ansible cannot gather facts, allegedly due to not having permissions to write to the temporary directory:

TASK [Gathering Facts] *********************************************************
fatal: [instance]: UNREACHABLE! => changed=false
  msg: 'Failed to create temporary directory.In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in "/tmp", for more error information use -vvv. Failed command was: ( umask 77 && mkdir -p "` echo /tmp `"&& mkdir "` echo /tmp/ansible-tmp-1617853604.764122-263554-181361534476501 `" && echo ansible-tmp-1617853604.764122-263554-181361534476501="` echo /tmp/ansible-tmp-1617853604.764122-263554-181361534476501 `" ), exited with result 1'
  unreachable: true

The environment is as follows:

$ molecule --version
molecule 3.2.2 using python 3.8
    ansible:2.10.5
    delegated:3.2.2 from molecule
    docker:0.3.3 from molecule_docker

What is interesting is that when I run with two closely related OS Docker images, CentOS 7 and RHEL 7 in this case, the problem does not occur with CentOS 7 but does occur with RHEL 7. I've left remote_tmp set to /tmp but that does not seem to have any effect.

I'm mostly looking for any clues or guidance on how to dig further down and debug this issue.

[javanaut-de]

I have a similar running role tests with Jenkins using podman driver but it is also reported with docker, see #3818. I tried a lot, at least updating Python from 3.8 (Ubuntu 20.04 on a worker node) to latest source build (3.11.2) reduced reocurrance a lot but not completely.

Another sporadic error that occurs follows this scheme: containers/podman#13613. I cannot tell if there is a common root cause, but both types occur randomly with a low percentage of runs and seem not to be related to any variables of the tests, like used worker node or container image. They just happen with the same "felt probability". Example:

fatal: [redis-fedora37]: FAILED! => {
    "msg": "Failed to set execute bit on remote files (rc: 255, err: Error: open /var/lib/jenkins/.local/share/containers/storage/overlay-containers/731466a42e35f77c081daf18113118b4d855da25590c143040256586fb594ea4/userdata/68fb884c4c74b6aef936758fddcb1a90c8b0ab700274f48907f3495143adb7bc/exit/731466a42e35f77c081daf18113118b4d855da25590c143040256586fb594ea4: no such file or directory\n)"
}

[nre-ableton]

I'm also having problems with this. It fails on my Ubuntu 22.04 machine, but works fine on my 20.04 jumphost. I haven't had time to dig into it and figure out what the difference is in terms of Docker versions etc.