Permission denied from mkdir /tmp_docker in v4.2.0

[kenyon]

With v4.2.0 of this module, I get Permission denied - /tmp_docker from a puppet agent run.

Caused by #805: https://github.com/puppetlabs/puppetlabs-docker/pull/805/files#r847521695

[r-catania]

We are seeing the same behavior with 4.2.0

error during compilation: Evaluation Error: Error while evaluating a Virtual Query, Could not autoload puppet/type/docker_compose: Could not autoload puppet/provider/docker_compose/ruby: Permission denied @ dir_s_mkdir - /tmp_docker

[canihavethisone]

I have commented on the pull request. Requested investigation.

[canihavethisone]

I've done some testing with this and confirmed that the pull request has an unexpected outcome in some environments on the first puppet run only. Observations are:

1. /tmp_docker dir creation neither attempts or errors if docker classes are not included in ENC
2. /tmp_docker dir creation error only occurs on first puppet run when any docker classes are included in ENC, does not error on subsequent runs. This could be a similar issue to docker_volume not created on first puppet run #782, Error: Could not find a suitable provider for docker_compose #742 and docker_network is not applied correctly on first run #703 where providers are not realized on the first puppet run?
3. /tmp_docker dir creation only actually occurs if the docker_compose is declared (this is the intended outcome)
4. observation 2 above occurs regardless of where the directory for TMPDIR env var is created

As per comments in the pull request, the intention of the dir and env var are to overcome known issue as also reported at docker/compose#8041.

I will investigate a way to overcome observation 2 above, and have also requested maintainer input on this. In the interim, the error does not appear to occur beyond the first puppet run.

[chelnak]

Hey all! Thanks for raising this. It's certainly strange that this has popped up. It's not something that we observed initially.

We will spin up a lab today and attempt to reproduce the issue.

Thanks!!

[chelnak]

OK I'm looking at this now 👀

Now I'm in the codebase, I'm starting to wonder if we should create a new property on the type. e.g tmpdir.

We can make our intention clear in the description and say that this can be used to override the TMPDIR setting for docker compose and should only be used on systems that are impacted by the referenced bug.

We should also probably avoid creating new directories in this situation and instead recommend something like /var/tmp as an alternative to /tmp.. It might also be worth noting that unlike /tmp, the contents of /var/tmp will persist between reboots.

With the above implementation it is the responsibility of the module consumer to ensure that thier custom tmpdir exists before docker compose attempts to use it.

What do you think?

[chelnak]

@canihavethisone @kenyon

Please could you share an example manifest that I can use to reproduce the error? I want to make sure that we are doing things in the same way.

[canihavethisone]

Sounds like a good approach. I'll check a CIS hardened system in a few mins with regard to the exec bit on the suggested /var/tmp. I also observed that the temporary files created by docker-compose are cleaned up after it runs.

[canihavethisone]

i tested with a slim wrapper such as:

class test (
  $composefiles = lookup( 'test::composefiles',    Optional[Hash],  'deep', {} ),
  $compose_defs = lookup( 'test::compose_defs',    Optional[Hash],  'deep', {} ),
){

  # Include the main docker class
  class { 'docker':
    log_driver => 'journald',
  }

  # Include classes as listed in hiera
  hiera_include('classes')

  # Create docker-compose files
  $composefiles.each  |$key, $value| {
    file { $key:
      * => $value
    }
  }

  # Process docker-compose resources
  $compose_defs.each  |$key, $value| {
    docker_compose { $key:
      * => $value
    }
  }

  # Ensure that iptables is installed before docker
  Package <| title == 'iptables' |> -> Package <| title =='docker' |>

  # Ensure that images are built, then containers run, then docker compose
  Docker::Image <| |> -> Docker::Run <| |> -> Docker_compose <| |> 

}

with hiera such as:

---
classes:
  - docker::compose

test::composefiles:
  /tmp/composefile1.yaml:
    source: 'http://192.168.1.60/composefile1.yaml'
  /tmp/composefile2.yaml:
    source: 'http://192.168.1.60/composefile2.yaml'

test::compose_defs:
  test:
    compose_files: ['/tmp/composefile1.yaml']
    subscribe:     'File[/tmp/composefile1.yaml]'
    ensure:        present
    scale:
      db:        1
      wordpress: 1      
  test2:
    compose_files: ['/tmp/composefile2.yaml']
    subscribe:     'File[/tmp/composefile2.yaml]'
    ensure:        present
    scale:
      my_test:    1
      my_test_2:  2
      redis:      3
      postgres:   4

The content of the compose files is very basic
composefile1.yaml:

version: "3.9"
    
services:
  db:
    image: mysql:5.7
    volumes:
      - db_data:/var/lib/mysql
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: somewordpress
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: wordpress
    
  wordpress:
    depends_on:
      - db
    image: wordpress:latest
    volumes:
      - wordpress_data:/var/www/html
    ports:
      - "8000:80"
    restart: always
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: wordpress
      WORDPRESS_DB_NAME: wordpress
volumes:
  db_data: {}
  wordpress_data: {}

composefile2.yaml:

version: '3'
services:
  my_test:
    build:
      context: .
      dockerfile: ./test.dockerfile
    command: >
      sh -c "echo hello from my_test_$$(hostname) > /etc/motd &&
             while true; do echo hello from my_test_$$(hostname); sleep 5; done"
    restart: unless-stopped
  my_test_2:
    image: my_test:35
    command: >
      sh -c "echo hello from my_test_2_$$(hostname) > /etc/motd &&
             while true; do echo hello from my_test_2_$$(hostname); sleep 5; done"
    restart: unless-stopped
  redis:
    image: redis:latest
    ports:
      - "6379-6400:639"
  postgres:
    image: postgres:latest
    restart: always
    entrypoint: ["/bin/sh","-c"]
    command: 
    - | 
       echo hello from postgres_$$(hostname) > /etc/motd
       while true; do echo hello from postgres_$$(hostname); sleep 5; done
    ports:
      - "6500-6510:5432"

[canihavethisone]

on CIS hardened systems, /var/tmp is also noexec. Perhaps another folder such as /usr/local/share/tmp_docker?

[canihavethisone]

if users prefer to just allow exec on /tmp and it doesn't conflict with puppet managed hardening, the following works but is yuck

  if $compose_defs {
    exec { 'allow exec on /tmp':
      command => 'mount /tmp -o remount,exec',
      unless  => 'mount | grep "on /tmp" | grep -v noexec',
      path    => ['/usr/bin', 'bin']
    }
  }

[chelnak]

I feel like we should allow the user to decide on the directory that is used to override the /tmp directory. We should just give the option to pass in a directory and not prescribe a certain behaviour.

Maybe in the documentation we can indicate that the directory used should not have the noexec bit.

[maltewhiite]

Experiencing the same thing. Happens on every puppet run.

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Virtual Query, Could not autoload puppet/type/docker_compose: Could not autoload puppet/provider/docker_compose/ruby: Permission denied - /tmp_docker (file: /etc/puppetlabs/code/environments/test/modules/docker/manifests/init.pp, line: 663, column: 8) on node 

What version of this puppet-module should I roll back to?

EDIT: rolling back to 4.1.2 from september seems to be working.

[canihavethisone]

v4.1.2 doesn't have this issue. Alternatively it doesn't occur on the second puppet run with the latest v4.2.0, and the /tmp_docker folder is only created if docker_compose resources are declared