Odo Dev not creating podman containers

[ucfchandra]

/kind bug

What versions of software are you using?

Operating System:

Mac OSX Darwin (not M-series)

Output of odo version:

===============================================
⚠ unable to fetch the cluster server version

odo v3.15.0 (10b5e8a-Homebrew)

Podman Client: 4.9.3

How did you run odo exactly?

Using the repository I have at github.com/UCF/podman-django-template

I tried
odo dev --platform podman

Actual behavior

odo dev --platform podman --debug
__
/ __ Developing using the "helloworld" Devfile
_/ \ Platform: podman
/ _/ odo version: v3.15.0 (10b5e8a-Homebrew)
__/

↪ Running on podman in Dev mode
✓ Web console accessible at http://localhost:20000/
✓ API Server started at http://localhost:20000/api/v1
✓ API documentation accessible at http://localhost:20000/swagger-ui/

↪ Building & Pushing Image: mysql
• Building image locally ...
STEP 1/4: FROM mysql:8.3
STEP 2/4: ENV MYSQL_DATABASE helloworld
--> Using cache 2360beccba9650a81e882c7be5726aa6aeac51e4698904f48039fdcde1094f50
--> 2360beccba96
STEP 3/4: ENV MYSQL_ROOT_PASSWORD password
--> Using cache a7cb649d7e1757a24546eac9e917415f893887e27ed6b018c285d4fc529c813a
--> a7cb649d7e17
STEP 4/4: ADD ./helloworld-db/setup.sql /docker-entrypoint-initdb.d
--> Using cache ab3f141610c78879ee2d8bf1717a41cd7947ff6f5636bd1a01ca26da664c2a60
COMMIT mysql
--> ab3f141610c7
Successfully tagged localhost/mysql:latest
ab3f141610c78879ee2d8bf1717a41cd7947ff6f5636bd1a01ca26da664c2a60
✓ Building image locally [845ms]
• Pushing image to container registry ...
Getting image source signatures
Copying blob sha256:33d3b3159e300d8885f72cd9c3e2018bda3ad36c3e50353ff03c1381adf8c32c
Copying blob sha256:750da67a1a6f352df5cbc534874390dcb990bc9280bbf3971f2884092cebd220
Copying blob sha256:55d310cc1793fd699f7389585b86d5ff0270ad8f516f81db295e51e12f6d0b5b
Copying blob sha256:2750a469bea2e73998aa5cba7a922964d205740bd99be4ad566ac58b73268343
Copying blob sha256:3bac5d5ee750914c514bb25d3379507daa1423faa1e247ba7064154a8a44d55f
Copying blob sha256:8b4846888b1ba0dfc608d6c6e18feab1374d098e36ada375fb7406a5db84fff1
Copying blob sha256:eb8536f7685914517d47c0061f9a372cdb3d73db4728a3bad0760f83708ee114
Copying blob sha256:4c8b4f38c0d5985aa109bcf0e6a3bde3b4ffc28b335f9952ed5b6f4077b3ac29
Copying blob sha256:1319321d32989d1c7fdc5d19e6ee0fb358cc7faf3850f21f8bc070e56900014f
Copying blob sha256:4d7acf174bccb90ea7b122ace3109147ff0f7a3527c141e9ef5c92152e3bdcec
Copying blob sha256:107ae001c125197b331115a405899598dc923ca1026e48730d892abcb46e1ac8
Error: writing blob: initiating layer upload to /v2/library/mysql/blobs/uploads/ in registry-1.docker.io: requested access to the resource is denied
✗ Pushing image to container registry [6s]
Error occurred on Push - error running podman command: exit status 125

↪ Dev mode
Status:
Watching for changes in the current directory /Users/ch807460/code/podman-django-template

Web console accessible at http://localhost:20000/

Expected behavior

Expected podman ps to show containers. Expected to go to localhost:8004 and see Django application

Any logs, error output, etc?

None else

Thank you

[rm3l]

Hi @ucfchandra ,

Thanks for reporting this.

The default behavior of odo when handling Devfile image components is to build and push them. In your case, it was trying to push the image, and by default, since it is a relative image name, the underlying container engine would try to push to the docker.io registry, I think.
It is possible to instruct odo not to push images built, by setting the ODO_PUSH_IMAGES environment variable to false prior to running odo dev. See https://odo.dev/docs/overview/configure#environment-variables-controlling-odo-behavior

I've added the instructions to our Troubleshooting page: https://odo.dev/docs/troubleshooting/#i-have-an-image-component-in-my-devfile-and-odo-dev-on-podman-errors-out-with-requested-access-to-the-resource-is-denied

Please take a look and let us know if you run into any other issues.

[ucfchandra]

Thank you, this has gotten me to the point where Odo is now building the images. However, I run into this new issue:

✗ Deploying pod [4s]
Error occurred on Push - exit status 125:
Complete Podman output:
Error: playing YAML file: initializing source docker://localhost/mysql:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused

I am in an enterprise environment, and ping localhost is giving me request timeouts. I have access to the internet. Is there a way to get around this step?

[rm3l]

Thank you, this has gotten me to the point where Odo is now building the images. However, I run into this new issue:

✗ Deploying pod [4s] Error occurred on Push - exit status 125: Complete Podman output: Error: playing YAML file: initializing source docker://localhost/mysql:latest: pinging container registry localhost: Get "localhost/v2": dial tcp [::1]:443: connect: connection refused

I am in an enterprise environment, and ping localhost is giving me request timeouts. I have access to the internet. Is there a way to get around this step?

Because the image pull policy of the Pod created by odo is Always, Podman tries to pull the image, and by default uses localhost as the search registry for relative image names.
You can override the image pull policy to IfNotPresent or Never in your Devfile container component.
Please take a look at https://odo.dev/docs/troubleshooting/#recommended-solution-5, where this is described (first item of the recommended solution).

[ucfchandra]

You are so helpful thank you. I think I only have one last bug and everything should be up and running.

The pod is up and running and it says:

• Forwarding from 127.0.0.1:20001 -> 3306
• Forwarding from 127.0.0.1:20002 -> 8000

But when I run podman ps, it shows

 CONTAINER ID  IMAGE                                    COMMAND               CREATED             STATUS                 PORTS                                                 NAMES
24cf7f0e24f9  localhost/podman-pause:4.9.3-1708357294                        About a minute ago  Up About a minute      127.0.0.1:20001->3306/tcp, 127.0.0.1:20002->8000/tcp  08bf1dee0c42-infra
22991abd9d93  localhost/mysql:latest                   mysqld                About a minute ago  Up About a minute      127.0.0.1:20001->3306/tcp, 127.0.0.1:20002->8000/tcp  template-app-mysql
9118b715aa70  localhost/web:latest                     sh -c python mana...  About a minute ago  Up Less than a second  127.0.0.1:20001->3306/tcp, 127.0.0.1:20002->8000/tcp  template-app-web

When I try to access port 8000 on my browser to see the app running, it does not show the application.

Thank you so much for helping our university create a template application for others to do local development with Odo with.

EDIT: progressed past the previous issue posted in this comment and updated

[ucfchandra]

For reference, here is my current devfile

schemaVersion: 2.2.0
metadata:
  description: A template for setting up a Django-MySQL application
  name: template
  version: 0.0.1
components:
  - name: mysqlimage
    image:
      autoBuild: true
      dockerfile:
        buildContext: .
        rootRequired: false
        uri: Containerfile-db
      imageName: mysqlimage
  - name: mysql
    container:
      image: mysqlimage
      endpoints:
      - name: http-3306
        targetPort: 3306
    attributes:
      container-overrides:
        imagePullPolicy: Never
  - name: webimage
    image:
      autoBuild: true
      dockerfile:
        buildContext: .
        rootRequired: false
        uri: Containerfile-web
      imageName: webimage
  - name: web
    container:
      image: webimage
      endpoints:
      - name: http-8000
        targetPort: 8000
    attributes:
      container-overrides:
        imagePullPolicy: Never

[rm3l]

The pod is up and running and it says:

Forwarding from 127.0.0.1:20001 -> 3306
Forwarding from 127.0.0.1:20002 -> 8000
[...]
When I try to access port 8000 on my browser to see the app running, it does not show the application.

From the odo dev output that you shared, port 8000 of your application is forwarded locally via port 20002. So you should try to open http://127.0.0.1:20002 instead on your browser.
If the application is still not reachable, you can inspect the dev session logs by running odo logs --platform=podman from a separate terminal..

[rm3l]

I quickly tried on your repo, and it seems that your web application is not able to connect to the mysql database.
Right now, all Devfile container components run in the same pod, which means that the database container can be reachable from the web container using localhost.

$ odo logs --platform=podman
[...]
--> Logs for template-app / web
web: Traceback (most recent call last):
web:   File "/usr/local/lib/python3.12/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection
web:     self.connect()
web:   File "/usr/local/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
web:     return func(*args, **kwargs)
web:            ^^^^^^^^^^^^^^^^^^^^^
web:   File "/usr/local/lib/python3.12/site-packages/django/db/backends/base/base.py", line 256, in connect
web:     self.connection = self.get_new_connection(conn_params)
web:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web:   File "/usr/local/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
web:     return func(*args, **kwargs)
web:            ^^^^^^^^^^^^^^^^^^^^^
web:   File "/usr/local/lib/python3.12/site-packages/django/db/backends/mysql/base.py", line 256, in get_new_connection
web:     connection = Database.connect(**conn_params)
web:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web:   File "/usr/local/lib/python3.12/site-packages/MySQLdb/__init__.py", line 121, in Connect
web:     return Connection(*args, **kwargs)
web:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
web:   File "/usr/local/lib/python3.12/site-packages/MySQLdb/connections.py", line 195, in __init__
web:     super().__init__(*args, **kwargs2)
web: MySQLdb.OperationalError: (2002, "Can't connect to server on 'mysql' (115)")
web: 

[ucfchandra]

I think it may be an issue of the web container starting before the mysql container is fully initialized. Is there a way to do healthchecks or depends_on in a devfile akin to how compose does it? Or better yet, a way for odo to use a compose file? I couldn't find anything like that in the API Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Armel Soro ***@***.***> Sent: Saturday, March 9, 2024 2:20:47 AM To: redhat-developer/odo ***@***.***> Cc: Chandra Reddy ***@***.***>; Mention ***@***.***> Subject: Re: [redhat-developer/odo] Odo Dev not creating podman containers (Issue #7224) I quickly tried on your repo, and it seems that your web application is not able to connect to the mysql database. Right now, all Devfile container components run in the same pod, which means that the database container can be reachable from the web container using localhost. $ odo logs --platform=podman [...] --> Logs for template-app / web web: Traceback (most recent call last): web: File "/usr/local/lib/python3.12/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection web: self.connect() web: File "/usr/local/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner web: return func(*args, **kwargs) web: ^^^^^^^^^^^^^^^^^^^^^ web: File "/usr/local/lib/python3.12/site-packages/django/db/backends/base/base.py", line 256, in connect web: self.connection = self.get_new_connection(conn_params) web: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ web: File "/usr/local/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner web: return func(*args, **kwargs) web: ^^^^^^^^^^^^^^^^^^^^^ web: File "/usr/local/lib/python3.12/site-packages/django/db/backends/mysql/base.py", line 256, in get_new_connection web: connection = Database.connect(**conn_params) web: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ web: File "/usr/local/lib/python3.12/site-packages/MySQLdb/__init__.py", line 121, in Connect web: return Connection(*args, **kwargs) web: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ web: File "/usr/local/lib/python3.12/site-packages/MySQLdb/connections.py", line 195, in __init__ web: super().__init__(*args, **kwargs2) web: MySQLdb.OperationalError: (2002, "Can't connect to server on 'mysql' (115)") web: — Reply to this email directly, view it on GitHub<#7224 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BD5YIMSWHYK7XSZPH526XG3YXKZ47AVCNFSM6AAAAABELQ7XSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWG43TIOJQGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[rm3l]

I think it may be an issue of the web container starting before the mysql container is fully initialized. Is there a way to do healthchecks or depends_on in a devfile akin to how compose does it?

I don't think this is possible right now using the Devfile. Not tested, but based on your current devfile, maybe in your devfile, you can work around this by overriding the command field of your web container component, such that it waits until the DB is ready before starting the application? It is a hack, but this way, odo would override the CMD defined in your image.
Something like this (feel free to adapt):

@@ -32,6 +32,13 @@ components:
   - name: web
     container:
       image: webimage
+      command: ['/bin/sh']
+      args:
+      - -c
+      - 'until nc -vz 127.0.0.1 3306; do echo Waiting for DB; sleep 2; done && python manage.py migrate && python manage.py runserver 0.0.0.0:8000'
       endpoints:
       - name: http-8000
         targetPort: 8000

If nc is not available in your image, I think we can check with Python directly. So it should work by replacing nc -vz 127.0.0.1 3306 with python -c "import socket;exit(0) if 0 == socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect_ex((''127.0.0.1'',3306)) else exit(1)".

Or better yet, a way for odo to use a compose file? I couldn't find anything like that in the API

Unfortunateley not possible yet at this time, as this would need to be supported in the Devfile specification.
There is an issue in the Devfile specification to be able to support Compose files along with an ongoing PR, but I don't know the status of this. See devfile/api#501. Feel free to vote for this.

[github-actions]

A friendly reminder that this issue had no activity for 90 days. Stale issues will be closed after an additional 30 days of inactivity.

[github-actions]

This issue was closed because it has been inactive for 30 days since being marked as stale.