Switch to PostgreSQL 16 image by ShimShtein · Pull Request #577 · theforeman/foremanctl

ShimShtein · 2026-06-18T13:24:54Z

Why are you introducing these changes? (Problem description, related links)

As part of being prepared to run on RHEL 10, we need to switch to postgres 16.

What are the changes introduced in this pull request?

Switch of the container to postgres 16 instead of 13.

How to test this pull request

If the system spins up successfully and works with content - it's good to go.

ShimShtein · 2026-06-18T13:28:40Z

Extracted out of #395

evgeni · 2026-06-18T14:04:53Z

Extracted out of #395

What for? (#395 already identified the problems with it)

ShimShtein · 2026-06-18T14:24:09Z

Because #395 has both postgres and valkey in the same PR?

ShimShtein · 2026-06-18T16:27:11Z

I see the tests are failing on password encryption algorithm detection: https://github.com/theforeman/foremanctl/actions/runs/27768839212/job/82162895170?pr=577#step:17:643
That's weird. Locally the same setup gives me:

[vagrant@quadlet ~]$ sudo podman exec postgresql psql -U postgres -c 'SHOW password_encryption'
 password_encryption 
---------------------
 scram-sha-256
(1 row)

Which is weird.

evgeni · 2026-06-22T09:41:41Z

I see the tests are failing on password encryption algorithm detection: https://github.com/theforeman/foremanctl/actions/runs/27768839212/job/82162895170?pr=577#step:17:643 That's weird. Locally the same setup gives me:
[vagrant@quadlet ~]$ sudo podman exec postgresql psql -U postgres -c 'SHOW password_encryption'
 password_encryption 
---------------------
 scram-sha-256
(1 row)
Which is weird.

It's not weird, as that was the ext DB and that wasn't using our image definition properly. Fixed.

ShimShtein · 2026-06-28T14:12:53Z

  ansible.builtin.include_role:
    name: debug_tools

+- name: 'Enable postgresql:16 module'


@evgeni, let me ask this question again: wouldn't it be more resilient to run the PG commands from inside a PG container instead of installing the client tooling directly on the host?

Would, sure. I just had none that I could integrate easily into the various workflows where we need Postgres tooling on the host.

You'd likely want to write a module that does this to be re-used because there are postgres actions that are run in a lot more places. I am not sure I see the value of it at the moment but it might be worth exploring after we get everything else.

@ehelms Debian 13 has PostgreSQL 17. That probably means you can't run pg_dump on the PostgreSQL 16 container we run, which breaks backups.

evgeni · 2026-06-29T12:43:10Z

The upgrade complains:

Jun 28 14:37:27 quadlet.example.com postgresql[123049]: Incompatible data directory.  This container image provides
Jun 28 14:37:27 quadlet.example.com postgresql[123049]: PostgreSQL '16', but data directory is of
Jun 28 14:37:27 quadlet.example.com postgresql[123049]: version '13'.
Jun 28 14:37:27 quadlet.example.com postgresql[123049]: 
Jun 28 14:37:27 quadlet.example.com postgresql[123049]: This image supports automatic data directory upgrade from
Jun 28 14:37:27 quadlet.example.com postgresql[123049]: '15', please _carefully_ consult image documentation
Jun 28 14:37:27 quadlet.example.com postgresql[123049]: about how to use the '$POSTGRESQL_UPGRADE' startup option.

The fact its supporting 15→16 only is encoded here:
https://github.com/sclorg/postgresql-container/blob/f1c89bca212c6a829ab74fdfdeec264f0d08a992/16/Dockerfile.c10s#L13-L14

Funnily, the container actually contains 13 binaries:

% podman run --rm -ti quay.io/sclorg/postgresql-16-c10s bash
…
bash-5.2$ rpm -q postgresql-upgrade
postgresql-upgrade-16.13-2.el10.x86_64
bash-5.2$ rpm -qf /usr/lib64/pgsql/postgresql-13/bin/postgres
postgresql-upgrade-16.13-2.el10.x86_64
bash-5.2$ /usr/lib64/pgsql/postgresql-13/bin/postgres --version
postgres (PostgreSQL) 13.23

I'd argue this is a bug in the image we should ask for a fix.

ehelms · 2026-07-03T11:04:00Z

    - "../../../src/vars/defaults.yml"
    - "../../../src/vars/flavors/{{ flavor }}.yml"
    - "../../../src/vars/database.yml"
+    - "../../../src/vars/images.yml"


Why is this needed now?

ehelms · 2026-07-03T11:04:17Z

  notify:
    - Restart postgresql

+- name: Check if PG_VERSION file exists


This set of upgrade tasks look good. Could they go in a dedicated task file?

ehelms · 2026-07-03T11:04:43Z

+  ansible.builtin.systemd:
+    name: postgresql.service
+    masked: false
+  when:


As this when pattern is repeated, you can put it in a block.

ekohl

Wouldn't it be easier to unconditionally set POSTGRESQL_UPGRADE so it will always run when needed?

Thinking about the logging aspect: where do the logs of the upgrade go now? Can we somehow debug it when it fails? When it's part of the regular service then you'll have it the journal and sosreport will pick it up already.

ekohl · 2026-07-03T11:10:51Z

+      POSTGRESQL_MAX_CONNECTIONS: "{{ postgresql_max_connections }}"
+      POSTGRESQL_SHARED_BUFFERS: "{{ postgresql_shared_buffers }}"
+      POSTGRESQL_EFFECTIVE_CACHE_SIZE: "{{ postgresql_effective_cache_size }}"
+      POSTGRESQL_PREV_VERSION: "{{ postgresql_version }}"


Please link why this workaround is needed

Suggested change

POSTGRESQL_PREV_VERSION: "{{ postgresql_version }}"

# https://github.com/sclorg/postgresql-container/issues/661

POSTGRESQL_PREV_VERSION: "{{ postgresql_version }}"

ekohl · 2026-07-03T11:11:38Z

+      POSTGRESQL_SHARED_BUFFERS: "{{ postgresql_shared_buffers }}"
+      POSTGRESQL_EFFECTIVE_CACHE_SIZE: "{{ postgresql_effective_cache_size }}"
+      POSTGRESQL_PREV_VERSION: "{{ postgresql_version }}"
+      POSTGRESQL_UPGRADE: "hardlink"


Can you please make this a variable so we could easily switch to copy?

ekohl · 2026-07-03T11:17:58Z

+    - "postgresql_version == '13'"
+  register: postgresql_upgrade_result
+
+- name: Stop PostgreSQL systemd service (Quadlet)


Why is it needed to stop the service again?

pr-processor Bot added the Not yet reviewed label Jun 18, 2026

ShimShtein marked this pull request as ready for review June 18, 2026 13:25

ShimShtein force-pushed the test_pg16 branch from 678ef27 to a24bcfa Compare June 18, 2026 15:03

evgeni force-pushed the test_pg16 branch 2 times, most recently from f9dc89d to f6f88a0 Compare June 22, 2026 09:40

evgeni force-pushed the test_pg16 branch from f6f88a0 to 14e033e Compare June 22, 2026 12:13

evgeni mentioned this pull request Jun 23, 2026

Use different foremanctl versions when testing upgrades #586

Merged

2 tasks

ShimShtein force-pushed the test_pg16 branch from 14e033e to 2c2c937 Compare June 28, 2026 14:05

Switch to postgres 16 image

08e4391

ShimShtein force-pushed the test_pg16 branch from 2c2c937 to 08e4391 Compare June 28, 2026 14:11

ShimShtein commented Jun 28, 2026

View reviewed changes

pr-processor Bot removed the Not yet reviewed label Jun 28, 2026

evgeni closed this Jun 29, 2026

evgeni reopened this Jun 29, 2026

ShimShtein closed this Jul 1, 2026

ShimShtein reopened this Jul 1, 2026

ekohl mentioned this pull request Jul 1, 2026

PostgreSQL 16 images based on EL10 have an incorrect POSTGRESQL_PREV_VERSION set sclorg/postgresql-container#661

Open

ShimShtein force-pushed the test_pg16 branch from 1ce6d8c to 72e983b Compare July 1, 2026 16:04

Add a data migration steps

7e0aadc

ShimShtein force-pushed the test_pg16 branch from 72e983b to 7e0aadc Compare July 2, 2026 23:02

ehelms reviewed Jul 3, 2026

View reviewed changes

ekohl changed the title ~~Switch to postgres 16 image~~ Switch to PostgreSQL 16 image Jul 3, 2026

ekohl reviewed Jul 3, 2026

View reviewed changes

	POSTGRESQL_PREV_VERSION: "{{ postgresql_version }}"
	# https://github.com/sclorg/postgresql-container/issues/661
	POSTGRESQL_PREV_VERSION: "{{ postgresql_version }}"

Uh oh!

Conversation

ShimShtein commented Jun 18, 2026

Why are you introducing these changes? (Problem description, related links)

What are the changes introduced in this pull request?

How to test this pull request

Uh oh!

ShimShtein commented Jun 18, 2026

Uh oh!

evgeni commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShimShtein commented Jun 18, 2026

Uh oh!

ShimShtein commented Jun 18, 2026

Uh oh!

evgeni commented Jun 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

evgeni commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ekohl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

evgeni commented Jun 18, 2026 •

edited

Loading

evgeni commented Jun 29, 2026 •

edited

Loading