Skip to content

Commit bf4412e

Browse files
pferatepferate
committed
Add Kerberos integration tests
Stands up a real KDC and Kerberos-configured ClickHouse instance via docker-compose (behind a `kerberos` profile) for end-to-end testing of use_kerberos, plus a dedicated CI job.
1 parent c9c28c1 commit bf4412e

12 files changed

Lines changed: 546 additions & 5 deletions

File tree

‎.docker/kerberos-kdc/Dockerfile‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
FROM rockylinux:9
2+
3+
RUN yum install -y ca-certificates krb5-server krb5-libs krb5-workstation
4+
5+
EXPOSE 88 749
6+
7+
RUN touch /config.sh
8+
# Overwritten via the docker-compose volume mount -- see tests/integration_tests/kerberos_conf/kerberos_image_config.sh
9+
10+
ENTRYPOINT ["/bin/bash", "/config.sh"]

‎.github/workflows/on_push.yml‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -287,9 +287,39 @@ jobs:
287287
COMPOSE_PROJECT_NAME: clickhouse-connect-ci
288288
run: docker compose -f docker-compose.yml down --volumes --remove-orphans
289289

290+
kerberos-integration-test:
291+
runs-on: ubuntu-latest
292+
needs: bare-import-test
293+
name: Kerberos Integration Tests
294+
steps:
295+
- name: Checkout
296+
uses: actions/checkout@v6
297+
- name: Set up Python 3.12
298+
uses: actions/setup-python@v6
299+
with:
300+
python-version: '3.12'
301+
- name: Install pip
302+
run: python -m pip install --upgrade pip
303+
- name: Install system Kerberos packages
304+
run: |
305+
sudo apt-get update
306+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y gcc python3-dev libkrb5-dev krb5-user
307+
- name: Add Kerberos ClickHouse instance to /etc/hosts
308+
run: sudo echo "127.0.0.1 server1.clickhouse.test" | sudo tee -a /etc/hosts
309+
- name: Install Test Dependencies
310+
run: pip install -r tests/test_requirements.txt
311+
- name: Build cython extensions
312+
run: python setup.py build_ext --inplace
313+
- name: "Add distribution info"
314+
run: python setup.py develop
315+
- name: Run tests
316+
env:
317+
CLICKHOUSE_CONNECT_TEST_KERBEROS: '1'
318+
run: pytest tests/integration_tests/test_kerberos.py
319+
290320
check-secret:
291321
runs-on: ubuntu-latest
292-
needs: [bare-import-test, tests, pandas-3x-compat-test, sqlalchemy-1x-compat-test]
322+
needs: [bare-import-test, tests, pandas-3x-compat-test, sqlalchemy-1x-compat-test, kerberos-integration-test]
293323
outputs:
294324
has_secrets: ${{ steps.has_secrets.outputs.HAS_SECRETS }}
295325
steps:

‎CONTRIBUTING.md‎

Lines changed: 46 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,9 +56,9 @@ python setup.py develop
5656

5757
### Add /etc/hosts entry
5858

59-
Required for TLS tests.
59+
Required for TLS and Kerberos tests.
6060
The generated certificates assume TLS requests use `server1.clickhouse.test` as the hostname.
61-
See [test_tls.py](tests/integration_tests/test_tls.py) for more details.
61+
See [test_tls.py](tests/integration_tests/test_tls.py) and [test_kerberos.py](tests/integration_tests/test_kerberos.py) for more details.
6262

6363
```bash
6464
sudo -- sh -c "echo 127.0.0.1 server1.clickhouse.test >> /etc/hosts"
@@ -120,6 +120,50 @@ Additionally, the TLS ClickHouse instance should be running (see [docker-compose
120120
CLICKHOUSE_CONNECT_TEST_TLS=1 pytest tests/integration_tests/test_tls.py
121121
```
122122

123+
### Run the Kerberos integration tests
124+
125+
These tests require the `CLICKHOUSE_CONNECT_TEST_KERBEROS` environment variable to be set to `1`; otherwise, they will be skipped.
126+
Unlike the other test instances, the Kerberos KDC and ClickHouse instance (the `kerberos_kdc` and `kerberos_clickhouse` services in
127+
[docker-compose.yml](docker-compose.yml)) are behind a `kerberos` Compose profile rather than started by a plain `docker compose up -d`,
128+
since they also need an extra host-side step (obtaining a real Kerberos ticket) that Docker Compose cannot do for you. This walks
129+
through setting them up from scratch.
130+
131+
Install the system Kerberos client and development packages (needed to build the `gssapi`/`krb5` Python packages):
132+
133+
```bash
134+
# Debian/Ubuntu
135+
sudo apt-get install gcc python3-dev libkrb5-dev krb5-user
136+
137+
# CentOS/RHEL/Fedora
138+
sudo dnf install gcc python3-devel krb5-devel krb5-workstation
139+
140+
# Arch Linux
141+
sudo pacman -S gcc krb5
142+
```
143+
144+
Make sure you've added the `server1.clickhouse.test` `/etc/hosts` entry from
145+
["Add /etc/hosts entry"](#add-etchosts-entry) above.
146+
147+
The rest (starting a KDC and a Kerberos-configured ClickHouse instance, obtaining a ticket, and tearing it all
148+
back down afterward) is handled automatically by a fixture in
149+
[`test_kerberos.py`](tests/integration_tests/test_kerberos.py), via
150+
[`kerberos_manage.py`](tests/integration_tests/kerberos_manage.py), which uses the fixtures vendored under
151+
[`tests/integration_tests/kerberos_conf`](tests/integration_tests/kerberos_conf).
152+
153+
Run from the repo root:
154+
155+
```bash
156+
CLICKHOUSE_CONNECT_TEST_KERBEROS=1 pytest tests/integration_tests/test_kerberos.py
157+
```
158+
159+
To stand up (or tear down) the same environment by hand, outside of pytest -- for example, to poke at it manually
160+
with `curl --negotiate` -- run:
161+
162+
```bash
163+
python -m tests.integration_tests.kerberos_manage setup
164+
python -m tests.integration_tests.kerberos_manage teardown
165+
```
166+
123167
### Running the integration tests with ClickHouse Cloud
124168

125169
If you want to run the tests using your ClickHouse Cloud instance instead of the local ClickHouse instance running in Docker, you will need a few additional environment variables.

‎docker-compose.yml‎

Lines changed: 43 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,4 +33,46 @@ services:
3333
volumes:
3434
- './.docker/clickhouse/single_node_tls/config.xml:/etc/clickhouse-server/config.xml'
3535
- './.docker/clickhouse/single_node_tls/users.xml:/etc/clickhouse-server/users.xml'
36-
- './.docker/clickhouse/single_node_tls/docker_related_config.xml:/etc/clickhouse-server/config.d/docker_related_config.xml'
36+
- './.docker/clickhouse/single_node_tls/docker_related_config.xml:/etc/clickhouse-server/config.d/docker_related_config.xml'
37+
38+
# Both kerberos services are opt-in (profiles: [kerberos]), started only by
39+
# tests/integration_tests/kerberos_manage.py (see test_kerberos.py's kerberos_env fixture),
40+
# since they need an extra host-side step (obtaining a real Kerberos ticket) that Docker Compose
41+
# cannot do for you. See "Run the Kerberos integration tests" in CONTRIBUTING.md.
42+
kerberos_kdc:
43+
build:
44+
context: ./
45+
dockerfile: .docker/kerberos-kdc/Dockerfile
46+
container_name: 'clickhouse-connect-kerberos-kdc'
47+
profiles: ['kerberos']
48+
ports:
49+
- '8088:88/udp'
50+
- '8088:88/tcp'
51+
volumes:
52+
- './tests/integration_tests/kerberos_conf/kerberos_image_config.sh:/config.sh:ro'
53+
- 'kerberos_kdc_keytabs:/tmp/keytab'
54+
- 'kerberos_kdc_data:/var/kerberos/krb5kdc'
55+
56+
kerberos_clickhouse:
57+
image: 'clickhouse/clickhouse-server:${CLICKHOUSE_CONNECT_TEST_CH_VERSION-25.6-alpine}'
58+
container_name: 'clickhouse-connect-kerberos-clickhouse-server'
59+
profiles: ['kerberos']
60+
depends_on:
61+
- kerberos_kdc
62+
environment:
63+
CLICKHOUSE_SKIP_USER_SETUP: 1
64+
ports:
65+
- '8124:8123'
66+
ulimits:
67+
nofile:
68+
soft: 262144
69+
hard: 262144
70+
volumes:
71+
- 'kerberos_kdc_keytabs:/tmp/keytab:ro'
72+
- './tests/integration_tests/kerberos_conf/kerberos_config.xml:/etc/clickhouse-server/config.d/kerberos_config.xml:ro'
73+
- './tests/integration_tests/kerberos_conf/kerberos_users.xml:/etc/clickhouse-server/users.d/kerberos_users.xml:ro'
74+
- './tests/integration_tests/kerberos_conf/kerberos_server_krb5.conf:/etc/krb5.conf:ro'
75+
76+
volumes:
77+
kerberos_kdc_keytabs:
78+
kerberos_kdc_data:

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,4 +65,4 @@ log_cli = true
6565
log_cli_level = "INFO"
6666
env_files = ["test.env"]
6767
asyncio_default_fixture_loop_scope = "session"
68-
addopts = "-n 4"
68+
addopts = "-n 4 --dist=loadgroup"

‎tests/integration_tests/kerberos_conf/kerberos_client_krb5.conf‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
[libdefaults]
2+
default_realm = TEST.CLICKHOUSE.TECH
3+
dns_lookup_realm = false
4+
dns_lookup_kdc = false
5+
rdns = false
6+
7+
[realms]
8+
TEST.CLICKHOUSE.TECH = {
9+
kdc = 127.0.0.1:8088
10+
admin_server = 127.0.0.1:8088
11+
}

‎tests/integration_tests/kerberos_conf/kerberos_config.xml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
<clickhouse>
2+
<kerberos>
3+
<realm>TEST.CLICKHOUSE.TECH</realm>
4+
<keytab>/tmp/keytab/server1.clickhouse.test.keytab</keytab>
5+
</kerberos>
6+
</clickhouse>

‎tests/integration_tests/kerberos_conf/kerberos_image_config.sh‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
#!/bin/bash
2+
3+
4+
set -x # trace
5+
6+
: "${REALM:=TEST.CLICKHOUSE.TECH}"
7+
: "${DOMAIN_REALM:=test.clickhouse.com}"
8+
: "${KERB_MASTER_KEY:=masterkey}"
9+
: "${KERB_ADMIN_USER:=admin}"
10+
: "${KERB_ADMIN_PASS:=admin}"
11+
12+
create_config() {
13+
: "${KDC_ADDRESS:=$(hostname -f)}"
14+
15+
cat>/etc/krb5.conf<<EOF
16+
[logging]
17+
default = FILE:/var/log/kerberos/krb5libs.log
18+
kdc = FILE:/var/log/kerberos/krb5kdc.log
19+
admin_server = FILE:/var/log/kerberos/kadmind.log
20+
21+
[libdefaults]
22+
default_realm = $REALM
23+
dns_lookup_realm = false
24+
dns_lookup_kdc = false
25+
ticket_lifetime = 15s
26+
renew_lifetime = 15s
27+
forwardable = true
28+
default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
29+
default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
30+
permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
31+
32+
[realms]
33+
$REALM = {
34+
kdc = $KDC_ADDRESS
35+
admin_server = $KDC_ADDRESS
36+
}
37+
38+
[domain_realm]
39+
.$DOMAIN_REALM = $REALM
40+
$DOMAIN_REALM = $REALM
41+
EOF
42+
43+
cat>/var/kerberos/krb5kdc/kdc.conf<<EOF
44+
[kdcdefaults]
45+
kdc_ports = 88
46+
kdc_tcp_ports = 88
47+
48+
[realms]
49+
$REALM = {
50+
acl_file = /var/kerberos/krb5kdc/kadm5.acl
51+
dict_file = /usr/share/dict/words
52+
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
53+
master_key_type = aes256-cts-hmac-sha1-96
54+
supported_enctypes = aes256-cts-hmac-sha1-96:normal aes128-cts-hmac-sha1-96:normal
55+
default_principal_flags = +preauth
56+
}
57+
EOF
58+
}
59+
60+
create_db() {
61+
/usr/sbin/kdb5_util -P $KERB_MASTER_KEY -r $REALM create -s
62+
}
63+
64+
start_kdc() {
65+
mkdir -p /var/log/kerberos
66+
67+
/usr/sbin/krb5kdc
68+
/usr/sbin/kadmind
69+
}
70+
71+
restart_kdc() {
72+
pkill krb5kdc
73+
pkill kadmind
74+
start_kdc
75+
}
76+
77+
create_admin_user() {
78+
kadmin.local -q "addprinc -pw $KERB_ADMIN_PASS $KERB_ADMIN_USER/admin"
79+
echo "*/admin@$REALM *" > /var/kerberos/krb5kdc/kadm5.acl
80+
}
81+
82+
create_keytabs() {
83+
rm /tmp/keytab/*.keytab
84+
85+
kadmin.local -q "addprinc -randkey kuser@${REALM}"
86+
kadmin.local -q "ktadd -norandkey -k /tmp/keytab/kuser.keytab kuser@${REALM}"
87+
88+
kadmin.local -q "addprinc -randkey HTTP/server1.clickhouse.test@${REALM}"
89+
kadmin.local -q "ktadd -norandkey -k /tmp/keytab/server1.clickhouse.test.keytab HTTP/server1.clickhouse.test@${REALM}"
90+
91+
chmod g+r /tmp/keytab/kuser.keytab
92+
chmod g+r /tmp/keytab/server1.clickhouse.test.keytab
93+
}
94+
95+
main() {
96+
97+
if [ ! -f /kerberos_initialized ]; then
98+
create_config
99+
create_db
100+
create_admin_user
101+
start_kdc
102+
103+
touch /kerberos_initialized
104+
fi
105+
106+
if [ ! -f /var/kerberos/krb5kdc/principal ]; then
107+
while true; do sleep 1000; done
108+
else
109+
start_kdc
110+
create_keytabs
111+
tail -F /var/log/kerberos/krb5kdc.log
112+
fi
113+
114+
}
115+
116+
[[ "$0" == "${BASH_SOURCE[0]}" ]] && main "$@"

‎tests/integration_tests/kerberos_conf/kerberos_server_krb5.conf‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
[logging]
2+
default = FILE:/var/log/kerberos/krb5libs.log
3+
kdc = FILE:/var/log/kerberos/krb5kdc.log
4+
admin_server = FILE:/var/log/kerberos/kadmind.log
5+
6+
[libdefaults]
7+
default_realm = TEST.CLICKHOUSE.TECH
8+
dns_lookup_realm = false
9+
dns_lookup_kdc = false
10+
ticket_lifetime = 15s
11+
renew_lifetime = 15s
12+
forwardable = true
13+
14+
[realms]
15+
TEST.CLICKHOUSE.TECH = {
16+
kdc = kerberos_kdc
17+
admin_server = kerberos_kdc
18+
}
19+
20+
[domain_realm]
21+
.test.clickhouse.com = TEST.CLICKHOUSE.TECH
22+
test.clickhouse.com = TEST.CLICKHOUSE.TECH

‎tests/integration_tests/kerberos_conf/kerberos_users.xml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
<clickhouse>
2+
<profiles>
3+
<default>
4+
</default>
5+
</profiles>
6+
<users>
7+
<kuser>
8+
<kerberos>
9+
<realm>TEST.CLICKHOUSE.TECH</realm>
10+
</kerberos>
11+
<access_management>1</access_management>
12+
<networks replace="replace">
13+
<ip>::/0</ip>
14+
</networks>
15+
<profile>default</profile>
16+
<quota>default</quota>
17+
</kuser>
18+
</users>
19+
</clickhouse>

0 commit comments

Comments
 (0)