Week 01 2025 routine

[kiwixbot]

Check nodes free space

df -h / && df -h /data

• create a report in issue comment

Nodes system upgrades

apt update && apt upgrade

• run systematically the upgrade on bastion, stats, services, storage, demo, mirrors-qa nodes
• check for and apply important security upgrade on worker nodes asap (imager-worker, ondemand, sisyphus)

(regular workers updates are done separately on a monthly basis for worker nodes to not impact production)

Backups

• Ensure all borg repositories are being updated

k8s cluster

• Check Pod errors or in CrashLoopBackoff

k get pods -A -o wide|grep -E 'Error|Crash'

• Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• Upgrade k8s if applicable/possible

Stats

matomo - stats.kiwix.org

• Ensure download.kiwix.org stats are being recorded
• Check whether matomo should be upgraded

Grafana

• Alert list is normal
• Zimfarm dashboard is normal
• Jobs durations dashboard is normal
• There is no abnormal behaviors on cluster resources consumption
• All mdadm RAID arrays are OK (check all instances in the top left combobox)

Projects

• UptimeRobot has no alert
• zimit backlog is reasonable
• Cloud Code Signing Certificate usage is OK (look for Unused Signings under END ENTITY CERTIFICATES): we have 1,200 per year (August to August)
• Analyze zimit failed tasks and document bad domains in our WIP blacklist
• PRs awaiting your review

Security

• Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	16G	20G	44%	-
stats	/	233G	131G	90G	60%	+20G
services	/	456G	221G	212G	52%	+1G
storage	/	33T	9.7T	22T	32%	+0.2T
imager-worker	/	1.9T	394G	1.4T	23%	don't care
sisyphus	/	233G	12G	209G	-	don't care
ondemand	/	25G	14G	11G	56%	+4G
ondemand	/data	216G	201M	205G	1%	don't care
demo	/	40G	12G	26G	32%	+1G
demo	/data	1.8T	920G	743G	56%	don't care

misc

• grafana-k8s-monitoring-opencost restarted once this morning (7 Jan 2024), never seen before
• k8s still needs to be upgraded
• Updated Matomo to 5.2.1 (minor, no DB changes)
• Unused Signings: 852 (was 898 three weeks ago)

zimit

• (14) x Main URL returned an unprocessable HTTP code: 403
• (5) x Main URL returned an unprocessable HTTP code: 404
• (1) x Main URL returned an unprocessable HTTP code: 405
• (1) x Main URL returned an unprocessable HTTP code: 406
• (1) x Main URL returned an unprocessable HTTP code: 451
• (1) x Main URL returned an unprocessable HTTP code: 503
• (1) x Main URL returned an unprocessable HTTP code: 522
• (2) x net::ERR_HTTP_RESPONSE_CODE_FAILURE (Seed Page Load Failed)
• (7) x net::ERR_HTTP2_PROTOCOL_ERROR (Seed Page Load Failed)
• (1) x net::ERR_EMPTY_RESPONSE (Seed Page Load Failed)
• (1) x net::ERR_NAME_NOT_RESOLVED (Seed Page Load Failed)
• (2) x net::ERR_TOO_MANY_REDIRECTS (Seed Page Load Failed)
• (1) x net::ERR_CONNECTION_REFUSED (Seed Page Load Failed)
• (8) x Navigation timeout of 90000 ms exceeded (Seed Page Load Failed)
• (3) x Validate that include and exclude parameters are valid regular expression openzim/zimit-frontend#108 (NEW: someone tried many times to use * as include rule, instead of .*)
• (2) x LookupError: unknown encoding: unicode openzim/warc2zim#331 (-8, unicode)
• (1) x memory consumption (exit code 137) openzim/warc2zim#383
• (1) x Memory issue when WARC item does not fit into memory openzim/warc2zim#429 (NEW)
• many wp1 issues (WP1: Fail to create zim openzim/mwoffliner#2123)