Skip to content

Commit fe87cef

Browse files
maxkadelclaudeCopilot
authored
Serve static files via nginx (#685)
* Increase timeout for staging fcrepo for cleaner startup * Fix nginx to serve /system static assets directly instead of proxying to Puma The Hyrax chart's nginx sidecar exists to keep Puma from serving static files out of its small thread pool, but staging had no nginx pod at all (nginx.enabled: false), and production's had no extraVolumeMounts, so /system/* branding uploads passed straight through to Puma either way. Enables nginx and mounts the existing uploads PVC's public-system subPath, matching production's config plus the actual fix. * Bump staging hyrax web pod CPU limit/request, leave memory unchanged Per calculate_limits_and_requests' r2-friends right-sizing report (2026-07-28): actual p95/p99 CPU usage (597-748m) already close to the old 1000m limit, and pals-stress-tests' before/after runs independently confirmed real CPU throttling (~29-38% of periods) at that limit, both before and after the nginx static-asset fix. Memory left alone -- p99 usage was ~869-895Mi against the existing 4Gi, no need there, and this cluster's nodes are memory- not CPU-constrained, so growing memory here would only make node bin-packing worse for no benefit. Deliberately not touching RAILS_MAX_THREADS or Puma workers in this change -- keeping this an isolated, single-variable test. HPA/autoscaling is a separate, later effort with its own stress-test validation. * Bake compiled assets/pdf.js/uv into a custom nginx image, matching HykuUp Real traffic showed /assets/* (fonts/css/js, far higher volume than the one tenant logo the /system fix covers) was still 100% proxied to Puma -- 18s+ responses seen live -- because public/assets is baked into the *web* image at build time, not on any EFS volume the stock bitnami-nginx image could see. Fixed the same way HykuUp already does it: a dedicated hyku-nginx Dockerfile stage copies these three paths from the exact same commit as the web image, and CI now builds an "nginx" component alongside web/worker/solr. * Fix nginx websocket proxying for ActionCable realtime notifications nginx's @rails proxy location stripped the Upgrade/Connection headers before forwarding to Puma, so every websocket handshake for Hyrax's realtime-notifications feature (/notifications/endpoint, mounted via ActionCable, enabled by default) fell back to a plain HTTP request that Rails' router 404s. Confirmed live: a raw handshake attempt through staging's nginx returned a 404 exactly as described. Since the client's ActionCable JS retries indefinitely with backoff, every open tab from a logged-in user repeatedly re-hit this failing endpoint, consuming Puma threads. Added the standard map $http_upgrade $connection_upgrade pattern plus proxy_http_version 1.1 and the Upgrade/Connection proxy headers on the @rails location, in both friends (staging) and production nginx serverBlocks -- only activates when a client actually sends an Upgrade header, so it's a no-op for regular requests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Reduce wordiness * Include uv and pdf.js in static files served by nginx * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Fix production image to be the built one --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
1 parent 0d1c3a4 commit fe87cef

4 files changed

Lines changed: 167 additions & 10 deletions

File tree

‎.github/workflows/build-test-lint.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ jobs:
2727
uses: notch8/actions/.github/workflows/build.yaml@v1.0.10
2828
secrets: inherit
2929
with:
30-
components: '["web","worker","solr"]'
30+
components: '["web","worker","solr","nginx"]'
3131

3232
test:
3333
needs: build

‎Dockerfile‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,16 @@ CMD ./bin/web
2828
FROM hyku-web AS hyku-worker
2929
CMD ./bin/worker
3030

31+
FROM registry.gitlab.com/notch8/scripts/bitnami-nginx:1.21.5-debian-10-r4 AS hyku-nginx
32+
# Copy assets & other public files into nginx so it can serve them as static files
33+
COPY --from=hyku-web /app/samvera/hyrax-webapp/public/assets /app/samvera/hyrax-webapp/public/assets
34+
COPY --from=hyku-web /app/samvera/hyrax-webapp/public/pdf.js /app/samvera/hyrax-webapp/public/pdf.js
35+
COPY --from=hyku-web /app/samvera/hyrax-webapp/public/uv /app/samvera/hyrax-webapp/public/uv
36+
3137
# Use a Solr version with patched Log4j to address CVE-2021-44228
3238
FROM solr:8.11.2 AS hyku-solr
3339
ENV SOLR_USER="solr" \
3440
SOLR_GROUP="solr"
3541
USER root
3642
COPY --chown=solr:solr solr/security.json /var/solr/data/security.json
37-
USER $SOLR_USER
43+
USER $SOLR_USER

‎ops/friends-deploy.tmpl.yaml‎

Lines changed: 144 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,10 @@ replicaCount: 2
33
resources:
44
limits:
55
memory: "4Gi"
6-
cpu: "1000m"
6+
cpu: "1250m"
77
requests:
88
memory: "2Gi"
9-
cpu: "300m"
9+
cpu: "900m"
1010

1111
livenessProbe:
1212
enabled: false
@@ -435,6 +435,12 @@ fcrepo:
435435
database: fcrepo
436436
containerPorts:
437437
postgresql: 5432
438+
livenessProbe:
439+
initialDelaySeconds: 90
440+
failureThreshold: 6
441+
readinessProbe:
442+
initialDelaySeconds: 90
443+
failureThreshold: 6
438444
s3:
439445
enabled: true
440446
bucket: besties-fcrepo-clone
@@ -447,6 +453,8 @@ global:
447453
username: fcrepo
448454
password: $DB_PASSWORD
449455
postgresPassword: $DB_PASSWORD
456+
# Required for the nginx serverBlock's `upstream rails_app` to resolve
457+
hyraxHostName: palni-palci-knapsack-friends-hyrax
450458
postgresql:
451459
enabled: false
452460
redis:
@@ -496,6 +504,138 @@ externalSolrCollection: "palni-palci-knapsack-friends"
496504
externalSolrPassword: $SOLR_ADMIN_PASSWORD
497505

498506
nginx:
499-
enabled: false
507+
enabled: true
500508
service:
501-
port: 80
509+
type: ClusterIP
510+
image:
511+
registry: ghcr.io
512+
repository: "$REPO_LOWER/nginx"
513+
tag: "$TAG"
514+
extraVolumes:
515+
- name: uploads
516+
persistentVolumeClaim:
517+
claimName: "{{ .Values.global.hyraxHostName }}-uploads"
518+
extraVolumeMounts:
519+
- name: uploads
520+
mountPath: /app/samvera/hyrax-webapp/public/system
521+
subPath: public-system
522+
serverBlock: |-
523+
upstream rails_app {
524+
server {{ .Values.global.hyraxHostName }};
525+
}
526+
527+
map ${DOLLAR}status ${DOLLAR}loggable {
528+
~^444 0;
529+
default 1;
530+
}
531+
532+
# Lets the ActionCable websocket handshake for Hyrax's realtime
533+
# notifications (/notifications/endpoint) actually upgrade instead of
534+
# silently falling through to a plain HTTP request that Rails 404s --
535+
# only kicks in when the client sends an Upgrade header, so it's a
536+
# no-op for every other request.
537+
map ${DOLLAR}http_upgrade ${DOLLAR}connection_upgrade {
538+
default upgrade;
539+
'' '';
540+
}
541+
542+
log_format loki 'host=${DOLLAR}host ip=${DOLLAR}http_x_forwarded_for remote_user=${DOLLAR}remote_user [${DOLLAR}time_local] '
543+
'request="${DOLLAR}request" status=${DOLLAR}status bytes=${DOLLAR}body_bytes_sent '
544+
'referer="${DOLLAR}http_referer" agent="${DOLLAR}http_user_agent" request_time=${DOLLAR}request_time upstream_response_time=${DOLLAR}upstream_response_time upstream_response_length=${DOLLAR}upstream_response_length';
545+
546+
error_log /opt/bitnami/nginx/logs/error.log warn;
547+
#tcp_nopush on;
548+
549+
# Cloudflare ips see for refresh
550+
# https://support.cloudflare.com/hc/en-us/articles/200170786-Restoring-original-visitor-IPs-logging-visitor-IP-addresses
551+
# update list https://www.cloudflare.com/ips/
552+
set_real_ip_from 103.21.244.0/22;
553+
set_real_ip_from 103.22.200.0/22;
554+
set_real_ip_from 103.31.4.0/22;
555+
set_real_ip_from 104.16.0.0/13;
556+
set_real_ip_from 104.24.0.0/14;
557+
set_real_ip_from 108.162.192.0/18;
558+
set_real_ip_from 131.0.72.0/22;
559+
set_real_ip_from 141.101.64.0/18;
560+
set_real_ip_from 162.158.0.0/15;
561+
set_real_ip_from 172.64.0.0/13;
562+
set_real_ip_from 173.245.48.0/20;
563+
set_real_ip_from 188.114.96.0/20;
564+
set_real_ip_from 190.93.240.0/20;
565+
set_real_ip_from 197.234.240.0/22;
566+
set_real_ip_from 198.41.128.0/17;
567+
set_real_ip_from 2400:cb00::/32;
568+
set_real_ip_from 2606:4700::/32;
569+
set_real_ip_from 2803:f800::/32;
570+
set_real_ip_from 2405:b500::/32;
571+
set_real_ip_from 2405:8100::/32;
572+
set_real_ip_from 2a06:98c0::/29;
573+
set_real_ip_from 2c0f:f248::/32;
574+
575+
real_ip_header X-Forwarded-For;
576+
real_ip_recursive on;
577+
include /opt/bitnami/nginx/conf/conf.d/*.conf;
578+
server {
579+
listen 8080;
580+
server_name _;
581+
root /app/samvera/hyrax-webapp/public;
582+
index index.html;
583+
584+
client_body_in_file_only clean;
585+
client_body_buffer_size 32K;
586+
client_max_body_size 0;
587+
access_log /opt/bitnami/nginx/logs/access.log loki;
588+
# if=${DOLLAR}loggable;
589+
590+
sendfile on;
591+
send_timeout 300s;
592+
593+
include /opt/bitnami/nginx/conf/bots.d/ddos.conf;
594+
include /opt/bitnami/nginx/conf/bots.d/blockbots.conf;
595+
596+
location ~ (\.php|\.aspx|\.asp) {
597+
return 404;
598+
}
599+
600+
# deny requests for files that should never be accessed
601+
location ~ /\. {
602+
deny all;
603+
}
604+
605+
location ~* ^.+\.(rb|log)${DOLLAR} {
606+
deny all;
607+
}
608+
609+
location ~ ^/(assets|packs|fonts|images|javascripts|stylesheets|swfs|system|uv|pdf\.js)/ {
610+
try_files ${DOLLAR}uri @rails;
611+
612+
# access_log off;
613+
gzip_static on; # to serve pre-gzipped version
614+
615+
expires max;
616+
add_header Cache-Control public;
617+
618+
# Some browsers still send conditional-GET requests if there's a
619+
# Last-Modified header or an ETag header even if they haven't
620+
# reached the expiry date sent in the Expires header.
621+
add_header Last-Modified "";
622+
add_header ETag "";
623+
break;
624+
}
625+
626+
# send non-static file requests to the app server
627+
location / {
628+
try_files ${DOLLAR}uri @rails;
629+
}
630+
631+
location @rails {
632+
proxy_set_header X-Real-IP ${DOLLAR}remote_addr;
633+
proxy_set_header X-Forwarded-For ${DOLLAR}proxy_add_x_forwarded_for;
634+
proxy_set_header Host ${DOLLAR}http_host;
635+
proxy_http_version 1.1;
636+
proxy_set_header Upgrade ${DOLLAR}http_upgrade;
637+
proxy_set_header Connection ${DOLLAR}connection_upgrade;
638+
proxy_redirect off;
639+
proxy_pass http://rails_app;
640+
}
641+
}

‎ops/production-deploy.tmpl.yaml‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -449,6 +449,7 @@ fcrepo:
449449
access_key: $AWS_ACCESS_KEY_ID
450450
secret_key: $AWS_SECRET_ACCESS_KEY
451451
global:
452+
# Used by the nginx chart to calculate upstream rails_app
452453
hyraxHostName: palni-palci-knapsack-production
453454
# These global postgresql values are used by the fcrepo chart
454455
postgresql:
@@ -491,9 +492,9 @@ nginx:
491492
service:
492493
type: ClusterIP
493494
image:
494-
registry: registry.gitlab.com
495-
repository: notch8/scripts/bitnami-nginx
496-
tag: 1.21.5-debian-10-r4
495+
registry: ghcr.io
496+
repository: "$REPO_LOWER/nginx"
497+
tag: "$TAG"
497498
serverBlock: |-
498499
upstream rails_app {
499500
server {{ .Values.global.hyraxHostName }};
@@ -504,6 +505,13 @@ nginx:
504505
default 1;
505506
}
506507
508+
# For ActionCable websocket handshake for Hyrax's realtime
509+
# notifications (/notifications/endpoint)
510+
map ${DOLLAR}http_upgrade ${DOLLAR}connection_upgrade {
511+
default upgrade;
512+
'' '';
513+
}
514+
507515
log_format loki 'host=${DOLLAR}host ip=${DOLLAR}http_x_forwarded_for remote_user=${DOLLAR}remote_user [${DOLLAR}time_local] '
508516
'request="${DOLLAR}request" status=${DOLLAR}status bytes=${DOLLAR}body_bytes_sent '
509517
'referer="${DOLLAR}http_referer" agent="${DOLLAR}http_user_agent" request_time=${DOLLAR}request_time upstream_response_time=${DOLLAR}upstream_response_time upstream_response_length=${DOLLAR}upstream_response_length';
@@ -572,7 +580,7 @@ nginx:
572580
}
573581
574582
# serve static (compiled) assets directly if they exist (for rails production)
575-
location ~ ^/(assets|packs|fonts|images|javascripts|stylesheets|swfs|system)/ {
583+
location ~ ^/(assets|packs|fonts|images|javascripts|stylesheets|swfs|system|uv|pdf\.js)/ {
576584
try_files ${DOLLAR}uri @rails;
577585
578586
# access_log off;
@@ -598,6 +606,9 @@ nginx:
598606
proxy_set_header X-Real-IP ${DOLLAR}remote_addr;
599607
proxy_set_header X-Forwarded-For ${DOLLAR}proxy_add_x_forwarded_for;
600608
proxy_set_header Host ${DOLLAR}http_host;
609+
proxy_http_version 1.1;
610+
proxy_set_header Upgrade ${DOLLAR}http_upgrade;
611+
proxy_set_header Connection ${DOLLAR}connection_upgrade;
601612
proxy_redirect off;
602613
proxy_pass http://rails_app;
603614
}

0 commit comments

Comments
 (0)