Skip to content

Commit

Permalink
Strict patch
Browse files Browse the repository at this point in the history
  • Loading branch information
ktsakalozos committed Mar 17, 2022
1 parent 11da010 commit 9e378a5
Show file tree
Hide file tree
Showing 77 changed files with 1,048 additions and 268 deletions.
101 changes: 88 additions & 13 deletions .github/workflows/build-snap.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,17 @@
name: Build MicroK8s snap on PR and push to master

on:
push:
branches:
- master
pull_request:
branches:
- master
- push
- pull_request

### While we work on the strict feature we want the tests to run even if we do put PRs against the master.
### When this work get merged into master the following should be commented in.
# push:
# branches:
# - master
# pull_request:
# branches:
# - master

jobs:
build:
Expand Down Expand Up @@ -43,24 +48,94 @@ jobs:
- name: Running upgrade path test
run: |
set -x
sudo -E UPGRADE_MICROK8S_FROM=latest/edge UPGRADE_MICROK8S_TO=`pwd`/`ls microk8s*.snap` pytest -s ./tests/test-upgrade-path.py
sudo snap remove microk8s --purge
- name: Running addons tests
# Remove the snapd refresh as soon as v2.52 lands
sudo snap refresh snapd --channel=latest/edge
- name: Check branches
run: |
set -x
(cd tests; pytest -s verify-branches.py)
- name: Running addons tests in strict mode
run: |
set -x
sudo snap install *.snap --classic --dangerous
sudo snap install microk8s.snap --dangerous
for i in account-control docker-privileged kubernetes-support k8s-journald k8s-kubelet \
k8s-kubeproxy dot-kube network network-bind network-control network-observe \
firewall-control process-control kernel-module-observe mount-observe \
hardware-observe system-observe home opengl home-read-all \
login-session-observe log-observe dot-config-helm
do
sudo snap connect microk8s:$i
done
./tests/smoke-test.sh
export UNDER_TIME_PRESSURE="True"
export SKIP_OPENEBS="True"
export SKIP_PROMETHEUS="False"
(cd tests; pytest -s verify-branches.py)
sudo -E bash -c "cd /var/snap/microk8s/common/addons/core/tests; pytest -s -ra test-addons.py"
sudo microk8s enable community
sudo -E bash -c "cd /var/snap/microk8s/common/addons/community/tests; pytest -s -ra test-addons.py"
grep -Po "Report tarball is at \K.+" |
sudo xargs -I {} mv {} inspection-report-strict-${{ strategy.job-index }}.tar.gz
sudo snap remove microk8s --purge
- name: Running upgrade tests
sudo rm -rf $HOME/.kube
sudo rm -rf $HOME/.config/helm
sudo dmesg | grep 'apparmor="DENIED"' > ./denials-${{ strategy.job-index }}.log
- name: Upload strict inspect tarball
uses: actions/upload-artifact@v2
with:
name: inspection-report-strict-actions
path: ./inspection-report-strict-${{ strategy.job-index }}.tar.gz
- name: Upload AppArmor denials
uses: actions/upload-artifact@v2
with:
name: apparmor-denials
path: ./denials-${{ strategy.job-index }}.log
- name: Running addons tests in devmode
run: |
set -x
sudo snap install *.snap --classic --dangerous
################ Until devmode of docker-support is fixed we skip this part of the tests #######
exit 0
sudo snap install microk8s.snap --devmode --dangerous
for i in account-control docker-privileged kubernetes-support k8s-journald k8s-kubelet \
k8s-kubeproxy dot-kube network network-bind network-control network-observe \
firewall-control process-control kernel-module-observe mount-observe \
hardware-observe system-observe home opengl home-read-all \
login-session-observe log-observe dot-config-helm
do
sudo snap connect microk8s:$i
done
./tests/smoke-test.sh
export UNDER_TIME_PRESSURE="True"
sudo -E bash -c "cd /var/snap/microk8s/common/addons/core/ ; UPGRADE_MICROK8S_FROM=latest/edge UPGRADE_MICROK8S_TO=`pwd`/`ls microk8s*.snap` pytest -s ./tests/test-upgrade.py"
export SKIP_OPENEBS="False"
export SKIP_PROMETHEUS="False"
(cd tests; sudo -E pytest -s -ra test-addons.py)
sudo microk8s inspect |
grep -Po "Report tarball is at \K.+" |
sudo xargs -I {} mv {} inspection-report-devmode-${{ strategy.job-index }}.tar.gz
sudo snap remove microk8s --purge
- name: Upload devmode inspect tarball
uses: actions/upload-artifact@v2
with:
name: inspection-report-devmode-actions
path: ./inspection-report-devmode-${{ strategy.job-index }}.tar.gz
- name: Generate AppArmor on failure
run: sudo dmesg | grep 'apparmor="DENIED"' > ./denials-${{ strategy.job-index }}.log
if: failure()
- name: Upload AppArmor denials failure
uses: actions/upload-artifact@v2
with:
name: apparmor-denials
path: ./denials-${{ strategy.job-index }}.log
if: failure()
- name: Generate inspect tarball
run: >
sudo microk8s inspect |
grep -Po "Report tarball is at \K.+" |
sudo xargs -I {} mv {} inspection-report-fail-${{ strategy.job-index }}.tar.gz
if: failure()
- name: Upload inspect tarball
uses: actions/upload-artifact@v2
with:
name: inspection-report-actions
path: ./inspection-report-fail-${{ strategy.job-index }}.tar.gz
if: failure()
24 changes: 12 additions & 12 deletions build-scripts/patches/0000-Kubelite-integration.patch
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
From 6583648f325f98bf5aed35b3a90ef9132bead309 Mon Sep 17 00:00:00 2001
From 3198c607c243bd4363eadc7d9223e3a743f85215 Mon Sep 17 00:00:00 2001
From: Konstantinos Tsakalozos <[email protected]>
Date: Wed, 3 Mar 2021 18:19:37 +0200
Subject: [PATCH] Kubelite integration
Expand All @@ -19,10 +19,10 @@ Subject: [PATCH] Kubelite integration
create mode 100644 cmd/kubelite/kubelite.go

diff --git a/cmd/kube-apiserver/app/server.go b/cmd/kube-apiserver/app/server.go
index 8c2d2a94d99..2351c86fa78 100644
index 411567a4175..a871cc9c975 100644
--- a/cmd/kube-apiserver/app/server.go
+++ b/cmd/kube-apiserver/app/server.go
@@ -91,7 +91,7 @@ func checkNonZeroInsecurePort(fs *pflag.FlagSet) error {
@@ -96,7 +96,7 @@ func checkNonZeroInsecurePort(fs *pflag.FlagSet) error {
}

// NewAPIServerCommand creates a *cobra.Command object with default parameters
Expand All @@ -31,7 +31,7 @@ index 8c2d2a94d99..2351c86fa78 100644
s := options.NewServerRunOptions()
cmd := &cobra.Command{
Use: "kube-apiserver",
@@ -127,8 +127,11 @@ cluster's shared state through which all other components interact.`,
@@ -138,8 +138,11 @@ cluster's shared state through which all other components interact.`,
if errs := completedOptions.Validate(); len(errs) != 0 {
return utilerrors.NewAggregate(errs)
}
Expand All @@ -46,10 +46,10 @@ index 8c2d2a94d99..2351c86fa78 100644
Args: func(cmd *cobra.Command, args []string) error {
for _, arg := range args {
diff --git a/cmd/kube-scheduler/app/server.go b/cmd/kube-scheduler/app/server.go
index dc6cc055415..d7674db4b01 100644
index b65f4e74d09..0885de22da1 100644
--- a/cmd/kube-scheduler/app/server.go
+++ b/cmd/kube-scheduler/app/server.go
@@ -117,7 +117,11 @@ func runCommand(cmd *cobra.Command, opts *options.Options, registryOptions ...Op
@@ -120,7 +120,11 @@ func runCommand(cmd *cobra.Command, opts *options.Options, registryOptions ...Op
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go func() {
Expand All @@ -63,7 +63,7 @@ index dc6cc055415..d7674db4b01 100644
cancel()
}()
diff --git a/cmd/kubelet/app/server.go b/cmd/kubelet/app/server.go
index 59662e425bb..c3e4408685a 100644
index 43dfd8d4f45..015a3df23f3 100644
--- a/cmd/kubelet/app/server.go
+++ b/cmd/kubelet/app/server.go
@@ -112,7 +112,7 @@ const (
Expand All @@ -75,7 +75,7 @@ index 59662e425bb..c3e4408685a 100644
cleanFlagSet := pflag.NewFlagSet(componentKubelet, pflag.ContinueOnError)
cleanFlagSet.SetNormalizeFunc(cliflag.WordSepNormalizeFunc)
kubeletFlags := options.NewKubeletFlags()
@@ -279,7 +279,12 @@ HTTP server: The kubelet can also listen for HTTP and respond to a simple API
@@ -287,7 +287,12 @@ HTTP server: The kubelet can also listen for HTTP and respond to a simple API
klog.ErrorS(err, "kubelet running with insufficient permissions")
}
// set up signal context here in order to be reused by kubelet and docker shim
Expand All @@ -89,8 +89,8 @@ index 59662e425bb..c3e4408685a 100644

// make the kubelet's config safe for logging
config := kubeletServer.KubeletConfiguration.DeepCopy()
@@ -290,7 +295,7 @@ HTTP server: The kubelet can also listen for HTTP and respond to a simple API
klog.V(5).InfoS("KubeletConfiguration", "configuration", kubeletServer.KubeletConfiguration)
@@ -298,7 +303,7 @@ HTTP server: The kubelet can also listen for HTTP and respond to a simple API
klog.V(5).InfoS("KubeletConfiguration", "configuration", config)

// run the kubelet
- if err := Run(ctx, kubeletServer, kubeletDeps, utilfeature.DefaultFeatureGate); err != nil {
Expand Down Expand Up @@ -394,10 +394,10 @@ index 00000000000..667b24f68e6
+ println("Stopping kubelite")
+}
diff --git a/pkg/volume/csi/csi_plugin.go b/pkg/volume/csi/csi_plugin.go
index e48b3d6deb7..a64d9d045e0 100644
index 0ae6d084f0e..f27f9e1b812 100644
--- a/pkg/volume/csi/csi_plugin.go
+++ b/pkg/volume/csi/csi_plugin.go
@@ -237,20 +237,24 @@ func (p *csiPlugin) Init(host volume.VolumeHost) error {
@@ -243,20 +243,24 @@ func (p *csiPlugin) Init(host volume.VolumeHost) error {
}

// Initializing the label management channels
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
From 08607d16c6f9ef393e18e0f62fcd967e91c5f7c9 Mon Sep 17 00:00:00 2001
From: Alberto Mardegan <[email protected]>
Date: Wed, 16 Jun 2021 15:04:16 +0300
Subject: [PATCH 1/3] apparmor: change profile immediately, not on exec

---
libcontainer/apparmor/apparmor_linux.go | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/libcontainer/apparmor/apparmor_linux.go b/libcontainer/apparmor/apparmor_linux.go
index 5da14fb3..93ede183 100644
--- a/libcontainer/apparmor/apparmor_linux.go
+++ b/libcontainer/apparmor/apparmor_linux.go
@@ -49,9 +49,9 @@ func setProcAttr(attr, value string) error {
return err
}

-// changeOnExec reimplements aa_change_onexec from libapparmor in Go
-func changeOnExec(name string) error {
- if err := setProcAttr("exec", "exec "+name); err != nil {
+// changeProfile reimplements aa_change_profile from libapparmor in Go
+func changeProfile(name string) error {
+ if err := setProcAttr("current", "changeprofile "+name); err != nil {
return fmt.Errorf("apparmor failed to apply profile: %s", err)
}
return nil
@@ -64,5 +64,5 @@ func ApplyProfile(name string) error {
return nil
}

- return changeOnExec(name)
+ return changeProfile(name)
}
--
2.25.1

Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
From 66fd3c5129599834de8262ee90a1ab2bf6b68ff0 Mon Sep 17 00:00:00 2001
From: Alberto Mardegan <[email protected]>
Date: Wed, 16 Jun 2021 15:04:40 +0300
Subject: [PATCH 2/3] setns_init_linux: set the NNP flag after changing the
apparmor profile

With the current version of the AppArmor kernel module, it's not
possible to switch the AppArmor profile if the NoNewPrivileges flag is
set. So, we invert the order of the two operations.
---
libcontainer/setns_init_linux.go | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/libcontainer/setns_init_linux.go b/libcontainer/setns_init_linux.go
index 97987f1d..eec427a0 100644
--- a/libcontainer/setns_init_linux.go
+++ b/libcontainer/setns_init_linux.go
@@ -57,11 +57,6 @@ func (l *linuxSetnsInit) Init() error {
return err
}
}
- if l.config.NoNewPrivileges {
- if err := unix.Prctl(unix.PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); err != nil {
- return err
- }
- }
if err := selinux.SetExecLabel(l.config.ProcessLabel); err != nil {
return err
}
@@ -80,6 +75,11 @@ func (l *linuxSetnsInit) Init() error {
if err := apparmor.ApplyProfile(l.config.AppArmorProfile); err != nil {
return err
}
+ if l.config.NoNewPrivileges {
+ if err := unix.Prctl(unix.PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); err != nil {
+ return err
+ }
+ }
// Set seccomp as close to execve as possible, so as few syscalls take
// place afterward (reducing the amount of syscalls that users need to
// enable in their seccomp profiles).
--
2.25.1

Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
From 728d989c7643a87ca9d57e3135e35c7af833bae0 Mon Sep 17 00:00:00 2001
From: Alberto Mardegan <[email protected]>
Date: Thu, 17 Jun 2021 14:31:35 +0300
Subject: [PATCH 3/3] standard_init_linux: change AppArmor profile as late as
possible

---
libcontainer/standard_init_linux.go | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/libcontainer/standard_init_linux.go b/libcontainer/standard_init_linux.go
index d77022ad..6f43da5f 100644
--- a/libcontainer/standard_init_linux.go
+++ b/libcontainer/standard_init_linux.go
@@ -114,10 +114,6 @@ func (l *linuxStandardInit) Init() error {
return errors.Wrap(err, "sethostname")
}
}
- if err := apparmor.ApplyProfile(l.config.AppArmorProfile); err != nil {
- return errors.Wrap(err, "apply apparmor profile")
- }
-
for key, value := range l.config.Config.Sysctl {
if err := writeSystemProperty(key, value); err != nil {
return errors.Wrapf(err, "write sysctl key %s", key)
@@ -137,17 +133,21 @@ func (l *linuxStandardInit) Init() error {
if err != nil {
return errors.Wrap(err, "get pdeath signal")
}
- if l.config.NoNewPrivileges {
- if err := unix.Prctl(unix.PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); err != nil {
- return errors.Wrap(err, "set nonewprivileges")
- }
- }
// Tell our parent that we're ready to Execv. This must be done before the
// Seccomp rules have been applied, because we need to be able to read and
// write to a socket.
if err := syncParentReady(l.pipe); err != nil {
return errors.Wrap(err, "sync ready")
}
+ if err := apparmor.ApplyProfile(l.config.AppArmorProfile); err != nil {
+ return errors.Wrap(err, "apply apparmor profile")
+ }
+ if l.config.NoNewPrivileges {
+ if err := unix.Prctl(unix.PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); err != nil {
+ return errors.Wrap(err, "set nonewprivileges")
+ }
+ }
+
if err := selinux.SetExecLabel(l.config.ProcessLabel); err != nil {
return errors.Wrap(err, "set process label")
}
--
2.25.1

4 changes: 2 additions & 2 deletions build-scripts/set-env-variables.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ export K8S_DQLITE_TAG="${K8S_DQLITE_TAG:-v1.0.4}"
export KUBE_SNAP_ROOT="$(readlink -f .)"

export ADDONS_REPOS="
core,${CORE_ADDONS_REPO:-https://github.com/canonical/microk8s-core-addons},${CORE_ADDONS_REPO_BRANCH:-main}
community,${COMMUNITY_ADDONS_REPO:-https://github.com/canonical/microk8s-addons},${COMMUNITY_ADDONS_REPO_BRANCH:-main}
core,${CORE_ADDONS_REPO:-https://github.com/canonical/microk8s-core-addons},${CORE_ADDONS_REPO_BRANCH:-strict}
community,${COMMUNITY_ADDONS_REPO:-https://github.com/canonical/microk8s-addons},${COMMUNITY_ADDONS_REPO_BRANCH:-MK-390/split-core-addons}
"
export ADDONS_REPOS_ENABLED="core"

Expand Down
10 changes: 9 additions & 1 deletion docs/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,9 +83,17 @@ lxc file pull test-build/root/microk8s/microk8s_v1.9.6_amd64.snap .
After copying it, you can install it with:

```shell
snap install microk8s_*_amd64.snap --classic --dangerous
sudo snap install microk8s_latest_amd64.snap --dangerous
```

Finally, you need to connect the interfaces:

```shell
for i in account-control docker-privileged kubernetes-support k8s-journald k8s-kubelet k8s-kubeproxy dot-kube network network-bind network-control network-observe firewall-control process-control kernel-module-observe mount-observe hardware-observe system-observe home opengl dot-config-helm home-read-all log-observe login-session-observe; do sudo snap connect microk8s:$i; done

```


## Assembling the Calico CNI manifest

The calico CNI manifest can be found under `upgrade-scripts/000-switch-to-calico/resources/calico.yaml`.
Expand Down
Loading

0 comments on commit 9e378a5

Please sign in to comment.