AAW dev: system nodepools track resource usage

[Jose-Matsuda]

EPIC
Follow up to changing the workload sizes

We want to observe if there are any workloads being booted off / acting weirdly. This is to validate the work done in #1992

[jacek-dudek]

Here is a table of resource usage metrics that compares the discrepancies between actual usage and resource requests before request adjustments were made and after requests were adjusted. All workloads whose requests were adjusted track much better (in some cases by a factor of 10 or more) their new requests.
filtered-resource-utilization-on-aaw-dev-system-nodes.ods

[Jose-Matsuda]

Reviewed the pods in the .ods file and LGTM;
The metrics on grafana are fine(memory and CPU are well provisioned).
The pods on k9s have 0 restarts which is good meaning it's not getting resource starved and crashloopbackoffing.