Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Relic Monitoring Setup #4423

Merged
merged 1 commit into from
Feb 12, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions k8s/new-relic/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
1. Run this command on your host to install Kubernetes integration.

KSM_IMAGE_VERSION="v2.13.0" && helm repo add newrelic https://helm-charts.newrelic.com && helm repo update && kubectl create namespace newrelic-monitoring ; helm upgrade --install newrelic-bundle newrelic/nri-bundle --set global.licenseKey=<<LICENSE KEY>> --set global.cluster=staging --namespace=newrelic-monitoring --set newrelic-infrastructure.privileged=true --set global.lowDataMode=true --set kube-state-metrics.image.tag=${KSM_IMAGE_VERSION} --set kube-state-metrics.enabled=true --set kubeEvents.enabled=true --set newrelic-prometheus-agent.enabled=true --set newrelic-prometheus-agent.lowDataMode=true --set newrelic-prometheus-agent.config.kubernetes.integrations_filter.enabled=false --set k8s-agents-operator.enabled=true --set logging.enabled=true --set newrelic-logging.lowDataMode=true

2. Update the values file with any required configurations.
3. Upgrade the newrelic-bundle helm release using the values file.

helm repo add newrelic https://helm-charts.newrelic.com && helm repo update; helm upgrade --install newrelic-bundle newrelic/nri-bundle -n newrelic-monitoring --values values.yaml

4. Run this command to enable APM auto-instrumentation using the instrumentation file.

kubectl apply -f ./instrumentation.yaml -n newrelic-monitoring
33 changes: 33 additions & 0 deletions k8s/new-relic/instrumentation.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@

apiVersion: newrelic.com/v1alpha2
kind: Instrumentation
metadata:
name: newrelic-instrumentation
namespace: newrelic-monitoring
spec:
agent:
# Values supported: dotnet, java, nodejs, python, ruby, php
language: nodejs

# Values supported: newrelic-java-init:latest, newrelic-dotnet-init:latest, newrelic-node-init:latest, newrelic-python-init:latest, newrelic-ruby-init:latest, newrelic-php-init:latest, newrelic-php-init:musl
image: newrelic/newrelic-node-init:latest
env:
# Example overriding the appName configuration
# - name: NEW_RELIC_APP_NAME
# valueFrom:
# fieldRef:
# fieldPath: metadata.labels['app.kubernetes.io/name']

# Select a namespace with a specific name by using "kubernetes.io/metadata.name" label
namespaceLabelSelector:
matchExpressions:
- key: "kubernetes.io/metadata.name"
operator: "In"
values: ["staging"]

# Select pods containing a specific label and value
podLabelSelector:
matchExpressions:
- key: "app.kubernetes.io/name"
operator: "In"
values: ["airqo-stage-device-registry-api"]
192 changes: 192 additions & 0 deletions k8s/new-relic/values.prod.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,192 @@
newrelic-infrastructure:
# newrelic-infrastructure.enabled -- Install the [`newrelic-infrastructure` chart](https://github.com/newrelic/nri-kubernetes/tree/main/charts/newrelic-infrastructure)
enabled: true

nri-prometheus:
# nri-prometheus.enabled -- Install the [`nri-prometheus` chart](https://github.com/newrelic/nri-prometheus/tree/main/charts/nri-prometheus)
enabled: false

nri-metadata-injection:
# nri-metadata-injection.enabled -- Install the [`nri-metadata-injection` chart](https://github.com/newrelic/k8s-metadata-injection/tree/main/charts/nri-metadata-injection)
enabled: true

kube-state-metrics:
# kube-state-metrics.enabled -- Install the [`kube-state-metrics` chart](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-state-metrics) from the stable helm charts repository.
# This is mandatory if `infrastructure.enabled` is set to `true` and the user does not provide its own instance of KSM version >=1.8 and <=2.0. Note, kube-state-metrics v2+ disables labels/annotations
# metrics by default. You can enable the target labels/annotations metrics to be monitored by using the metricLabelsAllowlist/metricAnnotationsAllowList options described [here](https://github.com/prometheus-community/helm-charts/blob/159cd8e4fb89b8b107dcc100287504bb91bf30e0/charts/kube-state-metrics/values.yaml#L274) in
# your Kubernetes clusters.
enabled: true

nri-kube-events:
# nri-kube-events.enabled -- Install the [`nri-kube-events` chart](https://github.com/newrelic/nri-kube-events/tree/main/charts/nri-kube-events)
enabled: true

newrelic-logging:
# newrelic-logging.enabled -- Install the [`newrelic-logging` chart](https://github.com/newrelic/helm-charts/tree/master/charts/newrelic-logging)
enabled: true
# fluentBit:
# -- What path will be mounted to read logs from the node
# linuxMountPath: /var
# persistence:
# -- Fluent Bit persistence is needed to keep track of tailed logs, if set to none data loss or logs duplications could happen. Options are "hostPath", "none", "persistentVolume"
# mode: hostPath
# persistentVolume:
# -- When using persistent volume a storage class could be needed depending on the cluster. It should be a storage class that allows ReadWriteMany
# storageClass:


newrelic-pixie:
# newrelic-pixie.enabled -- Install the [`newrelic-pixie`](https://github.com/newrelic/helm-charts/tree/master/charts/newrelic-pixie)
enabled: false

pixie-chart:
# pixie-chart.enabled -- Install the [`pixie-chart` chart](https://docs.pixielabs.ai/installing-pixie/install-schemes/helm/#3.-deploy)
enabled: false

newrelic-infra-operator:
# newrelic-infra-operator.enabled -- Install the [`newrelic-infra-operator` chart](https://github.com/newrelic/newrelic-infra-operator/tree/main/charts/newrelic-infra-operator) (Beta)
enabled: false

newrelic-prometheus-agent:
# newrelic-prometheus-agent.enabled -- Install the [`newrelic-prometheus-agent` chart](https://github.com/newrelic/newrelic-prometheus-configurator/tree/main/charts/newrelic-prometheus-agent)
enabled: true

newrelic-eapm-agent:
# newrelic-eapm-agent.enabled -- Install the [`nr-eapm-agent`](https://github.com/newrelic/helm-charts/tree/master/charts/nr-ebpf-agent)
enabled: false

k8s-agents-operator:
# k8s-agents-operator.enabled -- Install the [`k8s-agents-operator` chart](https://github.com/newrelic/k8s-agents-operator/tree/main/charts/k8s-agents-operator)
enabled: false

Comment on lines +58 to +61
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Verify agent configuration differences between environments.

The k8s-agents-operator is disabled in production but enabled in staging. This inconsistency should be reviewed.


🏁 Script executed:

#!/bin/bash
# Compare agent configurations between environments
diff k8s/new-relic/values.stage.yaml k8s/new-relic/values.prod.yaml | grep "enabled:"

Length of output: 141


Agent Configuration Discrepancy Between Environments

I verified that in the staging configuration the k8s-agents-operator.enabled flag is set to true, whereas in production it is set to false. This inconsistency should be reviewed to ensure it is intentional—especially considering any security or observability implications. Please confirm if production is meant to disable the operator or if this was an oversight.

newrelic-k8s-metrics-adapter:
# newrelic-k8s-metrics-adapter.enabled -- Install the [`newrelic-k8s-metrics-adapter.` chart](https://github.com/newrelic/newrelic-k8s-metrics-adapter/tree/main/charts/newrelic-k8s-metrics-adapter) (Beta)
enabled: false


# -- change the behaviour globally to all the supported helm charts.
# See [user's guide of the common library](https://github.com/newrelic/helm-charts/blob/master/library/common-library/README.md) for further information.
# @default -- See [`values.yaml`](values.yaml)
global:
# -- The cluster name for the Kubernetes cluster.
cluster: production

# -- The license key for your New Relic Account. This will be preferred configuration option if both `licenseKey` and `customSecret` are specified.
licenseKey: <<LICENSE-KEY>>
# -- The license key for your New Relic Account. This will be preferred configuration option if both `insightsKey` and `customSecret` are specified.
insightsKey: ""
# -- Name of the Secret object where the license key is stored
customSecretName: ""
# -- Key in the Secret object where the license key is stored
customSecretLicenseKey: ""

# -- Additional labels for chart objects
labels: {}
# -- Additional labels for chart pods
podLabels: {}

images:
# -- Changes the registry where to get the images. Useful when there is an internal image cache/proxy
registry: ""
# -- Set secrets to be able to fetch images
pullSecrets: []

serviceAccount:
# -- Add these annotations to the service account we create
annotations: {}
# -- Configures if the service account should be created or not
create:
# -- Change the name of the service account. This is honored if you disable on this chart the creation of the service account so you can use your own
name:

# -- (bool) Sets pod's hostNetwork
# @default -- false
hostNetwork:
# -- Sets pod's dnsConfig
dnsConfig: {}

# -- Sets pod's priorityClassName
priorityClassName: ""
# -- Sets security context (at pod level)
podSecurityContext: {}
# -- Sets security context (at container level)
containerSecurityContext: {}

# -- Sets pod/node affinities
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: role
operator: In
values:
- high-mem
weight: 1
# -- Sets pod's node selector
nodeSelector: {}
# -- Sets pod's tolerations to node taints
tolerations: []

# -- Adds extra attributes to the cluster and all the metrics emitted to the backend
customAttributes: {}

# -- (bool) Reduces number of metrics sent in order to reduce costs
# @default -- false
lowDataMode: true

# -- (bool) In each integration it has different behavior. See [Further information](#values-managed-globally-3) but all aims to send less metrics to the backend to try to save costs |
# @default -- false
privileged: true

Comment on lines +134 to +141
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Review production security settings.

Similar to staging, production has privileged mode enabled. This should be carefully reviewed.

Consider implementing Pod Security Policies or OPA Gatekeeper policies to enforce security standards across both environments.

# -- (bool) Must be set to `true` when deploying in an EKS Fargate environment
# @default -- false
fargate:

# -- Configures the integration to send all HTTP/HTTPS request through the proxy in that URL. The URL should have a standard format like `https://user:password@hostname:port`
proxy:

# -- (bool) Send the metrics to the staging backend. Requires a valid staging license key
# @default -- false
nrStaging:
fedramp:
# fedramp.enabled -- (bool) Enables FedRAMP
# @default -- false
enabled:

# -- (bool) Sets the debug logs to this integration or all integrations if it is set globally
# @default -- false
verboseLog:


# To add values to the subcharts. Follow Helm's guide: https://helm.sh/docs/chart_template_guide/subcharts_and_globals

# If you wish to monitor services running on Kubernetes you can provide integrations
# configuration under `integrations_config` that it will passed down to the `newrelic-infrastructure` chart.
#
# You just need to create a new entry where the "name" is the filename of the configuration file and the data is the content of
# the integration configuration. The name must end in ".yaml" as this will be the
# filename generated and the Infrastructure agent only looks for YAML files.
#
# The data part is the actual integration configuration as described in the spec here:
# https://docs.newrelic.com/docs/integrations/integrations-sdk/file-specifications/integration-configuration-file-specifications-agent-v180
#
# In the following example you can see how to monitor a Redis integration with autodiscovery
#
#
# newrelic-infrastructure:
# integrations:
# nri-redis-sampleapp:
# discovery:
# command:
# exec: /var/db/newrelic-infra/nri-discovery-kubernetes --tls --port 10250
# match:
# label.app: sampleapp
# integrations:
# - name: nri-redis
# env:
# # using the discovered IP as the hostname address
# HOSTNAME: ${discovery.ip}
# PORT: 6379
# labels:
# env: test
Loading
Loading