Skip to content
BranchesTags

Repository files navigation

Centralized Log Collection for Windows Kubernetes Nodes with Fluentd on vSphere Kubernetes Service

A complete technical guide to deploying Fluentd on Windows Server 2022 nodes in VKS — covering the full pipeline from log collection and Kubernetes metadata enrichment through to Elasticsearch, VMware Aria Operations for Logs, and local file output.

Author: Julius Nicolescu Date: March 2026

Table of Contents

Overview

Running Windows workloads on Kubernetes means logs written to stdout/stderr by containers are captured by the Windows kubelet and stored as plain-text files on the node under C:\var\log\containers\. Without a log collection agent, those files are only accessible by SSHing to the node or running kubectl logs one pod at a time — neither of which scales.

This project deploys Fluentd as a Kubernetes DaemonSet (one pod per Windows node) to:

  1. Continuously tail all container log files on the node
  2. Parse the kubelet log format into structured fields
  3. Enrich each log record with Kubernetes metadata (pod, namespace, container, node)
  4. Forward enriched records to a configurable output backend

Four output backends are provided, each as a standalone Fluentd configuration file. Only one is active at a time and is selected by editing a single @include line.

A single-node Elasticsearch + Kibana stack and a log-generating test pod are included for end-to-end validation with no external dependencies.

Architecture

Windows Node (Windows Server 2022)
┌───────────────────────────────────────────────────────────────────┐
│                                                                   │
│  Kubelet writes container logs to:                                │
│  C:\var\log\containers\<pod>_<ns>_<container>-<hash>.log         │
│                         │                                         │
│               ┌─────────▼──────────┐                             │
│               │    Fluentd Pod     │  ← DaemonSet (1 per node)   │
│               │                   │                               │
│               │  [1] tail source  │  read log files               │
│               │  [2] regexp parse │  extract time/stream/message  │
│               │  [3] k8s_metadata │  inject pod/namespace/labels  │
│               │  [4] record_xform │  flatten + rename fields      │
│               └─────────┬──────────┘                             │
│                         │                                         │
│  Position file: C:\var\log\fluentd\fluentd-containers.pos        │
│  (tracks read offset — survives pod restarts)                     │
└─────────────────────────┼─────────────────────────────────────────┘
                          │
          ┌───────────────▼──────────────────────────────┐
          │      Output Backend (one active at a time)   │
          ├──────────────────────────────────────────────┤
          │  file-fluentd.conf              local file   │
          │  elastic-fluentd.conf           Elasticsearch │
          │  operations-for-logs-fluentd.conf  Aria HTTP  │
          │  loginsight-fluentd.conf     Aria native (⚠ deprecated) │
          └──────────────────────────────────────────────┘

Repository Structure

fluentd-namespace.yaml            Namespace "fluentd" with privileged pod-security labels
fluentd-rbac.yaml                 ServiceAccount, ClusterRole, ClusterRoleBinding
fluentd-configmap-windows.yaml    Complete Fluentd pipeline: all conf files in one ConfigMap
fluentd-daemonset-windows.yaml    DaemonSet — runs on Windows nodes only

counter-windows.yaml              Test pod — emits a numbered log line every 5 seconds

demo-elastic-kibana.yaml          Namespace "elastic-kibana" for the demo ELK stack
demo-elastic-7.17.29.yaml         Single-node Elasticsearch 7.17.29 + ClusterIP + LoadBalancer
demo-kibana-7.17.29.yaml          Kibana 7.17.29 + ClusterIP + LoadBalancer

Prerequisites

  • kubectl configured to reach the VKS cluster
  • vcf CLI installed for kubeconfig context management
  • At least one Windows Server 2022 worker node in the cluster
  • Node taint os=windows:NoSchedule applied to Windows nodes

Connecting to the VKS Cluster

Use the vcf CLI to create or refresh a kubeconfig context for the target cluster.

export SUPERVISOR_IP=192.168.231.7
export VCF_CLI_VSPHERE_PASSWORD='<password>'
export CLUSTER_NAMESPACE=lab-namespace-01
export CLUSTER_NAME=dev-cluster-05

# Delete and recreate a context (use this when the context has expired)
vcf context delete ${CLUSTER_NAME} -y
vcf context create ${CLUSTER_NAME} \
  --endpoint ${SUPERVISOR_IP} \
  --username administrator@vsphere.local \
  --workload-cluster-namespace ${CLUSTER_NAMESPACE} \
  --workload-cluster-name ${CLUSTER_NAME} \
  --insecure-skip-tls-verify

# List available contexts and switch to the target cluster
vcf context list
vcf context use ${CLUSTER_NAME}:${CLUSTER_NAME}

# Or simply refresh an existing context without recreating it
vcf context refresh ${CLUSTER_NAME} --insecure-skip-tls-verify

Verify cluster nodes:

kubectl get nodes -l kubernetes.io/os=linux
kubectl get nodes -l kubernetes.io/os=windows

Example output from dev-cluster-05:

NAME                         STATUS   ROLES           AGE   VERSION
dev-cluster-05-tl497-vg2dg   Ready    control-plane   22d   v1.34.1+vmware.1

NAME                                        STATUS   ROLES    AGE   VERSION
dev-cluster-05-win2022-ls7zk-vg5bz-fl6p6   Ready    <none>   32h   v1.34.1+vmware.1

Fluentd Deployment

Apply the manifests in order. The namespace and RBAC must exist before the ConfigMap and DaemonSet are created.

kubectl apply -f fluentd-namespace.yaml
kubectl apply -f fluentd-rbac.yaml
kubectl apply -f fluentd-configmap-windows.yaml
kubectl apply -f fluentd-daemonset-windows.yaml

Monitor the rollout:

kubectl -n fluentd rollout status ds/fluentd
kubectl -n fluentd get pods
kubectl -n fluentd logs -l k8s-app=fluentd-logging --tail=50

Switching output plugins requires editing fluent.conf in the ConfigMap, then restarting the pods. ConfigMap changes are not picked up automatically.

# After editing fluentd-configmap-windows.yaml:
kubectl apply -f fluentd-configmap-windows.yaml
kubectl -n fluentd rollout restart ds/fluentd
kubectl -n fluentd rollout status ds/fluentd

Remove Fluentd:

kubectl delete -f fluentd-daemonset-windows.yaml
kubectl delete -f fluentd-configmap-windows.yaml
kubectl delete -f fluentd-rbac.yaml
kubectl delete -f fluentd-namespace.yaml

Fluentd Configuration and Plugins

All configuration is stored in fluentd-configmap-windows.yaml. The ConfigMap is mounted into each Fluentd pod as a directory at C:\fluent\conf, where each ConfigMap key becomes a .conf file.

C:\fluent\conf\
  fluent.conf                      ← main entry point
  pods-fluentd.conf                ← shared collection + enrichment pipeline
  file-fluentd.conf                ← output: local file
  elastic-fluentd.conf             ← output: Elasticsearch
  operations-for-logs-fluentd.conf ← output: VCF Ops for Logs (HTTP)
  loginsight-fluentd.conf          ← output: Aria native plugin (deprecated)

fluent.conf — Main Entry Point

@include pods-fluentd.conf

# --- uncomment exactly one output plugin ---
# @include file-fluentd.conf
# @include elastic-fluentd.conf
# @include operations-for-logs-fluentd.conf
@include loginsight-fluentd.conf

pods-fluentd.conf is always included. It defines the log source and the two enrichment filters that run regardless of which output backend is active. Only one @include for an output plugin should be uncommented at any time — having two outputs active simultaneously will cause duplicate log delivery.

Plugin: tail — Log Collection

Type: built-in input plugin File: pods-fluentd.conf Docs: https://docs.fluentd.org/input/tail

The tail plugin continuously reads new lines appended to matching files, similar to tail -F. It is the entry point for all log data in this pipeline.

<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd/fluentd-containers.pos
  exclude_path ["/var/log/containers/fluent*"]
  path_key log_path
  read_from_head true
  tag kubernetes.*
  <parse> ... </parse>
</source>

Key settings:

Setting Value Purpose
path /var/log/containers/*.log Match all container log files on the node. Forward-slash paths work inside Windows containers because Ruby normalises path separators.
pos_file /var/log/fluentd/fluentd-containers.pos Stores the byte offset last read for each file. Mounted on a hostPath volume so the position survives pod restarts — prevents re-reading logs already delivered.
exclude_path ["/var/log/containers/fluent*"] Skips Fluentd's own log file to avoid a feedback loop where Fluentd logs become input to itself.
path_key log_path Adds the source file path as a record field. The file name encodes pod name, namespace, and container name, which the metadata filter uses.
read_from_head true Reads each file from the beginning on first encounter (e.g. when a new pod starts). Without this, only lines written after Fluentd starts are read.
tag kubernetes.* Tags all events with a prefix that downstream filters and outputs match against.

Log line format (kubelet container log format):

2024-01-15T10:23:45.123456789Z stdout F Hello World from counter-windows
│                             │      │ │
│                             │      │ └─ log message
│                             │      └─── logtag (F = full line, P = partial)
│                             └────────── stream (stdout or stderr)
└──────────────────────────────────────── RFC3339Nano timestamp

The regexp parser extracts these four fields:

expression: /^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$/
Field Description
time Event timestamp (parsed as %Y-%m-%dT%H:%M:%S.%N%Z)
stream stdout or stderr
logtag F (full line) or P (partial — CRI log rotation artifact)
log Raw log message from the container

Plugin: kubernetes_metadata — Metadata Enrichment

Type: filter plugin Gem: fluent-plugin-kubernetes_metadata_filter File: pods-fluentd.conf Docs: https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter

<filter **>
  @type  kubernetes_metadata
  @id    filter_kube_metadata
</filter>

This filter looks up the Kubernetes API to find the pod that produced each log record and injects a nested kubernetes hash into every event. The pod is identified from the log_path field set by the tail source — the container log filename follows the pattern <pod>_<namespace>_<container>-<hash>.log.

Fields injected into each record:

{
  "kubernetes": {
    "namespace_name": "default",
    "pod_name":       "counter-windows",
    "container_name": "count",
    "host":           "dev-cluster-05-win2022-ls7zk-vg5bz-fl6p6",
    "labels": {
      "app": "counter-windows"
    },
    "annotations": { ... }
  }
}

Why ClusterRole is required: the plugin queries the Kubernetes API for pod and namespace objects across all namespaces. A namespaced Role would only allow lookups within the fluentd namespace, missing pods in every other namespace. The ClusterRoleBinding in fluentd-rbac.yaml grants the fluentd ServiceAccount the necessary cluster-wide read permissions.

The plugin caches pod metadata in memory to avoid an API call per log line. Cached entries are refreshed when a pod changes or the cache TTL expires.

Plugin: record_transformer — Field Flattening

Type: built-in filter plugin File: pods-fluentd.conf Docs: https://docs.fluentd.org/filter/record_transformer

<filter **>
  @type        record_transformer
  enable_ruby  true
  <record>
    namespace   ${record.dig("kubernetes", "namespace_name")}
    pod         ${record.dig("kubernetes", "pod_name")}
    container   ${record.dig("kubernetes", "container_name")}
    node        ${record.dig("kubernetes", "host")}
    log_type    kubernetes
    k8s_cluster dev-cluster-05
    message     ${record["log"]}
  </record>
  remove_keys log,kubernetes,stream,logtag,log_path
</filter>

This filter runs after the metadata filter and does two things:

  1. Flatten the nested kubernetes hash into individual top-level fields. This produces a flat record that is easier to index and query in Elasticsearch or Operations for Logs without needing dot-notation field names.

  2. Remove intermediate fields that were needed for processing but are not useful in the final record (log, kubernetes, stream, logtag, log_path).

enable_ruby true is required to use record.dig(), which safely traverses the nested hash without raising an error if a key is absent.

Record shape after transformation:

{
  "@timestamp":  "2024-01-15T10:23:45.123456789Z",
  "namespace":   "default",
  "pod":         "counter-windows",
  "container":   "count",
  "node":        "dev-cluster-05-win2022-ls7zk-vg5bz-fl6p6",
  "log_type":    "kubernetes",
  "k8s_cluster": "dev-cluster-05",
  "message":     "42: WINDOWS TESTMSG 2024-01-15T10:23:45.1234567+00:00"
}

Plugin: file — Local File Output

Type: built-in output plugin File: file-fluentd.conf Docs: https://docs.fluentd.org/output/file

<match **>
  @type file
  path /var/log/fluentd/file-test-log
  append true
  <format>
    @type json
  </format>
  <buffer>
    flush_interval 10s
  </buffer>
</match>

Writes each log record as a JSON object to a date-suffixed file on the node:

/var/log/fluentd/file-test-log.20240115.log    (inside container)
C:\var\log\fluentd\file-test-log.20240115.log  (on the Windows host)

Use this plugin first when setting up or debugging the pipeline. It requires no external system — if records appear in the file, the tail source, metadata filter, and record transformer are all working correctly.

To activate:

# fluent.conf
@include pods-fluentd.conf
@include file-fluentd.conf

To verify from inside the pod:

# List generated output files
Get-ChildItem C:\var\log\fluentd\

# Read and pretty-print the latest output file
Get-Content C:\var\log\fluentd\file-test-log.20240115.log |
  ForEach-Object { $_ | ConvertFrom-Json }

Plugin: elasticsearch — Elasticsearch Output

Type: output plugin Gem: fluent-plugin-elasticsearch File: elastic-fluentd.conf Docs: https://github.com/uken/fluent-plugin-elasticsearch

<match **>
  @type      elasticsearch
  @id        out_es
  @log_level info

  host     "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
  port     "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
  user     "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
  password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
  scheme   http

  logstash_format true
  logstash_prefix fluentd-windows
  type_name       fluentd
  include_tag_key true

  <buffer> ... </buffer>
</match>

Sends log records to Elasticsearch using the Logstash-compatible index naming convention. With logstash_format true and logstash_prefix fluentd-windows, Fluentd creates a new index each day named fluentd-windows-YYYY.MM.DD. This matches the pattern expected by Kibana's default index management and allows time-based data management (e.g. deleting old indices by date).

Connection settings are read from environment variables set in the DaemonSet, so the Elasticsearch address can be changed without modifying the ConfigMap. All buffer parameters also accept environment variable overrides via FLUENT_ELASTICSEARCH_BUFFER_* variables.

Setting Environment variable / Default
Host FLUENT_ELASTICSEARCH_HOST — 192.168.231.11
Port FLUENT_ELASTICSEARCH_PORT — 9200
Username FLUENT_ELASTICSEARCH_USER — elastic
Password FLUENT_ELASTICSEARCH_PASSWORD
Index prefix fluentd-windows
Index name format fluentd-windows-YYYY.MM.DD
Scheme http
Flush threads FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT — 8
Flush interval FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL — 5s
Chunk size FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE — 2M
Queue length FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH — 32
Max retry interval FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL — 30
Retry forever

retry_forever true means Fluentd will keep retrying failed deliveries indefinitely rather than dropping records. This is appropriate for a log pipeline where losing data is undesirable and temporary Elasticsearch unavailability is expected.

To activate:

# fluent.conf
@include pods-fluentd.conf
@include elastic-fluentd.conf

To verify from outside the cluster:

# Cluster health
curl -u elastic:elastic http://<ES-EXTERNAL-IP>:9200/_cluster/health | jq

# List fluentd indices
curl -u elastic:elastic \
  "http://<ES-EXTERNAL-IP>:9200/_cat/indices/fluentd-windows-*?v&s=index"

# Fetch the 5 most recent log records
curl -u elastic:elastic \
  "http://<ES-EXTERNAL-IP>:9200/fluentd-windows-*/_search?size=5&sort=@timestamp:desc&pretty"

Plugin: http — VCF Operations for Logs Output

Type: built-in output plugin File: operations-for-logs-fluentd.conf Docs: https://docs.fluentd.org/output/http

<match **>
  @type        http
  endpoint     http://192.168.200.40:9000/api/v2/events
  http_method  post
  content_type application/json
  tls_verify_mode none
  json_array   true

  <format>
    @type json
  </format>
  <inject>
    time_key    timestamp
    time_type   string
    time_format %Y-%m-%dT%H:%M:%SZ
  </inject>
  <buffer>
    flush_interval   10s
    chunk_limit_size 256k
  </buffer>
</match>

Sends log records to VMware Aria Operations for Logs using the native REST ingestion API. This is the recommended method for VCF Operations for Logs integration — it uses the standard fluent-plugin-http (no additional gem required) and targets the stable /api/v2/events endpoint.

json_array true causes Fluentd to batch multiple records into a single JSON array per HTTP POST, which is required by the Operations for Logs v2 bulk ingest API.

The <inject> block adds a timestamp field in ISO-8601 format to each record so Operations for Logs can parse the event time correctly, rather than using the ingestion time.

Setting Value
Endpoint http://192.168.200.40:9000/api/v2/events
Method POST
Format JSON array (bulk ingest)
Time field timestamp (ISO-8601, injected per record)
TLS disabled (tls_verify_mode none)
Flush interval 10 seconds
Chunk size 256 KB

Note: HTTP on port 9000 must be explicitly enabled in the Operations for Logs appliance. Go to Configuration → SSL → API Server SSL and uncheck Require SSL Connection, then reboot the appliance.

To activate:

# fluent.conf
@include pods-fluentd.conf
@include operations-for-logs-fluentd.conf

To test the ingestion API directly:

# Step 1 — Obtain a session token
curl -sk -X POST https://192.168.200.40:9543/api/v2/sessions \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"<password>","provider":"Local"}' | jq
# Response:
# {
#   "userId":    "31530986-176b-41ca-b5e4-e47ae616787e",
#   "sessionId": "SQadVha/u8PV2gwk...",
#   "ttl":       1800
# }

# Step 2 — Generate a UUID (used as the agent/source identifier)
uuidgen
# e.g. 36a3891f-f6d4-4172-9b03-c769fa12ef78

# Step 3 — Send a test event over HTTPS (port 9543)
curl -sk -X POST \
  https://192.168.200.40:9543/api/v2/events/ingest/36a3891f-f6d4-4172-9b03-c769fa12ef78 \
  -H "Content-Type: application/json" \
  -d '{"events":[{"text":"Test Event https"}]}' | jq

# Step 4 — Send a test event over HTTP (port 9000, after disabling SSL requirement)
curl -sk -X POST \
  http://192.168.200.40:9000/api/v2/events/ingest/36a3891f-f6d4-4172-9b03-c769fa12ef78 \
  -H "Content-Type: application/json" \
  -d '{"events":[{"text":"Test Event http"}]}' | jq

# Step 5 — Query recent events (search for the last 5 minutes)
curl -sk -X POST https://192.168.200.40:9543/rest-api/internal/events/query \
  -H "Authorization: Bearer <sessionId>" \
  -H "Content-Type: application/json" \
  -d '{"query":"WINDOWS TESTMSG","durationMs":300000}' | jq

Operations for Logs UI navigation:

  • Log Sources → Containers → Kubernetes — verify that Fluentd is appearing as a log source
  • Configuration → SSL → API Server SSL — toggle HTTP access on port 9000

Plugin: vmware_loginsight — Native Aria Output (Deprecated)

Type: output plugin Gem: fluent-plugin-vmware-loginsight File: loginsight-fluentd.conf

Deprecated. fluent-plugin-vmware-loginsight is no longer maintained by VMware/Broadcom. Existing deployments continue to function, but new deployments should use operations-for-logs-fluentd.conf instead.

See: VMware Aria Operations for Logs 8.18.5 Release Notes

<match **>
  @type vmware_loginsight

  host            192.168.200.40
  port            9543
  scheme          https
  http_method     post
  ssl_verify      false
  raise_on_error  true
  include_tag_key true

  <buffer>
    chunk_limit_records 300
    flush_interval      3s
    retry_max_times     3
  </buffer>
</match>

The native plugin communicates with the Log Insight ingestion API using a proprietary format over HTTPS. It is kept here for backwards compatibility with deployments that were built before the HTTP plugin approach was recommended.

Setting Value
Host 192.168.200.40
Port 9543
Scheme HTTPS
SSL verification disabled
Chunk limit 300 records
Flush interval 3 seconds
Max retries 3 (then drop)

Unlike the elasticsearch plugin, this plugin does not retry forever — after 3 failed attempts it discards the buffer chunk. For production use, increase retry_max_times or switch to the HTTP plugin with retry_forever true.

Counter Test Application

counter-windows.yaml deploys a single pod to a Windows node. It runs a PowerShell loop that writes a numbered, timestamped line to stdout every 5 seconds.

kubectl apply -f counter-windows.yaml

The pod runs in the default namespace and requires the os=windows:NoSchedule toleration to be scheduled on the Windows node.

Live log view:

kubectl logs -f counter-windows

Expected stdout output:

0: WINDOWS TESTMSG 2024-01-15T10:23:45.1234567+00:00
1: WINDOWS TESTMSG 2024-01-15T10:23:50.1234567+00:00
2: WINDOWS TESTMSG 2024-01-15T10:23:55.1234567+00:00

After Fluentd collects and enriches these lines, each record delivered to the output backend will look like:

{
  "@timestamp":  "2024-01-15T10:23:45.123456789Z",
  "namespace":   "default",
  "pod":         "counter-windows",
  "container":   "count",
  "node":        "dev-cluster-05-win2022-ls7zk-vg5bz-fl6p6",
  "log_type":    "kubernetes",
  "k8s_cluster": "dev-cluster-05",
  "message":     "42: WINDOWS TESTMSG 2024-01-15T10:23:45.1234567+00:00"
}

This makes it easy to search for test records in any backend by filtering on pod: counter-windows or the literal string WINDOWS TESTMSG.

Remove the test pod when done:

kubectl delete -f counter-windows.yaml

Demo: Elasticsearch and Kibana

A single-node Elasticsearch 7.17.29 cluster and Kibana 7.17.29 are included to visualise the logs collected by Fluentd without requiring an external logging stack.

Deploy

kubectl apply -f demo-elastic-kibana.yaml
kubectl apply -f demo-elastic-7.17.29.yaml
kubectl apply -f demo-kibana-7.17.29.yaml

kubectl -n elastic-kibana get pods,services

Expected output once both pods are Running:

NAME                                 READY   STATUS    RESTARTS   AGE
pod/elasticsearch-6bf648d79d-j8md9   1/1     Running   0          4m17s
pod/kibana-78b4467cd4-hl4kh          1/1     Running   0          4m9s

NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)
service/elasticsearch      ClusterIP      10.105.70.170   <none>           9200/TCP
service/elasticsearch-lb   LoadBalancer   10.105.70.171   192.168.231.11   9200:31xxx/TCP
service/kibana             ClusterIP      10.106.92.41    <none>           5601/TCP
service/kibana-lb          LoadBalancer   10.102.12.56    192.168.231.11   5601:32022/TCP

Access

Service URL Credentials
Kibana UI http://<kibana-lb-external-ip>:5601 elastic / elastic
Elasticsearch http://<elasticsearch-lb-external-ip>:9200 elastic / elastic

Configure Fluentd to send to this Elasticsearch instance

Edit fluentd-daemonset-windows.yaml and set FLUENT_ELASTICSEARCH_HOST to the LoadBalancer external IP of the elasticsearch-lb service, then activate the Elasticsearch output plugin:

# fluent.conf
@include pods-fluentd.conf
@include elastic-fluentd.conf

Apply and restart:

kubectl apply -f fluentd-configmap-windows.yaml
kubectl -n fluentd rollout restart ds/fluentd

Configure Kibana to view logs

  1. Open Kibana at http://<kibana-lb-external-ip>:5601
  2. Navigate to Stack Management → Index Patterns → Create index pattern
  3. Enter fluentd-windows-* as the pattern
  4. Select @timestamp as the time field and click Create index pattern
  5. Go to Discover to search and filter log records

Verify Elasticsearch directly

# Check cluster health
curl -u elastic:elastic \
  http://<elasticsearch-lb-external-ip>:9200/_cluster/health | jq

# List all fluentd indices with document counts
curl -u elastic:elastic \
  "http://<elasticsearch-lb-external-ip>:9200/_cat/indices/fluentd-windows-*?v&s=index"

# Fetch the 5 most recent log records from all fluentd indices
curl -u elastic:elastic \
  "http://<elasticsearch-lb-external-ip>:9200/fluentd-windows-*/_search?size=5&sort=@timestamp:desc&pretty"

# Search for counter-windows test messages specifically
curl -u elastic:elastic \
  "http://<elasticsearch-lb-external-ip>:9200/fluentd-windows-*/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{"query":{"match":{"pod":"counter-windows"}},"size":10,"sort":[{"@timestamp":"desc"}]}'

Remove

kubectl delete -f demo-kibana-7.17.29.yaml
kubectl delete -f demo-elastic-7.17.29.yaml
kubectl delete -f demo-elastic-kibana.yaml

Verification and Troubleshooting

Open a shell inside the Fluentd pod

export NAMESPACE=fluentd
FluentdPod=$(kubectl -n ${NAMESPACE} get pods --no-headers | grep fluentd | tail -1 | awk '{print $1}')

# Open a PowerShell session inside the Windows container
kubectl -n ${NAMESPACE} exec -it ${FluentdPod} -- powershell

Check installed Fluentd gems

kubectl -n ${NAMESPACE} exec -it ${FluentdPod} -- powershell \
  -Command "fluent-gem list | Select-String 'kubernetes|elastic|loginsight|http'"

Validate the configuration (dry run — no output sent)

kubectl -n ${NAMESPACE} exec -it ${FluentdPod} -- powershell \
  -Command "fluentd --dry-run -c C:\fluent\conf\fluent.conf -v"

Run Fluentd manually with verbose output (full debug)

This is useful for seeing exactly what Fluentd is doing — every record parsed, every filter applied, every delivery attempt — without restarting the DaemonSet pod.

kubectl -n ${NAMESPACE} exec -it ${FluentdPod} -- powershell \
  -Command "fluentd -c C:\fluent\conf\fluent.conf -vv"

Inspect the configuration directory from inside the pod

Directory: C:\fluent\etc

Mode     LastWriteTime    Name
----     -------------    ----
-a---l   2/28/2026 7:26   elastic-fluentd.conf
-a---l   2/28/2026 7:26   file-fluentd.conf
-a---l   2/28/2026 7:26   fluent.conf
-a---l   2/28/2026 7:26   loginsight-fluentd.conf
-a---l   2/28/2026 7:26   operations-for-logs-fluentd.conf
-a---l   2/28/2026 7:26   pods-fluentd.conf

Inspect log files on the Windows node

# Container log symlinks — one symlink per container, named: <pod>_<ns>_<container>-<hash>.log
Get-ChildItem C:\var\log\containers\
# antrea-agent-windows-kqrxh_kube-system_antrea-agent-a001018663fa...log
# counter-windows_default_count-<hash>.log
# fluentd-njtlw_fluentd_fluentd-2e4683425ef32...log

# Pod log directories — one directory per pod, containing per-container subdirectories
Get-ChildItem C:\var\log\pods\
# fluentd_fluentd-njtlw_acc7aa7e-e12d-49f9-8a91-2423d92af123\
# kube-system_antrea-agent-windows-kqrxh_f0a4c9b0-58eb-4308-...\
# vmware-system-csi_vsphere-csi-node-windows-cx96w_349cb0fd-...\

# Fluentd position file and file-output logs
Get-ChildItem C:\var\log\fluentd\

Apply pod-security labels to all namespaces

If Fluentd or demo stack pods are rejected by the pod-security admission controller, label all namespaces as privileged:

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  kubectl label ns "$ns" \
    pod-security.kubernetes.io/enforce=privileged \
    pod-security.kubernetes.io/audit=privileged \
    pod-security.kubernetes.io/warn=privileged \
    --overwrite
done

References

About

Custom Fluentd Docker images for Windows Server 2019 and 2022 that extend the official Fluentd base images with Kubernetes metadata enrichment, multiline log parsing, and output plugins for Elasticsearch and VMware Aria Operations for Logs.

Topics

ansiblefluentdkuberneteskubernetes-image-builderpackertanzuvcfvcf9vksvks-image-buildervmware

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

1 fork
Report repository

Releases

Packages

Contributors