Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sniffer_class_name does not work #150

Open
1 of 2 tasks
wiizii opened this issue Jan 22, 2025 · 1 comment
Open
1 of 2 tasks

sniffer_class_name does not work #150

wiizii opened this issue Jan 22, 2025 · 1 comment

Comments

@wiizii
Copy link

wiizii commented Jan 22, 2025 •
edited
Loading

(check apply)

  • read the contribution guideline
  • (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Steps to replicate

Provide example config and message

I'm using fluent/fluentd-kubernetes-daemonset:v1.18-debian-opensearch-1 image
When I use the sniffer_class_name field as shown below and restart fluentd, an uninitialized constant error occurs as shown in the log.

  • flunetd config
<match lss.addon_**>
  @type opensearch
  host -
  port -
  reload_connections true
  sniffer_class_name "Fluent::Plugin::OpenSearchSimpleSniffer"
  reload_after 100
  scheme https
  ssl_version TLSv1_2
  user -
  password -
  index_name -
  logstash_format true
  logstash_prefix -
  logstash_prefix_separator -
  include_tag_key true
  tag_key @log_name
  ssl_verify true

  <buffer>
    @type file
    path -
    flush_thread_count 4
    flush_interval 5s
    chunk_limit_size 8M
    queue_limit_length 4
    flush_mode interval
    retry_max_interval 30
    retry_forever false
  </buffer>
</match>
  • fluentd log
2025-01-22 01:46:18 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2025-01-22 01:46:18 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2025-01-22 01:46:18 +0000 [info]: gem 'fluentd' version '1.17.1'
2025-01-22 01:46:18 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2025-01-22 01:46:18 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2025-01-22 01:46:18 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2025-01-22 01:46:18 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2025-01-22 01:46:18 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.5.1'
2025-01-22 01:46:18 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2025-01-22 01:46:18 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.5'
2025-01-22 01:46:18 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2025-01-22 01:46:18 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.1.0'
2025-01-22 01:46:18 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2025-01-22 01:46:18 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2025-01-22 01:46:18 +0000 [info]: gem 'fluent-plugin-systemd' version '1.1.0'
2025-01-22 01:46:19 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Could not load sniffer class Fluent::Plugin::OpenSearchSimpleSniffer: uninitialized constant Fluent::Plugin::OpenSearchSimpleSniffer"

Expected Behavior or What you need to ask

I hope fluentd to work properly with sniffer_class_name applied.

...

Using Fluentd and OpenSearch plugin versions

  • OS version
    • Debian GNU/Linux 12 (bookworm)
  • Bare Metal or within Docker or Kubernetes or others?
    • kubernetes
  • Fluentd v1.0 or later
    • paste result of fluentd --version or td-agent --version
    • fluentd 1.17.1
  • OpenSearch plugin version
    • paste boot log of fluentd or td-agent
    • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock

abbrev (default: 0.1.1)
addressable (2.8.7)
aws-eventstream (1.3.0)
aws-partitions (1.997.0)
aws-sdk-core (3.211.0)
aws-sigv4 (1.10.1)
base64 (0.2.0, default: 0.1.1)
benchmark (default: 0.2.1)
bigdecimal (default: 3.1.3)
bundler (default: 2.4.19, 2.4.17)
cgi (default: 0.3.6)
concurrent-ruby (1.3.4)
cool.io (1.9.0)
csv (3.3.0, default: 3.2.6)
date (default: 3.3.3)
delegate (default: 0.3.0)
did_you_mean (default: 1.6.3)
digest (default: 3.1.1)
domain_name (0.6.20240107)
drb (2.2.1, default: 2.1.1)
english (default: 0.7.2)
erb (default: 4.0.2)
error_highlight (default: 0.5.1)
etc (default: 1.4.2)
excon (1.0.0)
faraday (2.12.0)
faraday-excon (2.3.0)
faraday-net_http (3.3.0)
faraday_middleware-aws-sigv4 (1.0.1)
fcntl (default: 1.0.2)
ffi (1.17.0 x86_64-linux-gnu)
ffi-compiler (1.3.2)
fiddle (default: 1.1.1)
fileutils (default: 1.7.0)
find (default: 0.1.1)
fluent-config-regexp-type (1.0.0)
fluent-plugin-concat (2.5.0)
fluent-plugin-detect-exceptions (0.0.15)
fluent-plugin-grok-parser (2.6.2)
fluent-plugin-json-in-json-2 (1.0.2)
fluent-plugin-kubernetes_metadata_filter (3.5.1)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-opensearch (1.1.5)
fluent-plugin-parser-cri (0.1.1)
fluent-plugin-prometheus (2.1.0)
fluent-plugin-record-modifier (2.1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-systemd (1.1.0)
fluentd (1.17.1)
forwardable (default: 1.3.3)
getoptlong (default: 0.2.0)
http (5.2.0)
http-accept (1.7.0)
http-cookie (1.0.7)
http-form_data (2.3.0)
http_parser.rb (0.8.0)
io-console (default: 0.6.0)
io-nonblock (default: 0.2.0)
io-wait (default: 0.3.0)
ipaddr (default: 1.2.5)
irb (default: 1.6.2)
jmespath (1.6.2)
json (2.7.4, default: 2.6.3)
jsonpath (1.1.5)
kubeclient (4.12.0)
llhttp-ffi (0.5.0)
logger (1.6.1, default: 1.5.3)
lru_redux (1.1.0)
mime-types (3.6.0)
mime-types-data (3.2024.1001)
msgpack (1.7.3)
multi_json (1.15.0)
mutex_m (default: 0.1.2)
net-http (default: 0.4.1)
net-protocol (default: 0.2.1)
netrc (0.11.0)
nkf (default: 0.1.2)
observer (default: 0.1.1)
oj (3.15.1)
open-uri (default: 0.3.0)
open3 (default: 0.1.2)
opensearch-ruby (3.4.0)
openssl (default: 3.1.0)
optparse (default: 0.3.1)
ostruct (0.6.0, default: 0.5.5)
pathname (default: 0.2.1)
pp (default: 0.4.0)
prettyprint (default: 0.1.1)
prometheus-client (4.2.3)
pstore (default: 0.1.2)
psych (default: 5.0.1)
public_suffix (6.0.1)
racc (default: 1.6.2)
rake (13.2.1)
rdoc (default: 6.5.1.1)
readline (default: 0.0.3)
readline-ext (default: 0.1.5)
recursive-open-struct (1.3.1)
reline (default: 0.3.2)
resolv (default: 0.2.2)
resolv-replace (default: 0.1.1)
rest-client (2.1.0)
rexml (3.2.9)
rinda (default: 0.1.1)
ruby2_keywords (default: 0.0.5)
securerandom (default: 0.2.2)
serverengine (2.4.0)
set (default: 1.0.3)
shellwords (default: 0.1.0)
sigdump (0.2.5)
singleton (default: 0.1.1)
stringio (default: 3.0.4)
strptime (0.2.5)
strscan (3.1.0, default: 3.0.5)
syntax_suggest (default: 1.1.0)
syslog (default: 0.1.1)
systemd-journal (2.0.0)
tempfile (default: 0.1.3)
time (default: 0.2.2)
timeout (default: 0.3.1)
tmpdir (default: 0.1.3)
tsort (default: 0.1.1)
tzinfo (2.0.6)
tzinfo-data (1.2024.2)
un (default: 0.2.1)
uri (0.13.1, default: 0.12.2)
weakref (default: 0.1.2)
webrick (1.8.2)
yajl-ruby (1.4.3)
yaml (default: 0.2.1)
zlib (default: 3.0.0)
  • OpenSearch version (optional)
  • OpenSearch template(s) (optional)
@wiizii
Copy link
Author

wiizii commented Feb 4, 2025

Image

I ran it using fluent/fluentd-kubernetes-daemonset:v1.16-debian-opensearch-2 and I could use sniffer_class_name.

There was an option to use sniffer_class in entryPoint.sh in version 1.16, but it was removed in later versions.

1.18 source
1.16 source

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@wiizii