Skip to content

Commit d5f1782

Browse files
add api_key support (logstash-plugins#934)
bump version to 10.5.0
1 parent df32e53 commit d5f1782

File tree

7 files changed

+148
-17
lines changed

7 files changed

+148
-17
lines changed

CHANGELOG.md

+3
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
## 10.5.0
2+
- Added api_key support [#934](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/934)
3+
14
## 10.4.1
25
- [DOC] Added note about `_type` setting change from `doc` to `_doc` [#884](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/884)
36

docs/index.asciidoc

+14
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,9 @@ Elasticsearch] to take advantage of response compression when using this plugin
232232
For requests compression, regardless of the Elasticsearch version, users have to enable `http_compression`
233233
setting in their Logstash config file.
234234

235+
==== Authentication
236+
237+
Authentication to a secure Elasticsearch cluster is possible using one of the `user`/`password`, `cloud_auth` or `api_key` options.
235238

236239
[id="plugins-{type}s-{plugin}-options"]
237240
==== Elasticsearch Output Configuration Options
@@ -242,6 +245,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
242245
|=======================================================================
243246
|Setting |Input type|Required
244247
| <<plugins-{type}s-{plugin}-action>> |<<string,string>>|No
248+
| <<plugins-{type}s-{plugin}-api_key>> |<<password,password>>|No
245249
| <<plugins-{type}s-{plugin}-bulk_path>> |<<string,string>>|No
246250
| <<plugins-{type}s-{plugin}-cacert>> |a valid filesystem path|No
247251
| <<plugins-{type}s-{plugin}-cloud_auth>> |<<password,password>>|No
@@ -324,6 +328,16 @@ The Elasticsearch action to perform. Valid actions are:
324328

325329
For more details on actions, check out the http://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html[Elasticsearch bulk API documentation]
326330

331+
[id="plugins-{type}s-{plugin}-api_key"]
332+
===== `api_key`
333+
334+
* Value type is <<password,password>>
335+
* There is no default value for this setting.
336+
337+
Authenticate using Elasticsearch API key. Note that this option also requires enabling the `ssl` option.
338+
339+
Format is `id:api_key` where `id` and `api_key` are as returned by the Elasticsearch https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html[Create API key API].
340+
327341
[id="plugins-{type}s-{plugin}-bulk_path"]
328342
===== `bulk_path`
329343

lib/logstash/outputs/elasticsearch.rb

+12
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,10 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
122122
# Password to authenticate to a secure Elasticsearch cluster
123123
config :password, :validate => :password
124124

125+
# Authenticate using Elasticsearch API key.
126+
# format is id:api_key (as returned by https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html[Create API key])
127+
config :api_key, :validate => :password
128+
125129
# Cloud authentication string ("<username>:<password>" format) is an alternative for the `user`/`password` configuration.
126130
#
127131
# For more details, check out the https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html#_cloud_auth[cloud documentation]
@@ -255,6 +259,14 @@ def config_init(params)
255259
end
256260

257261
def build_client
262+
# the following 3 options validation & setup methods are called inside build_client
263+
# because they must be executed prior to building the client and logstash
264+
# monitoring and management rely on directly calling build_client
265+
# see https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/934#pullrequestreview-396203307
266+
validate_authentication
267+
fill_hosts_from_cloud_id
268+
setup_hosts
269+
258270
params["metric"] = metric
259271
if @proxy.eql?('')
260272
@logger.warn "Supplied proxy setting (proxy => '') has no effect"

lib/logstash/outputs/elasticsearch/common.rb

+22-14
Original file line numberDiff line numberDiff line change
@@ -20,10 +20,6 @@ def register
2020
@stopping = Concurrent::AtomicBoolean.new(false)
2121
# To support BWC, we check if DLQ exists in core (< 5.4). If it doesn't, we use nil to resort to previous behavior.
2222
@dlq_writer = dlq_enabled? ? execution_context.dlq_writer : nil
23-
24-
fill_hosts_from_cloud_id
25-
fill_user_password_from_cloud_auth
26-
setup_hosts # properly sets @hosts
2723
build_client
2824
setup_after_successful_connection
2925
check_action_validity
@@ -112,6 +108,28 @@ def event_action_tuple(event)
112108
[action, params, event]
113109
end
114110

111+
def validate_authentication
112+
authn_options = 0
113+
authn_options += 1 if @cloud_auth
114+
authn_options += 1 if (@api_key && @api_key.value)
115+
authn_options += 1 if (@user || (@password && @password.value))
116+
117+
if authn_options > 1
118+
raise LogStash::ConfigurationError, 'Multiple authentication options are specified, please only use one of user/password, cloud_auth or api_key'
119+
end
120+
121+
if @api_key && @api_key.value && @ssl != true
122+
raise(LogStash::ConfigurationError, "Using api_key authentication requires SSL/TLS secured communication using the `ssl => true` option")
123+
end
124+
125+
if @cloud_auth
126+
@user, @password = parse_user_password_from_cloud_auth(@cloud_auth)
127+
# params is the plugin global params hash which will be passed to HttpClientBuilder.build
128+
params['user'], params['password'] = @user, @password
129+
end
130+
end
131+
private :validate_authentication
132+
115133
def setup_hosts
116134
@hosts = Array(@hosts)
117135
if @hosts.empty?
@@ -135,16 +153,6 @@ def fill_hosts_from_cloud_id
135153
@hosts = parse_host_uri_from_cloud_id(@cloud_id)
136154
end
137155

138-
def fill_user_password_from_cloud_auth
139-
return unless @cloud_auth
140-
141-
if @user || @password
142-
raise LogStash::ConfigurationError, 'Both cloud_auth and user/password specified, please only use one.'
143-
end
144-
@user, @password = parse_user_password_from_cloud_auth(@cloud_auth)
145-
params['user'], params['password'] = @user, @password
146-
end
147-
148156
def parse_host_uri_from_cloud_id(cloud_id)
149157
begin # might not be available on older LS
150158
require 'logstash/util/cloud_setting_id'

lib/logstash/outputs/elasticsearch/http_client_builder.rb

+11-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
require 'cgi'
2+
require "base64"
23

34
module LogStash; module Outputs; class ElasticSearch;
45
module HttpClientBuilder
@@ -8,7 +9,7 @@ def self.build(logger, hosts, params)
89
:pool_max_per_route => params["pool_max_per_route"],
910
:check_connection_timeout => params["validate_after_inactivity"],
1011
:http_compression => params["http_compression"],
11-
:headers => params["custom_headers"]
12+
:headers => params["custom_headers"] || {}
1213
}
1314

1415
client_settings[:proxy] = params["proxy"] if params["proxy"]
@@ -56,6 +57,7 @@ def self.build(logger, hosts, params)
5657

5758
client_settings.merge! setup_ssl(logger, params)
5859
common_options.merge! setup_basic_auth(logger, params)
60+
client_settings[:headers].merge! setup_api_key(logger, params)
5961

6062
external_version_types = ["external", "external_gt", "external_gte"]
6163
# External Version validation
@@ -151,6 +153,14 @@ def self.setup_basic_auth(logger, params)
151153
}
152154
end
153155

156+
def self.setup_api_key(logger, params)
157+
api_key = params["api_key"]
158+
159+
return {} unless (api_key && api_key.value)
160+
161+
{ "Authorization" => "ApiKey " + Base64.strict_encode64(api_key.value) }
162+
end
163+
154164
private
155165
def self.dedup_slashes(url)
156166
url.gsub(/\/+/, "/")

logstash-output-elasticsearch.gemspec

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
Gem::Specification.new do |s|
22
s.name = 'logstash-output-elasticsearch'
3-
s.version = '10.4.1'
3+
s.version = '10.5.0'
44

55
s.licenses = ['apache-2.0']
66
s.summary = "Stores logs in Elasticsearch"

spec/unit/outputs/elasticsearch_spec.rb

+85-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
require_relative "../../../spec/es_spec_helper"
2+
require "base64"
23
require "flores/random"
34
require "logstash/outputs/elasticsearch"
45

@@ -142,6 +143,25 @@
142143

143144
include_examples("an authenticated config")
144145
end
146+
147+
context 'claud_auth also set' do
148+
let(:do_register) { false } # this is what we want to test, so we disable the before(:each) call
149+
let(:options) { { "user" => user, "password" => password, "cloud_auth" => "elastic:my-passwd-00" } }
150+
151+
it "should fail" do
152+
expect { subject.register }.to raise_error LogStash::ConfigurationError, /Multiple authentication options are specified/
153+
end
154+
end
155+
156+
context 'api_key also set' do
157+
let(:do_register) { false } # this is what we want to test, so we disable the before(:each) call
158+
let(:options) { { "user" => user, "password" => password, "api_key" => "some_key" } }
159+
160+
it "should fail" do
161+
expect { subject.register }.to raise_error LogStash::ConfigurationError, /Multiple authentication options are specified/
162+
end
163+
end
164+
145165
end
146166

147167
describe "with path" do
@@ -577,7 +597,15 @@
577597
let(:options) { { 'cloud_auth' => 'elastic:my-passwd-00', 'user' => 'another' } }
578598

579599
it "should fail" do
580-
expect { subject.register }.to raise_error LogStash::ConfigurationError, /cloud_auth and user/
600+
expect { subject.register }.to raise_error LogStash::ConfigurationError, /Multiple authentication options are specified/
601+
end
602+
end
603+
604+
context 'api_key also set' do
605+
let(:options) { { 'cloud_auth' => 'elastic:my-passwd-00', 'api_key' => 'some_key' } }
606+
607+
it "should fail" do
608+
expect { subject.register }.to raise_error LogStash::ConfigurationError, /Multiple authentication options are specified/
581609
end
582610
end
583611
end if LOGSTASH_VERSION > '6.0'
@@ -659,6 +687,62 @@
659687
end
660688
end
661689

690+
describe "API key" do
691+
let(:manticore_options) { subject.client.pool.adapter.manticore.instance_variable_get(:@options) }
692+
let(:api_key) { "some_id:some_api_key" }
693+
let(:base64_api_key) { "ApiKey c29tZV9pZDpzb21lX2FwaV9rZXk=" }
694+
695+
context "when set without ssl" do
696+
let(:do_register) { false } # this is what we want to test, so we disable the before(:each) call
697+
let(:options) { { "api_key" => api_key } }
698+
699+
it "should raise a configuration error" do
700+
expect { subject.register }.to raise_error LogStash::ConfigurationError, /requires SSL\/TLS/
701+
end
702+
end
703+
704+
context "when set without ssl but with a https host" do
705+
let(:do_register) { false } # this is what we want to test, so we disable the before(:each) call
706+
let(:options) { { "hosts" => ["https://some.host.com"], "api_key" => api_key } }
707+
708+
it "should raise a configuration error" do
709+
expect { subject.register }.to raise_error LogStash::ConfigurationError, /requires SSL\/TLS/
710+
end
711+
end
712+
713+
context "when set" do
714+
let(:options) { { "ssl" => true, "api_key" => ::LogStash::Util::Password.new(api_key) } }
715+
716+
it "should use the custom headers in the adapter options" do
717+
expect(manticore_options[:headers]).to eq({ "Authorization" => base64_api_key })
718+
end
719+
end
720+
721+
context "when not set" do
722+
it "should have no headers" do
723+
expect(manticore_options[:headers]).to be_empty
724+
end
725+
end
726+
727+
context 'user also set' do
728+
let(:do_register) { false } # this is what we want to test, so we disable the before(:each) call
729+
let(:options) { { "ssl" => true, "api_key" => api_key, 'user' => 'another' } }
730+
731+
it "should fail" do
732+
expect { subject.register }.to raise_error LogStash::ConfigurationError, /Multiple authentication options are specified/
733+
end
734+
end
735+
736+
context 'cloud_auth also set' do
737+
let(:do_register) { false } # this is what we want to test, so we disable the before(:each) call
738+
let(:options) { { "ssl" => true, "api_key" => api_key, 'cloud_auth' => 'foobar' } }
739+
740+
it "should fail" do
741+
expect { subject.register }.to raise_error LogStash::ConfigurationError, /Multiple authentication options are specified/
742+
end
743+
end
744+
end
745+
662746
@private
663747

664748
def stub_manticore_client!(manticore_double = nil)

0 commit comments

Comments
 (0)