Snowplow Elasticsearch Loader

Introduction

The Snowplow Elasticsearch Loader consumes Snowplow enriched events or failed events from an Amazon Kinesis stream or NSQ topic, transforms them to JSON, and writes them to Elasticsearch. Events which cannot be transformed or which are rejected by Elasticsearch are written to a separate Kinesis stream.

Building

Assuming you already have SBT installed:

$ git clone git://github.com/snowplow/snowplow-elasticsearch-loader.git
$ sbt compile

Usage

The Snowplow Elasticsearch Loader has the following command-line interface:

snowplow-elasticsearch-loader 2.1.3

Usage: snowplow-elasticsearch-loader [options]

  --config <filename>

Running

Create your own config file:

$ cp config/config.kinesis.reference.hocon my.conf

Update the configuration to fit your needs.

Next, start the loader, making sure to specify your new config file:

$ java -jar snowplow-elasticsearch-loader-2.1.3.jar --config my.conf

Find out more

Technical Docs	Setup Guide	Roadmap	Contributing
Technical Docs	Setup Guide	Roadmap	Contributing

Copyright and license

Copyright (c) 2014-present Snowplow Analytics Ltd. All rights reserved.

Licensed under the Snowplow Limited Use License Agreement. (If you are uncertain how it applies to your use case, check our answers to frequently asked questions.)