Skip to content
This repository was archived by the owner on Jan 29, 2024. It is now read-only.

GetLinkfire/bigdata-devops-assignment

Folders and files

NameName
Last commit message
Last commit date

Latest commit

author
Kristian Østergaard Martensen
Aug 29, 2018
bf27400 · Aug 29, 2018

History

7 Commits
Aug 29, 2018
Jun 8, 2018
Jun 8, 2018
Jun 8, 2018
Jun 8, 2018
Jun 15, 2018
Jun 8, 2018
Jun 8, 2018
Jun 8, 2018
Jun 8, 2018

Repository files navigation

Linkfire Big Data DevOps assignment

What to do?

First of all, clone this repository:

git clone git@github.com:getlinkfire/bigdata-devops-assignment.git

Now, given this Docker Compose file, stand up a Druid cluster.

The compose file is going to launch :

  • 1 zookeeper node
  • 1 MySQL database
  • 1 Kafka message broker

and the following druid services :

  • 1 broker
  • 1 middlemanager
  • 1 historical
  • 1 coordinator/overlord

This will take a couple of minutes, but once it is up, you will have:

Now that you have a running cluster, ingest the data found in data/wikiticker-2015-09-12-sampled.json.gz.
Once your ingestion job has succesfully executed, perform a simple query on the freshly created datasource.

The dataset is kindly borrowed from the Druid quickstart tutorial, you may find some inspiration there.
Read a little about the Druid indexing service if you like ;)

Extra assignments

Piece of cake was it?
Alright, while the above should get us off the ground talking about troubleshooting skills and JVM heap configuration (hint), if you feel courageous; go ahead and setup realtime ingestion through Kafka, into the wikiticker datasource, and query your data back out again with a select query.

Also, the attentive ones will notice that we have configured Druid to emit metrics into Kafka. Now create a new Druid datasource, and ingest those metrics back into Druid where we can query them. You may find a bit of inspiration in this blog post.

Came here by accident?

Well a good puzzle is hard to resist isn't it?
If you found your way through our little builtin quirks and gotchas out of pure curiosity, we would absolutely love to hear from you. Please write us an email on devops@linkfire.com, tell us what we misconfigured and what you did to make this setup shine.

About

Test assignment for DevOps engineers: A little something to talk about during our job interview

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published