📋 User story

As a system I want a service that is able to refine the data and cache it in an appropriate data store

Description

The system needs as service that is able to consume data from rabbitMQ queues. Then take that data and refine it by adding or removing data attributes, and generally restructuring data. When the data is processed, it then needs to be stored persistently in a database layer for caching purposes, but should allow for enough abstraction that developers can add any other type of DB adapter The purpose is to have a service that takes care of processing data and storing it in MongoDB. In this use-case, if ElasticSearch crashes, it needs to have a place where it can restore data independently of the original source, ie. a "cache" server for history logs.

Prerequisites

• Setup a MongoDB database.

Acceptance Criteria

• The refiner needs to be able to subscribe to queues in RabbitMQ
• Based on the queue(subscription) you should be able to add different processors to the type of queue, ie. creating two outputs for one message. For instance originalData[A, B ]->["Hello A, Hello B"] AND ["Bye A, Bye B"].
• It needs to be able to perform the actions based; storing it in database, sending it to a queue.
• It needs to have an abstract database adapter, that is used for a MongoDB that can has the details on how to structure the data in the key-value store.
• Errors needs to be logged along with actions in the system and then parsed by a Humio formatter, and put in var/main.log.
• The micro-service needs to follow a 12-factor app methodology.