Ingestion service that communicates with a message broker

[alex855k]

📋 Epic

As a system I want an API gateway that communicates with a with a message broker micro-service architecture, so that I can send events to different smaller services, to make it easier to manage, extend and develop new services.

Description

It necessary to have an API, where it's possible to interact with the system. It also needs to be able to broker the different type of messages to an event based architecture. Which will make it possible to decouple the services needed for the system and easier to maintain these micro services.
There needs to be an ingest service, which will renew data from all sources at a specific interval. To create the API the team will be using .Net Core 2.x and RabbitMQ as a message broker middle-ware.

List of requirements

• As a system I want a service that can ingest data from an endpoint at regular intervals from the latest cached data entry point, so I can update the data in a cache database with the latest logs entries Ingestion service that fetches data from endpoint at regular intervals #12
• As a system I want to be able to split messages into queues and distribute it asynchronously to different types of consumers, so it that will scale in a micro-service based cloud architecture as additional services are added on. #

Acceptance Criteria

• All system components needs to log errors and actions in /var/main.log with logs that is parsed with a Humio formatter. Prerequisite Format system log files in a Humio format #13
• Every part of the solution needs to be unit tested as a minimum.
• Create functional tests that test if all components of the system works in conjection to eachother.

Non-functional requirements

• Needs to be asynchronous.
• Needs to be OS agnostic.
• The solution should be elastic and make the consumers able to scale with the amount of workload from a minimum to infinite load, based on performance metrics. And have a workload balance in place.

[IulianBogdan]

Outdated