Server side

Server side

The main tasks carried out by the server are:

1. Define the standard of the federation
2. Create the database representing the standard
3. Provide a web service for easy mapping of the local data to the estandard of the federation
4. Provide robust services for data extraction, trasformation and loading
5. Provide services for data visualization, exploration and sharing

The whole framework is mainly based on Elastic Stack:

1. Filebeat
2. Logstash
3. ElasticSearch
4. Kibana

Logstash "listen" to each Fileabeat through a selected port (default: 5066). In order to process each data source, Logstash uses configuration files. Each data site will have its particular configuration according to the results of the mapping. The configuration files are automatically created in the server by the logstash-configurator-generator and placed in folders where Logstash finds the configuration files. The ElasticSearch index is created in two steps:

1. Logstash process each filebeat data according to each configuration file and temporarily stores the data in a database (-we suggest using MySql database)
2. Logstash extract the data from the database and create or update the ElasticSearch index

It means that 2 instances of Logstash are runing in the server separately, one for processing and storing the beats in the database and another for creating the ElasticSearch index from the database.

Kibana is the visualization tool used by this framework to allow users explore the data from all the data sites. Once the index is created, it is visualized in Kibana. Kibana requires to be configured to visualized the index (or indexes creted by ElasticSearch).

For more information, please get familiar with the Elastic products.