pipelines.md

Pipelines

monasca-docker is home for two pipelines, one for processing metrics and the other one for processing logs. Each pipeline requires a set of services to be complete.

Metric pipeline

Metrics pipeline requires the following services to be launched or available (providing they are hosted externally):

• agent-forwarder - launches monasca-agent-forwarder
• agent-collector - launches monasca-agent-collector
• zookeeper - launches Zookeeper
• influxdb - launches InfluxDB
• kafka - launches Kafka
• thresh - launches monasca-thresh
• monasca-sidecar - launches monasca-sidecar
• monasca-persister- launches monasca-persister
• monasca - launches monasca-api
• mysql - launches MySQL
• keystone - launches Keystone
• memcached - launches memcache
• monasca-notification - launches monasca-notification
• grafana - launches Grafana
• cadvisor - launched cadvisor

Some of the services require initialization prior to launching, hence the following are available:

• mysql-init for mysql
• kafka-init for kafka
• influxdb-init for influxdb
• grafana-init for grafana

There is also:

• alarms - that can be used to pre-create some alarm-definitions and notification-methods.

Log pipeline

Logs pipeline requires the following services to be launched or available (providing they are hosted externally):

• log-metrics - launches monasca-log-metrics
• log-api - launches monasca-log-api
• log-persister - launches monasca-log-persister
• kibana - launches Kibana
• log-transformer - launches monasca-log-transformer
• elasticsearch - launches ElasticSearch

Some of the services require initialization prior to launching, hence the following are available:

• elasticsearch-init for elasticsearch
• kafka-log-init for kafka

Dependencies

The majority of services that are part of monasca-docker require one or more additional services. The way to find out what dependencies a service has, is to open docker-compose.yml or log-pipeline.yml (depends on the service) and check if its definition contains depends_on. For example:

Kafka from docker-compose.yml definition is:

kafka:
  image: monasca/kafka:${MON_KAFKA_VERSION}
  depends_on:
    - zookeeper

It depends_on on Zookeeper. That means that either:

• Zookeeper needs to be launched with docker-compose

• Zookeeper should be provided externally and referenced with ZOOKEEPER_CONNECTION_STRING environmental variable

  If the components are hosted externally, you should remove depends_on. Otherwise docker-compose will try to launch the required services.