Skip to content

Latest commit

 

History

History
245 lines (177 loc) · 11.4 KB

pipeline_aws_kafka_etcd.md

File metadata and controls

245 lines (177 loc) · 11.4 KB

How to deploy Kafka on Kubernetes using etcd on AWS with Banzai Cloud's Pipeline

In this tutorial we will guide you through how to deploy a 'Kafta on Kubernetes - using etcd' application from the ground on AWS.

At first you have to deploy our Pipeline Control Plane which takes care all the hard work.

Pipeline Control Plane

Prerequisites of hosting the control plane on AWS

Hosting Pipeline Control Plane and creating Kubernetes clusters on AWS

  1. AWS account
  2. AWS EC2 key pair

Launch Pipeline Control Plane on AWS

The easiest way for running a Pipeline Control Plane is to use a Cloudformation template.

`https://s3-eu-west-1.amazonaws.com/cf-templates-grr4ysncvcdl-eu-west-1/2018026em9-new.templatee93ate9mob7`

 <a href="https://raw.githubusercontent.com/banzaicloud/pipeline/master/docs/images/howto/ControlPlaneFromTemplate.png" target="_blank"><img src="https://raw.githubusercontent.com/banzaicloud/pipeline/master/docs/images/howto/ControlPlaneFromTemplate.png" height="230"></a>

  • Fill in the following fields on the form:

    • Stack name

      • specify a name for the Control Plane deployment

    • AWS Credentials

    • Azure Credentials and Information - needed only for creating Kubernetes clusters on Azure

      • AzureClientId - see how to get Azure Client Id above
      • AzureClientSecret - see how to get Azure Client Secret above
      • AzureSubscriptionId - your Azure Subscription Id
      • AzureTenantId - see how to get Azure Client Tenant Id above

    • Control Plane Instance Config

      • InstanceName - name of the EC2 instance that will host the Control Plane
      • ImageId - pick the image id from the README
      • KeyName - specify your AWS EC2 key pair

    • Banzai Pipeline Credentials

      • Pipeline API Password - specify the password for accessing the Pipeline REST API exposed by the Pipeline PaaS. Take note of the user name and password as those will be required when setting the secrets for the GitHub repositories in the CI/CD workflow.

    • Banzai-Ci Credentials

      • Orgs - comma-separated list of Github organizations whose members to grant access to use Banzai Cloud Pipeline's CI/CD workflow
      • Github Client - GitHub OAuth Client Id
      • Github Secret - Github OAuth Client Secret

    • Grafana Dashboard

      • Grafana Dashboard Password - specify password for accessing Grafana dashboard with defaults specific to the application

    • Prometheus Dashboard

      • Prometheus Password - specify password for accessing Prometheus that collects cluster metrics

    • Advanced Pipeline Options

      • PipelineImageTag - specify 0.2.0 for using current stable Pipeline release.

    • Slack Credentials

      • this section is optional. Complete this section to receive cluster related alerts through a Slack push notification channel.

    • Alert SMTP Credentials

      • this section is optional. Fill this section to receive cluster related alerts through email.

  • Finish the wizard to create a Control Plane instance.

  • Take note of the PublicIP of the created Stack. We refer to this as the PublicIP of Control Plane.

  • Go back to the earlier created GitHub OAuth application and modify it. Set the Authorization callback URL field to http://{control_plane_public_ip}/authorize

Deploying Kafka on Kubernetes using Pipeline

Now our Pipeline Control Plane is ready to help us to deploy applications in a cloud native way.

We will communicate with the CP through it's REST interface. In the near future you will be able manage these steps on a fancy UI.

Let's start the process:

At first we have to create a Kubernetes cluster where our Kafka app will be deployed to:

curl --request POST \
  --url 'http://{PublicIPOfYourCP}/pipeline/api/v1/clusters' \
  --header 'Authorization: Basic {Base64EncodedStringofYourCPCredetial(username:password)}' \
  --header 'Content-Type: application/json' \
  --data '{
  "name":"awscluster-tutorial-1",
    "location": "eu-west-1",
    "cloud": "amazon",
    "nodeInstanceType": "m4.xlarge",
    "properties": {
        "amazon": {
            "node": {
                "spotPrice": "0.2",
                "minCount": 1,
                "maxCount": 2,
                "image": "ami-06d1667f"
            },
            "master": {
                "instanceType": "m4.xlarge",
                "image": "ami-06d1667f"
            }
        }
    }
}'

As you can see we are going to create our cluster in the "eu-west-1" location, the master and the worker nodes will also be deployed to an "m4.xlarge" instance. We are trying to run our worker nodes on a spot instance and our offer is USD 0.2 for them.

The output looks like this:

{"Ip":"x.x.x.x","message":"Cluster created successfully!","name":"awscluster-tutorial-1","resourceId":2,"status":201}

In the output you can find the IP address of the master node, the name and the id of the created cluster.

Now our Kubernetes cluster is being created and will be ready after couple of minutes.

You can check the Cluster status by the following API call:

curl --head \
  --url 'http://x.x.x.x/pipeline/api/v1/clusters/2' \
  --header 'Authorization: Basic YWRtaW46UGFzczEyMzQ=' \
  --header 'Content-Type: application/json'

The result should look like this:

 HTTP/1.1 200 OK

To deploy our Kafka solution to our newly created Kubernetes cluster use the following call:

curl --request POST \
   --url 'http://x.x.x.x/pipeline/api/v1/clusters/2/deployments' \
   --header 'Authorization: Basic YWRtaW46UGFzczEyMzQ=' \
   --header 'Content-Type: application/json' \
   --data '{"name": "banzaicloud-stable/kafka"}'

And the result:

{"notes":"","release_name":"wrinkled-aardwolf","status":201}

The following call help us to check our deployments on the given cluster:

 curl --head --url 'http://x.x.x.x/pipeline/api/v1/clusters/2/deployments'  \
  --header 'Authorization: Basic YWRtaW46UGFzczEyMzQ=' \
  --header 'Content-Type: application/json'

If everything is fine then we should see this result:

 HTTP/1.1 200 OK

Produce and Consume messages

At this point your Kafka cluster is only accessible inside the Kubernetes cluster, so you have to create a kafka-test pod with the following yaml:

By using Pipeline spotguides this process is automated

apiVersion: v1
kind: Pod
metadata:
  name: kafka-test
spec:
  containers:
  - name: kafka-test
    image: banzaicloud/kafka:2.12-1.2.0-etcd-0.0.1
    # Just spin & wait forever
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 3000; done;" ]

This creates a simple pod which will be available for trying out Kafka (kubectl create -f kafka-test.yaml). The next Pipeline release will contain the Kafka spotguide as well, thus Kafka will be accessible from outside as well. Now exec into this pod by using: kubectl exec -it kafka-test bash. Once you are inside the container you should create a topic first:

./bin/kafka-topics.sh --zookeeper etcd://etcd-cluster-client:2379 --create --topic kafka-test --partitions 1 --replication-factor 3
Created topic "kafka-test".

When you are done let's produce some messages:

root@kafka-test:/opt/kafka# ./bin/kafka-console-producer.sh --broker-list bootstrap:9092 --topic kafka-test
>welcome
>kafka
>on
>etcd
>good
>you
>are
>here

Let's consume these messages:

./bin/kafka-console-consumer.sh --bootstrap-server bootstrap:9092 --topic kafka-test --from-beginning
welcome
kafka
on
etcd
good
you
are
here

As you see all the messages arrived from the producer side.