aws_elastic_search.md

List ELK information

使用 AWS Elasticsearch Service

• Amazon Elasticsearch Service 入門 - 建立一個簡單的 ES 來測試

  • 建立的 command

aws es create-elasticsearch-domain --domain-name bruce --elasticsearch-version 7.1 --elasticsearch-cluster-config InstanceType=t2.small.elasticsearch,InstanceCount=1 --ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=10 --access-policies '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"*"},"Action":["es:*"],"Condition":{"IpAddress":{"aws:SourceIp":["60.251.40.253","118.163.170.73","35.165.85.252"]}}}]}'

* 查詢 es domain command

aws es describe-elasticsearch-domain --domain bruce

• 簽署 HTTP 請求到 Amazon Elasticsearch Service - 將資料寫入到 elastic search 中
• 將串流資料載入到 Amazon Elasticsearch Service
  • Streaming CloudWatch Logs Data to Amazon Elasticsearch Service
• Amazon Elasticsearch Service 中的 Identity and Access Management 有關 ES 的 ploicy 管理

將 cloudwatch 資料，餵到 ES 的設定

• Streaming CloudWatch Logs Data to Amazon Elasticsearch Service
• 將 CloudWatch Logs 資料串流至 Amazon Elasticsearch Service
• Function Execution Role for Service Principals
• AWS - subscribe multiple lambda logs to one elasticsearch service 設定多個 cloudWatch 到 aws es 時，會出現 error, 解法如文章所述
• Why can't create principal on aws iam for datadog? Principal 的設定是在建立 role 時，建立的

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "es:*"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:es:region:account-id:domain/target-domain-name/*"
        }
    ]
}

將 serverless log (lambda 資料) 寫入到 elastic search 中

• Lambda Logs in ELK - How to ingest AWS Lambda Log Streams from CloudWatch into an ELK monitoring stack ** 這篇來看看
• Streaming AWS CloudWatch Logs to your own ELK logging solution
• daniel-cottone/serverless-es-logs - A Serverless plugin to transport logs to ElasticSearch
• AWS Lambda and ELK

取得 cloudwatch 的資料

• Functionbeat: Serverless Ingestion for Elasticsearch
• Functionbeat overview

Query String

• Elasticsearch/Kibana Queries - In Depth Tutorial

kibana 說明文件

• Kibana Guide 7.4 » Getting Started
  • 由這份開始來試試 kibana
• Build your own dashboard - 這份滿好的，由匯入 data 開始，到建立 dashboard **

insert elasticSearch index

• How to use Python helpers to bulk load data into an Elasticsearch index
• How to use Bulk API to store the keywords in ES by using Python
• 要加入 timestamp 的方式
  • Add Timestamp to ElasticSearch with Elasticsearch-py using Bulk-API
  • Bulk APIedit

Amazon Elasticsearch Service Slow Logs

• Analyzing Amazon Elasticsearch Service Slow Logs Using Amazon CloudWatch Logs Streaming and Kibana ** 這篇值得一讀
• Viewing Amazon Elasticsearch Service Slow Logs

ELK course

• Complete Guide to Elasticsearch

  • codingexplained/complete-guide-to-elasticsearch
  • kibana demo

• console command on kibana

  • Nodes info API

將 mysql 轉換成 json

• sethblack/mysql-to-json Connects to a MySQL database and exports selected data to JSON.
  • pre-install pkg

$ sudo yum install mysql-devel
$ sudo yum list | grep python3 | grep dev
$ sudo yum install python36-devel

* example

$ mysql-to-json -e 'SELECT * FROM information_schema.tables' > tables.json

• 用自己改的一版來由 sql result to elasticsearch bulk json
  • wokuang/mysql-to-json

建立 date format 的方式

• 先建立 index, then 再用 mapping 的方式，先設定好 date format
• 範例

# 建立 index
PUT /odc

# 設定 index mapping
PUT /odc/_mapping
{
    "properties": {
      "updated_at": {
        "type":   "date",
        "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
      },
      "created_at": {
        "type":   "date",
        "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
      }
    }
}

用 logstash 傳資料到，在 AWS ES 上

• Setup a Logstash Server for Amazon Elasticsearch Service and Auth With IAM ** 這篇要看看
• Logstash tutorial: A quick start guide ** 這篇要看看
• Kibana and Logstash
• How do I integrate Logstash with Amazon's Elasticsearch Service (ES)?
• awslabs/logstash-output-amazon_es
• plugin 安裝

$ logstash-plugin install logstash-output-amazon_es

關於 index 操作的一些文件

• Index APIs ** 重要
  • Create index
  • Delete index
  • Put mapping index

一些常用到的 kibana console command

• query index

GET /_cat/indices?v

• 取得 index 資料

GET /index
GET /odc

• 取得 index stats

GET /_stats

geographical location

• Kibana GeoIP example: How to index geographical location of IP addresses into Elasticsearch
• GeoIP in the Elastic Stack
• How To Map User Location with GeoIP and ELK (Elasticsearch, Logstash, and Kibana)

使用 serverless 將資料放到 ES 中

• miztiik/serverless-s3-to-elasticsearch-ingester

在 AWS EC2 上安裝 docker - 目前改用 AWS 的 elasticsearch service 來玩

• Docker and Docker-Compose Setup on AWS EC2 Instance
• How to install Docker on AWS EC2 instance with AMI (CE/EE Update)

收集 AWS 使用資料

• Metricbeat overview 使用 metricbeat 來收集

收集 MySQL 資料

• AWS RDS Mysql SlowQuery monitoring on Kibana using Logstash

收集 lambda 資料

• 將 serverless log (lambda 資料) 寫入到 elastic search 中
• Centralized Logging

中文說明

• Elasticsearch权威指南（中文版） - 有空來看看

public cloud 的比較

• AWS Elasticsearch Service 对比：Amazon 和 Elastic
• Elasticsearch Service on Elastic Cloud AWS marketplace

將資料 feed 到 elastic search 的方式

• jrbeilke/logstash-lambda
  • AWS lambda function to ship ELB, S3, CloudTrail, VPC, CloudFront logs to Logstash
• Functionbeat: Serverless Ingestion for Elasticsearch

log

• Viewing Amazon Elasticsearch Service Error Logs

monitoring

• Monitoring a critical part of your infrastructure: Amazon Elasticsearch domain
• Monitoring AWS Lambda Invocations with the ELK Stack

reference

• Elasticsearch Tutorial
• Elasticsearch - Query DSL
• Elasticsearch Demo
• AWS Centralized Logging Guide