Firehose Quickstart

Summary

• Explain some components, including why you'd want SSL
• -kentik_email field (optional)
• -how to get new relic api key
• -how to get nr account id

Pre-reqs

• Linux host
• Access to internet
• Docker running

Installation

Download the ktranslate image:

docker pull kentik/ktranslate:v2

Send flow data with:

docker run \
 -e NEW_RELIC_API_KEY=NRII-$NR_TOKEN \
 -e KENTIK_API_TOKEN=$KENTIK_TOKEN \
 -p 8082:8082 \
 --net=host \
 -name ktranslate \
 kentik/ktranslate:v2 \
  -nr_account_id $NR_ACCOUNT_ID \
  -kentik_email [email protected] \
  -metrics=jchf \
  -tee_logs=true \
  -nf.source=netflow5 \
  -nf.port=9995 \
  nr1.flow

Sample Output:

2021-07-06T21:51:19.581 ktranslate [Info] flow Netflow listener running on 0.0.0.0:9995 for format netflow5 and a batch size of 10000
2021-07-06T21:51:19.581 ktranslate [Info] flow Netflow listener sending fields Type,TimeReceived,SequenceNum,SamplingRate,SamplerAddress,TimeFlowStart,TimeFlowEnd,Bytes,Packets,SrcAddr,DstAddr,Etype,Proto,SrcPort,DstPort,InIf,OutIf,SrcMac,DstMac,SrcVlan,DstVlan,VlanId,IngressVrfID,EgressVrfID,IPTos,ForwardingStatus,IPTTL,TCPFlags,IcmpType,IcmpCode,IPv6FlowLabel,FragmentId,FragmentOffset,BiFlowDirection,SrcAS,DstAS,NextHop,NextHopAS,SrcNet,DstNet,HasMPLS,MPLSCount,MPLS1TTL,MPLS1Label,MPLS2TTL,MPLS2Label,MPLS3TTL,MPLS3Label,MPLSLastTTL,MPLSLastLabel
2021-07-06T21:51:19.581 ktranslate [Info] KTranslate System running with format new_relic, compression gzip, max flows: 10000, sample rate 1000:1 after 100

Optional: Pass in a MaxMind GeoIP DB to the --geo flag. For example: -geo /etc/country/GeoLite2-Country.mmdb

---

Send VPC logs to New Relic with:

docker run \
 -e NEW_RELIC_API_KEY=NRII-$NR_TOKEN \
 -e KENTIK_API_TOKEN=$KENTIK_TOKEN \
 -p 8082:8082 \
 kentik/ktranslate:v2 \
  -nr_account_id $NR_ACCOUNT_ID \
  -metrics=jchf \
  -tee_logs=true \
  -kentik_email [email protected] \
  nr1.vpc

Optional: generate SSL cert files via Lets Encrypt or other service. An example using Certbot would be sudo certbot certonly --standalone.

Run ktranslate, writing out into the influx format:

docker run \
 -v /etc/letsencrypt/live/$MYHOST:/ssl \
 -p 8082:8082
 kentik/ktranslate:v2 \
 -format influx \
 -log_level=info \
 -ssl_cert_file /ssl/fullchain.pem \
 -ssl_key_file /ssl/privkey.pem 

Point firehose at this endpoint:

https://kb.kentik.com/v0/Fc19.htm#Fc19-Firehose_Data_Sources

And you should see output. Next steps include sending data into New Relic.