diff --git a/README.md b/README.md index 52f0680..399e212 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,20 @@ # grpcreplay [![golang-ci](https://github.com/vearne/grpcreplay/actions/workflows/golang-ci.yml/badge.svg)](https://github.com/vearne/grpcreplay/actions/workflows/golang-ci.yml) -GrpcReplay 是一个网络监控工具，可以记录您的 grpc流量(Unary RPC)，并将其用于灰度测试、压测或者流量分析。 +GrpcReplay is a network monitoring tool that can record your grpc traffic (Unary RPC) +and use it for grayscale testing, stress testing or traffic analysis. +* [中文 README](https://github.com/vearne/grpcreplay/blob/main/README_zh.md) -* [English README](https://github.com/vearne/grpcreplay/blob/main/README_en.md) +## Feature +* support filter +* support to parse Protobuf, requires grpc reflection [GRPC Server Reflection Protocol](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md#grpc-server-reflection-protocol) +* Supports various input and output plugins +* Supports multiple encoding forms of gRPC requests (can be easily extended) +* Support gRPC request replay -## 特性 -* 支持过滤器 -* 可以解析Protobuf,需要grpc反射,参考[GRPC Server Reflection Protocol](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md#grpc-server-reflection-protocol) -* 支持多种input和output -* 支持多种gRPC请求的编码形式(可以方便的扩展) -* 支持gRPC请求重放 - -## 编译 -### 安装libpcap +## Compile +### install libpcap Ubuntu ``` apt-get install -y libpcap-dev @@ -27,59 +27,61 @@ Mac ``` brew install libpcap ``` -### 编译 +### compile ``` make build ``` -## 原理 -1. 由于gRPC使用的Hpack来压缩头部，为了解决这个问题，使用了类似于tcpkill的机制，杀死旧连接，迫使client端发起新连接 -2. 使用gRPC的反射机制来获取Message的定义，以便能够解析gRPC请求 +## Principle +1. Since gRPC uses Hpack to compress the header, in order to solve this problem, a mechanism similar to tcpkill +is used to kill the old connection and force the client to initiate a new connection. +2. Use gRPC's reflection mechanism to get the definition of Message so that gRPC requests can be parsed. -## 架构图 +## Architecture ![architecture](https://github.com/vearne/grpcreplay/raw/main/img/grpc.svg) -## 注意（请务必阅读一下） -1. 暂时只支持h2c, 不支持h2 -2. 目前gRPC的编码只支持Protobuf。 -参考[encoding](https://github.com/grpc/grpc-go/blob/master/Documentation/encoding.md) -3. gPRC服务端需要开启反射 [GRPC Server Reflection Protocol](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md#grpc-server-reflection-protocol) -4. 只支持Unary RPC，不支持Streaming RPC -5. macOS上需要sudo +## Notice +1. Temporarily only supports h2c, not h2 +2. The current gRPC encoding only supports Protobuf. + refer to [encoding](https://github.com/grpc/grpc-go/blob/master/Documentation/encoding.md) +3. The gRPC server needs to enable reflection [GRPC Server Reflection Protocol](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md#grpc-server-reflection-protocol) +4. Only supports Unary RPC, not Streaming RPC +5. Root permissions required on macOS ``` sudo -s ``` -## 用法 -捕获"0.0.0.0:35001"上的gRPC请求，并打印在控制台中 +## Usage +Capture gRPC request on "0.0.0.0:35001" and print in console ``` ./grpcr --input-raw="0.0.0.0:35001" --output-stdout ``` -捕获"127.0.0.1:35001"上的gRPC请求，并打印在控制台中 +Capture gRPC request on "127.0.0.1:35001" and print in console ``` ./grpcr --input-raw="127.0.0.1:35001" --output-stdout ``` -捕获"127.0.0.1:35001"上的gRPC请求，发往"127.0.0.1:35002"， 同时打印在控制台中 +Capture the gRPC request on "127.0.0.1:35001", send it to "127.0.0.1:35002", and print it in the console ``` ./grpcr --input-raw="127.0.0.1:35001" --output-stdout --output-grpc="grpc://127.0.0.1:35002" ``` -指定codec 可选值: "simple" | "json" +Set the value of codec, optional value: "simple" | "json" ``` ./grpcr --input-raw="127.0.0.1:35001" --output-stdout --codec="simple" ``` - -捕获"127.0.0.1:35001"上的gRPC请求，并记录在某个文件夹中, 文件按照最大500MB的限制进行切分，并有压缩。 -注意: 目录必须已经存在且可以执行写入操作 +Capture the gRPC request on "127.0.0.1:35001" and record it in a folder. +The file is divided according to the maximum limit of 500MB and compressed. +Note: The directory must already exist and be writeable. ``` ./grpcr --input-raw="127.0.0.1:35001" --output-file-directory="/tmp/mycapture" --output-file-max-size=500 ``` -从某个文件夹中读取gRPC请求，进行重放。gRPC请求会被发往"127.0.0.1:35002"，同时打印在控制台中 +Read gRPC requests from a folder and replay them. gRPC requests will be sent to "127.0.0.1:35002" +and printed in the console ``` ./grpcr --input-file-directory="/tmp/mycapture" --output-stdout --output-grpc="grpc://127.0.0.1:35002" ``` -### 捕获的请求形如 +### The captured request looks like ``` { "headers": { @@ -99,31 +101,31 @@ sudo -s ``` -## 调试 -设置日志级别 -可选值: debug | info | warn | error +## Debug +Set the log level +Optional value: debug | info | warn | error ``` export SIMPLE_LOG_LEVEL=debug ``` -## 依赖 -本项目使用了[google/gopacket](https://github.com/google/gopacket)，因而依赖`libpcap` +## Dependency +This project uses [google/gopacket](https://github.com/google/gopacket) and therefore depends on `libpcap` -### 感谢 -受到 [fullstorydev/grpcurl](https://github.com/fullstorydev/grpcurl) -和 [buger/goreplay](https://github.com/buger/goreplay)的启发 +### Thanks +inspired by [fullstorydev/grpcurl](https://github.com/fullstorydev/grpcurl) +and [buger/goreplay](https://github.com/buger/goreplay) ## TODO -* [x] 1)发现与目标端口关联的所有连接 -* [x] 2)使用旁路阻断逐个结束这些连接 -* [x] 3)抓取目标端口上的请求并解析 -* [x] 4)GRPC请求重放 -* [x] 5)支持将GRPC请求写入控制台 -* [x] 6)支持将GRPC请求写入文件 -* [ ] 7)支持将GRPC请求写入kafka -* [ ] 8)支持将GRPC请求写入RocketMQ -* [x] 9)支持从文件中读取GRPC请求 -* [ ] 10)支持从kafka中读取GRPC请求 -* [ ] 11)支持从RocketMQ中读取GRPC请求 -* [ ] 12)支持自定义filter -* [ ] 13)支持TLS +* [x] 1)Discover all connections associated with the target port +* [x] 2)Use bypass blocking to end these connections one by one +* [x] 3)Grab the request on the target port and parse it +* [x] 4)GRPC request replay +* [x] 5)Supports writing GRPC requests to the console +* [x] 6)Support for writing GRPC requests to files +* [ ] 7)Support for writing GRPC requests to kafka +* [ ] 8)Support for writing GRPC requests to RocketMQ +* [x] 9)Support for reading GRPC requests from files +* [ ] 10)Support reading GRPC requests from kafka +* [ ] 11)Support for reading GRPC requests from RocketMQ +* [ ] 12)Support custom filter +* [ ] 13)support TLS diff --git a/README_en.md b/README_en.md deleted file mode 100644 index cd886ca..0000000 --- a/README_en.md +++ /dev/null @@ -1,129 +0,0 @@ -# grpcreplay -[![golang-ci](https://github.com/vearne/grpcreplay/actions/workflows/golang-ci.yml/badge.svg)](https://github.com/vearne/grpcreplay/actions/workflows/golang-ci.yml) - -GrpcReplay is a network monitoring tool that can record your grpc traffic (Unary RPC) -and use it for grayscale testing, stress testing or traffic analysis. - -## Feature -* support filter -* support to parse Protobuf, requires grpc reflection [GRPC Server Reflection Protocol](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md#grpc-server-reflection-protocol) -* Supports various input and output plugins -* Supports multiple encoding forms of gRPC requests (can be easily extended) -* Support gRPC request replay - -## Compile -### install libpcap -Ubuntu -``` -apt-get install -y libpcap-dev -``` -Centos -``` -yum install -y libpcap-devel -``` -Mac -``` -brew install libpcap -``` -### compile -``` -make build -``` - -## Principle -1. Since gRPC uses Hpack to compress the header, in order to solve this problem, a mechanism similar to tcpkill -is used to kill the old connection and force the client to initiate a new connection. -2. Use gRPC's reflection mechanism to get the definition of Message so that gRPC requests can be parsed. - -## Architecture -![architecture](https://github.com/vearne/grpcreplay/raw/main/img/grpc.svg) - -## Notice -1. Temporarily only supports h2c, not h2 -2. The current gRPC encoding only supports Protobuf. - refer to [encoding](https://github.com/grpc/grpc-go/blob/master/Documentation/encoding.md) -3. The gRPC server needs to enable reflection [GRPC Server Reflection Protocol](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md#grpc-server-reflection-protocol) -4. Only supports Unary RPC, not Streaming RPC -5. Root permissions required on macOS -``` -sudo -s -``` - -## Usage -Capture gRPC request on "0.0.0.0:35001" and print in console -``` -./grpcr --input-raw="0.0.0.0:35001" --output-stdout -``` -Capture gRPC request on "127.0.0.1:35001" and print in console -``` -./grpcr --input-raw="127.0.0.1:35001" --output-stdout -``` -Capture the gRPC request on "127.0.0.1:35001", send it to "127.0.0.1:35002", and print it in the console -``` -./grpcr --input-raw="127.0.0.1:35001" --output-stdout --output-grpc="grpc://127.0.0.1:35002" -``` - -Set the value of codec, optional value: "simple" | "json" -``` -./grpcr --input-raw="127.0.0.1:35001" --output-stdout --codec="simple" -``` -Capture the gRPC request on "127.0.0.1:35001" and record it in a folder. -The file is divided according to the maximum limit of 500MB and compressed. -Note: The directory must already exist and be writeable. -``` -./grpcr --input-raw="127.0.0.1:35001" --output-file-directory="/tmp/mycapture" --output-file-max-size=500 -``` -Read gRPC requests from a folder and replay them. gRPC requests will be sent to "127.0.0.1:35002" -and printed in the console -``` -./grpcr --input-file-directory="/tmp/mycapture" --output-stdout --output-grpc="grpc://127.0.0.1:35002" -``` - -### The captured request looks like -``` -{ - "headers": { - ":authority": "localhost:35001", - ":method": "POST", - ":path": "/SearchService/Search", - ":scheme": "http", - "content-type": "application/grpc", - "te": "trailers", - "testkey1": "testvalue1", - "testkey2": "testvalue2", - "user-agent": "grpc-go/1.48.0" - }, - "method": "/SearchService/Search", - "request": "{\"staffName\":\"zhangsan\",\"gender\":true,\"age\":405084}" -} -``` - - -## Debug -Set the log level -Optional value: debug | info | warn | error -``` -export SIMPLE_LOG_LEVEL=debug -``` - -## Dependency -This project uses [google/gopacket](https://github.com/google/gopacket) and therefore depends on `libpcap` - -### Thanks -inspired by [fullstorydev/grpcurl](https://github.com/fullstorydev/grpcurl) -and [buger/goreplay](https://github.com/buger/goreplay) - -## TODO -* [x] 1)Discover all connections associated with the target port -* [x] 2)Use bypass blocking to end these connections one by one -* [x] 3)Grab the request on the target port and parse it -* [x] 4)GRPC request replay -* [x] 5)Supports writing GRPC requests to the console -* [x] 6)Support for writing GRPC requests to files -* [ ] 7)Support for writing GRPC requests to kafka -* [ ] 8)Support for writing GRPC requests to RocketMQ -* [x] 9)Support for reading GRPC requests from files -* [ ] 10)Support reading GRPC requests from kafka -* [ ] 11)Support for reading GRPC requests from RocketMQ -* [ ] 12)Support custom filter -* [ ] 13)support TLS diff --git a/README_zh.md b/README_zh.md new file mode 100644 index 0000000..2a33432 --- /dev/null +++ b/README_zh.md @@ -0,0 +1,129 @@ +# grpcreplay +[![golang-ci](https://github.com/vearne/grpcreplay/actions/workflows/golang-ci.yml/badge.svg)](https://github.com/vearne/grpcreplay/actions/workflows/golang-ci.yml) + +GrpcReplay 是一个网络监控工具，可以记录您的 grpc流量(Unary RPC)，并将其用于灰度测试、压测或者流量分析。 + + +* [English README](https://github.com/vearne/grpcreplay/blob/main/README.md) + +## 特性 +* 支持过滤器 +* 可以解析Protobuf,需要grpc反射,参考[GRPC Server Reflection Protocol](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md#grpc-server-reflection-protocol) +* 支持多种input和output +* 支持多种gRPC请求的编码形式(可以方便的扩展) +* 支持gRPC请求重放 + +## 编译 +### 安装libpcap +Ubuntu +``` +apt-get install -y libpcap-dev +``` +Centos +``` +yum install -y libpcap-devel +``` +Mac +``` +brew install libpcap +``` +### 编译 +``` +make build +``` + +## 原理 +1. 由于gRPC使用的Hpack来压缩头部，为了解决这个问题，使用了类似于tcpkill的机制，杀死旧连接，迫使client端发起新连接 +2. 使用gRPC的反射机制来获取Message的定义，以便能够解析gRPC请求 + +## 架构图 +![architecture](https://github.com/vearne/grpcreplay/raw/main/img/grpc.svg) + +## 注意（请务必阅读一下） +1. 暂时只支持h2c, 不支持h2 +2. 目前gRPC的编码只支持Protobuf。 +参考[encoding](https://github.com/grpc/grpc-go/blob/master/Documentation/encoding.md) +3. gPRC服务端需要开启反射 [GRPC Server Reflection Protocol](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md#grpc-server-reflection-protocol) +4. 只支持Unary RPC，不支持Streaming RPC +5. macOS上需要sudo +``` +sudo -s +``` + +## 用法 +捕获"0.0.0.0:35001"上的gRPC请求，并打印在控制台中 +``` +./grpcr --input-raw="0.0.0.0:35001" --output-stdout +``` +捕获"127.0.0.1:35001"上的gRPC请求，并打印在控制台中 +``` +./grpcr --input-raw="127.0.0.1:35001" --output-stdout +``` +捕获"127.0.0.1:35001"上的gRPC请求，发往"127.0.0.1:35002"， 同时打印在控制台中 +``` +./grpcr --input-raw="127.0.0.1:35001" --output-stdout --output-grpc="grpc://127.0.0.1:35002" +``` + +指定codec 可选值: "simple" | "json" +``` +./grpcr --input-raw="127.0.0.1:35001" --output-stdout --codec="simple" +``` + +捕获"127.0.0.1:35001"上的gRPC请求，并记录在某个文件夹中, 文件按照最大500MB的限制进行切分，并有压缩。 +注意: 目录必须已经存在且可以执行写入操作 +``` +./grpcr --input-raw="127.0.0.1:35001" --output-file-directory="/tmp/mycapture" --output-file-max-size=500 +``` +从某个文件夹中读取gRPC请求，进行重放。gRPC请求会被发往"127.0.0.1:35002"，同时打印在控制台中 +``` +./grpcr --input-file-directory="/tmp/mycapture" --output-stdout --output-grpc="grpc://127.0.0.1:35002" +``` + +### 捕获的请求形如 +``` +{ + "headers": { + ":authority": "localhost:35001", + ":method": "POST", + ":path": "/SearchService/Search", + ":scheme": "http", + "content-type": "application/grpc", + "te": "trailers", + "testkey1": "testvalue1", + "testkey2": "testvalue2", + "user-agent": "grpc-go/1.48.0" + }, + "method": "/SearchService/Search", + "request": "{\"staffName\":\"zhangsan\",\"gender\":true,\"age\":405084}" +} +``` + + +## 调试 +设置日志级别 +可选值: debug | info | warn | error +``` +export SIMPLE_LOG_LEVEL=debug +``` + +## 依赖 +本项目使用了[google/gopacket](https://github.com/google/gopacket)，因而依赖`libpcap` + +### 感谢 +受到 [fullstorydev/grpcurl](https://github.com/fullstorydev/grpcurl) +和 [buger/goreplay](https://github.com/buger/goreplay)的启发 + +## TODO +* [x] 1)发现与目标端口关联的所有连接 +* [x] 2)使用旁路阻断逐个结束这些连接 +* [x] 3)抓取目标端口上的请求并解析 +* [x] 4)GRPC请求重放 +* [x] 5)支持将GRPC请求写入控制台 +* [x] 6)支持将GRPC请求写入文件 +* [ ] 7)支持将GRPC请求写入kafka +* [ ] 8)支持将GRPC请求写入RocketMQ +* [x] 9)支持从文件中读取GRPC请求 +* [ ] 10)支持从kafka中读取GRPC请求 +* [ ] 11)支持从RocketMQ中读取GRPC请求 +* [ ] 12)支持自定义filter +* [ ] 13)支持TLS