Performance degradation in a global NATS super-cluster

[Choi-Sung-Hoon]

Hello there,

I have a NATS super-cluster with multiple local clusters around the world and there's a performance degradation when I test nats bench pub and nats bench sub for clusters.

My cluster topology is as below:

• South Korea (IDC 1 + IDC 2)
• Singapore
• US West
• US East
• Germany

Each cluster has 3 NATS server nodes and they are configured as a cluster in each local region and the clusters are configured as a super-cluster in a global environment.

Case A

• nats bench pub --size="4KB" perf-test --context korea-idc1
• nats bench sub --size="4KB" perf-test --context korea-idc2

In this case, NATS bench results in about 70,000 ~ 80,000 msgs / sec.

They are in the same region(= South Korea) and the benchmark result looks good.

Case B

• nats bench pub --size="4KB" perf-test --context korea-idc1
• nats bench sub --size="4KB" perf-test --context use

In this case, NATS bench results in 2,000 msgs / sec.

I assume that this is because of physical distance between South Korea and US.

Case C

• nats bench pub --size="4KB" perf-test --context korea-idc1
• nats bench sub --size="4KB" perf-test --context korea-idc2
• nats bench sub --size="4KB" perf-test --context use

In this case, NATS pub/sub consists of few clusters.

The delivery rate between IDCs in South Korea should be around 70,000 ~ 80,000 msgs / sec like case A, as they are in the same region, but the actual result is as below:

The message delivery rate in the same region is expected to be 70,000 ~ 80,000 msgs / sec as case A, but it's actually around 2,000 / sec.

It seems like a cluster is affected by other clusters when they are subscribing same subject.

Is this an expected behavior? or is there something that I'm missing?