Describe the bug
I'm testing geo-replication with subscription enabled. In my testing environment, there are 2 clusters (ClusterA and ClusterB) with a global zookeeper on a dedicated server host.
Each cluster also has its own producer and consumer client application which is based on Pulsar (Java) producer API and consumer Sync API.
Below is the summary of my testing steps:
- ProducerA publishes some messages (e.g. 10 messages 0 ~ 9) to a topic that is geo-replication enabled.
- ConsumerA receives the first half of the messages (e.g. msg 0 ~ 4) from the same topic and gets killed.
- After a short while (> 10s), starts ConsumerB in ClusterB to receive messages from the same topic.
I'm expecting ConsumerB to receive the second half of the messages (e.g. msg 5 ~ 9) if subscription replication is working properly. But the results shows that for most time, ConsumerB receives all 10 messages (e.g. msg 0 ~ 9). There are also several cases where ConsumerB receives 6 or 7 messages (e.g. msg 3 ~ 9 or 4 ~ 9).
My testing is based on Pulsar 2.6.3 with default configuration, with key geo-replication/subscription replication settings below:
broker.conf
enableReplicatedSubscriptions=true
replicatedSubscriptionsSnapshotFrequencyMillis=1000
The consumer application also has the following code added:
.replicateSubscriptionState(true)
To Reproduce
See "Description" section
Expected behavior
See "Description" section
Screenshots
Below is one example of the testing results;
- ProducerA publishes 6 messages to a geo-replication enabled topic "georep/default/t1"
$ ./producerA.sh -n 6 -t georep/default/t1
2021-03-24 06:51:54.965
---------------------------------------------------------
Cluster Name: ClusterA
Producer : [P]0090276c-8cd2-11eb-b297-d555334bf2d0
Topic : persistent://georep/default/t1
Number of messages to publish: 6
---------------------------------------------------------
message published: msg-key=[0] 20210324-065155-016, msg-payload=mN8etPUo9l0PDTW2cpa0
message published: msg-key=[1] 20210324-065155-089, msg-payload=z4tGKNSz4vmnqWIwIKO6kiclyhhso
message published: msg-key=[2] 20210324-065155-144, msg-payload=8T0O88q9upN1bqvEtIiEko208fHM4R0HMP1it1NycVkavxaR5b41OR7NX
message published: msg-key=[3] 20210324-065155-199, msg-payload=AlMs3I3C2xGa0BJu8MIv1o
message published: msg-key=[4] 20210324-065155-253, msg-payload=1CcO0W4qInSLYhOJrjvVMw04Ti32x8sk8xoyt4wwHGHL6GfctcvFKcf
message published: msg-key=[5] 20210324-065155-308, msg-payload=GtzTIPuBtsBG21sSiHv2CBEv0gdP4k
- ConsumerA receives 3 messages from this topic
$ ./consumerA.sh -sn mysub -t georep/default/t1 -n 3
2021-03-24 06:52:17.993
---------------------------------------------------------
Cluster Name: ClusterA
Consumer : [C]0e4d002b-8cd2-11eb-8164-69da960415c9
Topic : persistent://georep/default/t1
Subscription: mysub
---------------------------------------------------------
message received: msg-key=[0] 20210324-065155-016, msg-properties={cluster=ClusterA}, msg-payload=mN8etPUo9l0PDTW2cpa0, publish-time: 2021-03-24 06:51:55.021, receive-time: 2021-03-24 06:52:18.956
message received: msg-key=[1] 20210324-065155-089, msg-properties={cluster=ClusterA}, msg-payload=z4tGKNSz4vmnqWIwIKO6kiclyhhso, publish-time: 2021-03-24 06:51:55.089, receive-time: 2021-03-24 06:52:18.957
message received: msg-key=[2] 20210324-065155-144, msg-properties={cluster=ClusterA}, msg-payload=8T0O88q9upN1bqvEtIiEko208fHM4R0HMP1it1NycVkavxaR5b41OR7NX, publish-time: 2021-03-24 06:51:55.144, receive-time: 2021-03-24 06:52:18.957
- Wait for a short while and starts ConsumerB in ClusterB and tries to receive all remaining messages. But actually it receives all 6 messages, the first 3 of which have already been consumed and acknowledged by ClusterA
./consumerB.sh -sn mysub -t georep/default/t1
2021-03-24 06:52:54.635
---------------------------------------------------------
Cluster Name: ClusterB
Consumer : [C]24216084-8cd2-11eb-b01c-bff91b5df0bd
Topic : persistent://georep/default/t1
Subscription: mysub
---------------------------------------------------------
message received: msg-key=[0] 20210324-065155-016, msg-properties={cluster=ClusterA}, msg-payload=mN8etPUo9l0PDTW2cpa0, publish-time: 2021-03-24 06:51:55.021, receive-time: 2021-03-24 06:52:55.028
message received: msg-key=[1] 20210324-065155-089, msg-properties={cluster=ClusterA}, msg-payload=z4tGKNSz4vmnqWIwIKO6kiclyhhso, publish-time: 2021-03-24 06:51:55.089, receive-time: 2021-03-24 06:52:55.028
message received: msg-key=[2] 20210324-065155-144, msg-properties={cluster=ClusterA}, msg-payload=8T0O88q9upN1bqvEtIiEko208fHM4R0HMP1it1NycVkavxaR5b41OR7NX, publish-time: 2021-03-24 06:51:55.144, receive-time: 2021-03-24 06:52:55.029
message received: msg-key=[3] 20210324-065155-199, msg-properties={cluster=ClusterA}, msg-payload=AlMs3I3C2xGa0BJu8MIv1o, publish-time: 2021-03-24 06:51:55.199, receive-time: 2021-03-24 06:52:55.029
message received: msg-key=[4] 20210324-065155-253, msg-properties={cluster=ClusterA}, msg-payload=1CcO0W4qInSLYhOJrjvVMw04Ti32x8sk8xoyt4wwHGHL6GfctcvFKcf, publish-time: 2021-03-24 06:51:55.254, receive-time: 2021-03-24 06:52:55.030
message received: msg-key=[5] 20210324-065155-308, msg-properties={cluster=ClusterA}, msg-payload=GtzTIPuBtsBG21sSiHv2CBEv0gdP4k, publish-time: 2021-03-24 06:51:55.308, receive-time: 2021-03-24 06:52:55.030
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Describe the bug
I'm testing geo-replication with subscription enabled. In my testing environment, there are 2 clusters (ClusterA and ClusterB) with a global zookeeper on a dedicated server host.
Each cluster also has its own producer and consumer client application which is based on Pulsar (Java) producer API and consumer Sync API.
Below is the summary of my testing steps:
I'm expecting ConsumerB to receive the second half of the messages (e.g. msg 5 ~ 9) if subscription replication is working properly. But the results shows that for most time, ConsumerB receives all 10 messages (e.g. msg 0 ~ 9). There are also several cases where ConsumerB receives 6 or 7 messages (e.g. msg 3 ~ 9 or 4 ~ 9).
My testing is based on Pulsar 2.6.3 with default configuration, with key geo-replication/subscription replication settings below:
broker.conf
The consumer application also has the following code added:
To Reproduce
See "Description" section
Expected behavior
See "Description" section
Screenshots
Below is one example of the testing results;
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.