Polices On Topic

[codelipenghui]

This is the master ticket for tracking all the tasks for polices on topic

• 1. MessageTTL [Get, Set, Remove] Support set message TTL on topic level. #7738
• 2. Retention Policy [Get, Set, Remove] [Issue 2688]Support set retention on topic level. #7747
• 3. Message Dispatch Rate [Get, Set, Remove] support message dispatch rate on topic level #7863
• 4. Backlog Quota [Get, Set, Remove] [Issue 2689] Support set backlog quota on topic level. #7646
• 5. Persistence Policy [Get, Set, Remove] [ISSUE 7757] Support Persistence Policies on topic level #7817
• 6. Max Producer [Get, Set, Remove] [ISSUE 7758] Support set Max Producer on topic level. #7914
• 7. Max Consumer [Get, Set, Remove] [Issue 7759] Support set Max Consumer on topic level. #7968
• 8. Compaction Threshold [Get, Set, Remove] add compaction threshold for topic level #7881
• 9. Deduplication [Get, Set, Remove] Support deduplication on topic level #7821
• 10. Offload Policy [Get, Set, Remove] Support topicl level offload #7883
• 11. MaxUnackMessagesPerConsumer [Get, Set, Remove] Support MaxUnackedMessagesOnConsumer on topic level #7818
• 12. MaxUnackMessagesPerSubscription [Get, Set, Remove] Support set MaxUnackMessagesPerSubscription on topic level #7802
• 13. DelayedDeliveryPolicies [Get, Set, Remove] support topic level delayed delivery policy #7784
• 14. Inactive topic policy Support topic-level inactiveTopicPolicies #7986

[ivankelly]

@codelipenghui

This change will cause more load on zookeeper. For each topic loaded, we would have to do a read for the policy of that topic when loading the topic policy, even if no topic policy exists.

How does the broker get notified of updated to the policy for a topic? If we add a watcher, that's a watcher per topic. We should consider if there's another way of doing this without placing more load on ZK, such as in a 'system' topic.

[codelipenghui]

@ivankelly

This change will cause more load on zookeeper. For each topic loaded, we would have to do a read for the policy of that topic when loading the topic policy, even if no topic policy exists.

How does the broker get notified of updated to the policy for a topic? If we add a watcher, that's a watcher per topic. We should consider if there's another way of doing this without placing more load on ZK, such as in a 'system' topic.

Yes, this is a very necessary concern. I think 'system' topic is an excellent solution.

1. For policy changed, send message to the 'system' topic and write to zookeeper, then 'system' topic consumer update policy cache in broker. So that just need to read the cache.
2. For topic owner changed, load the namespace policy and topic policy into broker cache.

For 'system' topic design. I'm not sure the 'system' topic border.

1. A 'system' topic in a cluster
2. Namespace 'system' topic / Topic 'system' topic
3. 'system' topic for each namespace / 'system' topic for each topic

I'm prone to 1 or 2, for 3 too much topic will be created.

@ivankelly Can you give me some idea more ? Thanks

[ivankelly]

I would put it at the namespace level, so 2. It would be good to have a generic model for system topics, as currently it would require you wiring up something yourself. Maybe it would be better to put it at the managed ledger layer also, since if we put it at the pulsar level we would have a circular dependency of sorts.

[sijie]

@ivankelly I am a bit confused about using managed ledger. managed ledger doesn't have tailing facility, how do you propagate the policy changes using managed ledger?

[ivankelly]

You shouldn't need to propagate. When you update the policy, you're hitting the broker that owns the topic.

[sijie]

@ivankelly

How does the broker get notified of updated to the policy for a topic? If we add a watcher, that's a watcher per topic.

You shouldn't need to propagate. When you update the policy, you're hitting the broker that owns the topic.

I am lost now. Isn't the existence of system topic to address the concern of zookeeper notifications? so we convert the zookeeper notifications, to consume from system topic. penghui's previous comment was using system topic, which is making sense, since pulsar already has producer and consumer. but in your last comment, you suggest using mledger to avoid circular dependency, then my question is "how is possible that the other brokers get notified with policy changes, because mledger doesn't have tailing facility".

[ivankelly]

Isn't the existence of system topic to address the concern of zookeeper notifications

From what I can tell, system topics don't actually exist as an actual concept in pulsar, they're just normal topics. But as a conceptual concept, they don't exist just to replace notifications. They also exist as a place to store metadata which can scale.

Ok, so, to recap where the suggestion to use managed ledger came from...

The first suggestion was to put topic policies in zookeeper. This was discarded because of zookeeper scale issues.

So then it was suggested to use a system topic in pulsar. I'm questioning whether we should use a topic for policies on topics. What happens if the topic that contains the policy so has a topic policy? Maybe we need to prevent system topics from having topic level policies.

Otherwise, we put the "system topics" a lower layer and we wouldn't even have this problem. I suggested managed ledger, but dlog would also work here if we want notifications.

But stepping back, what I really want is that we define what system topics are and make them a reusable concept within the system.