Unable to get web service url when split bundle

[BewareMyPower]

Describe the bug
When I tried to split a bundle range in a single node cluster, the weird error logs Unable to get web service url occurred.

To Reproduce
Deploy a single node cluster.

Changes to conf/bookie.conf:

advertisedAddress=0.0.0.0
prometheusStatsHttpPort=8001
httpServerPort=8001

Changes to conf/broker.conf:

zookeeperServers=localhost:2181
configurationStoreServers=localhost:2181
managedLedgerDefaultEnsembleSize=1
managedLedgerDefaultWriteQuorum=1
managedLedgerDefaultAckQuorum=1
loadBalancerAutoBundleSplitEnabled=false
clusterName=standalone
defaultNumberOfNamespaceBundles=4

1. Start a ZooKeeper

./bin/pulsar zookeeper

2. Initialize metadata

bin/pulsar initialize-cluster-metadata \
--cluster standalone \
--zookeeper localhost:2181 \
--configuration-store localhost:2181 \
--web-service-url http://localhost:8080 \
--broker-service-url pulsar://localhost:6650

3. Start a Bookie

./bin/pulsar bookie

4. Start a Broker

./bin/pulsar broker

(Here we use 3 terminals for step above)

Finally run following application:

    public static void main(String[] args) throws PulsarClientException, PulsarAdminException {
        final String namespace = "public/default";
        final String topic = "my-topic";
        final int numPartitions = 16;
        try (PulsarAdmin admin = PulsarAdmin.builder().serviceHttpUrl("http://localhost:8080").build()) {
            System.out.println(admin.namespaces().getBundles(namespace).getBoundaries());
            admin.topics().createPartitionedTopic(topic, numPartitions);
            try (PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build()) {
                for (int i = 0; i < numPartitions; i++) {
                    client.newProducer().topic(topic + TopicName.PARTITIONED_TOPIC_SUFFIX + i)
                            .create().send("1".getBytes());
                }
            }
            final BundlesData data = admin.namespaces().getBundles(namespace);
            System.out.println(data.getBoundaries());
            final String bundle = data.getBoundaries().get(0) + "_" + data.getBoundaries().get(1);
            admin.namespaces().splitNamespaceBundle(namespace, bundle, true, null);
            System.out.println(admin.namespaces().getBundles(namespace).getBoundaries());
        }
    }

We can see the topic creation, produce and get bundles operations all succeeded. However, the splitNamespaceBundle failed with

2022-03-13 01:28:31:736%   [AsyncHttpClient-7-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:8080/admin/v2/namespaces/public/default/0x00000000_0x10000000/split?unload=true] Failed to perform http put request: org.apache.pulsar.shade.javax.ws.rs.ClientErrorException: HTTP 412 Precondition Failed
Exception in thread "main" org.apache.pulsar.client.admin.PulsarAdminException$PreconditionFailedException: Failed to find ownership for ServiceUnit:public/default/0x00000000_0x10000000

The broker's warning log is weird:

2022-03-13T01:28:31,722+0800 [pulsar-web-36-12] INFO  org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [13/三月/2022:01:28:31 +0800] "GET /admin/v2/namespaces/public/default/bundles HTTP/1.1" 200 253 "-" "Pulsar-Java-v2.9.1" 3
2022-03-13T01:28:31,728+0800 [pulsar-web-36-14] INFO  org.apache.pulsar.broker.admin.impl.NamespacesBase - [null] Split namespace bundle public/default/0x00000000_0x10000000
2022-03-13T01:28:31,730+0800 [pulsar-web-36-14] WARN  org.apache.pulsar.broker.web.PulsarWebResource - Unable to get web service url
2022-03-13T01:28:31,735+0800 [pulsar-web-36-14] INFO  org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [13/三月/2022:01:28:31 +0800] "PUT /admin/v2/namespaces/public/default/0x00000000_0x10000000/split?unload=true HTTP/1.1" 412 90 "-" "Pulsar-Java-v2.9.1" 9

The standard output is also weird:

[0x00000000, 0x10000000, 0x20000000, 0x30000000, 0x40000000, 0x50000000, 0x60000000, 0x70000000, 0x80000000, 0x90000000, 0xa0000000, 0xb0000000, 0xc0000000, 0xd0000000, 0xe0000000, 0xf0000000, 0xffffffff]

The defaultNumberOfNamespaceBundles is 4, but the initial number of bundles is 16.

It works well in a Pulsar standalone.

Expected behavior
It should succeed as it runs in a Pulsar standalone. Or the broker should log the meaningful message.

[Technoboy-]

We need to make a little change in splitNamespaceBundle

[BewareMyPower]

@Technoboy- I see. But do you know what makes the difference between standalone and the cluster mode? The same application runs well in standalone mode.

[codelipenghui]

@BewareMyPower In standalone, only have one broker, so no matter the readOnly is false or true, the broker will return the correct owner. I'm not sure if this is a breaking change in 2.8.x, 2.9.x, 2.10.x, or for all versions. Which pulsar version you have encountered this issue?

[codelipenghui]

Oh, sorry, If the bundle does not been loaded before, the issue also happens.

[BewareMyPower]

I'm not sure if this is a breaking change in 2.8.x, 2.9.x, 2.10.x, or for all versions. Which pulsar version you have encountered this issue?

I tried 2.9.1. I'll test other versions soon.

[BewareMyPower]

In standalone, only have one broker, so no matter the readOnly is false or true, the broker will return the correct owner.

But we also have only one broker in my test env. I deployed only one broker, one bookie, one zookeeper.