solidfire primary capacity is reset to zero bytes

[DaanHoogland]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Primary iSCSI solidfire

CLOUDSTACK VERSION

4.15.0 (after upgrade from 4.11.3)

CONFIGURATION

zone wide storage

OS / ENVIRONMENT

SUMMARY

a 4.11.3 environment with solidfire zone wide primary storage is upgraded to 4.15.0. After the upgrade the capacity of the storage is set to 0 bytes. After adjusting it it will be reset within a few minutes to 0 bytes again.

STEPS TO REPRODUCE

EXPECTED RESULTS

This happened in 4.11.3 and was expected in 4.15.0 as well

2021-04-12 10:01:25,843 DEBUG c.c.s.StorageManagerImpl (logid:) Found storage pool London Primary Storage of type Iscsi
2021-04-12 10:01:25,843 DEBUG c.c.s.StorageManagerImpl (logid:) Total over provisioned capacity of the pool London Primary Storage id: 2 is 17592186044416
2021-04-12 10:01:25,844 DEBUG c.c.s.StorageManagerImpl (logid:) Successfully set Capacity - 17592186044416 for capacity type - 3 , DataCenterId - 2, HostOrPoolId - 2, PodId null

ACTUAL RESULTS

2021-04-12 09:17:26,370 DEBUG c.c.s.StorageManagerImpl (logid:) Found storage pool London Primary Storage of type Iscsi
2021-04-12 09:17:26,371 DEBUG c.c.s.StorageManagerImpl (logid:) Total over provisioned capacity of the pool London Primary Storage id: 2 is (0 bytes) 0
2021-04-12 09:17:26,372 DEBUG c.c.s.StorageManagerImpl (logid:) Successfully set Capacity - (0 bytes) 0 for capacity type - 3 , DataCenterId - 2, HostOrPoolId - 2, PodId null

[yadvr]

Hi @mike-tutkowski - do you have any guidance for us?

[mike-tutkowski]

Hi @rhtyd: In the log lines above, it looks like the capacity was already at 0 by the time the createCapacityEntry method (which is where those log lines come from) is invoked.

I don't currently have a CloudStack environment up and running. Would you (or someone else) be able to re-set the capacity to the desired value, wait for it to be set back to 0, and then update this ticket with the log file?

Thanks!

[DaanHoogland]

I found this telling bit: probably why further calculation is stopped and 0 is passed to createCapacityEntry

2021-04-12 00:00:33,676 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-b9094408) (logid:fc1ff0ae) Seq 2-8552898642330477068: Received:  { Ans: , MgmtId: 2482650231502, via: 2(pool1-kvm1.sys.lon1.cloudsys.tmcs), Ver: v1, Flags: 10, { Answer } }
2021-04-12 00:00:33,676 DEBUG [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-b9094408) (logid:fc1ff0ae) Details from executing class com.cloud.agent.api.GetVolumeStatsCommand: com.cloud.utils.exception.CloudRuntimeException: Wrong format for iScsi path: f7bf6389-cb9a-4360-8116-2f2ed85b6220. It should be formatted as '/targetIQN/LUN'.
	at com.cloud.hypervisor.kvm.storage.IscsiAdmStorageAdaptor.getComponent(IscsiAdmStorageAdaptor.java:272)
	at com.cloud.hypervisor.kvm.storage.IscsiAdmStorageAdaptor.getIqn(IscsiAdmStorageAdaptor.java:257)
	at com.cloud.hypervisor.kvm.storage.IscsiAdmStorageAdaptor.getByPath(IscsiAdmStorageAdaptor.java:217)
	at com.cloud.hypervisor.kvm.storage.IscsiAdmStorageAdaptor.getPhysicalDisk(IscsiAdmStorageAdaptor.java:222)
	at com.cloud.hypervisor.kvm.storage.IscsiAdmStoragePool.getPhysicalDisk(IscsiAdmStoragePool.java:109)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)
	at com.cloud.agent.Agent.processRequest(Agent.java:640)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1053)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[mike-tutkowski]

I'm not really sure what this createCapacityEntry is exactly supposed to be doing, but does it apply for managed storage?

[yadvr]

cc @swill @skattoju4 - can you have a look at advise?

[yadvr]

@DaanHoogland when you say 'adjusting' it, was the storage pool capacity value manually updated using the API http://cloudstack.apache.org/api/apidocs-4.15/apis/updateStoragePool.html ?

Looking into the codebase, it looks if the capacity value is getting reset to 0, it's triggered via a thread, most likely on pool connect/add (when the mgmt server starts or kvm host connects I think) or likely by the stat collector:
https://github.com/apache/cloudstack/blob/4.15/server/src/main/java/com/cloud/server/StatsCollector.java#L1026

If it's the stats collector, then the GetStorageStatsCommand answer has capacity 0 returned. In case of solidfire on KVM, it could be resulting from https://github.com/apache/cloudstack/blob/4.15/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtGetStorageStatsCommandWrapper.java#L39 which calls over few methods/logic:
https://github.com/apache/cloudstack/blob/4.15/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/IscsiAdmStoragePool.java#L63

The code from the above class returns capacity as 0. @mike-tutkowski @swill @skattoju4 can you check/advise?

[yadvr]

I found some changes in code from 4.11.1 to 4.15, the logic in stat collector may end up updating for storage plugin it shouldn't which don't implement the total/used value to be returned. Proposed PR https://github.com/apache/cloudstack/pull/4918/files