server: create DB entry for storage pool capacity when create storage pool by ustcweizhou · Pull Request #4805 · apache/cloudstack

ustcweizhou · 2021-03-12T14:08:30Z

Description

This PR fixes the issue that new storage pool can not be used in the first few minute after adding to cloudstack.

steps to reproduce the issue
(1) change global setting vm.allocation.algorithm to 'firstfitleastconsumed' (this allocator orders host/storage by capacity, maybe other algorithim has same issue).
(2) create a zone and enable it
(3) systemvm cannot be created, until Capacity Checker runs (it runs every 5 minutes).

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

… pool

harikrishna-patnala · 2021-03-15T04:41:47Z

            throw new CloudRuntimeException("Failed to add data store: " + e.getMessage(), e);
        }

+        // create entry for storage pool capacity


May I know if this specific to any storage type ?
Because I already see the same code creating capacity entries for cluster scoped storage pool and local storage pools.
cluster storage :

cloudstack/engine/storage/src/main/java/org/apache/cloudstack/storage/volume/datastore/PrimaryDataStoreHelper.java

Line 159 in 75fb1d9

storageMgr.createCapacityEntry(pool.getId());

Local storage :

cloudstack/engine/storage/src/main/java/org/apache/cloudstack/storage/volume/datastore/PrimaryDataStoreHelper.java

Line 152 in 75fb1d9

this.storageMgr.createCapacityEntry(pool, Capacity.CAPACITY_TYPE_LOCAL_STORAGE, pool.getUsedBytes());

@harikrishna-patnala
I tested with zone-wide storage pools.
do you think it is better to add storageMgr.createCapacityEntry(pool.getId()); at line 176 below ?

cloudstack/engine/storage/src/main/java/org/apache/cloudstack/storage/volume/datastore/PrimaryDataStoreHelper.java

Line 176 in 75fb1d9

pool.setScope(ScopeType.ZONE);

Yes @weizhouapache, that's the only missing piece. Other storage scopes are already handled.

@harikrishna-patnala done

… storage pool" This reverts commit e790167.

…de storage pools

harikrishna-patnala

LGTM

harikrishna-patnala · 2021-03-15T11:27:46Z

@blueorangutan package

blueorangutan · 2021-03-15T11:29:02Z

@harikrishna-patnala a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. [S]

shwstppr · 2021-03-18T05:41:59Z

@blueorangutan package

blueorangutan · 2021-03-18T05:43:02Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. [S]

blueorangutan · 2021-03-18T06:17:11Z

Packaging result: ✖️ centos7 ✖️ centos8 ✖️ debian. SL-JID 167

blueorangutan · 2021-03-18T12:41:00Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 171

yadvr · 2021-03-22T06:46:27Z

@blueorangutan test

blueorangutan · 2021-03-22T06:47:03Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-03-23T03:44:46Z

Trillian test result (tid-199)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 73266 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4805-t199-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_extra_config_data.py
Intermittent failure detected: /marvin/tests/smoke/test_diagnostics.py
Intermittent failure detected: /marvin/tests/smoke/test_human_readable_logs.py
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_loadbalance.py
Intermittent failure detected: /marvin/tests/smoke/test_network.py
Intermittent failure detected: /marvin/tests/smoke/test_password_server.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 58 look OK, 28 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_create_lb_rule_src_nat	`Failure`	308.90	test_loadbalance.py
test_02_create_lb_rule_non_nat	`Failure`	34.55	test_loadbalance.py
test_assign_and_removal_lb	`Failure`	34.51	test_loadbalance.py
test_01_verify_libvirt	`Error`	602.65	test_deploy_virtio_scsi_vm.py
test_02_verify_libvirt_after_restart	`Error`	609.89	test_deploy_virtio_scsi_vm.py
test_03_verify_libvirt_attach_disk	`Error`	605.80	test_deploy_virtio_scsi_vm.py
test_04_verify_guest_lspci	`Error`	602.06	test_deploy_virtio_scsi_vm.py
test_05_change_vm_ostype_restart	`Error`	612.45	test_deploy_virtio_scsi_vm.py
test_06_verify_guest_lspci_again	`Error`	602.07	test_deploy_virtio_scsi_vm.py
ContextSuite context=TestAddConfigtoDeployVM>:setup	`Error`	0.00	test_deploy_vm_extra_config_data.py
test_01_port_fwd_on_src_nat	`Failure`	604.85	test_network.py
test_02_port_fwd_on_non_src_nat	`Failure`	610.95	test_network.py
ContextSuite context=TestPrivateVlansL2Networks>:setup	`Error`	1232.19	test_network.py
test_reboot_router	`Failure`	358.90	test_network.py
test_network_rules_acquired_public_ip_1_static_nat_rule	`Error`	608.45	test_network.py
test_network_rules_acquired_public_ip_2_nat_rule	`Error`	609.05	test_network.py
test_network_rules_acquired_public_ip_3_Load_Balancer_Rule	`Error`	612.09	test_network.py
test_isolate_network_password_server	`Failure`	157.51	test_password_server.py
test_02_vpc_privategw_static_routes	`Failure`	756.07	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	764.15	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	852.53	test_privategw_acl.py
test_01_disableHumanReadableLogs	`Error`	602.79	test_human_readable_logs.py
test_02_enableHumanReadableLogs	`Error`	602.73	test_human_readable_logs.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80	`Failure`	287.55	test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	`Failure`	326.10	test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces	`Error`	142.87	test_internal_lb.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	`Error`	181.99	test_internal_lb.py
test_01_deploy_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_02_invalid_upgrade_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_03_deploy_and_upgrade_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_04_deploy_and_scale_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_05_delete_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_06_deploy_invalid_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_01_add_delete_kubernetes_supported_version	`Error`	0.01	test_kubernetes_supported_versions.py
test_router_dhcphosts	`Failure`	157.05	test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPHosts>:teardown	`Error`	168.35	test_router_dhcphosts.py
test_router_dhcp_opts	`Error`	609.15	test_router_dhcphosts.py
test_router_dns_guestipquery	`Failure`	455.02	test_router_dns.py
test_router_dns_guestipquery	`Failure`	455.03	test_router_dnsservice.py
test_02_routervm_iptables_policies	`Error`	661.48	test_routers_iptables_default_policy.py
test_01_single_VPC_iptables_policies	`Error`	721.93	test_routers_iptables_default_policy.py
test_01_isolate_network_FW_PF_default_routes_egress_true	`Failure`	209.44	test_routers_network_ops.py
test_02_isolate_network_FW_PF_default_routes_egress_false	`Failure`	211.22	test_routers_network_ops.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true	`Failure`	252.03	test_routers_network_ops.py
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false	`Failure`	251.64	test_routers_network_ops.py
test_03_RVR_Network_check_router_state	`Error`	697.61	test_routers_network_ops.py
test_01_router_internal_basic	`Error`	602.80	test_routers.py
test_02_router_internal_adv	`Error`	602.75	test_routers.py
test_04_restart_network_wo_cleanup	`Error`	604.80	test_routers.py
test_01_sys_vm_start	`Failure`	0.06	test_secondary_storage.py
test_01_service_offering_cpu_limit_use	`Error`	100.48	test_service_offerings.py
test_04_change_offering_small	`Failure`	714.15	test_service_offerings.py
test_01_snapshot_root_disk	`Error`	604.94	test_snapshots.py
test_03_ssvm_internals	`Error`	602.82	test_ssvm.py
test_04_cpvm_internals	`Error`	602.70	test_ssvm.py
test_05_stop_ssvm	`Error`	649.35	test_ssvm.py
test_06_stop_cpvm	`Error`	653.42	test_ssvm.py
test_07_reboot_ssvm	`Error`	720.25	test_ssvm.py
test_08_reboot_cpvm	`Error`	627.13	test_ssvm.py
test_09_destroy_ssvm	`Error`	668.28	test_ssvm.py
test_10_destroy_cpvm	`Error`	656.38	test_ssvm.py
ContextSuite context=TestDeployVM>:setup	`Error`	0.00	test_vm_life_cycle.py
test_01_migrate_VM_and_root_volume	`Error`	1.18	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	1.16	test_vm_life_cycle.py
test_01_secure_vm_migration	`Error`	1.07	test_vm_life_cycle.py
test_02_unsecure_vm_migration	`Error`	1.08	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	1.08	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	1.08	test_vm_life_cycle.py
ContextSuite context=TestVMLifeCycle>:setup	`Error`	1.73	test_vm_life_cycle.py
ContextSuite context=TestVmSnapshot>:setup	`Error`	1.60	test_vm_snapshots.py
ContextSuite context=TestCreateVolume>:setup	`Error`	0.00	test_volumes.py
ContextSuite context=TestVolumes>:setup	`Error`	0.00	test_volumes.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Error`	6.67	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	`Error`	6.64	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Error`	7.71	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Error`	7.69	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	5.77	test_vpc_redundant.py
test_01_VPC_nics_after_destroy	`Error`	5.66	test_vpc_router_nics.py
test_02_VPC_default_routes	`Error`	5.64	test_vpc_router_nics.py
test_01_redundant_vpc_site2site_vpn	`Failure`	7.35	test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options	`Failure`	4.32	test_vpc_vpn.py
test_01_vpc_remote_access_vpn	`Failure`	3.17	test_vpc_vpn.py
test_01_vpc_site2site_vpn	`Failure`	4.29	test_vpc_vpn.py
test_01_cancel_host_maintenace_with_no_migration_jobs	`Error`	0.05	test_host_maintenance.py
test_02_cancel_host_maintenace_with_migration_jobs	`Error`	0.04	test_host_maintenance.py
test_03_cancel_host_maintenace_with_migration_jobs_failure	`Error`	0.03	test_host_maintenance.py
test_01_cancel_host_maintenance_ssh_enabled_agent_connected	`Error`	0.01	test_host_maintenance.py
test_03_cancel_host_maintenance_ssh_disabled_agent_connected	`Error`	0.01	test_host_maintenance.py
test_04_cancel_host_maintenance_ssh_disabled_agent_disconnected	`Error`	0.01	test_host_maintenance.py
test_hostha_enable_ha_when_host_disconected	`Error`	605.09	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	302.54	test_hostha_kvm.py

yadvr · 2021-03-24T06:42:51Z

@blueorangutan test

blueorangutan · 2021-03-24T06:44:03Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-03-24T16:52:29Z

Trillian test result (tid-244)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34614 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4805-t244-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 84 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_migrate_VM_and_root_volume	`Error`	85.00	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	51.99	test_vm_life_cycle.py
test_hostha_enable_ha_when_host_disabled	`Error`	2.38	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	301.54	test_hostha_kvm.py

server: create DB entry for storage pool capacity when create storage…

e790167

… pool

harikrishna-patnala reviewed Mar 15, 2021

View reviewed changes

ustcweizhou added 2 commits March 15, 2021 08:40

Revert "server: create DB entry for storage pool capacity when create…

4140cea

… storage pool" This reverts commit e790167.

server: create DB entry for storage pool capacity when create zone-wi…

f4f5fec

…de storage pools

harikrishna-patnala approved these changes Mar 15, 2021

View reviewed changes

weizhouapache closed this Mar 15, 2021

weizhouapache reopened this Mar 15, 2021

DaanHoogland added this to the 4.15.1.0 milestone Mar 15, 2021

DaanHoogland added Severity:Minor type:improvement labels Mar 15, 2021

DaanHoogland approved these changes Mar 15, 2021

View reviewed changes

apache deleted a comment from blueorangutan Mar 18, 2021

GabrielBrascher approved these changes Mar 19, 2021

View reviewed changes

yadvr self-assigned this Mar 24, 2021

yadvr merged commit b8884ef into apache:4.15 Mar 29, 2021

Uh oh!

Conversation

ustcweizhou commented Mar 12, 2021

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

harikrishna-patnala Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!

weizhouapache Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!

harikrishna-patnala Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!

weizhouapache Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

harikrishna-patnala commented Mar 15, 2021

Uh oh!

blueorangutan commented Mar 15, 2021

Uh oh!

shwstppr commented Mar 18, 2021

Uh oh!

blueorangutan commented Mar 18, 2021

Uh oh!

blueorangutan commented Mar 18, 2021

Uh oh!

blueorangutan commented Mar 18, 2021

Uh oh!

yadvr commented Mar 22, 2021

Uh oh!

blueorangutan commented Mar 22, 2021

Uh oh!

blueorangutan commented Mar 23, 2021

Uh oh!

yadvr commented Mar 24, 2021

Uh oh!

blueorangutan commented Mar 24, 2021

Uh oh!

blueorangutan commented Mar 24, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants