uservmjoindaoimpl: Set free memory to zero if greater than total memory by davidjumani · Pull Request #4571 · apache/cloudstack

davidjumani · 2021-01-07T04:53:13Z

Description

Sets memoryintfreekbs to zero if it is greater than memorykbs. Caused by KVM returning the RSS memory of the process running the VM rather than the free memory inside the VM

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?

Before

(localcloud) SBCM5> > list virtualmachines filter=memoryintfreekbs,memorykbs,
{
  "count": 2,
  "virtualmachine": [
    {
      "memoryintfreekbs": 2138672,
      "memorykbs": 2097152
    },
    {
      "memoryintfreekbs": 2143292,
      "memorykbs": 2097152
    }
  ]
}

After

(localcloud) SBCM5> > list virtualmachines filter=memoryintfreekbs,memorykbs,
{
  "count": 2,
  "virtualmachine": [
    {
      "memoryintfreekbs": 0,
      "memorykbs": 2097152
    },
    {
      "memoryintfreekbs": 0,
      "memorykbs": 2097152
    }
  ]
}

davidjumani · 2021-01-07T04:53:28Z

@blueorangutan package

blueorangutan · 2021-01-07T04:54:43Z

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-01-07T05:31:45Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2528

weizhouapache · 2021-01-07T12:08:56Z

set to 0 for all kvm vm ?
is the value correct on xenserver/vmware ?

davidjumani · 2021-01-08T06:40:01Z

@weizhouapache It works fine on Xen and VMware. I haven't specifically changed it for KVM, but instead, set it to zero if the value returned for the internal free memory is greater than the total memory allocated to the VM

weizhouapache · 2021-01-08T08:33:05Z

@weizhouapache It works fine on Xen and VMware. I haven't specifically changed it for KVM, but instead, set it to zero if the value returned for the internal free memory is greater than the total memory allocated to the VM

@davidjumani since the value is incorrect for kvm vms, would it be better to set it to 0 for all kvm vms and mention it in the api response ?

davidjumani · 2021-01-12T05:35:34Z

@weizhouapache I think that checking whether it is greater than the total memory and setting it to zero is a better option since it'll cover any case on other hypervisors if it were to happen. I'll change the docs to match

blueorangutan · 2021-01-12T12:41:10Z

Packaging result: ?centos7 ?debian. JID-5

blueorangutan · 2021-01-12T13:08:02Z

Packaging result: ?centos7 ?debian. JID-6

blueorangutan · 2021-01-12T13:44:28Z

Packaging result: ?centos7 ?debian. JID-7

blueorangutan · 2021-01-21T08:52:29Z

Packaging result: ✔centos7 ✔centos8 ✖debian. JID-2559

yadvr · 2021-02-18T12:50:51Z

@blueorangutan package

blueorangutan · 2021-02-18T12:52:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-02-18T14:12:18Z

Packaging result: ✔centos7 ✖centos8 ✖debian. JID-2737

yadvr · 2021-02-19T09:01:28Z

LGTM
@blueorangutan package

blueorangutan · 2021-02-19T09:03:01Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-02-19T11:31:27Z

Packaging result: ✔centos7 ✖centos8 ✔debian. JID-2752

davidjumani · 2021-02-22T04:54:10Z

@blueorangutan package

blueorangutan · 2021-02-22T04:55:02Z

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-02-22T05:30:52Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2773

DaanHoogland · 2021-02-23T08:25:53Z

@blueorangutan test

blueorangutan · 2021-02-23T08:27:07Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-02-23T18:33:05Z

Trillian test result (tid-3600)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34153 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4571-t3600-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 86 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

davidjumani · 2021-03-06T06:15:23Z

@blueorangutan package

blueorangutan · 2021-03-06T06:15:50Z

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-03-06T06:50:40Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2874

Co-authored-by: dahn <daan.hoogland@gmail.com>

davidjumani · 2021-03-12T04:54:00Z

@blueorangutan package

blueorangutan · 2021-03-12T04:57:02Z

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress. [S]

blueorangutan · 2021-03-12T05:21:17Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 96

shwstppr

LGTM

shwstppr · 2021-03-18T09:14:13Z

@blueorangutan test

blueorangutan · 2021-03-18T09:15:02Z

@shwstppr a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests [S]

blueorangutan · 2021-03-19T14:07:14Z

[S] Trillian test result (tid-171)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 101848 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4571-t171-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_extra_config_data.py
Intermittent failure detected: /marvin/tests/smoke/test_human_readable_logs.py
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_loadbalance.py
Intermittent failure detected: /marvin/tests/smoke/test_network.py
Intermittent failure detected: /marvin/tests/smoke/test_password_server.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 60 look OK, 26 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_create_lb_rule_src_nat	`Failure`	308.72	test_loadbalance.py
test_02_create_lb_rule_non_nat	`Failure`	35.98	test_loadbalance.py
test_assign_and_removal_lb	`Failure`	34.94	test_loadbalance.py
test_01_verify_libvirt	`Error`	602.74	test_deploy_virtio_scsi_vm.py
test_02_verify_libvirt_after_restart	`Error`	612.16	test_deploy_virtio_scsi_vm.py
test_03_verify_libvirt_attach_disk	`Error`	604.90	test_deploy_virtio_scsi_vm.py
test_04_verify_guest_lspci	`Error`	621.96	test_deploy_virtio_scsi_vm.py
test_05_change_vm_ostype_restart	`Error`	610.01	test_deploy_virtio_scsi_vm.py
test_06_verify_guest_lspci_again	`Error`	602.08	test_deploy_virtio_scsi_vm.py
ContextSuite context=TestAddConfigtoDeployVM>:setup	`Error`	0.00	test_deploy_vm_extra_config_data.py
test_01_port_fwd_on_src_nat	`Failure`	605.06	test_network.py
test_02_port_fwd_on_non_src_nat	`Failure`	608.23	test_network.py
ContextSuite context=TestPrivateVlansL2Networks>:setup	`Error`	1229.44	test_network.py
test_reboot_router	`Failure`	359.04	test_network.py
test_network_rules_acquired_public_ip_1_static_nat_rule	`Error`	607.38	test_network.py
test_network_rules_acquired_public_ip_2_nat_rule	`Error`	609.01	test_network.py
test_network_rules_acquired_public_ip_3_Load_Balancer_Rule	`Error`	613.08	test_network.py
test_isolate_network_password_server	`Failure`	157.00	test_password_server.py
test_02_vpc_privategw_static_routes	`Failure`	758.91	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	754.83	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	842.42	test_privategw_acl.py
test_01_disableHumanReadableLogs	`Error`	602.74	test_human_readable_logs.py
test_02_enableHumanReadableLogs	`Error`	602.69	test_human_readable_logs.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80	`Failure`	270.57	test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	`Failure`	321.81	test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces	`Error`	138.74	test_internal_lb.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	`Error`	180.22	test_internal_lb.py
test_01_deploy_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_02_invalid_upgrade_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_03_deploy_and_upgrade_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_04_deploy_and_scale_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_05_delete_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_06_deploy_invalid_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_01_add_delete_kubernetes_supported_version	`Error`	0.01	test_kubernetes_supported_versions.py
test_router_dhcphosts	`Failure`	157.27	test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPHosts>:teardown	`Error`	167.54	test_router_dhcphosts.py
test_router_dhcp_opts	`Error`	609.05	test_router_dhcphosts.py
test_router_dns_guestipquery	`Failure`	454.58	test_router_dns.py
test_router_dns_guestipquery	`Failure`	454.63	test_router_dnsservice.py
test_02_routervm_iptables_policies	`Error`	662.43	test_routers_iptables_default_policy.py
test_01_single_VPC_iptables_policies	`Error`	722.81	test_routers_iptables_default_policy.py
test_01_isolate_network_FW_PF_default_routes_egress_true	`Failure`	211.14	test_routers_network_ops.py
test_02_isolate_network_FW_PF_default_routes_egress_false	`Failure`	206.96	test_routers_network_ops.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true	`Failure`	244.96	test_routers_network_ops.py
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false	`Failure`	243.73	test_routers_network_ops.py
test_03_RVR_Network_check_router_state	`Error`	692.12	test_routers_network_ops.py
test_01_router_internal_basic	`Error`	602.73	test_routers.py
test_02_router_internal_adv	`Error`	602.71	test_routers.py
test_04_restart_network_wo_cleanup	`Error`	604.77	test_routers.py
test_01_service_offering_cpu_limit_use	`Error`	100.52	test_service_offerings.py
test_04_change_offering_small	`Failure`	713.89	test_service_offerings.py
test_01_snapshot_root_disk	`Error`	604.79	test_snapshots.py
test_03_ssvm_internals	`Error`	602.78	test_ssvm.py
test_04_cpvm_internals	`Error`	602.77	test_ssvm.py
test_05_stop_ssvm	`Error`	647.38	test_ssvm.py
test_06_stop_cpvm	`Error`	655.30	test_ssvm.py
test_07_reboot_ssvm	`Error`	720.19	test_ssvm.py
test_08_reboot_cpvm	`Error`	623.96	test_ssvm.py
test_09_destroy_ssvm	`Error`	669.31	test_ssvm.py
test_10_destroy_cpvm	`Error`	660.45	test_ssvm.py
test_01_migrate_VM_and_root_volume	`Error`	69.20	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	50.11	test_vm_life_cycle.py
test_01_secure_vm_migration	`Error`	702.23	test_vm_life_cycle.py
test_02_unsecure_vm_migration	`Error`	637.17	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	637.23	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	638.19	test_vm_life_cycle.py
test_10_attachAndDetach_iso	`Failure`	607.71	test_vm_life_cycle.py
test_01_create_vm_snapshots	`Failure`	604.54	test_vm_snapshots.py
test_02_revert_vm_snapshots	`Failure`	601.54	test_vm_snapshots.py
test_03_delete_vm_snapshots	`Failure`	0.01	test_vm_snapshots.py
test_01_create_volume	`Failure`	607.94	test_volumes.py
test_02_attach_volume	`Failure`	605.77	test_volumes.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Error`	831.12	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	`Error`	830.15	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Error`	746.12	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Error`	707.95	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	801.30	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	828.28	test_vpc_redundant.py
test_01_VPC_nics_after_destroy	`Failure`	721.75	test_vpc_router_nics.py
test_02_VPC_default_routes	`Failure`	721.43	test_vpc_router_nics.py
test_01_redundant_vpc_site2site_vpn	`Failure`	355.58	test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options	`Error`	251.25	test_vpc_vpn.py
test_01_vpc_site2site_vpn	`Error`	284.83	test_vpc_vpn.py
test_hostha_enable_ha_when_host_disabled	`Error`	2.38	test_hostha_kvm.py
test_hostha_enable_ha_when_host_disconected	`Error`	605.15	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	304.10	test_hostha_kvm.py

yadvr · 2021-03-24T07:07:49Z

@blueorangutan package

blueorangutan · 2021-03-24T07:09:05Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-03-24T07:20:40Z

Packaging result: ✖️ centos7 ✖️ centos8 ✖️ debian. SL-JID 225

yadvr · 2021-03-24T07:26:39Z

@blueorangutan package

blueorangutan · 2021-03-24T07:27:02Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-03-24T07:37:01Z

Packaging result: ✖️ centos7 ✖️ centos8 ✖️ debian. SL-JID 226

yadvr · 2021-03-24T11:21:17Z

@blueorangutan package

yadvr · 2021-03-24T11:21:38Z

@blueorangutan package

blueorangutan · 2021-03-24T11:22:05Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-03-24T12:12:39Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 234

davidjumani · 2021-03-24T12:15:36Z

@blueorangutan test

blueorangutan · 2021-03-24T12:16:03Z

@davidjumani a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-03-25T02:16:13Z

Trillian test result (tid-248)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 46247 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4571-t248-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 84 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	71.80	test_kubernetes_clusters.py
test_01_migrate_VM_and_root_volume	`Error`	67.22	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	50.05	test_vm_life_cycle.py

shwstppr added this to the 4.16.0.0 milestone Feb 8, 2021

davidjumani marked this pull request as ready for review February 22, 2021 04:54

DaanHoogland reviewed Feb 24, 2021

View reviewed changes

Comment thread api/src/main/java/org/apache/cloudstack/api/response/UserVmResponse.java Outdated

yadvr reviewed Mar 6, 2021

View reviewed changes

Comment thread api/src/main/java/org/apache/cloudstack/api/response/UserVmResponse.java Outdated

Fix typo

c74024b

Co-authored-by: dahn <daan.hoogland@gmail.com>

davidjumani force-pushed the fix-metrics branch from 2287b88 to c74024b Compare March 12, 2021 04:53

davidjumani modified the milestones: 4.16.0.0, 4.15.1.0 Mar 16, 2021

shwstppr approved these changes Mar 17, 2021

View reviewed changes

shwstppr requested a review from weizhouapache March 24, 2021 04:30

yadvr assigned davidjumani Mar 24, 2021

yadvr closed this Mar 24, 2021

yadvr reopened this Mar 24, 2021

yadvr merged commit 03ad702 into apache:4.15 Mar 30, 2021

yadvr mentioned this pull request Mar 30, 2021

Incorrect memoryintfreekbs in listVirtualMachines for KVM #4566

Closed

Uh oh!

Conversation

davidjumani commented Jan 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

How Has This Been Tested?

Before

After

Uh oh!

davidjumani commented Jan 7, 2021

Uh oh!

blueorangutan commented Jan 7, 2021

Uh oh!

blueorangutan commented Jan 7, 2021

Uh oh!

weizhouapache commented Jan 7, 2021

Uh oh!

davidjumani commented Jan 8, 2021

Uh oh!

weizhouapache commented Jan 8, 2021

Uh oh!

davidjumani commented Jan 12, 2021

Uh oh!

blueorangutan commented Jan 12, 2021

Uh oh!

blueorangutan commented Jan 12, 2021

Uh oh!

blueorangutan commented Jan 12, 2021

Uh oh!

blueorangutan commented Jan 21, 2021

Uh oh!

yadvr commented Feb 18, 2021

Uh oh!

blueorangutan commented Feb 18, 2021

Uh oh!

blueorangutan commented Feb 18, 2021

Uh oh!

yadvr commented Feb 19, 2021

Uh oh!

blueorangutan commented Feb 19, 2021

Uh oh!

blueorangutan commented Feb 19, 2021

Uh oh!

davidjumani commented Feb 22, 2021

Uh oh!

blueorangutan commented Feb 22, 2021

Uh oh!

blueorangutan commented Feb 22, 2021

Uh oh!

DaanHoogland commented Feb 23, 2021

Uh oh!

blueorangutan commented Feb 23, 2021

Uh oh!

blueorangutan commented Feb 23, 2021

Uh oh!

Uh oh!

davidjumani commented Mar 6, 2021

Uh oh!

blueorangutan commented Mar 6, 2021

Uh oh!

blueorangutan commented Mar 6, 2021

Uh oh!

Uh oh!

davidjumani commented Mar 12, 2021

Uh oh!

blueorangutan commented Mar 12, 2021

Uh oh!

blueorangutan commented Mar 12, 2021

Uh oh!

shwstppr left a comment

Choose a reason for hiding this comment

Uh oh!

shwstppr commented Mar 18, 2021

Uh oh!

blueorangutan commented Mar 18, 2021

Uh oh!

blueorangutan commented Mar 19, 2021

Uh oh!

yadvr commented Mar 24, 2021

davidjumani commented Jan 7, 2021 •

edited

Loading