Incorrect `memoryintfreekbs` in listVirtualMachines for KVM

[davidjumani]

The memoryintfreekbs retuned by listVirtualMachines for KVM is always greater than the total memory allocated for the VM. This is because it returns the RSS of the process running the VM (https://libvirt.org/manpages/virsh.html#dommemstat)
The RSS shows how much memory is allocated to that process and is in RAM. It is returned by memoryStats over at https://github.com/apache/cloudstack/blob/master/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java#L3712-L3718 when trying to fetch the free memory instead

Need to discuss how to handle this as well as the impact on usage

ISSUE TYPE

• Bug Report

CLOUDSTACK VERSION

4.11+ (tested)

OS / ENVIRONMENT

KVM

STEPS TO REPRODUCE

(localcloud) SBCM5> > list virtualmachines filter=memory,memoryintfreekbs,memorykbs,memorytargetkbs,
{
  "count": 3,
  "virtualmachine": [
    {
      "memory": 512,
      "memoryintfreekbs": 552836, <---
      "memorykbs": 524288,
      "memorytargetkbs": 524288
    },

EXPECTED RESULTS

The used memory

ACTUAL RESULTS

Total memory used by the process running the VM

[davidjumani]

Related to #3456

[DaanHoogland]

@PaulAngus @andrijapanicsb or any other functional eyes on this (@weizhouapache, @wido maybe from your camp), This one raises a question about billing, should an operator be able to see both memory used by the vm and by the containing process for their accounting? what is the best un-ambiguous way to define this?

[PaulAngus]

This issue jumps between FREE memory and USED memory - which one are we talking about here?

[DaanHoogland]

the reported issue is about how free mem is calculated but the real issue is whether used memory as used by the VM-process as opposed to the memory used inside the vm should be attributed to the user or reported in some way at all

[PaulAngus]

thanks. can we change the title to match the problem please. Anyone looking back through commits will be misled.

I don't think that the overhead to run a VM is the end-users problem, I think that if I created a VM with 8GB RAM, I would expect to get charged for 8GB.

[wido]

@PaulAngus I agree. Overhead is something the operator should take into account. We at PCextreme do that. We take a safety margin on top of all the memory we allocate to VMs and which we then reserve for the Operating System, buffers and overhead.

So yes, the end-user should never be aware of the RSS on the hypervisor.

VMs using Ceph for example have a much higher overhead to due librados and librbd consuming memory under the hood.

[andrijapanicsb]

agree with Paul and Wido.

[davidjumani]

I agree as well so I gather that we have a consensus on it. The other issue is that the memoryintfreekbs parameter returns the wrong result (the rss instead) and it might not be possible to retrieve the free memory in the VM. What would be a good approach to go about handling it ? @PaulAngus @wido @andrijapanicsb @DaanHoogland

[wido]

Inside the VM depends on the hypervisor I guess. With KVM for example you have the Qemu Guest Agent which is needed to look inside the VM and get these results.

I don't think we can safely fetch that information from inside the VM.

[andrijapanicsb]

+1 on what Wido said, as this is public cloud people will be (mostly/soemtimes) running, so it's impossible to ensure specific hypervisor tools are running.
I don't see the value of monitoring the amount of RAM used inside the VM - let me user monitor his own VMs if needed (not the operator) using whatever tools they use.

[weizhouapache]

thanks. can we change the title to match the problem please. Anyone looking back through commits will be misled.

I don't think that the overhead to run a VM is the end-users problem, I think that if I created a VM with 8GB RAM, I would expect to get charged for 8GB.

Totally agree.

by the way, there is a way to get accurate memory usage: memory ballooning.
see https://libvirt.org/formatdomain.html#memory-balloon-device

  <devices>
    <memballoon model='virtio'>
      <stats period='10'/>
    </memballoon>
  </devices>

we could add a global setting to enable/disable it.