Broadcast URI not set to vxlan, but vlan

[wido]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Management Server

CLOUDSTACK VERSION

master / 4.12

CONFIGURATION

Advanced Networking with KVM and VXLAN

OS / ENVIRONMENT

• Ubuntu 18.04
• KVM

SUMMARY

Advanced Network zone with VXLAN is created. When creating a guest network and setting the VNI this is set for the network:

• broadcast_uri: vlan://1000
• isolation_uri: vlan://1000

STEPS TO REPRODUCE

• Create a Advanced Zone with VXLAN
• Create a guest network
• See that broadcast is set to vlan and not vxlan

[yadvr]

@wido I tested with 4.11.2.0 I could n't reproduce the issue, likely a configuration issue on your end. What did you configure for the guest network?
My config from UI:

[wido]

@rhtyd The difference I see is that I did not set "VLAN/VNI Range(s)" under the Physical Network.

But it's still odd that if the zone is set to VXLAN that vlan:// isolation is used. That should default to VXLAN.

[yadvr]

@wido when you created the shared network did you pass a custom just vid/vni but without the vxlan:// scheme? Try again to create a shared network passing vxlan://id. I think somewhere it must be assuming vlan:// by default when scheme is not passed.

[wido]

When trying to set the VNI range I get this error:

Can't add vnet range to the physical network in the zone that supports Advanced network, Security Group enabled: true

Checking NetworkServiceImpl.java I see:

        if (vnetRange != null) {
            // Verify zone type
            if (zoneType == NetworkType.Basic || (zoneType == NetworkType.Advanced && zone.isSecurityGroupEnabled())) {
                throw new InvalidParameterValueException("Can't add vnet range to the physical network in the zone that supports " + zoneType
                        + " network, Security Group enabled: " + zone.isSecurityGroupEnabled());
            }
        }

Why wouldn't this be supported with Security Grouping enabled?

[wido]

@wido when you created the shared network did you pass a custom just vid/vni but without the vxlan:// scheme? Try again to create a shared network passing vxlan://id. I think somewhere it must be assuming vlan:// by default when scheme is not passed.

I just supplied the VNI number without a scheme.

Although I understand you reasoning, if the zone is set to VXLAN, we should default to that?

[yadvr]

@wido looks like you hit a presentational bug. I see you are using adv zone with SG, so I'm not sure if you need vxlan/vlan at all or how should such a deployment work unless you've made the default public/private bridge device to carry traffic over a vlan enabled nic. With SG enabled, you already got L3 isolation with host level firewall (ebtables), why would you need vxlans?

[wido]

Network	Broadcast	IPv4	IPv6	Account
Guest Network 1	vxlan://1000	109.72.91.0/26	2a00:f10:114:1000::/64	NULL
Guest Network 2	vxlan://2000	109.72.91.64/26	2a00:f10:114:2000::/64	NULL
Guest Network 3	vxlan://3000	109.72.91.128/26	2a00:f10:114:3000::/64	NULL
Guest Network 4	vxlan://4000	109.72.91.192/26	2a00:f10:114:4000::/64	57191

On our public cloud every VM gets public IPs. The reason we want to use VXLAN is the flexibility it provides on a network level to separate our network using Layer 3.

Layer 2 VLANs are not robust and flexible enough to route on large scale.

The 'Guest Networks' are shared. Multiple different customers can have a Virtual Machine in that network.

For this example I used real-life IPs which we are using now in a PoC.

Security Grouping is needed to prevent IP-spoofing, but also provide firewalling for Virtual Machines.

'Guest Network 4' is dedicated to a account. This way the customer knows that all VMs he/she deploys will be in that network and on for example his office connection he/she can safely allow that specific subnet in the firewall.

[yadvr]

@wido the current network model is not designed and implemented that way wrt KVM. The vlan/vxlan based isolation is for isolated/vpc networks implemented by the VR which does NAT-ing from public network end; and typically a bridge device is created where guest domain/VM's nic (via tap devices) are plugged and any vxlan tunnel end point (vtep) or vlan enable ethernet device is plugged to implement the (isolated/vpc-tier) network. With SG you get massive-scalable L3 isolation by having host-level firewall implemented/enforced by ebtables (firewall rules on bridge) rules and ipset config, and so there is no need of a VR to do network isolation (however VR does only provide dhcp and dns).

What you've described is a new feature where you want the guest VMs to still receive public IPs and network access, but the (public) bridge device to provide isolation by plugging in a vxlan (or vlan) enabled ethernet device and carry public traffic over vxlan for a tenant. This means on your core router side, for all such guest networks isolation should terminate (vxlan). If there is no need of isolated network and guest networks are still shared, so the question is -- do you only want isolation between these shared networks which carry public traffic only in your datacenter? You already have L3/SGs isolation.

I've not tested but you can try adding public address ranges with SG enabled and some isolation (vlan:// or vxlan:// tagging), dedicate that range to an account and see if that works.

Lastly, L2/vlan isolation implementation and usage are quite robust, widely used in enterprise but not massively scalable.

[yadvr]

@wido are you btw exploring something like https://www.exoscale.com/syslog/introducing-private-networks/ https://community.exoscale.com/documentation/compute/private-networks/ ?

[svenvogel]

@wido can i ask what you plan do use kernel bridge or openvswitch?

[wido]

@rhtyd Let me respectfully disagree with you there.

With VXLAN this works. I can create a Shared Network with Security Grouping where my Instance obtains a public IPv4/IPv6 address without the VR in the middle. Our own routers handle the gateway.

VLANs are fun, but L2 is not useful in large scale deployments. It creates many problems which are solved by VXLAN. For CloudStack there is also no difference between VXLAN and VLAN. It's just a slightly different way of creating the bridges on the KVM hypervisor.

@svenvogel At the moment we are looking into using Linux Kernel Bridges.

Openvswitch is something we might consider later.