4 types of OpenStack Neutron networks you must understand

If your OpenStack hosted virtual instances need network connectivity you’re going to have to create a network. There are multiple kinds of networks and in order to make the right choice you will need to understand at least two very important network attributes: ‘router:external’ and ‘shared.’ Unless you know what these attributes and their combinations mean, it will be difficult to make the optimal network choice.

In this article, we’ll explain the four types of networks these attributes dictate and how to configure them and briefly explain some typical use cases. Some of these exercises apply to tenants and others are for administrators only. Examples are given to work through this exercise on a KVM-hosted Red Hat or RDO all-in-one OpenStack instance. This has been tested on Red Hat OpenStack Platform 8, 9, 10 and RDO Mitaka and Newton. 

The four types explained

The table below gives an overview of the types of networks we’re going to create and their names.  

Logical network topology

OpenStack all-in-one build

An all-in-one instance is handy for understanding new concepts where separate roles are not required. Troubleshooting and configuration are considerably simplified, and that is what we’re going to use for our examples. If you have a Red Hat subscription, you can follow these instructions. You can also sign up for a 60 day Red Hat OpenStack Platform evaluation.

For an RDO instance, go here for a quick installation guide, and be sure to enable the additional repositories on this page.

When your packstack build is complete, log in as root and run the following commands to verify your OpenStack deployment is working:

. ./keystonerc_admin
openstack catalog list
neutron agent-list
nova service-list
nova boot --flavor m1.tiny --image cirros --nic net-name=private admin01

Once you have established the health of OpenStack, delete the virtual instance you just created along with the ‘Public’ and ‘Private’ networks (and their subnets) and ‘router1’ created by the packstack installation. 

KVM host configuration

The KVM host for this exercise will have three networks:

Your KVM host probably already comes with a default network on eth0 that with bridge virbr0. We’ll define and then create the other two external networks. Be sure to do this as root:

cat > /tmp/external1.xml << EOF
<network>
 <name>external1</name>
 <forward mode='nat'>
   <nat>
     <port start='1024' end='65535'/>
   </nat>
 </forward>
 <bridge name='virbr1' stp='on' delay='0'/>
 <mac address='52:54:00:73:a0:8e'/>
 <ip address='172.16.0.1' netmask='255.255.255.192'>
 </ip>
</network>
EOF

cat > /tmp/external2.xml << EOF
<network>
 <name>external2</name>
 <forward mode='nat'>
   <nat>
     <port start='1024' end='65535'/>
   </nat>
 </forward>
 <bridge name='virbr2' stp='on' delay='0'/>
 <mac address='52:54:00:61:98:8c'/>
 <ip address='172.16.0.65' netmask='255.255.255.192'>
 </ip>
</network>
EOF

Now create these networks and start them:

virsh net-define /tmp/external1.xml; virsh net-define /tmp/external2.xml
virsh net-autostart external1; virsh net-autostart external2
virsh net-start external1; virsh net-start external2

Now add a VNIC to the OpenStack all-in-one instance for each external network. Do this from the KVM host:

dom=<your Red Hat Enterprise Linux or RDO instance name from ‘virsh list’>
virsh attach-interface --domain $dom --type network --source external1 --model virtio --config --live
virsh attach-interface --domain $dom --type network --source external2 --model virtio --config --live   

Verify the KVM host

$ virsh net-list
Name              State   Autostart Persistent
----------------------------------------------------------
default           active yes        yes
external1         active yes        yes
external2         active yes        yes

$ brctl show
bridge name    bridge id    STP enabled    interfaces
virbr0    8000.525400ce5983    yes    virbr0-nic
virbr1    8000.52540073a08e    yes    virbr1-nic
virbr2    8000.52540061988c    yes    virbr2-nic
  
$ route
Kernel IP routing table
Destination  Gateway      Genmask      Flags Metric Ref Use Iface
172.16.0.0   0.0.0.0      255.255.255.192 U 0   0     0 virbr1
172.16.0.64  0.0.0.0      255.255.255.192 U 0   0     0 virbr2
192.168.122.0  0.0.0.0            255.255.255.0     U      0          0          0 virbr0

OpenStack all-in-one configuration

We need to make some changes to the packstack delivered Neutron configuration to remove the single Linux bridge ‘br-ex’ and re-add new bridges br-ex and br-ex2 and the flat networks we’ll attach them to:

yum install crudini -y
crudini --set /etc/neutron/l3_agent.ini DEFAULT external_network_bridge #blank this
crudini --set /etc/neutron/l3_agent.ini DEFAULT gateway_external_network_id  #blank this
crudini --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 type_drivers  vxlan,flat
crudini --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 network_vlan_ranges physnet1,physnet2
crudini --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs bridge_mappings physnet1:br-ex,physnet2:br-ex2

Note that it is up to you what you want to call your network_vlan_ranges when using ‘flat’ networks. These will be the ‘--provider:physical_network’ names when you create your external networks. External bridge names (br-ex*) are also your choice.  

Now create the interface configurations for Open vSwitch:

cat >  /etc/sysconfig/network-scripts/ifcfg-br-ex << EOF
DEVICE=br-ex
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=172.16.0.1
NETMASK=255.255.255.192
EOF

cat >  /etc/sysconfig/network-scripts/ifcfg-eth1 << EOF
DEVICE=eth1
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-ex
BOOTPROTO=none
EOF

cat >  /etc/sysconfig/network-scripts/ifcfg-br-ex2 << EOF
DEVICE=br-ex2
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=172.16.0.65
NETMASK=255.255.255.192
EOF

cat >  /etc/sysconfig/network-scripts/ifcfg-eth2 << EOF
DEVICE=eth2
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
DEVICETYPE=ovs
TYPE=OVSPort
OVS_BRIDGE=br-ex2
BOOTPROTO=none
EOF

Verify Open vSwitch external bridge

Once you have completed those configuration steps you will want to reboot your virtual host. The bridges created should look like the following after the reboot:

ovs-vsctl list-ports br-ex

eth1
phy-br-ex

ovs-vsctl list-ports br-ex2

eth2
phy-br-ex2

Neutron networks, subnets, and routers

Finally, we’re ready to get down to business and create the networks we’ve been talking about. The following series of commands will create networks, subnets, routers, security group rules, instances and floating IPs. When this is complete you’ll have the four different types of networks ready for use.  

External networks

The following series creates our external networks. Note that both networks use ‘router:external=true’ and they are both ‘network_type=flat’, but each network is on a different physical_network and the 2nd one has the ‘--shared’ attribute:

. ~/keystonerc_admin
neutron net-create external1  --router:external=True  --provider:network_type=flat  --provider:physical_network=physnet1

neutron subnet-create --name external1-subnet --disable-dhcp external1 172.16.0.0/26 --allocation-pool start=172.16.0.2,end=172.16.0.62

neutron net-create external2-shared --shared --router:external=True --provider:network_type=flat --provider:physical_network=physnet2   

neutron subnet-create external2-shared --name external2-shared-subnet --allocation-pool\
 start=172.16.0.66,end=172.16.0.126  --gateway 172.16.0.65 172.16.0.64/26

Tenant network and router

Tenant networks are where you can create your own private rfc 1918 address spaces. Tenants can provision any combination of networks, subnets, and routers to meet their needs. They can define their own broadcast domains and determine which instances are externally accessible through floating IP addresses.  

The following creates a private or ‘tenant’ network and subnet. It also creates a router to attach it to external network external1 Note that we are switching to the ‘demo’ project for these networks:

. ~/keystonerc_demo
neutron net-create private1-demo

neutron subnet-create private1-demo 10.0.1.0/24 --name private1-demo-subnet    

Note that this network was created with these by default:

neutron net-show -c 'router:external' -c shared private1-demo
+-----------------+-------+
| Field           | Value |
+-----------------+-------+
| router:external | False |
| shared          | False |
+-----------------+-------+

neutron router-create router1-demo

neutron router-gateway-set router1-demo external1

neutron router-interface-add router1-demo private1-demo-subnet

The router gateways should be pingable. Perform this command and ping the gateway IPs it returns:

neutron router-list -c external_gateway_info | grep -o "172.16.0.[0-9]*"

Security group rules, virtual instances, and floating IPs

Before creating virtual instances, let’s open the ports for ICMP and SSH in the default security group so we will be able to connect to them:

. ~/keystonerc_demo
neutron security-group-rule-create --direction ingress \
   --ethertype IPv4 --protocol tcp --port-range-min 22 \
   --port-range-max 22 default

neutron security-group-rule-create --direction ingress \
   --ethertype IPv4 --protocol icmp default

nova boot --flavor m1.tiny --image cirros --nic\
 net-id=$(neutron net-show private1-demo -c id -f value) demo01-private1

nova floating-ip-create external1

nova floating-ip-associate demo01-private1 $(neutron floatingip-list -c floating_ip_address -f value)

nova list  #the floating IP should be pingable and you should be able to ssh to this instance through it. 

What did you just do? You created external networks and an instance on a private tenant network with the following characteristics:

Outbound access: This instance can access the external1 network through its router (router1-demo). Its source address will be 'SNATed' to that of its floating IP address. That 'SNATing' takes place in the router namespace. If this instance did not have a floating IP, however, it would still have external1 network access. The router, in this case, will SNAT the source address to be that of its own public IP address.  

Inbound access: This instance is known to the outside world by it’s floating IP address, which router1-demo will respond to ARP requests for. That IP is DNAT’ed to that of its local IP address in the router namespace.  

Note: to see these NATing rules dump them from the router namespace:

. ~/keystonerc_demo
ip netns exec qrouter-$(neutron router-show router1-demo -c id -f value) iptables -S -t nat

Tenant network access: This instance is in the same broadcast domain as any other instance on the private1-demo network. Because the vxlan overlay network abstracts the underlying physical network, this broadcast domain extends across compute nodes where these instances are hosted.  

View the network topology from the Horizon console

. ~/keystonerc_admin
echo $OS_AUTH_URL
echo $OS_USERNAME
echo $OS_PASSWORD

Point your browser at the URL above. Log in with the OS_USERNAME and OS_PASSWORD from ~/keystonerc_admin. From the ‘Project’ tab select the ‘demo’ project (you will need to add admin as a member of the demo project first). Now select ‘Network/Network Topology.’ Here you’ll see the networks we created; be sure to try the two views - topology and graph. Keep this view in front of you as we try to understand what we are doing. 

Shared tenant network

A tenant network can be created with the ‘--shared’ attribute which allows other tenants to attach their own instances to it. By default, only the admin tenant can create a shared tenant network but it is possible for other tenants to do so with RBAC. See Role-Based Access Control for networks. This type of network can be useful when two or more projects have instances that benefit from being in the same broadcast domain, thus bypassing the need to share with floating IP addresses.  

. ~/keystonerc_admin
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0 #allow icmp through default security group

nova secgroup-add-rule default tcp 22 22 0.0.0.0/0 #allow ssh through default security group

neutron net-create --shared --router:external=false admin1-shared

neutron subnet-create admin1-shared 10.0.9.0/24 --name admin1-shared-subnet  

Now create an instance on the shared tenant network from the admin project:

nova boot --flavor m1.tiny --image cirros --nic\
 net-id=$(neutron net-show admin1-shared -c id -f value) admin01-admin1-shared

Now create an instance on the shared tenant network from the demo project:

. ~/keystonerc_demo
nova boot --flavor m1.tiny --image cirros --nic\
 net-id=$(neutron net-show admin1-shared -c id -f value) demo02-admin1-shared

What did you just do? If you log into either of these instances using the instance/console option from Horizon you will notice that they are on the same subnet, even though they are in different projects. This type of network can be useful for sharing instances between projects; the only other choice for doing so is with floating IPs. The shared network (admin1-shared) can be seen and joined by any tenant by default. Note that there is no external network access from this network unless you add a router and set the gateway to external1.  

Shared external network

On a shared, external network tenants can connect their instances directly to an external network and get a floating IP address assigned automatically. We’ll use the external2-shared network already created above.  

. ~/keystonerc_demo
nova boot --flavor m1.tiny --image cirros --nic net-name=external2-shared demo03-external2-shared

nova console-log demo03-external2-shared 

You will probably notice from the console log output that the instance failed to contact the metadata service:

checking http://169.254.169.254/2009-04-04/instance-id
failed 1/20: up 3.03. request failed
failed 2/20: up 8.09. request failed
failed 3/20: up 11.09. request failed
etc…

You may be able to ping the instance, but if you want instances on external2-shared to have access to the metadata service then in /etc/neutron/dhcp_agent.ini set:

crudini --set /etc/neutron/dhcp_agent.ini DEFAULT enable_isolated_metadata True

systemctl restart neutron-dhcp-agent

This is necessary because you are not connecting to this network through a router, which is normally where metadata service access comes from. Now reboot the instance and check the console log again:

nova reboot demo03-external2-shared

nova console-log dem03-external2-shared 

Now you should see this in the log:

checking http://169.254.169.254/2009-04-04/instance-id
successful after 1/20 tries: up 2.04. iid=i-000000ef

What did we just do? By creating an instance on this shared, external network you bypass the need for using a router for external access. Additionally, you get what amounts to a floating IP by default. Lastly, the broadcast domain is the external subnet. So, any instance on this subnet is in the same broadcast domain regardless of the tenant, the same characteristic of the shared tenant networks we looked at previously. If a tenant is accessing instances by floating IPs only on their private network then they might be better off with a shared, external network.

Use the right network attributes for the right job

By understanding the characteristics of these four types of networks you’ll be off to a good start in harnessing the flexibility of Neutron. Floating IP’s can get to be a scarce IPv4 resource, for example. As a workaround, you could rely on NAT, or use shared private networks to save yourself some headaches. By understanding your use-cases you’ll be able to select the right network type for yourself and/or your customers.

This article was originally published on the Red Hat blog as a part of a series from Red Hat Technical Account Managers (TAM).