Kubernetes is designed to integrate with major cloud providers' load balancers to provide public IP addresses and direct traffic into a cluster. Some professional network equipment manufacturers also offer controllers to integrate their physical load-balancing products into Kubernetes installations in private data centers. For an enthusiast running a Kubernetes cluster at home, however, neither of these solutions is very helpful.
Kubernetes does not have a built-in network load-balancer implementation. A bare-metal cluster, such as a Kubernetes cluster installed on Raspberry Pis for a private-cloud homelab, or really any cluster deployed outside a public cloud and lacking expensive professional hardware, needs another solution. MetalLB fulfills this niche, both for enthusiasts and large-scale deployments.
MetalLB is a network load balancer and can expose cluster services on a dedicated IP address on the network, allowing external clients to connect to services inside the Kubernetes cluster. It does this via either layer 2 (data link) using Address Resolution Protocol (ARP) or layer 4 (transport) using Border Gateway Protocol (BGP).
While Kubernetes does have something called Ingress, which allows HTTP and HTTPS traffic to be exposed outside the cluster, it supports only HTTP or HTTPS traffic, while MetalLB can support any network traffic. It is more of an apples-to-oranges comparison, however, because MetalLB provides resolution of an unassigned IP address to a particular cluster node and assigns that IP to a Service, while Ingress uses a specific IP address and internally routes HTTP or HTTPS traffic to a Service or Services based on routing rules.
MetalLB can be set up in just a few steps, works especially well in private homelab clusters, and within Kubernetes clusters, it behaves the same as public cloud load-balancer integrations. This is great for education purposes (i.e., learning how the technology works) and makes it easier to "lift-and-shift" workloads between on-premises and cloud environments.
ARP vs. BGP
As mentioned, MetalLB works via either ARP or BGP to resolve IP addresses to specific hosts. In simplified terms, this means when a client attempts to connect to a specific IP, it will ask "which host has this IP?" and the response will point it to the correct host (i.e., the host's MAC address).
With ARP, the request is broadcast to the entire network, and a host that knows which MAC address has that IP address responds to the request; in this case, MetalLB's answer directs the client to the correct node.
With BGP, each "peer" maintains a table of routing information directing clients to the host handling a particular IP for IPs and the hosts the peer knows about, and it advertises this information to its peers. When configured for BGP, MetalLB peers each of the nodes in the cluster with the network's router, allowing the router to direct clients to the correct host.
More on Kubernetes
- What is Kubernetes?
- eBook: Storage Patterns for Kubernetes
- Test drive OpenShift hands-on
- eBook: Getting started with Kubernetes
- An introduction to enterprise Kubernetes
- How to explain Kubernetes in plain terms
- eBook: Running Kubernetes on your Raspberry Pi homelab
- Kubernetes cheat sheet
- eBook: A guide to Kubernetes for SREs and sysadmins
- Latest Kubernetes articles
In both instances, once the traffic has arrived at a host, Kubernetes takes over directing the traffic to the correct pods.
For the following exercise, you'll use ARP. Consumer-grade routers don't (at least easily) support BGP, and even higher-end consumer or professional routers that do support BGP can be difficult to set up. ARP, especially in a small home network, can be just as useful and requires no configuration on the network to work. It is considerably easier to implement.
Installing MetalLB is straightforward. Download or copy two manifests from MetalLB's GitHub repository and apply them to Kubernetes. These two manifests create the namespace MetalLB's components will be deployed to and the components themselves: the MetalLB controller, a "speaker" daemonset, and service accounts.
Install the components
Once you create the components, a random secret is generated to allow encrypted communication between the speakers (i.e., the components that "speak" the protocol to make services reachable).
(Note: These steps are also available on MetalLB's website.)
The two manifests with the required MetalLB components are:
They can be downloaded and applied to the Kubernetes cluster using the
kubectl apply command, either locally or directly from the web:
# Verify the contents of the files, then download and pipe then to kubectl with curl # (output omitted) $ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml $ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
After applying the manifests, create a random Kubernetes secret for the speakers to use for encrypted communications:
# Create a secret for encrypted speaker communications $ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
Completing the steps above will create and start all the MetalLB components, but they will not do anything until they are configured. To configure MetalLB, create a configMap that describes the pool of IP addresses the load balancer will use.
Configure the address pools
MetalLB needs one last bit of setup: a configMap with details of the addresses it can assign to the Kubernetes Service LoadBalancers. However, there is a small consideration. The addresses in use do not need to be bound to specific hosts in the network, but they must be free for MetalLB to use and not be assigned to other hosts.
In my home network, IP addresses are assigned by the DHCP server my router is running. This DHCP server must not attempt to assign the addresses that MetalLB uses. Most consumer routers allow you to decide the size of your subnet, and can be configured to assign only a subset of IPs in that subnet to hosts.
For MetalDB, you can specify a list of addresses as a range:
You can alternately use CIDR notation. In my network, for instance, I use the subnet
192.168.2.1/24, and I decided to give half the IPs to MetalLB. The first half of the subnet consists of IP addresses from
192.168.2.126. This range can be represented by a
192.168.2.1/25. The second half of the subnet can similarly be represented by a
192.168.2.128/25. Each half contains 126 IPs—more than enough for the hosts and Kubernetes services. Make sure to decide on subnets appropriate to your own network and configure your router and MetalLB appropriately.
After configuring the router to ignore addresses in the
192.168.2.128/25 subnet (or whatever subnet you are using), create a configMap to tell MetalLB to use that pool of addresses:
# Create the config map $ cat <<EOF | kubectl create -f - apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | address-pools: - name: address-pool-1 protocol: layer2 addresses: - 192.168.2.128/25 EOF
Once the configMap is created, MetalLB is active. Time to try it out!
You can test the new MetalLB configuration by creating an example web service, and you can use one from a previous article in this series: Kube Verify. Use the same image to test that MetalLB is working as expected:
quay.io/clcollins/kube-verify:01. This image contains an Nginx server listening for requests on port 8080. You can view the Containerfile used to create the image. If you want, you can instead build your own container image from the Containerfile and use that for testing.
If you previously created a Kubernetes cluster on Raspberry Pis, you may already have a Kube Verify service running and can skip to the section on creating a LoadBalancer-type of service.
If you need to create a kube-verify namespace
If you do not already have a
kube-verify namespace, create one with the
# Create a new namespace $ kubectl create namespace kube-verify # List the namespaces $ kubectl get namespaces NAME STATUS AGE default Active 63m kube-node-lease Active 63m kube-public Active 63m kube-system Active 63m metallb-system Active 21m kube-verify Active 19s
With the namespace created, create a deployment in that namespace:
# Create a new deployment $ cat <<EOF | kubectl create -f - apiVersion: apps/v1 kind: Deployment metadata: name: kube-verify namespace: kube-verify labels: app: kube-verify spec: replicas: 3 selector: matchLabels: app: kube-verify template: metadata: labels: app: kube-verify spec: containers: - name: nginx image: quay.io/clcollins/kube-verify:01 ports: - containerPort: 8080 EOF deployment.apps/kube-verify created
Now expose the deployment by creating a LoadBalancer-type Kubernetes service. If you already have a service named
kube-verify, this will replace that one:
# Create a LoadBalancer service for the kube-verify deployment cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Service metadata: name: kube-verify namespace: kube-verify spec: selector: app: kube-verify ports: - protocol: TCP port: 80 targetPort: 8080 type: LoadBalancer EOF
You could accomplish the same thing with the
kubectl expose command:
kubectl expose deployment kube-verify -n kube-verify --type=LoadBalancer --target-port=8080 --port=80
MetalLB is listening for services of type LoadBalancer and immediately assigns an external IP (an IP chosen from the range you selected when you set up MetalLB). View the new service and the external IP address MetalLB assigned to it with the
kubectl get service command:
# View the new kube-verify service $ kubectl get service kube-verify -n kube-verify NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-verify LoadBalancer 10.105.28.147 192.168.2.129 80:31491/TCP 4m14s # Look at the details of the kube-verify service $ kubectl describe service kube-verify -n kube-verify Name: kube-verify Namespace: kube-verify Labels: app=kube-verify Annotations: <none> Selector: app=kube-verify Type: LoadBalancer IP: 10.105.28.147 LoadBalancer Ingress: 192.168.2.129 Port: <unset> 80/TCP TargetPort: 8080/TCP NodePort: <unset> 31491/TCP Endpoints: 10.244.1.50:8080,10.244.1.51:8080,10.244.2.36:8080 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal IPAllocated 5m55s metallb-controller Assigned IP "192.168.2.129" Normal nodeAssigned 5m55s metallb-speaker announcing from node "gooseberry"
In the output from the
kubectl describe command, note the events at the bottom, where MetalLB has assigned an IP address (yours will vary) and is "announcing" the assignment from one of the nodes in your cluster (again, yours will vary). It also describes the port, the external port you can access the service from (80), the target port inside the container (port 8080), and a node port through which the traffic will route (31491). The end result is that the Nginx server running in the pods of the
kube-verify service is accessible from the load-balanced IP, on port 80, from anywhere on your home network.
For example, on my network, the service was exposed on
http://192.168.2.129:80, and I can
curl that IP from my laptop on the same network:
# Verify that you receive a response from Nginx on the load-balanced IP $ curl 192.168.2.129 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title>Test Page for the HTTP Server on Fedora</title> (further output omitted)
MetalLB is a great load balancer for a home Kubernetes cluster. It allows you to assign real IPs from your home network to services running in your cluster and access them from other hosts on your home network. These services can even be exposed outside the network by port-forwarding traffic through your home router (but please be careful with this!). MetalLB easily replicates cloud-provider-like behavior at home on bare-metal computers, Raspberry Pi-based clusters, and even virtual machines, making it easy to "lift-and-shift" workloads to the cloud or just familiarize yourself with how they work. Best of all, MetalLB is easy and convenient and makes accessing the services running in your cluster a breeze.
Have you used MetalLB, or do you use another load-balancer solution? Are you primarily using Nginx or HAProxy Ingress? Let me know in the comments!