Build k8s Cluster on Hyper-V

Robert Wang - Apr 22 - - Dev Community

Intro

I used Virtualbox to launch Linux vm using Vagrant for many years, have to switch to Hyper-V due to it by default enabled in win10 host.

The outstanding point comparing to Virtualbox is that it supports nested virtualization, means you can use kvm hypervisor on top of Hyper-V Linux VM, to launch nested vm easily.

The drawback or learning curve with Hyper-V is that network model is different than Virtualbox.

I will address few pain points and go through completely k8s lab deployment on top of Hyper-V on win10 host in this post.

  • static ip for Hyper-V vm and permanent ip for k8s API server
  • experiment k8s setup automation with Vagrant

Lab env

  • Win10 32GB RAM
  • Hyper-V
  • vagrant 2.4.1 windows
  • WSL2 optional as terminal

k8s hyperv tool set

We heavy use handy scripts to make k8s cluster on Hyper-V automatically, you can clone to local or download them manually,

git clone git@github.com:robertluwang/hands-on-nativecloud.git
cd ./hands-on-nativecloud/src/k8s-cri-dockerd
Enter fullscreen mode Exit fullscreen mode

Hyper-V network

3 types:

  • private - vm<->vm only
  • internal - vm<->vm, vm<->host, vm->Internet if NAT enabled
  • external - kind of bridge, vm<->host, vm->Internet

private network
It is isolated network, vm can access each other only.

internal network
It is private network plus host communication due to virtual NIC on host.

It is similar with Host-only in Virtualbox.

It could be further as NAT to allow vm to access Internet by adding internal network in NetNat, which enabling the routing traffic to outside.

The "Default Switch" is default NAT switch, the only issue is switch ip always changed after reboot; also DNS not working well. As remedy you can manually change vm ip and DNS inside vm, to match new change on Default Switch.

The good news is that it is possible to setup own static NAT network, to have stable test lab env, which is host-only + NAT if comparing with Virtualbox.

external switch
External switch will bind with physical NIC on host, sharing same network with host to access outside, ip assigned from DHCP.

In my pc there is only Wi-Fi adaptor, so external switch will bind to Wi-Fi adaptor, it always crashes and lost all Wi-Fi Internet access during creating, have to recover by Network Reset.

Let's forget about external switch for wi-fi at all, since it will directly impact host network access.

So Internal network + NAT is best and stable option for static IP, it allow host and vm talk each other and also access to Internet.

Create own NAT switch

We need to create new Hyper-V switch for example myNAT as type of Internal either from Hyper-V Manager GUI or Powershell as admin role, then create vNIC network like 192.168.120.0/24, and enable as NAT.

You can run PowerShell term as admin on win10 host, then run below script to complete vm switch, vNIC and NAT setup.

Please change your prefer values in script:

  • hyperv switch name: myNAT
  • vEthernet ip: 192.168.120.1 it is gateway on win host
  • NetNAT name: myNAT
  • NetNAT network: 192.168.120./24

myNAT.ps1

If ("myNAT" -in (Get-VMSwitch | Select-Object -ExpandProperty Name) -eq $FALSE) {
    'Creating Internal-only switch "myNAT" on Windows Hyper-V host...'

    New-VMSwitch -SwitchName "myNAT" -SwitchType Internal

    New-NetIPAddress -IPAddress 192.168.120.1 -PrefixLength 24 -InterfaceAlias "vEthernet (myNAT)"

    New-NetNAT -Name "myNAT" -InternalIPInterfaceAddressPrefix 192.168.120.0/24
}
else {
    'Internal-only switch "myNAT" for static IP configuration already exists; skipping'
}

If ("192.168.120.1" -in (Get-NetIPAddress | Select-Object -ExpandProperty IPAddress) -eq $FALSE) {
    'Registering new IP address 192.168.120.1 on Windows Hyper-V host...'

    New-NetIPAddress -IPAddress 192.168.120.1 -PrefixLength 24 -InterfaceAlias "vEthernet (myNAT)"
}
else {
    'IP address "192.168.120.1" for static IP configuration already registered; skipping'
}

If ("192.168.120.0/24" -in (Get-NetNAT | Select-Object -ExpandProperty InternalIPInterfaceAddressPrefix) -eq $FALSE) {
    'Registering new NAT adapter for 192.168.120.0/24 on Windows Hyper-V host...'

    New-NetNAT -Name "myNAT" -InternalIPInterfaceAddressPrefix 192.168.120.0/24
}
else {
    '"192.168.120.0/24" for static IP configuration already registered; skipping'
}
Enter fullscreen mode Exit fullscreen mode

Here is sample of running result,

.\myNAT.ps1
Creating Internal-only switch named "myNAT" on Windows Hyper-V host...

Name   SwitchType NetAdapterInterfaceDescription
----   ---------- ------------------------------
myNAT Internal                                 

IPv4Address              : 192.168.120.1
IPv6Address              : 
IPVersionSupport         : 
PrefixLength             : 24
SubnetMask               : 
AddressFamily            : IPv4
AddressState             : Tentative
InterfaceAlias           : vEthernet (myNAT)
InterfaceIndex           : 109
IPAddress                : 192.168.120.1
PreferredLifetime        : 10675199.02:48:05.4775807
PrefixOrigin             : Manual
SkipAsSource             : False
Store                    : ActiveStore
SuffixOrigin             : Manual
Type                     : Unicast
ValidLifetime            : 10675199.02:48:05.4775807
PSComputerName           : 
ifIndex                  : 109

IPv4Address              : 192.168.120.1
IPv6Address              : 
IPVersionSupport         : 
PrefixLength             : 24
SubnetMask               : 
AddressFamily            : IPv4
AddressState             : Invalid
InterfaceAlias           : vEthernet (myNAT)
InterfaceIndex           : 109
IPAddress                : 192.168.120.1
PreferredLifetime        : 10675199.02:48:05.4775807
PrefixOrigin             : Manual
SkipAsSource             : False
Store                    : PersistentStore
SuffixOrigin             : Manual
Type                     : Unicast
ValidLifetime            : 10675199.02:48:05.4775807
PSComputerName           : 
ifIndex                  : 109

Caption                          : 
Description                      : 
ElementName                      : 
InstanceID                       : myNAT;0
Active                           : True
ExternalIPInterfaceAddressPrefix : 
IcmpQueryTimeout                 : 30
InternalIPInterfaceAddressPrefix : 192.168.120.0/24
InternalRoutingDomainId          : {00000000-0000-0000-0000-000000000000}
Name                             : myNAT1
Store                            : Local
TcpEstablishedConnectionTimeout  : 1800
TcpFilteringBehavior             : AddressDependentFiltering
TcpTransientConnectionTimeout    : 120
UdpFilteringBehavior             : AddressDependentFiltering
UdpIdleSessionTimeout            : 120
UdpInboundRefresh                : False
PSComputerName                   : 

IP address "192.168.120.1" for static IP configuration already registered; skipping
"192.168.120.0/24" for static IP configuration already registered; skipping
Enter fullscreen mode Exit fullscreen mode

Also there is a handy PowerShell script to delete hyper-v switch myNAT, vNIC and associated NAT, in case tear down NAT switch or reset it.
myNAT-delete.ps1

If (Get-VMSwitch | ? Name -Eq myNAT) {
    'Deleting Internal-only switch named myNAT on Windows Hyper-V host...'
    Remove-VMSwitch -Name myNAT
}
else {
    'VMSwitch "myNAT" not exists; skipping'
}

If (Get-NetIPAddress | ? IPAddress -Eq "192.168.120.1") {
    'Deleting IP address 192.168.120.1 on Windows Hyper-V host...'
    Remove-NetIPAddress -IPAddress "192.168.120.1"
}
else {
    'IP address "192.168.120.1" not existing; skipping'
}

If (Get-NetNAT | ? Name -Eq myNAT) {
    'Deleting NAT adapter myNAT on Windows Hyper-V host...'
    Remove-NetNAT -Name myNAT
}
else {
    'NAT adapter myNAT not existing; skipping'
}
Enter fullscreen mode Exit fullscreen mode

Bring up static host-only ip with Vagrant

Until now the Vagrant not supports well with Hyper-V network configuration like Virtualbox in Vagrantfile.

https://developer.hashicorp.com/vagrant/docs/providers/hyperv/limitations

host-only NIC for Virtualbox,

        vm.network :private_network, ip: "192.168.99.30"
Enter fullscreen mode Exit fullscreen mode

then what should we do in hyper-v Vagrantfile? we only can choice network switcher, not chance to assign static ip.

Good news is that Vagrant can ssh to new launched vm via ipv6 IP, means we have chance to run any provision script, so as remedy we can run a shell inline or offline in provision stage, to setup static ip to eth0 manually, also correct /etc/resolv.conf for DNS setting.

Here is sample to launch Ubuntu 22.04 vm with 192.168.120.20 with Vagrant, to demo how static ip setup works for Ubuntu vm using Vagrantfile.

oldhorse@wsl2:$ cat Vagrantfile
$nic = <<SCRIPT

echo === $(date) Provisioning - nic $1 by $(whoami) start

SUBNET=$(echo $1 | cut -d"." -f1-3)

cat <<EOF | sudo tee /etc/netplan/01-netcfg.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: no
      dhcp6: no
      addresses: [$1/24]
      routes:
      - to: default
        via: ${SUBNET}.1
      nameservers:
        addresses: [8.8.8.8,1.1.1.1]
EOF

sudo unlink /etc/resolv.conf
sudo rm /etc/resolv.conf
cat << EOF | sudo tee /etc/resolv.conf
nameserver 8.8.8.8
nameserver 1.1.1.1
EOF

sudo chattr +i /etc/resolv.conf

cat /etc/netplan/01-netcfg.yaml
cat /etc/resolv.conf

sudo netplan apply
sleep 30 
echo eth0 setting

ip addr
ip route
ping -c 2 google.ca

echo === $(date) Provisioning - nic $1 by $(whoami) end
SCRIPT

Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu2204"
  config.vm.provision "shell", inline: "sudo timedatectl set-timezone America/Montreal", privileged: false, run: "always"
  config.ssh.insert_key = false
  config.vm.box_check_update = false
  config.vm.network "public_network", bridge: "myNAT"

  config.vm.define "master" do |master|
      master.vm.hostname = "ub22hpv"
      master.vm.provider "hyperv" do |v|
          v.vmname = "ub22hpv"
          v.memory = 1024
      end
      master.vm.provision "shell", inline: $nic, args: "192.168.120.20", privileged: false
  end
end
Enter fullscreen mode Exit fullscreen mode

We can see vagrant trying to ssh to vm via ipv6 with private key,

    master: SSH address: fe80::215:5dff:fec9:606c:22
    master: SSH username: vagrant
    master: SSH auth method: private key
Enter fullscreen mode Exit fullscreen mode

Full vagrant up log as below,

oldhorse@wsl2: vagrant.exe up
Bringing machine 'master' up with 'hyperv' provider...
==> master: Verifying Hyper-V is enabled...
==> master: Verifying Hyper-V is accessible...
==> master: Importing a Hyper-V instance
    master: Creating and registering the VM...
    master: Successfully imported VM
    master: Configuring the VM...
    master: Setting VM Enhanced session transport type to disabled/default (VMBus)
==> master: Starting the machine...
==> master: Waiting for the machine to report its IP address...
    master: Timeout: 120 seconds
    master: IP: fe80::215:5dff:fec9:606c
==> master: Waiting for machine to boot. This may take a few minutes...
    master: SSH address: fe80::215:5dff:fec9:606c:22
    master: SSH username: vagrant
    master: SSH auth method: private key
==> master: Machine booted and ready!
==> master: Setting hostname...
==> master: Running provisioner: shell...
    master: Running: inline script
==> master: Running provisioner: shell...
    master: Running: inline script
    master: === Sun Apr 21 05:36:13 PM EDT 2024 Provisioning - nic 192.168.120.20 by vagrant start
    master: network:
    master:   version: 2
    master:   renderer: networkd
    master:   ethernets:
    master:     eth0:
    master:       dhcp4: no
    master:       dhcp6: no
    master:       addresses: [192.168.120.20/24]
    master:       routes:
    master:       - to: default
    master:         via: 192.168.120.1
    master:       nameservers:
    master:         addresses: [8.8.8.8,1.1.1.1]
    master: rm: cannot remove '/etc/resolv.conf': No such file or directory
    master: nameserver 8.8.8.8
    master: nameserver 1.1.1.1
    master: network:
    master:   version: 2
    master:   renderer: networkd
    master:   ethernets:
    master:     eth0:
    master:       dhcp4: no
    master:       dhcp6: no
    master:       addresses: [192.168.120.20/24]
    master:       routes:
    master:       - to: default
    master:         via: 192.168.120.1
    master:       nameservers:
    master:         addresses: [8.8.8.8,1.1.1.1]
    master: nameserver 8.8.8.8
    master: nameserver 1.1.1.1
    master: 
    master: ** (generate:3029): WARNING **: 17:36:14.124: Permissions for /etc/netplan/00-installer-config.yaml are too open. Netplan configuration should NOT be accessible by others.

    master: eth0 setting
    master: 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    master:     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    master:     inet 127.0.0.1/8 scope host lo
    master:        valid_lft forever preferred_lft forever
    master: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    master:     link/ether 00:15:5d:c9:60:6c brd ff:ff:ff:ff:ff:ff
    master:     inet 192.168.120.20/24 brd 192.168.120.255 scope global eth0
    master:        valid_lft forever preferred_lft forever
    master:     inet6 fe80::215:5dff:fec9:606c/64 scope link
    master:        valid_lft forever preferred_lft forever
    master: default via 192.168.120.1 dev eth0 proto static
    master: 192.168.120.0/24 dev eth0 proto kernel scope link src 192.168.120.20
    master: PING google.ca (142.250.65.163) 56(84) bytes of data.
    master: 64 bytes from lga25s71-in-f3.1e100.net (142.250.65.163): icmp_seq=1 ttl=105 time=30.8 ms
    master: 64 bytes from lga25s71-in-f3.1e100.net (142.250.65.163): icmp_seq=2 ttl=105 time=69.3 ms
    master:
    master: --- google.ca ping statistics ---
    master: 2 packets transmitted, 2 received, 0% packet loss, time 1004ms
    master: rtt min/avg/max/mdev = 30.791/50.024/69.258/19.233 ms
    master: === Sun Apr 21 05:36:45 PM EDT 2024 Provisioning - nic 192.168.120.20 by vagrant end
Enter fullscreen mode Exit fullscreen mode

Build up k8s cluster with Hyper-V using Vagrant

After we make hyper-v vm with static ip works, then it is possible to install Docker, k8s package and build up k8s cluster via provision inline script.

We assume it is 2 nodes k8s cluster:

  • Ubuntu 22.04
  • master: 2GB RAM 1 cpu 192.168.120.20
  • worker: 1GB RAM 1 cpu 192.168.120.30
  • Docker: latest version
  • k8s: latest version , it is 1.30 here

It spent around 26 mins to build up k8s 2 nodes cluster very smoothly,

oldhorse@wsl2: date; vagrant.exe up ; date 
Sun Apr 21 18:05:06 EDT 2024
Bringing machine 'master' up with 'hyperv' provider...
Bringing machine 'worker' up with 'hyperv' provider...
...
master: kubeadm join 192.168.120.20:6443 --token y10r7h.1vycfilas9kpovzp \
    master:     --discovery-token-ca-cert-hash sha256:64e627afc4ed15beb607adf4073ae856400097fb4fed248094fb679e235cc597

...
worker: This node has joined the cluster:
worker: * Certificate signing request was sent to apiserver and a response was received.
...
Sun Apr 21 18:31:33 EDT 2024
Enter fullscreen mode Exit fullscreen mode

check k8s cluster, it is running as 2 nodes cluster.

We can ssh to master vm using vagrant, or directly ssh to 192.168.120.20 master and 192.168.120.30 worker.

vagrant ssh master
Enter fullscreen mode Exit fullscreen mode

It looks good,

vagrant@master:~$ k get node
NAME     STATUS   ROLES           AGE   VERSION
master   Ready    control-plane   31m   v1.30.0
worker   Ready    <none>          19m   v1.30.0

vagrant@master:~$ k get pod -A
NAMESPACE          NAME                                       READY   STATUS    RESTARTS   AGE
calico-apiserver   calico-apiserver-7868f6b748-glqdx          1/1     Running   0          11m
calico-apiserver   calico-apiserver-7868f6b748-kc4gq          1/1     Running   0          11m
calico-system      calico-kube-controllers-78788579b8-vvrdh   1/1     Running   0          30m
calico-system      calico-node-dlrs8                          1/1     Running   0          19m
calico-system      calico-node-vvxm6                          1/1     Running   0          30m
calico-system      calico-typha-65bf4f686-mlpbx               1/1     Running   0          30m
kube-system        coredns-76f75df574-65mk8                   1/1     Running   0          31m
kube-system        coredns-76f75df574-z982z                   1/1     Running   0          31m
kube-system        etcd-master                                1/1     Running   0          31m
kube-system        kube-apiserver-master                      1/1     Running   0          31m
kube-system        kube-controller-manager-master             1/1     Running   0          31m
kube-system        kube-proxy-dcfcc                           1/1     Running   0          19m
kube-system        kube-proxy-x7qtl                           1/1     Running   0          31m
kube-system        kube-scheduler-master                      1/1     Running   0          31m
tigera-operator    tigera-operator-6fbc4f6f8d-h5wcm           1/1     Running   0          

vagrant@master:~$ k run test --image=nginx $do 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: test
  name: test
spec:
  containers:
  - image: nginx
    name: test
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

vagrant@master:~$ k run test --image=nginx 
pod/test created
vagrant@master:~$ k get pod 
NAME   READY   STATUS              RESTARTS   AGE
test   0/1     ContainerCreating   0          4s

vagrant@master:~$ k get pod -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP               NODE     NOMINATED NODE   READINESS GATES
test   1/1     Running   0          42s   192.168.171.67   worker   <none>           <none>

vagrant@master:~$ curl 192.168.171.67
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
</html>
Enter fullscreen mode Exit fullscreen mode

k8s setup tool set

In above example, we demo the quick way to launch k8s cluster using Vagrant, which uses inline scripts to install docker, setup cri-dockerd, install k8s packages, init k8s cluster or join to cluster on worker node.

In fact we can use scripts for handy setup k8s cluster when you already have running Linux node, no matter it is physical Linux, Linux vm, or WSL2.

Just run script as sudo user,

master node

docker-server.sh
k8s-install.sh
cri-dockerd.sh
k8s-init.sh "192.168.120.20"
Enter fullscreen mode Exit fullscreen mode

worker node

docker-server.sh
k8s-install.sh
cri-dockerd.sh
k8s-join-worker.sh "192.168.120.20"
Enter fullscreen mode Exit fullscreen mode

Conclusion

We make the k8s 2 nodes cluster setup automation smoothly using Vagrant:

  • it is possible to make k8s cluster setup on Hyper-V full automatically
  • it is ideal local k8s lab for production like testing and education

About Me

Hey! I am Robert Wang, live in Montreal.

More simple and more efficient.

Image
Our prime Term regarding PTPRH Is assigned to Very poor Diagnosis of Individual Lung Adenocarcinoma.
Image
Researching Intra-Arterial, Auscultatory, and Oscillometric Measurement Means of Arterial Blood Pressure.
Image
Ovarian damaged tissues soon after grafting: methodical overview of methods to increase hair follicle final results.
Image
98win charity
Image
[Electroretinography in evaluation of neuroprotective result in an trial and error model of glaucoma].
Image
Pb2+-Containing Metal-Organic Rotaxane Frameworks (MORFs).
Image
Assessing the grade of Expertise Performance Statistic with regard to UAV-Based Cpa networks.
Image
Heat-treated malware inactivation price depends highly upon therapy method: case in point using SARS-CoV-2.
Image
5 Tools That Everyone Who Works In The Lightweight Folding Wheelchairs Industry Should Be Making Use Of
Image
Top 10 Reasons to Hire a Full-Time Blockchain Developer from Vietnam

blockchain, recruitment, developer, development
Image
The Top Reasons Why People Succeed With The Bi Fold Door Repairs Near Me Industry
Image
Do You Think Bifold Doors Repair Always Rule The World?
Image
Pavel Durov's Arrest: Controversial Charges and Impacts on Free Speech

paveldurov, telegram, freespeech, legalissues
Image
HR management system

softwaredevelopment
Image
Adaptable Turned Handle regarding Related Car Platoon Along with Unidentified Insight Waiting times.
Image
What Is Oxblood Chesterfield? History Of Oxblood Chesterfield
Image
Bilateral Pseudo-hypopyon since Delivering Manifestation of Serious Monocytic Leukemia in an 8-Month-Old Child.
Image
Biochemical Investigation regarding 3-Sulfopropionaldehyde Reductase HpfD.
Image
Projecting unplanned hospital sessions in old home care individuals: a new cross-country outer approval study.
. . . .
Terabox Video Player