feat(networking): ipv6-based routing

[markovejnovic]

This is a feature branch which implements ipv6-based routing. At the time of writing, this PR completely tears out all of hyper's networking logic, with the explicit aim of rewriting the networking stack from the ground up.

Requirements

Whatever kind of networking solution we come up with needs to support the following.

Routing at L3

@jrimmer has proposed in #74 that we stick a reverse http proxy on the bare metal node. This is rejected because using such a system prevents VMs from serving any other protocol. VMs should feel as close to native as we can make them, so and I want to be able to support anything on IP. This includes UDP, TCP, but also more less known protocols such as SCTP, UDP Lite, etc. Explicitly doing so allows practical application protocols that aren't based on HTTP, including:

• SSH -- of course you should be able to ssh into a VM.
• RDP -- why not share the screen of a VM?
• Kafka, AQMP allow people to spin up VMs for their ephemeral workloads in Kafka and RMQ workloads. That's ideal for spot behavior.
• SMTP, IMAP, POP3 enable email.
• DNS, DHCP, NTP are all protocols that run over UDP.

The list is effectively infinite, and limiting Hyper to just HTTP feels like a cheap way out of a more generally applicable solution.

Sandboxing and Isolation

TODO

Latency & Bandwidth

TODO

Options

Luckily we've got IPv6 which rocks. Based on my research we have the following options.

Bridge

A Linux bridge interface is an interface that acts like an L2 switch. RedHat has a nice introduction. The effective idea is relatively simple:

flowchart TD
  router["external router (could be the internet)"]

  subgraph node
    if0["if0, public interface"]
    br0["br0, bridge interface"]
    
    subgraph vm1
      vif0["vif0, vm1's interface"]
    end

    subgraph vm2
      vif1["vif1, vm2's interface"]
    end

    if0 <--> br0
    br0 <---> vif0
    br0 <---> vif1
  end

  router <---> if0

Loading

This kind of configuration implies:

• Each VM is responsible for allocating its own MAC and IP. The bridge maintains a forwarding database (FDB) which maps between the VM's MAC address to the appropriate specific interface. A misbehaving VM can identify itself as a different MAC/IP and then sniff on traffic aimed for a neighbor VM. libvirt has a clean-network filter and we can mitigate this with nftables rules too. I suspect the things I listed are not the only possible concerns.
• The CPU of the hypervisor will be spending some time routing packets. Necessary evil.

Host-managed tunnel adapters

flowchart TD
  router["external router (could be the internet)"]

  subgraph node
    if0["if0, public interface 2001:19f0:5c00:19b9:5400:6ff:fe12:1a00/64 and proxy NDP on 111:1::1"]
    
	tap1["tap1, ...:1::2/128"]
    subgraph vm1
      vif0["vif0, vm1's interface 2001:19f0:5c00:19b9:1::2/128"]
    end
	
	tap2["tap2, ...:1::3/128"]
    subgraph vm2
      vif1["vif1, vm2's interface 2001:19f0:5c00:19b9:1::3/128"]
    end

    if0 <--->|L3 routing| tap1
    if0 <--->|L3 routing| tap2
    tap1 <--> vif0
    tap2 <--> vif1
  end

  router <---> if0

Loading

This idea builds on top of the aforementioned bridge idea, however it allocates an additional tunnel adapter on the host, managed by Hyper. Hyper is responsible for allocating the interfaces, their MACs and their IPs. This makes me feel a lot better, since the hypervisor has total control over the IP addresses.

This relies on NDP, which the Hypervisor would be responsible for setting up.

System Components

• Hyper.Node.Network.Tap.Registry is a process responsible for creating and issuing the TAP network interfaces. In my head, this is a Registry that stores the currently active TAP devices. A really clean design would be to keep all the processes necessary to boot one VM up in one supervisor, but that's not the system we have right now. Consequently, I'm thinking that just using a plain registry that acquires/releases TAP devices on the hypervisor node is the appropriate 80-20.
• suidhelper will receive a brand new set of CLI subcommands net for managing the network interfaces. I'm thinking suidhelepr net tap add and suidhelper net tap down will be enough.

[codecov]

Codecov Report

❌ Patch coverage is 38.46154% with 16 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
lib/hyper/suid_helper/network.ex	0.00%	6 Missing ⚠️
lib/hyper/cfg/network.ex	0.00%	4 Missing ⚠️
lib/hyper/node/fire_vmm/net.ex	0.00%	4 Missing ⚠️
lib/hyper/node/fire_vmm/state.ex	66.66%	1 Missing ⚠️
native/suidhelper/src/tools/network/mod.rs	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

Test Results

    4 files  ± 0  216 suites   - 11   3m 20s ⏱️ -3s
  631 tests  - 44  631 ✅  - 43    0 💤 ± 0  0 ❌  - 1 
1 088 runs   - 63  631 ✅  - 43  457 💤  - 19  0 ❌  - 1 

Results for commit 3ed1a5a. ± Comparison against base commit 3ac24e1.

This pull request removes 54 and adds 10 tests. Note that renamed tests count towards both.

Elixir.Hyper.Cfg.NetworkTest ‑ test clone_pool/0 defaults when unset
Elixir.Hyper.Cfg.NetworkTest ‑ test configured?/0 false when no uplink configured
Elixir.Hyper.Cfg.NetworkTest ‑ test uplink/0 raises ArgumentError when network.uplink is not a string
Elixir.Hyper.Cfg.NetworkTest ‑ test uplink/0 raises Hyper.Cfg.MissingError when network.uplink is unset
Elixir.Hyper.E2e.NetworkTest ‑ test a booted guest can reach the internet over its NIC
Elixir.Hyper.Node.FireVMM.BootSpecTest ‑ test every VM gets a NIC and the ip=/hyper.resolver= cmdline
Elixir.Hyper.Node.FireVMM.DaemonTest ‑ test netns lifecycle (mandatory) start_link prepares the netns before launching the jailer
Elixir.Hyper.Node.FireVMM.DaemonTest ‑ test netns lifecycle (mandatory) terminate/2 tears down both the network and the chroot jail
Elixir.Hyper.Node.FireVMM.JailerTest ‑ test Checks.network_ready/0 fails when uplink iface is absent
Elixir.Hyper.Node.FireVMM.JailerTest ‑ test Checks.network_ready/0 refuses to start when no [network] table is configured
…

Elixir.Hyper.Node.FireVMM.BootSpecTest ‑ test a VM boots with no NIC-related kernel cmdline
Elixir.Hyper.Node.FireVMM.DaemonTest ‑ test jail lifecycle start_link launches the jailer without any network setup
Elixir.Hyper.Node.FireVMM.DaemonTest ‑ test jail lifecycle terminate/2 removes the chroot jail without any network teardown
Elixir.Hyper.Node.Reaper.PlanTest ‑ test orphans/3 a cgroup-leaf-only orphan is a candidate
Elixir.Hyper.Node.Reaper.PlanTest ‑ test orphans/3 a dm-only orphan is a candidate
Elixir.Hyper.Node.Reaper.PlanTest ‑ test orphans/3 an id present in live is never a candidate, even if it also has resources
Elixir.Hyper.Node.Reaper.PlanTest ‑ test orphans/3 an id seen in both sources is a single candidate
Elixir.Hyper.Node.Reaper.PlanTest ‑ test orphans/3 only the non-live ids survive as candidates
hyper-suidhelper::config_network ‑ legacy_network_keys_are_accepted_and_ignored
hyper-suidhelper::e2e_jailer ‑ execs_jailer_without_netns_with_legacy_network_table_as_root