What is a microVM?

What is a microVM?

A microVM is a lightweight virtual machine (VM) that runs one workload on its own kernel and starts much faster than a full VM. Hardware virtualization separates that kernel from the host, so the workload is more isolated than in a container, which shares the host's kernel.

Firecracker is an open-source virtual machine monitor, the program that creates and runs virtual machines. Amazon Web Services (AWS) built it for AWS Lambda. Its design document says microVMs "combine the security and workload isolation properties of traditional VMs" with the speed and resource efficiency of containers.

A microVM is one way to build a sandbox, an isolated place that limits what code can reach. It is a common choice for code that nobody has reviewed, e.g. the commands a coding agent runs. In desktop security, the name micro VM also means a small VM that isolates one risky task, e.g. opening a downloaded file.

How does a microVM work?

With a virtual machine monitor, e.g. Firecracker, a microVM starts in five steps:

1. A host program starts one monitor process for each microVM, usually through a jailer that confines the process and drops its privileges.
2. The host program sends the monitor its settings through an API or a config file, including a Linux kernel file and a disk image that holds the guest's files.
3. The monitor asks the host's Kernel-based Virtual Machine (KVM) module to create the VM, with one thread for each virtual processor.
4. The guest kernel boots from the kernel file and starts the workload, e.g. a shell.
5. The guest kernel sends disk and network traffic to emulated devices. The monitor passes it to the disk image file or to a TAP device, a virtual network interface on the host.

The monitor leaves out most of what a full VM emulates. Firecracker offers only a few devices, e.g. a VirtIO network device, and its keyboard controller exists only so the guest can stop the microVM. Its design document treats the guest's processor threads as "running malicious code as soon as they have been started." Seccomp filters limit the monitor's own system calls to the host, and the jailer runs it without privileges in a restricted folder.

Other open-source monitors build microVMs in a similar way. Cloud Hypervisor keeps device emulation minimal, and QEMU has a microvm machine type modeled on Firecracker's. Kata Containers uses these monitors to start each container or Kubernetes pod in its own lightweight VM.

What is an example of a microVM?

Here is an illustrative example. A developer at Acme Co. runs a coding agent's task in a Firecracker microVM on a Linux build server. The task is "Let customers edit their delivery address during checkout." The run takes seven steps:

1. The developer confirms that the task's user can read and write /dev/kvm on the server.
2. The developer copies the base image acme-base.ext4, which holds the tools and a clean checkout of the web store, to task-17.ext4.
3. A config file names the guest kernel and task-17.ext4, and it gives the microVM 2 virtual processors and 2 GiB of memory.
4. The developer creates the TAP device tap17. Firewall rules on it allow the DNS server, the package registry, the Git server, and the agent's model API, and drop other traffic.
5. The developer starts the monitor, and the guest kernel boots.
6. Inside the guest, the agent edits the checkout code and runs npm install, and one package's install script fails to reach metrics.example.com.
7. The agent pushes the branch address-edit, and the developer stops the microVM and deletes task-17.ext4 and tap17.

The config file and the start commands look like this:

{
  "boot-source": {
    "kernel_image_path": "vmlinux",
    "boot_args": "console=ttyS0 reboot=k panic=1"
  },
  "drives": [{
    "drive_id": "rootfs",
    "path_on_host": "task-17.ext4",
    "is_root_device": true,
    "is_read_only": false
  }],
  "machine-config": { "vcpu_count": 2, "mem_size_mib": 2048 },
  "network-interfaces": [{ "iface_id": "eth0", "host_dev_name": "tap17" }]
}

cp acme-base.ext4 task-17.ext4
sudo ip tuntap add dev tap17 mode tap
sudo ip link set dev tap17 up
firecracker --api-sock /tmp/task-17.sock \
  --config-file task-17.json

The agent's commands changed only the task's disk copy, and the host firewall stopped the unknown host. This example is simplified. A real setup would start the monitor through the jailer, not directly.

What changes when a coding agent writes the code?

Firecracker was built so that AWS Lambda could run code from many customers on shared servers, as its authors describe in a 2020 paper. The threat there is code that attacks the host or other customers. A coding agent adds a threat inside the task. Its commands change as it works, and prompt injection can change them, because the agent reads files and web pages that other people wrote.

Those commands can read anything in the guest, e.g. a Git token the task holds, and send it out. Firecracker's design document says that Firecracker "does not perform any network traffic filtering." Filtering belongs on the host, as egress control with an allowlist of hosts.

Firecracker also offers no shared folder device, so the developer's working folder cannot be mounted into the guest by mistake. A practical setup copies the project in on a disk image. The agent's changes come back as a branch that someone reviews before they run on the host.

Why do agent sandboxes use microVMs?

Agent sandboxes, especially hosted ones, often use microVMs for four reasons:

• A kernel for each task. A command that exploits a kernel bug reaches the guest kernel of its own task, not the host kernel that other tasks share.
• A small attack surface. The monitor emulates few devices and runs confined, so guest code can reach little host code.
• Fast starts. A microVM starts fast enough to give each task a fresh machine and delete it afterwards. That suits background coding agents, which often run several tasks at once.
• A normal Linux kernel. Package installs and test runners usually behave as they do on a Linux server. A container runtime can also run inside the guest when its image includes one.

Continuous integration (CI) jobs that build an agent's changes also run code nobody has read, so the same reasons apply to CI for coding agents. A microVM can also hold an ephemeral environment, a short-lived copy of the app for one change.

What are the limits of microVMs?

A microVM's boundary is strong, but it has these limits:

• It needs hardware virtualization. Firecracker runs only on a Linux host that exposes KVM, e.g. a physical server. A cloud VM exposes KVM only when nested virtualization is on. Without KVM, gVisor adds a separate kernel layer that needs no hardware virtualization.
• It can still be escaped. A bug in KVM or in the monitor can let guest code reach the host.
• It protects only what stays outside. Anything let in, e.g. a secret, is open to every command inside.
• Each guest has a cost. Each microVM holds its own kernel in memory, and someone has to patch the guest images.
• Snapshots copy state. Firecracker can save a paused microVM as a snapshot for later runs. Its documentation calls resuming one snapshot more than once insecure, because values that should be unique repeat, e.g. a cryptographic token.
• It does not test the code. A change can run safely in a microVM and still be wrong. Evidence that it works comes from running the software, which developers also call runtime verification.

How is a microVM different from a full virtual machine?

A microVM and a full VM both run a guest kernel behind hardware virtualization, so their isolation is of the same kind. They differ in how much of a computer the monitor emulates. A full VM gets firmware and a full set of devices, e.g. a graphics card, so it can boot most operating systems from its own disk. A microVM's monitor offers a few devices and usually loads a Linux kernel file directly. That cuts start time and memory, and it leaves less monitor code for guest code to attack.

A full VM suits agent tasks that need another operating system, e.g. Windows, or a desktop session. The comparison of Docker and dev containers with VMs covers a full VM for agents.

FAQs

Does a microVM need hardware virtualization?

A microVM needs hardware virtualization, because its monitor creates the VM through the host's hypervisor, e.g. KVM on Linux. In the cloud, that rules out an ordinary cloud VM unless nested virtualization is on. A host without hardware virtualization can use gVisor, which isolates code without a VM.

Can a microVM run containers inside it?

A microVM can run containers inside it when its guest image includes a container runtime, because the guest runs a normal Linux kernel. Kata Containers works from the container side instead. It starts each container or Kubernetes pod inside its own lightweight VM.

Which open-source projects build microVMs?

Open-source projects that build microVMs include the monitors Firecracker, which AWS built for AWS Lambda, and Cloud Hypervisor, which keeps device emulation minimal. QEMU also has a microvm machine type modeled on Firecracker's.

Can a microVM keep state between runs?

A microVM keeps state between runs only when the team keeps its disk image or a snapshot. Deleting the task's copy of the image deletes what the task installed and changed. A Firecracker snapshot saves a paused microVM, but resuming one snapshot more than once can repeat values that should be unique.