What is the difference between containers, gVisor, and microVMs?

What is the difference between containers, gVisor, and microVMs?

A container shares the host's kernel, gVisor answers system calls with its own kernel in user space, and a microVM boots a separate kernel. Each later option puts more between the code and the host kernel, and costs more to start or to run.

All three are ways to build a sandbox, an isolated place that limits what code can reach. A container relies on features of the host kernel to hide other processes and files. The other two add a kernel that the code talks to instead, so the code has fewer ways to reach a bug in the host kernel.

The choice sets how much damage one bad command can do. Reviewed code with vetted, pinned dependencies usually runs safely in a container. Code that nobody has read, e.g. the install script of a package that a coding agent added, calls for a deeper boundary.

What is a container?

A container is a process that runs from a packaged image, with its own filesystem and resource limits, on the host's own kernel. Because it shares that kernel, it is lighter than a virtual machine and less isolated.

The boundary comes from features of the host kernel, which a container engine, e.g. Docker, sets up:

• Namespaces. Each container gets its own view of the system, e.g. its own list of processes.
• Control groups. The kernel limits the resources a container uses, e.g. memory.
• Capabilities and filters. Docker drops most of root's privileges by default and blocks some system calls with a seccomp profile.

On a Mac, Docker Desktop runs Linux containers inside a Linux virtual machine. Apple's container tool instead runs each Linux container in its own lightweight virtual machine.

What is gVisor?

gVisor is an open-source application kernel that sits between a container and the host kernel. Its main process, written in Go, implements Linux system calls itself, in user space. That process receives each system call from the program and makes a smaller set of calls to the host kernel.

Its runtime, runsc, follows the Open Container Initiative (OCI) specification, so Docker and Kubernetes can use it in place of their default runtime. Docker needs sudo runsc install and a daemon restart first, and Kubernetes selects it through a RuntimeClass. A gVisor container starts nearly as fast as a plain container, because its kernel is an ordinary process, not a virtual machine.

gVisor's documentation describes what the design gains and costs:

• No hardware virtualization. It needs a Linux host but no virtualization support, so it runs inside an ordinary cloud virtual machine.
• Partial compatibility. It does not implement every Linux system call, so a test that passes on a laptop can fail in continuous integration (CI) under gVisor.
• Slower system calls. Programs that make many calls or do heavy file and network I/O slow down. Programs that mostly compute do not, because gVisor does not emulate CPU instructions.

What is a microVM?

A microVM is a small virtual machine with its own kernel and only the devices a server workload needs. A virtual machine monitor creates it through hardware virtualization. Firecracker is one such monitor, and it uses the Linux Kernel-based Virtual Machine (KVM).

Firecracker emulates only a few devices, which leaves less code for a guest to attack. The guest kernel answers the program's system calls, so code has to get through that kernel and the hardware boundary to reach the host.

A microVM starts faster than a full virtual machine, but it still boots a kernel first. It runs a normal Linux kernel, so programs that fail under gVisor usually work. The costs are a guest image to maintain and memory for each guest's kernel.

Firecracker needs a Linux host with KVM, which means bare metal or a virtual machine that allows nested virtualization. A runtime, e.g. Kata Containers, can run each Kubernetes pod in its own lightweight virtual machine.

When should a team use each one?

Here is an illustrative example. Acme Co. sells furniture online. A developer at Acme picks an isolation option for three kinds of work, based on who wrote the code:

• The team's own tests. Reviewed code, with vetted dependencies pinned in a lockfile, runs its test suite in a container built from the production image. The same image lets a developer run CI checks locally.
• A coding agent's checkout task. The agent works on "Let customers edit their delivery address during checkout." It installs packages that run scripts, so its container runs under gVisor with --runtime=runsc.
• Agent tasks on shared build servers. Tasks from several repositories share one host, so each gets its own microVM and kernel.

Isolation does not test AI-generated code. The change can run safely under gVisor and still be wrong, so the developer also starts the web store there and tries the address form. This example is simplified. A real team would also check what its CI runners support, e.g. nested virtualization for microVMs.

What changes when a coding agent writes the code?

A coding agent runs code that nobody has read, including its own changes each time it runs the tests. Prompt injection, text planted in a file or web page that the agent reads, can also change the commands it runs.

The commands an agent runs are often package installs and test runs, which make many system calls and do heavy file I/O. Those are the programs that gVisor slows, so time them under runsc before choosing it.

Run the agent's workspace under gVisor or in a microVM, with test credentials under least privilege. The agent still needs the network for packages, so allow only approved hosts with egress control. Then check the setup with adversarial testing, e.g. a command inside the sandbox that tries to read a host file.

Why is a plain container not a sandbox by default?

A plain Docker container, including a dev container, is a partial sandbox. Docker's security documentation says that only trusted users should control the Docker daemon, which runs as root. A container started with docker run and a mounted project folder has these gaps:

• It shares the host kernel. Every system call reaches the host kernel, so one kernel bug can allow a sandbox escape.
• It can reach the internet. Docker's default network allows outbound connections to any host, so code inside can send out anything it can read.
• Root inside has root's user ID. Without user namespaces, which Docker leaves off by default, root in the container has the host's root user ID, with fewer capabilities.
• A mounted folder is the real folder. A command that deletes files there deletes the developer's files.
• Some settings remove the boundary. A container that mounts the Docker socket can start another container with the host's files mounted. The flag --privileged lifts most limits.

Docker settings close some gaps, e.g. rootless mode, which runs the daemon without root, but the kernel stays shared. Neither gVisor nor a microVM filters outbound traffic on its own, so egress rules live outside the sandbox.

How do containers, gVisor, and microVMs compare?

The three options differ on these attributes:

FAQs

Is Firecracker the same as a virtual machine?

Firecracker is not a virtual machine but a virtual machine monitor, the program that creates and runs microVMs through KVM. Each microVM is a virtual machine with its own kernel and only a few devices, so it starts faster than a full virtual machine.

Does gVisor slow programs down?

gVisor slows programs that make many system calls or do heavy file and network I/O, because its own kernel handles each call first. Programs that mostly compute run at close to normal speed, since gVisor does not emulate CPU instructions.

Which isolation option starts fastest?

A container usually starts fastest, because the host kernel is already running and only a process starts. A gVisor container starts nearly as fast, since its kernel is an ordinary process. A microVM boots a guest kernel first, and a full virtual machine takes longer still.

Do containers, gVisor, and microVMs work on a Mac?

Containers and gVisor need a Linux kernel, so on a Mac they run only inside a Linux virtual machine. Firecracker also needs KVM, so it runs only in a Linux virtual machine with nested virtualization. For containers, Docker Desktop starts a Linux virtual machine, and Apple's container tool starts a lightweight one for each container.