Inspecting guests

This page covers what is common to every flow that drives a guest, independent of the test suite or benchmark it runs.

Windmill is the primary view

A flow that runs work on a guest polls it from a wait (or equivalent) step, and that step streams the guest’s combined unit and kernel journal into the Windmill job log as the run proceeds. So the normal way to watch progress, dmesg, and the final verdict is the Windmill UI: the job log already carries it, live.

The command-line recipes on the per-flow pages are the out-of-band view. Reach for them to inspect or intervene on the guest directly, independently of the job: when a run looks stuck, when you want to drive systemd on the guest by hand, or when you are working on the guest without a job in flight at all.

Reaching a guest from the command line

Flows reach a guest over vsock-SSH, using the kdevops-managed SSH config that f/workbench/init generates: a Host <vm> block per booted VM, dialing its vsock/<cid> through systemd-ssh-proxy. Add the one-line Include of that config to your ~/.ssh/config (see Nix) and the VM name becomes an ordinary ssh alias, so ssh <vm> … works.

With that alias in place, systemd’s own remote flag reaches the guest’s systemd over the same ssh transport. Most systemd client tools (systemctl, timedatectl, loginctl, systemd-analyze) accept -H/--host <vm> and work this way, talking to the guest’s system bus:

$ systemctl --host <vm> status <unit>      # units, services, scopes
$ timedatectl --host <vm>                  # clock and timezone
$ loginctl --host <vm> list-sessions       # sessions and seats
$ systemd-analyze --host <vm> blame        # boot timing

This is the form to prefer for anything bus-based: it is the same mechanism the step code uses, and it keeps you in systemd’s own vocabulary.

journalctl is the exception. Its --host transport speaks to systemd-journal-gatewayd over HTTP, which the guests do not run, so reading the journal goes through plain ssh, running journalctl as a remote command on the guest:

$ ssh <vm> journalctl --unit=<unit> --follow

Running a tool as a remote ssh command is the universal fallback in general: use it whenever -H cannot reach a target, or for a guest-side tool that has no -H flag at all.

Machine and VM status

Distinct from reaching into a guest, these commands act on the VM as it looks from the host: a systemd user service, qemu-system@<vm>.service, a per-VM instance of the qemu-system@.service template unit. The qemu-system-units project that supplies the template documents the full command set, summarised here.

The VM registers itself with systemd-machined, so it also shows up as a machine. That registry is per-user, so machinectl needs --user to reach it (systemd v259 or newer):

$ machinectl --user list             # registered machines (VMs, containers)
$ machinectl --user status <vm>      # one VM's PID, cgroup, service
$ machinectl --user terminate <vm>   # emergency kill, not graceful

For a dependable list of the VMs running on the host, read the user manager’s units directly; the unit list is authoritative, independent of the machined registry:

$ systemctl --user list-units 'qemu-system@*'   # VMs on this host
$ systemctl --user status qemu-system@<vm>      # one VM's service state

Read a VM’s host-side log (boot console on ttyS0, QEMU and virtiofsd messages) through the user journal:

$ journalctl --user-unit=qemu-system@<vm>.service --follow   # one VM, live
$ journalctl --user-unit='qemu-system@*.service' --no-hostname --follow

A persistent VM also exposes a virtio console socket for interactive access; attach with socat (disconnect with Ctrl-], the VM keeps running):

$ socat -,raw,echo=0,escape=0x1d \
    UNIX-CONNECT:$XDG_RUNTIME_DIR/qemu-system/<vm>/console.sock

Stopping is graceful by default: a clean guest powerdown, with systemd escalating to SIGKILL if it overruns the stop timeout; restart does that stop, then boots a fresh guest (it is how a VM is re-launched):

$ systemctl --user restart qemu-system@<vm>               # fresh reboot
$ systemctl --user stop qemu-system@<vm>                  # graceful stop
$ systemctl --user kill qemu-system@<vm> --signal=SIGKILL # force kill
$ systemctl --user stop machines.target                   # stop every VM
$ systemctl --user reset-failed qemu-system@<vm>          # clear failed

To check a guest’s own health from the host instead, go through --host into its system bus:

$ systemctl --host <vm> is-system-running   # guest systemd state
$ hostnamectl --host <vm>                   # guest identity