Debugging problems which mean init goes crazy is tricky, especially so on modern Intel hardware that doesn't have anything resembling a serial port you can connect to.
Luckily this isn't a new problem, as Linux supports a network console which will send the console messages over UDP packets to a specific machine. This is mostly easy to use but there are some caveats that are not obvious.
The prerequisites are that netconsole support is enabled, and your
ethernet driver is built in to the kernel and not a module. Luckily, the
stock Yocto kernels have netconsole enabled and the
integrates the driver for my hardware.
Then, on the target machine, you pass
netconsole=... to the kernel.
The kernel documentation explains this quite well:
netconsole=[src-port]@[src-ip]/,[tgt-port]@/[tgt-macaddr] where src-port source for UDP packets (defaults to 6665) src-ip source IP to use (interface address) dev network interface (eth0) tgt-port port for logging agent (6666) tgt-ip IP address for logging agent tgt-macaddr ethernet MAC address for logging agent (broadcast)
The biggest gotcha is that you (obviously) need a source IP address, and netconsole starts before the networking normally comes up. Apart from that you can generally get away with minimal settings:
Note that apparently some routers may not forward the broadcast packets correctly, so you may need to specify the target MAC address.
On the target machine run something like netcat to capture the packets:
$ netcat -l -u -p 6666 | tee console.log
If you get the options wrong the kernel will tell you why, so if you don't get any logging iterate a working argument on an image that works, using dmesg to see what the problem is.
Finally instead of typing in this argument every time you boot, you can
add it to the boot loader in your
APPEND += "firstname.lastname@example.org/,@192.168.1.17/"
(APPEND being the name of the variable that is passed to the kernel by the boot loader.)
Update: when the journal starts up systemd will stop logging to the
console, so if you want to get all systemd messages also pass