dmesg and kernel messages

Before any log file exists, before there is even a filesystem to write one on, the kernel is already talking. It writes its messages into a fixed-size area of memory called the ring buffer: when it fills up, the oldest lines are overwritten. dmesg (*display messages*) prints that buffer. It is the first place to look when hardware misbehaves, a disk is not detected, or the machine did something odd at boot.

Reading it

~% dmesg | head -3
[    0.000000] Linux version 4.16.13 (fabian@nyu) (gcc version 7.3.0 (Buildroot 2018.08-git-00153-g8ac477681f)) #13 SMP Sat Jul 21 13:56:18 CST 2018
[    0.000000] x86/fpu: x87 FPU will use FXSAVE
[    0.000000] e820: BIOS-provided physical RAM map:
~% dmesg | wc -l
291
~% dmesg | tail -n 3
[    2.531828] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2
[    3.865661] ISO 9660 Extensions: Microsoft Joliet Level 3
[    3.867278] ISO 9660 Extensions: RRIP_1991A

Each line starts with a timestamp in brackets: seconds since the kernel started, with microseconds. Reading top to bottom you watch the boot happen: CPU features, the memory map, PCI devices, then drivers claiming hardware, then filesystems being mounted. On this VM the whole story is under three hundred lines, so dmesg | less (or more) is a comfortable read. On a laptop it is thousands.

Searching it

Nobody reads dmesg end to end twice. You pipe it into grep -i with the name of the thing you care about: a device (sr0, eth0, sda), a driver (ne2k), or a worry (error, fail). tail -n 20 shows what the kernel said most recently, which is the right move after plugging something in: the new lines are at the bottom.

~% dmesg | grep -i sr0
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/bzImage root=/dev/sr0
[    3.701488] sr 1:0:0:0: [sr0] scsi-1 drive
[    3.756374] sr 1:0:0:0: Attached scsi CD-ROM sr0
~% dmesg | grep -i eth
[    3.395302] ne2k-pci 0000:00:05.0 eth0: RealTek RTL-8029 found at 0xc000, IRQ 10, 00:22:15:e2:f8:aa.

Two useful facts fell out of those searches: the CD-ROM is a SCSI-emulated drive attached as sr0, and the network card is a RealTek 8029 clone driven by ne2k-pci, with its MAC address. That is the kind of detail you need when a device does not work: did the kernel see it at all, and which driver took it?

Writing to it, and clearing it

Root can append to the kernel log by writing to /dev/kmsg. That sounds exotic but has a practical use: drop a marker line before you try something (*echo 'test 3 starts' > /dev/kmsg*), and afterwards everything below the marker in dmesg is what your test caused. The logger command does the same for the system logger, when one is running.

~% echo 'lab67 marker' > /dev/kmsg
~% dmesg | tail -n 1
[   12.299248] lab67 marker