The log was three hours behind, and the clock was fine

2026-09-08

A small service I wrote in Go runs on the router. It logs four or five lines a day: it started, it reloaded, someone typed the wrong password. After a firmware update I noticed its log said 22:19 while the shell on the same box said 01:19. Same machine, same second, three hours apart.

The clock was not wrong. date was right, NTP was synced, the router's own web interface showed the right time. Only my process disagreed, and it disagreed by exactly the offset of my time zone, which is the tell: this is not drift, it is a program that never learned the zone and is printing UTC.

Where a program gets "local time"

On Linux the answer is usually /etc/localtime, which is a TZif file — a compiled binary of transition times. Here is what it is on this router:

/etc/localtime -> /var/TZ
$ cat /var/TZ
MSK-3

Five ASCII characters. That is a POSIX TZ string, the old format: name, then the offset west of UTC. Busybox understands it, which is why date was right. Go's time package opens /etc/localtime, tries to parse it as TZif, fails, and falls back to UTC — silently, because there is nothing sensible to say when the box has no zone database at all.

Two programs on one machine, two answers, both defensible.

The fix

Entware ships a real zoneinfo tree, and its installer leaves a symlink pointing at whichever zone you picked:

/opt/etc/localtime -> /opt/share/zoneinfo/Europe/Moscow

So the service gets its zone from an environment variable that points straight at that file. The leading colon is the POSIX convention for "this is a path, not a zone name", and Go honours it:

TZ=:/opt/etc/localtime

One line in the init script, and the log and the shell agree again. I deliberately did not hardcode Europe/Moscow anywhere: the symlink already knows, and a box that moves zones should move with it.

What I tried first, and what didn't work, in case you land here from a search engine:

attempt	result
nothing set	UTC
`TZ=Europe/Moscow`	UTC — no zone database in the default search paths
`ZONEINFO=/opt/share/zoneinfo TZ=Europe/Moscow`	UTC on this build
`TZ=:/opt/etc/localtime`	correct local time

The part worth remembering

Timestamps are the one thing you trust without checking. When two logs on the same machine disagree, the instinct is to suspect the clock, then NTP, then the container — and to spend an hour there. The cheaper test is to ask the program itself what it thinks, next to the shell:

date '+%H:%M:%S %Z'; my-service --version-and-time

If they differ by a whole number of hours, nothing is wrong with the clock. Something in the chain never found the zone, and the fallback is always UTC, because UTC is the only answer that is never wrong, only useless.