Crash loop after unclean shutdown: DuckDB WAL replay fails with GetDefaultDatabase (WAL never checkpointed) #63

After an unclean host shutdown (VPS hard reset, no graceful stop of the pod), liwan no longer starts: it exits with code 1 about 2 seconds after start, every time (618 restarts before I noticed).

Error: Failed to load DuckDB Database

Caused by:
    0: INTERNAL Error: Failure while replaying WAL file "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/data/liwan-events.duckdb.wal": Calling DatabaseManager::GetDefaultDatabase with no default database set
       This error signals an assertion failure within DuckDB. This usually occurs due to unexpected conditions or errors in the program's logic.

This is the same error as #36, but on an existing database, not a fresh one.

Environment

• Image ghcr.io/explodingcamera/liwan:1.8.0-rc.2 (bundles DuckDB 1.5.5), k3s, data on a local-path PVC
• LIWAN_DUCKDB_MEMORY_LIMIT=512MB, pod memory limit 1Gi, disk 11% used, no OOM kill in the kernel log

What the data directory looked like

liwan-events.duckdb       3420160  Oct  2 12:00
liwan-events.duckdb.wal   2609449  Oct 18 11:06

The pod had been running since Oct 2 (the day migration 8 event_properties was applied). In those 16 days the main database file was never checkpointed: all events from Oct 2 to Oct 18 were only in the WAL. The host went down on Oct 18.

Reproduced outside liwan

Opening a copy of the two files with the DuckDB CLI v1.5.6 (the version bundled in liwan 1.8.0) gives the same internal error:

duckdb liwan-events.duckdb
INTERNAL Error: Failure while replaying WAL file ".../liwan-events.duckdb.wal": Calling DatabaseManager::GetDefaultDatabase with no default database set

So upgrading to 1.8.0 would not fix it.

Workaround that recovered all data

Attaching the file from an in-memory session replays the WAL without errors, because a default database already exists:

-- duckdb :memory:
ATTACH 'liwan-events.duckdb' AS ev;
CHECKPOINT ev;
DETACH ev;

After this the file opens directly with no WAL, all 21901 events are there (last one at the time of the crash), the storage version is unchanged (v1.0.0+), and liwan 1.8.0-rc.2 starts normally on it.

My guess is that the WAL contains the ALTER TABLE from migration 8 (or the default now() on created_at), and replaying it needs a default database that isn't set yet when DuckDB opens the file as the main database. I haven't verified which entry triggers it.

Suggestions

1. Run a periodic CHECKPOINT (or lower checkpoint_threshold), so a crash doesn't leave weeks of data only in the WAL. 2.6 MB of WAL never reached the default 16 MB threshold here.
2. Optionally, open the events database through ATTACH from an in-memory connection, or fall back to it when WAL replay fails with this error, so the app recovers by itself.

The underlying replay bug is probably worth reporting to DuckDB as well. I can share the files privately if useful.