Skip to content

fix(persistence): merge data.json with changes made on other devices instead of overwriting them - #388

Merged
chhoumann merged 17 commits into
masterfrom
fix/sync-safe-persistence
Oct 11, 2026
Merged

chhoumann merged 17 commits into
masterfrom
fix/sync-safe-persistence

Conversation

@chhoumann

@chhoumann chhoumann commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Summary

Changes made on another device (Obsidian Sync, iCloud, a second Obsidian on the vault) now survive PodNotes' next save and show up without a restart. The player in a popout window saves its position when that window is hidden or closed. Two same-titled episodes from different podcasts can both stay in the queue, and the reorder modal lists both.

Fixes #357
Fixes #356

How it works, ported from QuickAdd (d418f01f, 46753e48, 40dd5b4f, 0142db3d):

  • Merge, not overwrite. Every save reads data.json right before writing and merges what PodNotes last loaded or wrote, memory and disk three ways. Maps (played episodes, feeds, playlists, downloads) merge per key, and episode lists per episode (podcastName::title). An entry both devices changed is taken whole from this device: a progress record, a saved feed (its URL and private-feed secret belong together), a note mapping, a current episode, and each episode in the queue, favorites, Local Files, playlists and downloads (so a stream URL, a file path and the "downloaded by PodNotes" mark never come from two devices). An entry PodNotes had to drop while reading an invalid file counts as kept, not deleted; an entry the file does not have is a deletion. Unknown fields at every level survive.
  • Changes made elsewhere. onExternalSettingsChange merges the synced file into memory and updates the feed, playlist, queue, favorites, downloads and played stores. It never switches the playing episode, seeks, or touches the queue order.
  • When saves happen. One promise chain. A change saves at once, as before; changes made while a save runs share the next one. Only the playing episode's periodic position is a checkpoint, written at most every 10 s. Pausing, hiding the player's window (popout included) and closing the player save the position at once. Quit starts no write and waits for one in flight, as in fix(persistence): never overwrite an unreadable data.json or write during window unload #376.
  • What PodNotes never writes over. Obsidian's saveData swallows write errors, so PodNotes reads the file back after every write and treats a mismatch as a failed save. It never writes over a non-empty data.json it cannot read, or cannot check (a permission error, say): saves stop with a Notice and resume once the file reads again, or is missing or empty. An empty data.json, which a write cut off on a full disk leaves, holds nothing to lose and is treated like a missing one; the next save writes it again with what the failed write held, changes from other devices included. It never writes over a data.json from an older version (one synced in from another device): saves stop with a Notice until a restart, whose migration moves the file's API keys into SecretStorage. A data.json that could not be read at startup is taken over only once it reads without any repair.
  • Queue identity. The queue and its reorder modal identify episodes by podcast and title, like played state and downloads already do, not by title alone and not by stream URL (hosts rewrite enclosure URLs).
#356 before #356 after

Limits of sync support:

  • When both devices changed the same thing (the same setting, or the same feed, progress record or queued episode) before either saw the other's change, this device's version wins.
  • Obsidian offers no compare-and-swap. A change another device writes between PodNotes' read and its write is overwritten; the read happens right before the write to keep that window short.
  • PodNotes writes data.json in place, not atomically. A sync client that reads the file during a write can see it half written.
  • An older PodNotes on another device writes the whole file from its own memory and can drop this device's changes. PodNotes stops saving when it sees such a file.
  • If a write fails and leaves data.json partly written, PodNotes does not repair it: saves stop with a Notice, and only a readable copy, such as one synced back, recovers. A write that leaves it empty is repaired by the next save.

Scope: data.json stays schema v2 with no format change. Not included: atomic file replace (deferred), and two grid issues found while testing (a podcast whose artwork has not loaded, or fails to load, is hidden from the grid, and the card keeps a failed state when the artwork URL changes).

Verification

Runtime: Obsidian 1.13.7 (installer 1.13.7), Linux, headless isolated worktree instance. Feeds: the verify-skill fixtures PodNotes Fixture and the new PodNotes Fixture B (fixtures.mjs feed-b, same episode titles), 60 s WAV episodes, added through Settings and played from the episode list. The "other device" is a script that rewrites data.json with a temp file and rename, like a sync client.

  • Synced data.json changes from another device are overwritten #357 before (master). While Fixture Episode 1 played, the other device added progress, a feed and a playlist. All three were on disk right after its write and gone 8 s later.
  • Synced data.json changes from another device are overwritten #357 after. Same script: all three stay on disk, local progress keeps advancing, and memory has the new feed after one onExternalSettingsChange.
  • Same-titled episodes from different podcasts collapse into one queue entry #356 before (master). With Fixture's "Fixture Episode 1" queued, Fixture B's "Fixture Episode 1" offered "Remove from Queue", and clicking it removed Fixture's episode. After: "Add to Queue", both stay queued, and the reorder modal lists and moves both.
  • Closing the main window 50 ms after a change (head). Real electronWindow.close() 50 ms after a volume change, three traced runs: the save started about 1 ms after the change and finished before quit; no write at or after pagehide; data.json complete with the new volume.
  • Popout. Seek to 42 s, then hide the popout: 42 on disk within 0.5 s. Seek to 42 s, then close the visible popout: 42 on disk 2.5 s later, and still 42 after an immediate reload. On the previous head the close cases kept about 15 s.
  • Full disk (head). A write that left data.json at 0 bytes is reported as failed. Once space is back, the next save writes the file again (2296 bytes, feed kept, new volume), with no Notice; a feed another device added before the failed write is kept too. Starting Obsidian on a 0-byte data.json proceeds as a new install. A half-written, non-empty file from another writer is left unchanged, with the Notice "PodNotes could not read .obsidian/plugins/podnotes/data.json, so it leaves the file unchanged and saves nothing until the file can be read again."
  • Writes during 60 s of playback (head): 8 writes: at the start, a checkpoint every 10.0 s, and two when the episode finished. Each write reads data.json before and reads it back after. Master: 15 writes.
  • E2E. New tests/e2e/sync-persistence.test.ts (QuickAdd's external-change cases with PodNotes data, the playing-episode case, unload during a read, adoption, Same-titled episodes from different podcasts collapse into one queue entry #356), tests/e2e/player-popout-save.test.ts (hide with an observed save first and a read before the next checkpoint; visible close; close then reload) and tests/e2e/quit-save.test.ts (closes the real main window 50 ms after a change and reads data.json from the test process while Obsidian is down). The journey suite's #357 known failure J-EDGE-12 now passes and is a plain test. npm run test:e2e passes all 53 tests on the head commit, from a fresh vault.
  • Unit. Merge tests ported from QuickAdd's suite with PodNotes shapes, writer tests and data.json integration tests. Each path rule and each refusal has a test that fails without it, including the saved-feed trace, the download ownership mark, a same-size failed write, a file a failed write cut short or left empty (with another device's feed), an empty file at startup, a refusal lifting when the file turns empty, a read that fails, a foreign unreadable file, a newer-schema file, an older-schema file with API keys, a lossy file at startup, and a synced podcast name that is not text.

Checks on the head commit (Node 22.23.2):

  • npm run lint: pass
  • npm run format:check: pass
  • npm run typecheck: pass
  • npm run build: pass
  • npm run size: pass (raw 314,794 of 330,000 bytes, gzip 99,368 of 100,000)
  • npm run size:data: pass
  • npm run test: pass (svelte-check, then Vitest 86 files, 1386 tests)
  • npm run docs:build: pass (Astro Starlight)
  • npm run test:e2e: pass (11 files, 53 tests)

Release impact

Storage: no schema or format change. data.json is read before each write and read back after it. Changes save at once; the playing position is written at most every 10 s, and at once on pause, hide and close. PodNotes stops saving, with a Notice, while data.json is unreadable and not empty, or comes from an older version. Queue membership now tells same-titled episodes of different podcasts apart; existing queues keep their entries. No API or URI changes. Docs: the feature list mentions sync support. Bundle: main.js gzip is 99,368 of the 100,000-byte budget.

Note

Merge synced data.json changes from other devices instead of overwriting them

  • Replaces the old pending-snapshot save loop in main.ts with a DataFileWriter that merges external data.json changes (from Obsidian Sync and similar tools) into local settings instead of overwriting them, without a restart
  • Adds a three-way settings merge so local edits and external changes to feeds, playlists, progress, and volume are both retained; repaired or older-schema external documents are refused rather than silently overwritten
  • Saves are now verified: a write that Obsidian reports as successful but that leaves the file truncated or unchanged is treated as a failure, and unloading hides any refusal Notice
  • Fixes queue identity: episodes are matched by full identity (isSameEpisode in episodeKey.ts) instead of title alone, so same-titled episodes from different podcasts coexist in the queue across the store, context menus, and reorder actions
  • EpisodePlayer in EpisodePlayer.svelte now persists the playback position immediately when its window becomes hidden or the component is destroyed, and flushes pending saves at that point
  • Risk: broad behavioral change to persistence — PodNotes.saveSettings, saveSettingsStrict, checkpoint, and flushSaves in main.ts now delegate to DataFileWriter and reject with typed refusal messages when the file is unreadable or older-schema; external settings changes go through onExternalSettingsChange → DataFileWriter.onExternalChange. Existing consumers of these APIs should be checked

Macroscope summarized 27610ad.

…ickAdd

Port QuickAdd's settingsPersistMerge to PodNotes' data.json document:
equal sides and unchanged-side shortcuts, plain objects merged per key,
and local winning a conflict. Episode lists (queue, favorites, Local
Files, playlists, per-podcast downloads) merge per episode by its
podcastName and title key. A current episode, a progress record, and an
episode entry both sides changed are taken whole from this device, so no
merged record mixes fields from two devices.
…instead of overwriting them

PodNotes rewrote all of data.json about every 5 seconds during playback
from memory, so progress, feeds and playlists another device synced in
were lost at the next save. It had no onExternalSettingsChange, and the
player's save on hide listened to the main window even in a popout.

Saves now go through a writer ported from QuickAdd: one promise chain, a
1 s debounce on the main window's timers, a disk read right before each
write, and a three-way merge of what was last loaded or written, memory
and disk. Memory takes the merged changes only after the write lands.
onExternalSettingsChange merges a synced file into memory without
changing the loaded episode, its position or the queue, and adopts a
data.json that could not be read at load once it becomes readable. A
value the codec had to repair is not merged as a deletion. Saves flush
on hide of the player's own window (also after a move into a popout), on
hide of the app, and on unload; on quit the pending save is written
through Obsidian's awaited quit task and nothing starts after it.

Fixes #357
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Deploying podnotes with  Cloudflare Pages  Cloudflare Pages

Latest commit: 27610ad
Status: ✅  Deploy successful!
Preview URL: https://714a4f20.podnotes.pages.dev
Branch Preview URL: https://fix-sync-safe-persistence.podnotes.pages.dev

View logs

Comment thread src/ui/ReorderQueue.svelte
Comment thread src/main.ts
Comment thread src/persistence/merge.ts
Same-titled episodes from two podcasts can share the queue now, and a
title-keyed each block failed with each_key_duplicate and rendered no rows.
…on that cannot be taken as is

- Settings changes save as soon as they happen, as before this branch; only
  the player's periodic position is a checkpoint, written at most every 10 s.
  Closing the main window right after a change keeps it, and quit goes back to
  starting no write and waiting for one in flight (#370).
- Closing the player, in a popout too, saves its position at once.
- No recovery that writes over a data.json: a file cut short by a failed
  write, or one another writer left unreadable, is never written over, and
  a file that could not be read at startup is taken only once it reads
  without repair.
- A data.json an older version synced in is never written over: saves stop
  with a Notice until a restart moves its API keys into SecretStorage.
- A synced episode whose title or podcast name is not text no longer breaks
  every save.
…anges save at once

The harness read data.json while the edit's immediate save was writing it.
…readable, and resume once it reads

Without the recovery that rewrote a file this device cut short, every save
failed silently until a restart, by which time the file was lost.
Comment thread src/persistence/merge.ts
Comment thread src/persistence/writer.ts
A write cut off on a full disk leaves 0 bytes, which hold nothing to lose:
the next save writes everything in memory, and a start proceeds as on a new
install. A non-empty file that cannot be read is still never written over.
…file, and refuse on read errors

- A failed write that left data.json empty held the changes made on other
  devices; the retry writes them again, with the edits made since.
- An unreadable-file refusal lifts once the file is missing or empty.
- A read that fails, as on a permission error, refuses saves with the Notice
  instead of treating the file as empty, and lifts on the next good read.
… and leave loadData errors as they were

A refusal ends with the plugin instance, but its sticky Notice outlived a
disable or reload. Obsidian's loadData already resolves undefined when it
cannot read the file, so its rejections keep their own error.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Synced data.json changes from another device are overwritten Same-titled episodes from different podcasts collapse into one queue entry

1 participant