Repository navigation
feat(transcripts): use transcripts that feeds publish before paid transcription - #420
Conversation
Excerpts of VTT, SRT, JSON, HTML and text transcripts from Buzzsprout, Transistor, Podnews and Adventures in DevOps, plus first-item excerpts of three feeds that publish them. The verify skill serves a transcripts feed, logs the requests PodNotes makes, and answers OpenAI and Deepgram with a fixture provider.
…or audio transcription
…eferences Rename FORMATS to FORMATS_BY_PREFERENCE and PublishedTranscript.link to reference, matching FeedRepository.itemUrl(key, reference). Replace three em-dashes in files this branch touches.
… fetching it whole
…ript findPublishedTranscript escalated to a complete load whenever the cached load found no item, so a feed that could not be fetched was requested twice. FeedRepository.load now resolves to false when a request it started or joined failed, and the lookup escalates only when it did not. Reading FeedState.error instead would keep an earlier failure, so a retry after a failed refetch would never ask the feed again.
TemplateEngine passes everything after the tag's colon as the format, so
{{linktime: H:mm}} formats with " H:mm". timestampStyle dropped the
leading whitespace and formatted those timestamps differently from
timestamps the template engine inserts.
The guard against a page served in place of a text transcript rejected any body starting with "<", including a transcript whose first line is a sound cue. Only a body that opens as an HTML or XML document is now unreadable.
Both service test files declared the same audio, fetch and Deepgram mock handles and the same reset hooks. They now live in tests/mocks/transcriptionService.ts behind installTranscriptionMocks(), and each file keeps only its vi.mock calls. The published test file drops its outer describe that held only those hooks.
…plate engine does
# Conflicts: # src/main.activateView.test.ts
Deploying podnotes with
|
| Latest commit: |
07b4070
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://e201e636.podnotes.pages.dev |
| Branch Preview URL: | https://feat-feed-transcripts.podnotes.pages.dev |
| const clocks = timing < 0 ? null : TIMING_LINE.exec(lines[timing]); | ||
| if (!clocks) continue; | ||
| const raw = lines.slice(timing + 1).join(" "); | ||
| const voice = VOICE_TAG.exec(raw); |
There was a problem hiding this comment.
🟡 Medium publishedTranscript/parse.ts:56
A cue containing multiple <v> spans is stored as one cue attributed entirely to the first speaker, so Bob’s words in <v Alice>Hello.</v> <v Bob>Hi.</v> are saved as Alice’s. VOICE_TAG.exec(raw) captures only the first voice tag, while the later tag is stripped from the text; preserve speaker attribution for each voice span instead of assigning one speaker to the whole payload.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/services/publishedTranscript/parse.ts around line 56:
A cue containing multiple `<v>` spans is stored as one cue attributed entirely to the first speaker, so Bob’s words in `<v Alice>Hello.</v> <v Bob>Hi.</v>` are saved as Alice’s. `VOICE_TAG.exec(raw)` captures only the first voice tag, while the later tag is stripped from the text; preserve speaker attribution for each voice span instead of assigning one speaker to the whole payload.
| if (cues.length > 0) return timed(cues); | ||
| return untimed( | ||
| Array.from(body.querySelectorAll("p"), (p) => | ||
| collapseWhitespace(p.textContent ?? ""), |
There was a problem hiding this comment.
🟡 Medium publishedTranscript/parse.ts:104
HTML transcripts containing <br> lose word boundaries: <p>Hello<br>world</p> is parsed as Helloworld in both timed cues and the untimed fallback, then written as successful transcript text. textContent omits separators for <br>, so collapseWhitespace cannot restore them; convert line breaks to whitespace before extracting paragraph text.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/services/publishedTranscript/parse.ts around line 104:
HTML transcripts containing `<br>` lose word boundaries: `<p>Hello<br>world</p>` is parsed as `Helloworld` in both timed cues and the untimed fallback, then written as successful transcript text. `textContent` omits separators for `<br>`, so `collapseWhitespace` cannot restore them; convert line breaks to whitespace before extracting paragraph text.
…teral angle brackets A 64,000-character cue of unmatched < takes about 1.7 s to parse and one of unclosed <v tags about 9 s, and 'a < b and c > d' loses everything between the brackets. The cross-format case pins that entity-encoded markup decodes once to literal text and is never stripped as a tag.
Cue text was stripped with /<[^>]*>/g and decoded afterwards. The regex rescans the rest of the cue from every unmatched <, so a 64,000-character malformed cue blocked the app for seconds, and a literal 'a < b and c > d' lost everything between the brackets. A scanner now walks each cue once: a < opens a tag only before a letter, / or digit and only when a > closes it, tags are dropped, a v tag's annotation names the voice, and entities are decoded once, between tags. This also removes the incomplete multi-character sanitization that CodeQL reported on the regex.
…ript notes A '^id' in a cue or speaker and a 'key:: value' reach the note unescaped, so Obsidian registers a block and Dataview indexes a field.
Transcript text and speakers now escape every ^ and every colon that follows a colon, so a feed's '^id' registers no block and its 'key:: value' no Dataview field, while the note reads the same characters.
# Conflicts: # .claude/skills/verify/SKILL.md # tests/e2e/journey/library.ts
| let from = 0; | ||
| let open = raw.indexOf("<"); | ||
| while (open >= 0) { | ||
| if (!TAG_START.test(raw.charAt(open + 1))) { |
There was a problem hiding this comment.
🟡 Medium publishedTranscript/parse.ts:76
Literal cue text x<5 and y>2 is returned as x2, silently dropping 5 and y. readCueText treats any < followed by a letter or digit as a tag and skips through the next >; recognize actual cue markup and timestamp tags before removing an angle-bracket span.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/services/publishedTranscript/parse.ts around line 76:
Literal cue text `x<5 and y>2` is returned as `x2`, silently dropping `5 and y`. `readCueText` treats any `<` followed by a letter or digit as a tag and skips through the next `>`; recognize actual cue markup and timestamp tags before removing an angle-bracket span.
Obsidian parses a line in time superlinear in its escapes and emphasis marks, and cubic in its pipes, so one hostile cue could freeze the editor on every launch that reopened the note. Paragraphs now split at a word boundary once their written text passes 4,000 characters, each piece carrying the time of the cue it starts in, and pipes are escaped.
…ive Preview or on reopen
…prose A speaker label repeats on each of its paragraphs, so a feed's 64,000-character speaker rebuilt the frozen line the paragraph cap removed. Labels keep at most 100 written characters; real names stay under 20. A split could also start an untimed paragraph with a list marker, quote or thematic break, so that start is escaped.
A 64,000-character speaker was escaped again for every paragraph, so 2,000 paragraphs took 5.6 s; each distinct speaker is now escaped once (10 ms). A paragraph start is escaped only where it would make a list, quote or thematic break, so a leading **Host:** stays bold.
One deadline covered both, and an indexing vault could spend it on the metadata and leave the Reading view check an empty page.
At 4,000, a body-only transcript of 64,000 escaped dollar signs stalled Live Preview at 2560x1440 for 4 to 4.6 s, and once for 7.1 s. At 2,000 it opens in about 1.4 s. The longest paragraph of 154 real transcripts is 1,874 characters, so none of them changes.
… wide body-only dollar journey
| let written = labels.get(speaker); | ||
| if (written === undefined) { | ||
| const name = splitText(escapeTranscriptText(speaker), MAX_SPEAKER_LENGTH)[0]; | ||
| written = formatSpeakerLabel(speakerTemplate, name); |
There was a problem hiding this comment.
🟡 Medium publishedTranscript/render.ts:116
A speaker ending in : can still inject a Dataview inline field into the rendered transcript: [field: becomes **\[field::**, and Dataview recognizes the escaped opening bracket and indexes the following text as metadata. escapeTranscriptText runs before formatSpeakerLabel adds its own colon, so protect the colon boundary when composing the label.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/services/publishedTranscript/render.ts around line 116:
A speaker ending in `:` can still inject a Dataview inline field into the rendered transcript: `[field:` becomes `**\[field::**`, and Dataview recognizes the escaped opening bracket and indexes the following text as metadata. `escapeTranscriptText` runs before `formatSpeakerLabel` adds its own colon, so protect the colon boundary when composing the label.
# Conflicts: # tests/e2e/journey-resilience.test.ts
Summary
Fixes #348. "Transcribe current episode" now writes the transcript a subscribed feed publishes (
<podcast:transcript>) as a timestamped note, with no API key and no call to OpenAI or Deepgram. Episodes without a published transcript keep today's paid path and notices. The new command "Transcribe current episode from audio" is the explicit way to pay for audio transcription, also after a published transcript fails.{{linktime}}, with no capture offset. Speaker labels use Speaker label format. Feed text is escaped so it cannot become an embed, link, HTML, code, Dataview query or inline field, tag, block ID, table, comment or math, and an untimed paragraph cannot start a list, quote or thematic break.Scope
languageis ignored.Tradeoffs
api.transcript.<costs the editor little but shows raw in Live Preview, and one escaping mode keeps the renderer simple.Blast radius
The transcribe command and the
TranscriptionServicequeue;FeedRepository(newitemUrl, transcript links validated when the cache is read,loadnow resolves to whether its requests succeeded, which no other caller reads);API.getPodcastTimeFormatted, which now calls the extractedformatEpisodeTimestampwith unchanged output.Verification
Obsidian 1.13.7, Linux headless, isolated vault, verify skill with the new
PodNotes Transcriptsfixture feed andnet.mjsnetwork log, through the palette and the direct command:00:00:32seeks to 32.5 s. The untranscribed episode keeps the key notice, and with a key the paid path. The 404 episode shows the failure notice and sends no audio; the audio command then uploads it. After an in-app reload the command does not refetch the cached feed, and an unreachable feed costs one request.Obsidian 1.14.4, the V-420 findings:
podnotes-private-feed:V420%20Private, and clicking the note's00:00:32link seeks to 32.7 s. The root fix is fix(uri): keep a subscribed private feed's secret out of the live episode #423, merged here.^v420-blockandv420-field:: injected valuegives no block ID, no Dataview field (Dataview 0.5.68) and reads unchanged (J-TR-05).<parses in 2 ms (was 1.5 s) and<v× 64,000 in 6 ms (was 9.5 s). CodeQL passes.Hostile notes on Obsidian 1.14.4, longest stall of a 10 ms heartbeat over 5 s after a fresh open, worst of three, at 1024x800 and 2560x1440:
<: Live Preview / Source / Reading|: Live Preview / Source / Reading$, body-only{{transcript}}template: Live Preview / Source / Reading<as a JSON speaker: Transcribe opens the note</|/$<The cost is per line: the 64,000 escapes split into 32 lines open in 0.47 s, so no cap at the parser was needed. Unescaped punctuation at 4,000 characters stays under 0.33 s except pipes, which are roughly cubic (2,000 take 3.5 s), so they are escaped too. Across 154 real transcripts from seven hosts the longest written paragraph is 1,874 characters and the longest speaker 17, and the 2,000 cap renders all of them byte for byte as before.
npm run test:e2e: 82 of 82, including:<cue. Transcribe's own Live Preview open stalls 0.21 s, a fresh tab 0.22 s, and the longest task after a reload is 0.39 s. Without the cap it fails on an 18,148-character line, without the pipe escape on unescaped pipes.$through a body-only template at 2560x1440. Transcribe's open stalls 0.83 s, a fresh tab 1.43 s, and the reload's longest task is 1.72 s, each bounded at 5 s. At 4,000 it fails on a 4,000-character paragraph; its fresh tab took 4.64 s there.Unit tests on real transcripts from Buzzsprout, Transistor, Podnews, Podcasting 2.0 and Adventures in DevOps (CRLF, leading spaces, raw
&, word-level JSON, a 30 s cue, overlapping cues; BOM and entities derived from them), selection on three real feeds, relative URL resolution, private-URL redaction, and splitting (word boundaries, escape pairs, surrogate pairs, paragraph starts). A 64,000-character speaker over 2,000 paragraphs renders in 10 ms (5.6 s when each label was escaped again).lint,format:check,typecheck,build,test(105 files, 1,846 tests),test:toolinganddocs:buildpass on Node 22.23.2.main.jsagainst masteredb9447, both throughnpm run size: 285,622 → 293,365 bytes raw, 90,884 → 93,318 bytes gzip (+2,434).Screen recording: VTT episode transcribed with no key
Release impact
None to settings, storage, API or URIs. Adds the command
podnotes-transcribe-audio.Docs at release
transcripts.md, "Transcribe an episode": when the subscribed feed publishes a transcript, the command writes it with no key and no charge; otherwise it transcribes the audio as before. Format order VTT, SRT, JSON, HTML, text; relative transcript URLs work. Paragraphs start with a timestamp in Capture timestamp format (a link with{{linktime}}, plain with{{time}}), and speakers use Speaker label format. Add Transcribe current episode from audio to force the audio.transcripts.md, "What transcription costs": a published transcript costs nothing.transcripts.md, "Limits and error messages": replace "Transcripts have no timestamps" with: transcripts from a feed have timestamp links, audio transcripts do not. Add the published-transcript failure notice, which names the audio command when a key is set, and that unsubscribed feeds, renamed subscriptions and changed enclosure URLs fall back to the audio.commands.md: "Transcribe current episode" uses the feed's transcript first; add a row and a section for Transcribe current episode from audio (podnotes:podnotes-transcribe-audio, same availability), which always transcribes the audio with the configured provider and key.faq.mdline 98: PodNotes reads transcripts a subscribed feed publishes, at no cost; it still does not import transcripts from Apple Podcasts.templates.mdline 178:{{transcript}}is the transcript text from the feed or the provider; feed transcripts carry timestamp links.Note
Use feed-published transcripts before paid audio transcription
TranscriptionService.transcribeCurrentEpisodenow searches the feed for a published transcript first. If found, it fetches and renders it without downloading audio or requiring an API key. It falls back to queued audio transcription only when no published transcript exists.podnotes-transcribe-audiocommand that runs audio transcription directly, skipping feed lookup (commands.ts).publishedTranscriptmodule parses VTT, SRT, JSON, HTML, and plain-text formats, ranks feed links by format preference, and renders output with timestamp and speaker templates (parse.ts, render.ts).FeedRepository.loadnow resolves to a boolean aggregate result instead of void, and cached transcript metadata that is not an array of valid links is dropped while the episode itself is kept.FeedRepository.loadno longer rejects on failed requests (check callers ofloadandensureAllin FeedRepository.ts).Macroscope summarized 07b4070.