Skip to content

logseq#12539: reduces memory allocation for graph file query to avoid crash; adds debug output - #4

Open
vr-one wants to merge 1 commit into
logseq:masterfrom
vr-one:fix-sync-crashes-app
Open

logseq#12539: reduces memory allocation for graph file query to avoid crash; adds debug output#4
vr-one wants to merge 1 commit into
logseq:masterfrom
vr-one:fix-sync-crashes-app

Conversation

@vr-one

@vr-one vr-one commented May 29, 2026

Copy link
Copy Markdown

Fix OOM crash in get_all_files_meta + improve error messages and logging

Problem

Logseq Desktop would crash with EXC_BREAKPOINT (SIGTRAP) on the tokio-runtime-worker thread during file sync on large graphs. The stack trace pointed to an allocation failure inside get_file_meta at graph.rs.

Root cause: get_all_files_meta collects one get_file_meta future per file and passes them all to future::join_all, which polls every future concurrently. Each future immediately allocated a Vec::with_capacity(1024 * 1024) (1 MB) read buffer and held it live for the duration of the I/O loop. With hundreds of files this meant hundreds of MB were reserved simultaneously, exhausting available memory and triggering a Rust OOM panic — which on ARM64 manifests as EXC_BREAKPOINT.

Changes

graph.rs — core fix + diagnostics

  • Reduced the read buffer in get_file_meta from 1 MB to 64 KB. The buffer is reset to len = 0 after each chunk, so only capacity is relevant to memory use; 64 KB is more than sufficient for efficient I/O and cuts peak concurrent allocation by 16×.
  • Replaced silent unwrap() calls in is_page_file_path with unwrap_or_else(|| panic!(...)) that include the offending path, making future panics immediately actionable.
  • Added log::trace! at the entry of get_file_meta to aid per-file debugging.

lib.rs — observability

  • get_local_all_files_meta: added a wall-clock timer; logs file count and elapsed milliseconds at INFO level after each full scan (e.g. "get file meta of "/path/to/graph": 347 files in 412ms").
  • get_local_files_meta: added log::trace! for the file list being requested.
  • rename_local_file: added log::info! showing source and destination paths.

sync.rs — panic quality

  • Replaced five bare unwrap() calls with .expect(…) or unwrap_or_else(|| panic!(…)) carrying descriptive context:
    • Proxy::https(proxy) — identifies invalid HTTPS_PROXY URL
    • builder.build() — identifies HTTP client construction failure
    • strip_prefix(bucket()) — shows the mismatched prefix and bucket values
    • self.credentials.as_ref() in upload_tempfile — clarifies that refresh_temp_credential was not called
    • self.s3_prefix.clone() — same

Testing

Verified on macOS arm64 with a ~350-file graph that previously triggered the crash. The app now completes the initial file-sync scan without crashing. The INFO log line confirms the correct file count and a reasonable scan time.

@vr-one

vr-one commented May 29, 2026

Copy link
Copy Markdown
Author

@andelf @tiensonqin @RCmerci Please review this PR to build a new version of the raspi library and release a maintenance version of logseq - cause the db-version still lacks features to switch for most users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant