Dirty pages deduplication between diff and base layer for memory by bchalios · Pull Request #2590 · e2b-dev/runtime

bchalios · 2026-05-07T14:10:53Z

Adds 4 KiB memfile diff dedup behind memfile-diff-dedup to avoid storing pages unchanged from the base when Firecracker reports dirty memory at 2 MiB granularity.

Also fixes related page-granular restore correctness for empty mappings and chunk reads, and runs integration pause/resume coverage across base, best-effort, and direct I/O dedup modes.

Stacked on #2522.

cursor · 2026-05-07T14:11:00Z

PR Summary

High Risk
Touches the sandbox pause/restore path and low-level memory snapshot I/O (page mapping, chunk fetching, UFFD reads), so mistakes could corrupt snapshots or break resume in production.

Overview
Adds an optional memfile-diff-dedup feature flag that rewrites memfile diffs at 4KiB page granularity by comparing dirty pages against the base memfile (with best-effort and O_DIRECT modes), wiring this through Firecracker export, snapshot header generation, and new metrics, while also tightening restore correctness around empty mappings/page-size handling and ensuring chunk fetches and UFFD faults fully cover multi-block ranges; CI integration tests now exercise the different dedup modes via a new dedup_mode matrix input.

^{Reviewed by Cursor Bugbot for commit 0ecb16e. Bugbot is set up for automated code reviews on this repo. Configure here.}

codecov · 2026-05-07T14:13:30Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
2723	2	2721	5

View the full list of 2 ❄️ flaky test(s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 60.45% (Passed 560 times, Failed 856 times)

Stack Traces | 62.9s run time

=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:27: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (62.91s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 60.63% (Passed 550 times, Failed 847 times)

Stack Traces | 111s run time

=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1266}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\nUsed memory before tmpfs mount: 189 MB\nFree memory before tmpfs mount: 794 MB\nMemory to use in integrity test (80% of free, min 64MB): 635 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"635+0 records in\n635+0 records out\n665845760 bytes (666 MB, 635 MiB) copied, 2.78637 s, 239 MB/s\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"C"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"o"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"a"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"d being timed: \"dd if=/dev/urandom of=/mnt/testfile bs=1M count=635\"\n\tUser time (seconds): 0.00\n\tS"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"y"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"i"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"("}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"conds): 2.77\n\tPercent of CPU this job got: 99%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 0:02.79\n\tAverag"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e shared tex"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t size"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" (kby"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"tes):"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tAver"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"age u"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"nsha"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"red "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"data "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"size (kby"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"tes"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"): "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"0\n\t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"Ave"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"rage"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" stack size"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" (kb"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 2676\n\tAverag"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e resid"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ent s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"et s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ize"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" (k"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"bytes"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"): 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\n\tMa"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"jor ("}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"requ"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"irin"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"g I/"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"O) p"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"age faults: "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"3\n\tMi"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"nor "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"(rec"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"laim"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ing "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"a fra"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"me) "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"page"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" fau"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"lts: 341\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tVolu"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"nta"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 829 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ry context switches: 4\n\tInvoluntary context switches: 14\n\tSwaps: 0\n\tFile system inputs: 176\n\tFile system outputs: 0\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox iscaikmiuflooq1xhlwfp
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1282}}
Executing command bash in sandbox izlrh9rckwggiccpasvvf (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{data:{stdout:"02405baab1299fb7f54698bf56521b7aeb8d4ad30f7c8f3c582bf0863ef54e5c\n"}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:80: Command [bash] completed successfully in sandbox iscaikmiuflooq1xhlwfp
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1286}}
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
Executing command bash in sandbox iscaikmiuflooq1xhlwfp (user: root)
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:81
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox iscaikmiuflooq1xhlwfp: unavailable: HTTP status 502 Bad Gateway
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:78
        	            				.../tests/orchestrator/sandbox_memory_integrity_test.go:110
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (110.69s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

gemini-code-assist

Code Review

This pull request implements memfd-backed guest memory support for Firecracker, enabling the orchestrator to retrieve dirty pages via pread on a shared file descriptor instead of process_vm_readv. Key changes include a new 4KiB page deduplication routine in the block cache, updates to the UFFD handler to manage the received memfd, and integration into the sandbox lifecycle via a new feature flag. I have no feedback to provide as the implementation is robust and no issues were identified.

linear-code · 2026-05-11T12:45:50Z

ENG-3091

Introduce a Cacher interface which abstracts the memory cache implementation, as seen by the diff/upload layer. Currently, it's only the Cache type that implements that. This is preparation for introducing a second cache type which is backed by the memfd used to map guest memory. Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

We are changing Firecracker to, optionally, back the guest memory using a memfd object. When enabled, Firecracker passes over the memfd file descriptor over the UFFD UDS, alongside the UFFD file descriptor, using SCM_RIGHTS. Change the UFFD serve logic to also parse the memfd file descriptor. When present, wrap the descriptor in a Memfd object. The object itself provides an interface that lets users access the guest memory from the memfd. UFFD logic exposes the Memfd object over a newly added method of the MemoryBackend interface, called Memfd(). The noop memory backend always returns nil for now, as Firecracker might only use memfd when resuming from a snapshot. Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

Change the ExportMemory() logic to export the memory via a MemfdCache when Firecracker has sent us a memfd file descriptor. Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

Add unit tests for memfd and MemfdCache functionality. Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

Add a feature flag that controls whether the orchestrator will instruct Firecracker to use memfd for backing the guest memory. Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

Trim verbose comments and drop trivial tests so the PR is easier to review. The behavioral changes are: - copyFromMemfd uses a fixed 2 MiB chunk (memfdCopyChunkSize) matching the source hugepage size, decoupled from cache.blockSize (which remains the dirty-tracking unit). - NewCacheFromMemfd no longer logs-and-swallows the memfd close error; it returns it. The size==0 fast path drops; the loop is a no-op when ranges sum to 0. - UseMemFdFlag comment matches the mmap-based implementation.

# Conflicts: # packages/orchestrator/pkg/sandbox/snapshot_metrics.go

arkamar

Noticed few nit things, but wasn't able to finish it completely.

jakubno

shouldn't we add IsCached to localDiff returning true?

Reuse the existing build diff cache lookup path and inline a single-use zero mapping check to keep the dedup diff smaller.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9017e0cf15

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Guard dedup source lookups, preserve fast-path read errors in logs, and avoid pooling hugepage buffers for 4 KiB UFFD faults.

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9a06f5f. Configure here.}

Enable memfd and memfile diff dedup in the local integration orchestrator so existing pause/resume tests cover the dedup path.

Enable memfile dedup across integration shards with base, best-effort, and direct I/O variants, and address remaining review feedback.

Skip empty iovecs before pwritev so zero-length batches complete cleanly.

Remove the unused memfd test override now that integration dedup coverage runs through the default memory export path.

Classify zero source pages before comparing against the base so zero bytes always map through Empty instead of uploading dirty zero pages.

Use a single integration dedup mode setting and remove unnecessary CLI flag wiring.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 76e9d6b072

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Avoid buffering one iovec per dirty page by flushing bounded pwritev batches while walking the dirty bitmap.

Install UFFD MISSING faults by writing source bytes into the FC-shared memfd and calling UFFDIO_WAKE, instead of UFFDIO_COPY. Read faults arm WP before populate so the kernel retry installs a write-protected PTE. Behind `use-memfd-wake` (sub-flag of `use-memfd`). `resume-build -resume-bench` compares `default` / `memfd-copy` / `memfd-wake` on the same template. Stacked on #2590. --------- Signed-off-by: Babis Chalios <babis.chalios@e2b.dev> Co-authored-by: Babis Chalios <babis.chalios@e2b.dev> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Nikita Kalyazin <nikita.kalyazin@e2b.dev>

cla-bot Bot added the cla-signed label May 7, 2026

e2b-request-same-site-reviewers Bot assigned arkamar May 7, 2026

bchalios assigned bchalios and unassigned arkamar May 7, 2026

bchalios changed the title ~~Dirty pages deduplication between diff and base layer~~ Dirty pages deduplication between diff and base layer for memory May 7, 2026

gemini-code-assist Bot reviewed May 7, 2026

View reviewed changes

cursor Bot reviewed May 11, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/sandbox/sandbox.go Outdated

bchalios force-pushed the feat/compact-diff branch from c75c871 to 32d7e1d Compare May 11, 2026 12:45

bchalios force-pushed the feat/compact-diff branch 9 times, most recently from 9ff4eab to 20e59d2 Compare May 15, 2026 11:05

bchalios added 5 commits May 15, 2026 14:08

feat(memfd): use memfd to export sandbox memory

17bffdf

Change the ExportMemory() logic to export the memory via a MemfdCache when Firecracker has sent us a memfd file descriptor. Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

test(memfd): add unit tests for memfd functionality

631fefc

Add unit tests for memfd and MemfdCache functionality. Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

feat(memfd): add feature flag for memfd

6d430ef

Add a feature flag that controls whether the orchestrator will instruct Firecracker to use memfd for backing the guest memory. Signed-off-by: Babis Chalios <babis.chalios@e2b.dev>

cursor Bot reviewed May 15, 2026

View reviewed changes

Comment thread packages/shared/pkg/storage/header/metadata.go Outdated

bchalios force-pushed the feat/compact-diff branch 2 times, most recently from cce6428 to 1bf6df8 Compare May 15, 2026 11:38

cursor Bot reviewed May 15, 2026

View reviewed changes

Comment thread packages/shared/pkg/storage/header/metadata.go

ValentaTomas added 2 commits May 20, 2026 19:57

Merge branch 'main' into feat/compact-diff

6ff7e88

Merge remote-tracking branch 'origin/main' into feat/compact-diff

14d4b7e

# Conflicts: # packages/orchestrator/pkg/sandbox/snapshot_metrics.go

arkamar reviewed May 22, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/sandbox/build/cache.go Outdated

Comment thread packages/orchestrator/pkg/sandbox/build/build.go Outdated

Comment thread packages/orchestrator/pkg/sandbox/build/cache.go Outdated

Comment thread packages/orchestrator/pkg/sandbox/block/memfd.go

jakubno reviewed May 22, 2026

View reviewed changes

ValentaTomas mentioned this pull request May 22, 2026

Use memfd wake path for UFFD installs #2742

Merged

ValentaTomas added 2 commits May 22, 2026 11:57

Reduce duplicate cache lookup helpers

6674c57

Reuse the existing build diff cache lookup path and inline a single-use zero mapping check to keep the dedup diff smaller.

Merge remote-tracking branch 'origin/main' into HEAD

9017e0c

cursor Bot reviewed May 22, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/sandbox/block/cache.go

chatgpt-codex-connector Bot reviewed May 22, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/sandbox/uffd/userfaultfd/userfaultfd.go

Address dedup review feedback

9a06f5f

Guard dedup source lookups, preserve fast-path read errors in logs, and avoid pooling hugepage buffers for 4 KiB UFFD faults.

cursor Bot reviewed May 22, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/sandbox/block/cache.go

Comment thread packages/orchestrator/pkg/sandbox/build/build.go

ValentaTomas and others added 8 commits May 22, 2026 15:54

Exercise memfile dedup in integration tests

fef9278

Enable memfd and memfile diff dedup in the local integration orchestrator so existing pause/resume tests cover the dedup path.

chore: auto-commit generated changes

1b0fc4e

Exercise dedup variants in integration tests

ebc1936

Enable memfile dedup across integration shards with base, best-effort, and direct I/O variants, and address remaining review feedback.

Handle empty pwritev batches

fa130e4

Skip empty iovecs before pwritev so zero-length batches complete cleanly.

Trim unused integration flag override

9f9bcc2

Remove the unused memfd test override now that integration dedup coverage runs through the default memory export path.

Route zero dedup pages to Empty

7746ac7

Classify zero source pages before comparing against the base so zero bytes always map through Empty instead of uploading dirty zero pages.

Fix dedup lint spacing

6887222

Minimize dedup test wiring

76e9d6b

Use a single integration dedup mode setting and remove unnecessary CLI flag wiring.

chatgpt-codex-connector Bot reviewed May 23, 2026

View reviewed changes

Comment thread packages/orchestrator/pkg/sandbox/block/cache.go Outdated

ValentaTomas added 3 commits May 23, 2026 01:30

Stream dedup dirty page writes

ba28b2e

Avoid buffering one iovec per dirty page by flushing bounded pwritev batches while walking the dirty bitmap.

Wake zero-filled faults on WP failure

22991cc

Fix userfaultfd lint spacing

0ecb16e

ValentaTomas enabled auto-merge (squash) May 24, 2026 07:07

ValentaTomas approved these changes May 24, 2026

View reviewed changes

ValentaTomas merged commit fd42d73 into main May 24, 2026
51 checks passed

ValentaTomas deleted the feat/compact-diff branch May 24, 2026 07:07

Conversation

bchalios commented May 7, 2026 • edited by ValentaTomas Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

codecov Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

linear-code Bot commented May 11, 2026

Uh oh!

Uh oh!

Uh oh!

arkamar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jakubno left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bchalios commented May 7, 2026 •

edited by ValentaTomas

Loading

cursor Bot commented May 7, 2026 •

edited

Loading

codecov Bot commented May 7, 2026 •

edited

Loading