fix(pause): preserve sandbox when snapshot fails, behind a flag

[AdaAibaby]

Summary

Fixes the pause half of #3658: a failed pause could destroy the sandbox with no recoverable snapshot.

When a pause's snapshot fails after the guest has been suspended — e.g. a rootfs-diff fsync returning EIO, or mmap memfd: cannot allocate memory under host memory/hugepage pressure — Server.Pause armed an unconditional deferred stop before snapshotting, so the error path tore down the already-suspended VM. Combined with the API having already removed the routing entry and store record, the sandbox was left unrecoverable with no snapshot.

Change (behind PauseRefusalRestoreFlag, off by default)

With the flag off, behaviour is unchanged (destroy-on-failure). With it on, a failed pause keeps the sandbox recoverable:

• sandbox: decouple the resume-on-error cleanup from WithMaintainSandbox via a new WithResumeOnFailure() option. It arms the same resume-in-place cleanup (health checks restarted, guest clock re-synced) that the in-place checkpoint already uses — but only on failure. A successful pause still suspends and leaves the VM for the caller to stop. (The top-level cleanup only runs on the error path, so success semantics are untouched.)
• Server.Pause: pass resumeOnFailure when the flag is on, and arm the deferred stop only after the snapshot succeeds. On snapshot error the guest was resumed in place, so re-register it in the live map (MarkRunning) and return FailedPrecondition with a stable sandbox preserved after snapshot failure marker. ErrSandboxLost (the resume itself failed) falls through to the destroy path and returns Internal unchanged.
• api: classify that FailedPrecondition+marker as ErrPausePreservedSandbox (pause_instance.go) and map it in DeleteInstance to the existing restoreRefusedPause path — restoring the store record and route instead of removing them, gated by the same flag.

Testing

• TestSnapshotInstance_PreservedSandboxIsClassified — a FailedPrecondition+marker is classified as ErrPausePreservedSandbox.
• TestSnapshotInstance_PlainFailedPreconditionNotPreserved — a plain FailedPrecondition (e.g. envd-version) is not misclassified.
• Both pass; all three packages (pkg/sandbox, pkg/server, api/internal/orchestrator) build; the full internal/orchestrator suite passes with no regressions.

Not yet done (follow-up)

End-to-end validation with the flag enabled against real Firecracker — inject a snapshot fault (fsync EIO / memfd ENOMEM), assert the sandbox survives and stays routable — has not been run. It requires a KVM host. The flag defaults off, so this is inert in production as merged; that integration run should happen before flipping the flag on.

Related: checkpoint half of #3658 is #3666.

[cla-bot]

We require contributors to sign our Contributor License Agreement, and we don't have @ada on file. You can sign our CLA at https://e2b.dev/docs/cla . Once you've signed, post a comment here that says '@cla-bot check'