You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Safe mode breaks Tinker training, loopback-only hosts crash at deploy, checkpoints unusable from stock Tinker SDK #298
I have searched existing issues, and this is a new bug report.
Bug Description
Three problems when running Twinkle 0.5.1 in an isolated container and using it from the stock Tinker SDK (source links at v0.5.1):
Safe mode breaks Tinker training.
With TWINKLE_TRUST_REMOTE_CODE=0, every forward_backward fails with Twinkle does not support Callable or Type inputs in safe mode. The check is check_unsafe on the remote-call path (infra/init.py#L573).
So operators can't turn remote-code trust off, and it defaults to on.
Loopback-only hosts crash at deploy.
Under e.g. docker run --network none, find_node_ip() skips every 127.* address and returns None (utils/network.py#L15-L28).
That None becomes MASTER_ADDR (ray_helper.py#L336), and the deployment fails with TypeError: runtime_env['env_vars'] must be of type Dict[str, str].
Also, .locks is created in the CWD (utils/parallel.py#L9), which fails when the CWD is read-only.
Checkpoints can't be used from the stock Tinker SDK.
save_weights_for_sampler() returns twinkle://... paths, which the unpatched SDK rejects (ValueError: Invalid tinker path). There's also no archive/download endpoint.
name is ignored, and each save deletes the previous sampler weights (checkpoint_manager.py#L241-L285), so earlier returned paths go stale.
How to Reproduce
Set TWINKLE_TRUST_REMOTE_CODE: "0" in the model deployment's env_vars, then call forward_backward(data, "cross_entropy").
Start ray start --head --node-ip-address=127.0.0.1 and twinkle-server launch inside a --network none container.
With tinker==0.16.1, call get_checkpoint_archive_url_from_tinker_path(tc.save_weights_for_sampler(name="x").result().path).
Additional Information
Suggested fixes:
Exempt Twinkle's built-in loss and template classes from check_unsafe.
In find_node_ip(), fall back to 127.0.0.1 (or the Ray node IP), and put .locks under TWINKLE_HOME or the temp dir.
Return tinker://-compatible paths, add the Tinker archive endpoint, and honour name. That would let separate evaluators load trained adapters without patching the SDK.
Environment: twinkle-kit 0.5.1, ray 2.58.0, Ubuntu 24.04 container, tinker 0.16.1.
Activity