Skip to content

Safe mode breaks Tinker training, loopback-only hosts crash at deploy, checkpoints unusable from stock Tinker SDK #298

Open

Description

@rycerzes
  • I have searched existing issues, and this is a new bug report.

Bug Description

Three problems when running Twinkle 0.5.1 in an isolated container and using it from the stock Tinker SDK (source links at v0.5.1):

  1. Safe mode breaks Tinker training.
    • With TWINKLE_TRUST_REMOTE_CODE=0, every forward_backward fails with Twinkle does not support Callable or Type inputs in safe mode. The check is check_unsafe on the remote-call path (infra/init.py#L573).
    • So operators can't turn remote-code trust off, and it defaults to on.
  2. Loopback-only hosts crash at deploy.
    • Under e.g. docker run --network none, find_node_ip() skips every 127.* address and returns None (utils/network.py#L15-L28).
    • That None becomes MASTER_ADDR (ray_helper.py#L336), and the deployment fails with TypeError: runtime_env['env_vars'] must be of type Dict[str, str].
    • Also, .locks is created in the CWD (utils/parallel.py#L9), which fails when the CWD is read-only.
  3. Checkpoints can't be used from the stock Tinker SDK.
    • save_weights_for_sampler() returns twinkle://... paths, which the unpatched SDK rejects (ValueError: Invalid tinker path). There's also no archive/download endpoint.
    • name is ignored, and each save deletes the previous sampler weights (checkpoint_manager.py#L241-L285), so earlier returned paths go stale.

How to Reproduce

  1. Set TWINKLE_TRUST_REMOTE_CODE: "0" in the model deployment's env_vars, then call forward_backward(data, "cross_entropy").
  2. Start ray start --head --node-ip-address=127.0.0.1 and twinkle-server launch inside a --network none container.
  3. With tinker==0.16.1, call get_checkpoint_archive_url_from_tinker_path(tc.save_weights_for_sampler(name="x").result().path).

Additional Information

Suggested fixes:

  1. Exempt Twinkle's built-in loss and template classes from check_unsafe.
  2. In find_node_ip(), fall back to 127.0.0.1 (or the Ray node IP), and put .locks under TWINKLE_HOME or the temp dir.
  3. Return tinker://-compatible paths, add the Tinker archive endpoint, and honour name. That would let separate evaluators load trained adapters without patching the SDK.

Environment: twinkle-kit 0.5.1, ray 2.58.0, Ubuntu 24.04 container, tinker 0.16.1.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions