Skip to content

swift: run tailscale_up off the TailscaleNode actor - #58

Open
prakashrj wants to merge 1 commit into
tailscale:mainfrom
indiagrams:localapi-no-proxy-hop
Open

swift: run tailscale_up off the TailscaleNode actor#58
prakashrj wants to merge 1 commit into
tailscale:mainfrom
indiagrams:localapi-no-proxy-hop

Conversation

@prakashrj

@prakashrj prakashrj commented Aug 26, 2026

Copy link
Copy Markdown

Revised twice. The first version of this PR diagnosed the hang as SOCKS5 proxy routing and quoted a "before" figure I had not actually run — both wrong. The second fixed the diagnosis but worked around the stall with a nonisolated memoized read. This version takes @barnstar's suggestion instead, which is a better fix, and the measurements below say why.

Symptom

LocalAPIClient is unusable during node bring-up. startLoginInteractive(), backendStatus() and watchIPNBus() all block until up() completes — minutes for an interactive login, indefinitely if the user never finishes it. startLoginInteractive() is precisely the call you want in that window.

After up() returns, every one of these works normally. That is why the failure is easy to misread: measured from the healthy state, nothing is wrong.

Cause

up() ran tailscale_up inline on the TailscaleNode actor. That C call blocks until the node is usable, so the actor was held for the whole login and every other actor-isolated call queued behind it.

That is broader than LocalAPIClient. It also means:

  • statusJSON() (added in 8077131) is unreachable during login, for the same reason.
  • close() is unreachable during login — so the cancellation path tailscale.h documents ("To cancel an in-progress call to tailscale_up, use tailscale_close") could not be used from TailscaleKit at all.

Fix

let res = await Task.detached { tailscale_up(tailscale) }.value

The await suspends up() and releases the actor, so concurrent calls are serviced. TailscaleHandle is Int32, so the capture is trivially Sendable. One line of behaviour change, plus a regression test.

Measurements

Same machine, same session, same test, baseline first. macOS arm64, Xcode 26.1.1, Swift 6.2.1. The node points at a control URL that never completes registration, so it stays in NeedsLogin and up() is genuinely in flight for the whole measurement — the state the defect lives in.

Build loopback() called while up() is in flight
stock (main) HUNG — no return within 10007 ms
this PR returned in 22 ms, addr 127.0.0.1:52042

Both runs assert that up() had not returned, before and after, so neither samples the healthy state.

Why releasing the actor is sufficient on its own. The Go layer never serialised these: tsnet.Server.Up() blocks on an IPN bus watcher rather than a lock, and Loopback() only takes the brief servers map lock. Measured directly against tsnet with testcontrol{RequireAuth: true}, so Up() was stuck in NeedsLogin:

MEASURED: Up() still blocked after 3.001599667s
MEASURED: Loopback() returned in 433.584µs addr=127.0.0.1:51590 proxyCredLen=32 localCredLen=32
MEASURED: Up() confirmed still blocked at end of measurement

This is what makes your suggestion strictly better than the nonisolated memoized read I had proposed. That version only worked when the loopback config had already been resolved before up() was called, and it needed a caveat telling callers to do so. Freeing the actor needs no such caveat: the first loopback() call of a node's life now succeeds during login.

Notes

  • loopback() stays actor-isolated on purpose. tsnet's Loopback() lazily initialises s.loopbackListener with no mutex, so the actor's serialisation is load-bearing for concurrent Swift callers.
  • Task.detached parks a cooperative-pool thread for the duration of the login. That is the tradeoff of a blocking C call; if you would rather not hold a pool thread for minutes, the same effect is available via a withCheckedContinuation over a dedicated DispatchQueue. Happy to switch — I went with your version as written.
  • The full existing suite (testV4, testV6, testProxy, testStatus) passes with this change.

Tracking issue: tailscale/tailscale#20997
Related: #57, from the same work.

prakashrj added a commit to indiagrams/tunnelless that referenced this pull request Aug 26, 2026
Calls the real LocalAPIClient.backendStatus() rather than reproducing its
request by hand, so the measurement is of the shipped code path rather than an
approximation of it.

Bounded by a 20s race because the failure mode is an indefinite hang with no
error: without a cap, "broken" and "slow" are indistinguishable, and the probe
would itself hang rather than report.

Measured on a physical iPhone 12, node Running, 65 peers:

  stock TailscaleKit    -> no return within 20s
  patched TailscaleKit  -> 200 OK in 6ms, 65 peers

The patched framework was built from tailscale/libtailscale#58 via
build-tailscalekit.sh, so this is the shipped wrapper under test, not a
hand-rolled stand-in.

Requires TailscaleNodeManager.currentNode(), which is documented as safe only
after up() has returned — up() holds the actor for the whole login flow, so any
earlier access queues behind it.

Updates #19
@prakashrj
prakashrj force-pushed the localapi-no-proxy-hop branch from aac64ae to b336159 Compare August 26, 2026 18:19
prakashrj added a commit to indiagrams/tunnelless that referenced this pull request Aug 26, 2026
…et (#18)

* docs(agents): record two silent-failure release tooling traps

Both cost real debugging time today, and both share the property that makes
them worth writing down: the command reports success while acting on the
wrong target, so the mistake surfaces later as an unrelated-looking failure.

1. gh resolves the base repo through the `upstream` remote (apple-shipkit),
   and `gh repo set-default` does not reliably override it. `gh variable set`
   wrote APP_NAME, BUNDLE_ID, and TSKIT_RELEASE_REPO to apple-shipkit; the
   verifying `gh variable list` read the same wrong repo and looked correct
   because apple-shipkit already had a DEPENDABOT_AUTOMERGE row. This fork's
   CI then fell back to the template default and every app matrix cell failed
   with `app/TailnetDemo.xcodeproj does not exist` — a message that points at
   the rename, not at the variable. The rule is to pass -R explicitly and to
   verify through the REST API rather than gh's own repo resolution.

2. .bootstrap.env stores ASC_API_KEY_P8_PATH, but the Fastfile's asc_api_key
   helper reads ASC_API_KEY_P8_BASE64. Neither is wrong — bin/lib/bootstrap.rb
   derives the base64 form when `make ship` exports the environment — but a
   direct `fastlane <lane>` skips that derivation and fails with
   "Authentication credentials are missing or invalid", which reads as a bad
   key rather than an unset variable.

Placed under Critical invariants as a new "Release tooling" subsection, in the
same Rule/Why table form as the Tailscale-specific invariants above it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: correct the LocalAPIClient finding — the wrapper hangs, not tsnet

TAILSCALE.md item 1 and README both blamed tsnet's loopback listener for the
LocalAPIClient hang on physical iOS devices. That was wrong, and it mattered:
it made peer and status data look unreachable on device, which is the data any
non-trivial feature here would be built on.

Two measurements on a physical iPhone 12, node Running:

  SOCKS5 CONNECT to a tailnet peer, then HTTP/1.1
    -> 200 OK in 76ms, correct body
  GET http://<loopback>/localapi/v0/status directly
    -> 200 OK in 69ms, 79,919 bytes, 65 peers

Both paths are served by the same loopback listener, so the listener is fine.

Root cause is in TailscaleKit's Swift wrapper. URLSession+Tailscale.swift's
proxyVia(_:) sets

    self.proxyConfigurations = [ProxyConfiguration(socksv5Proxy: endpoint)]

with endpoint being the loopback address itself, and LocalAPIClient calls it on
every request. Each LocalAPI call therefore asks the SOCKS5 proxy to dial the
address that proxy is listening on: the handshake succeeds, the CONNECT never
resolves, and the caller sees an indefinite hang with no error and no timeout.
The proxy hop is also unnecessary — tailscale.h states the same HTTP server
serves the LocalAPI on /localapi, so it is directly reachable given the
documented Sec-Tailscale: localapi header and basic auth with local_api_cred.

The LogPipeLogger workaround stays. It predates this finding, needs no
credentials, and is the only signal available before the node is up.

Also adds the reproduction harness, so the corrected docs cite code that exists:

- SOCKS5Client, a minimal SOCKS5 client over NWConnection. URLSession could not
  serve here: connectionProxyDictionary ignores the SOCKS keys on iOS, so a
  URLSession "through the proxy" silently exits over the normal interface and
  looks like it works while never touching the tailnet.
- -probe <host:port> and -localapi launch arguments reproducing both numbers.

Validation: make check passes; make format-check clean; both probes run green
against a Mac peer on the tailnet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Updates #19

* test(probe): add -localapiclient, the before/after for the upstream fix

Calls the real LocalAPIClient.backendStatus() rather than reproducing its
request by hand, so the measurement is of the shipped code path rather than an
approximation of it.

Bounded by a 20s race because the failure mode is an indefinite hang with no
error: without a cap, "broken" and "slow" are indistinguishable, and the probe
would itself hang rather than report.

Measured on a physical iPhone 12, node Running, 65 peers:

  stock TailscaleKit    -> no return within 20s
  patched TailscaleKit  -> 200 OK in 6ms, 65 peers

The patched framework was built from tailscale/libtailscale#58 via
build-tailscalekit.sh, so this is the shipped wrapper under test, not a
hand-rolled stand-in.

Requires TailscaleNodeManager.currentNode(), which is documented as safe only
after up() has returned — up() holds the actor for the whole login flow, so any
earlier access queues behind it.

Updates #19

* fix(lint): satisfy swiftlint in the probe harness

CI's swiftlint job failed on three violations that make check does not cover —
make format-check runs swiftformat, which is a different tool with a different
rule set, so a locally clean tree still failed in CI.

- optional_data_string_conversion x2: String(decoding:as:) replaced with the
  failable String(bytes:encoding:), which is what the rule asks for. The
  ContentView call falls back to "<non-utf8>" rather than silently producing
  mojibake for a body that is not valid UTF-8.

- cyclomatic_complexity: get(host:port:path:) was doing connection setup, the
  SOCKS5 handshake, RFC 1929 auth, CONNECT, reply parsing, and HTTP in one
  function. Split into openTunnel(host:port:timeout:), which returns a
  connected socket positioned at the payload, and get(), which speaks HTTP over
  it. Destination encoding and BND.ADDR draining moved to named helpers.

The split is not only for the linter: a raw tunnel is what a future in-app
viewer needs, and HTTP-over-SOCKS5 was the wrong seam to hard-code. openTunnel
cancels the socket if the handshake fails, since no caller can own it then.

Validation: swiftlint clean, make check passes, and the -probe run still
returns 200 OK in 74ms with the expected token from a Mac peer — so the
refactor preserves behaviour on device, not just compilation.

Updates #19

---------

Co-authored-by: Prakash Rudraraju <1471544+prakashrj@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@prakashrj
prakashrj force-pushed the localapi-no-proxy-hop branch from b336159 to d25ed30 Compare August 26, 2026 18:57
@prakashrj prakashrj changed the title swift: don't route LocalAPI requests through the SOCKS5 proxy swift: don't require actor entry to read the memoized loopback config Aug 26, 2026
@prakashrj

Copy link
Copy Markdown
Author

Correction — I got the diagnosis wrong, and one number in the original table was not measured.

The first version of this PR claimed the hang was caused by proxyVia() routing LocalAPI requests through the SOCKS5 proxy at the loopback address the proxy is itself bound to, and showed:

before (stock) LocalAPIClient.backendStatus() → never returns (>20s cap)

That "before" figure was never measured. I added the probe after applying my patch, so its first run was against the patched build; the stock row was carried over from a downstream note rather than produced by me. Presenting it in a measurement table was my error.

When I did measure stock, it returned 200 OK in 8 ms — the proxy-routing theory was wrong, and proxying a loopback request through its own proxy works fine.

What is actually true, from a run with the node left in NeedsLogin so up() blocked for ~53 s, probing concurrently with up():

Arm Touches the node actor? Result during up()
direct HTTP GET /localapi/v0/status, loopback config captured before up() no 200 OK in 32 ms, BackendState=NeedsLogin
LocalAPIClient.backendStatus() — stock yes, await node.loopback() hung, resolved only when up() returned
LocalAPIClient.backendStatus() — patched no (memoized read) 200 OK in 2 ms

So the real cause is that up() is a blocking C call holding the TailscaleNode actor, while every LocalAPIClient request awaits an actor-isolated accessor for a value that is already memoized and immutable.

The PR has been force-pushed with the correct fix and the body rewritten. The earlier commit is gone from the branch; this comment is the record of what it claimed.

Apologies for the noise — I should have measured the baseline before asserting it.

Comment thread swift/TailscaleKit/TailscaleNode.swift Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proper fix is probably:

let res = await Task.detached {
      return tailscale_up(tailscale)
}.value

The await should yield there and free up the actors context so you can concurrent fetch the loopback config.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopted — thank you, this is the better fix and the measurements back it.

I had assumed the actor hop was the only thing I could safely change, so I worked around it by making the memoized read nonisolated. That only helps when the loopback config was already resolved before up(), which is why the PR carried a caveat telling callers to resolve it first. Your version removes the caveat: the first loopback() of a node's life now succeeds during login.

Measured before and after, same machine and session, node held in NeedsLogin so up() is genuinely in flight:

build loopback() during in-flight up()
stock HUNG — no return within 10007 ms
await Task.detached { tailscale_up(tailscale) }.value returned in 22 ms

I also checked that the Go side can actually serve the call rather than just accepting it — Up() blocks on the IPN bus watcher, not a lock, and against tsnet directly with testcontrol{RequireAuth: true}, Loopback() returned in 433 µs while Up() was still blocked. So releasing the actor is sufficient on its own.

Two things it fixes beyond LocalAPIClient: statusJSON() was stalled for the same reason, and so was close() — which meant the cancellation path tailscale.h documents ("to cancel an in-progress tailscale_up, use tailscale_close") could not be reached from TailscaleKit.

Two notes:

  • I kept loopback() actor-isolated. tsnet's Loopback() lazily initialises s.loopbackListener with no mutex, so the actor's serialisation is doing real work for concurrent Swift callers.
  • Task.detached parks a cooperative-pool thread for the length of the login. I went with your version as written; if you would rather not hold a pool thread for minutes, the same effect is available via a withCheckedContinuation over a dedicated DispatchQueue. Say the word and I will switch it.

Added a regression test that measures in the blocked state, since measured after up() returns this defect is invisible. Full suite green.

tailscale_up blocks until the node is usable — for an interactive login
that is however long the user takes. Running it inline held the actor for
that entire window, so every other actor-isolated call queued behind it:
loopback(), statusJSON() and close() were all unreachable during login.
That is what made LocalAPIClient unusable during bring-up, and it also made
the documented cancellation path unreachable, since tailscale.h says to
cancel an in-progress tailscale_up with tailscale_close.

Awaiting a detached task suspends up() and releases the actor instead.
TailscaleHandle is Int32, so the capture is trivially Sendable.

Measured on macOS arm64 against a control URL that never completes
registration, so up() is genuinely in flight (Xcode 26.1.1, Swift 6.2.1):

  stock     loopback() HUNG — no return within 10007 ms
  patched   loopback() returned in 22 ms

The Go layer never serialized these: tsnet's Up() blocks on an IPN bus
watcher, not a lock, and Loopback() only takes the brief servers map lock.
Measured directly against tsnet with testcontrol RequireAuth, Loopback()
returned in 433 µs while Up() was blocked in NeedsLogin. Releasing the
actor is therefore sufficient on its own.

loopback() stays actor-isolated deliberately: tsnet's Loopback() lazily
initialises s.loopbackListener with no mutex, so actor serialisation is
load-bearing for concurrent Swift callers.

Adds a regression test that measures in the blocked state — after up()
returns, everything looks healthy and the defect is invisible.
@prakashrj
prakashrj force-pushed the localapi-no-proxy-hop branch from d25ed30 to 6595187 Compare August 27, 2026 00:28
@prakashrj prakashrj changed the title swift: don't require actor entry to read the memoized loopback config swift: run tailscale_up off the TailscaleNode actor Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants