test: add the UTS harness for derived REST unit tests

.claude/skills/uts-to-python/SKILL.md

@@ -0,0 +1,189 @@
+---
+description: Translate Universal Test Specifications from ably/specification into ably-python tests. Use when deriving, updating or evaluating tests under test/uts.
+allowed-tools: Bash, Read, Edit, Write, WebFetch
+---
+
+# Translating UTS specs into ably-python tests
+
+## Sources
+
+Fetch both fresh at the start of every run; do not work from memory.
+
+```bash
+gh api repos/ably/specification/contents/uts/docs/writing-derived-tests.md --jq '.content' | base64 -d
+gh api repos/ably/specification/contents/uts/rest/unit/<spec>.md --jq '.content' | base64 -d
+```
+
+`writing-derived-tests.md` governs. This file covers only what is particular to ably-python.
+
+## Layout
+
+A spec at `uts/<tier path>/<name>.md` becomes `test/uts/<tier path>/<name>_test.py`, so
+`uts/rest/unit/auth/token_renewal.md` becomes `test/uts/rest/unit/auth/token_renewal_test.py`.
+Every directory needs an `__init__.py`, as `test` is a package.
+
+`test/uts/rest/unit/time_test.py` is the reference example. Follow its shape.
+
+## Anatomy of a derived test
+
+```python
+"""Derived from uts/rest/unit/time.md in ably/specification.
+
+Spec points: RSC16
+"""
+
+# UTS: rest/unit/RSC16/returns-server-time-0
+async def test_rsc16_time_returns_server_time():
+    captured_requests = []
+
+    def on_request(request):
+        captured_requests.append(request)
+        request.respond_with(200, [SERVER_TIME_MS])
+
+    mock_http = MockHttpClient(
+        on_connection_attempt=lambda conn: conn.respond_with_success(),
+        on_request=on_request,
+    )
+    client = rest_client(mock_http)
+
+    result = await client.time()
+
+    assert result == SERVER_TIME_MS
+```
+
+- The `# UTS:` comment carries the spec's Test ID verbatim, immediately above the function.
+- The function name is the spec point plus the Test ID slug: `rest/unit/RSC16/returns-server-time-0`
+  becomes `test_rsc16_time_returns_server_time`. Drop the trailing index.
+- `pytest` runs with `asyncio_mode = "auto"`, so async tests take no decorator.
+- Keep `captured_requests` a local list appended from the handler, as the specs do. Do not
+  reach for `mock_http.captured_requests`.
+- Always pass `on_connection_attempt`, even where it only succeeds.
+- Do not close clients. `test/uts/conftest.py` closes them after every test.
+
+## Mapping pseudocode to ably-python
+
+| Pseudocode | ably-python |
+|---|---|
+| `install_mock(m)` + `Rest(options: ClientOptions(...))` | `rest_client(m, ...)` from `test.uts.helpers.client` |
+| `uninstall_mock()` | nothing; the fixture closes clients |
+| `ClientOptions(key: "app.key:secret")` | the default in `rest_client`; pass nothing |
+| `useBinaryProtocol` / `useTokenAuth` / `tls` | `use_binary_protocol` / `use_token_auth` / `tls` |
+| `AWAIT client.time()` | `await client.time()` |
+| `... FAILS WITH error` | `with pytest.raises(AblyException) as excinfo:` |
+| `error.code` / `error.statusCode` / `error.message` | `excinfo.value.code` / `.status_code` / `.message` |
+| `request.url.queryParams` / `queryParameters` | `request.url.query_params` (spec drift; one concept) |
+| `parse_json(request.body)` | `json.loads(request.body)` |
+| `msgpack_decode(x)` / `msgpack_encode(x)` | `msgpack.unpackb(x)` / `msgpack.packb(x, use_bin_type=False)` |
+| `process_pending_events()` | `await asyncio.sleep(0)` |
+| `enable_fake_timers()` / `ADVANCE_TIME(ms)` | no equivalent; see Timers below |
+
+Client options are snake_case throughout. Check the actual signature in
+`ably/types/options.py` before assuming an option exists.
+
+## The mock
+
+`test/uts/helpers/mock_http.py`, matching `uts/rest/unit/helpers/mock_http.md`. Read it.
+Names match the pseudocode exactly.
+
+Every call raises a `PendingConnection`, then a `PendingRequest` only if the connection
+succeeded. A failed connection records nothing in `captured_requests`.
+
+`PendingConnection`: `host`, `port`, `tls`, `timestamp`; `respond_with_success()`,
+`respond_with_refused()`, `respond_with_timeout()`, `respond_with_dns_error()`.
+
+`PendingRequest`: `method`, `url` (`scheme`, `host`, `port`, `path`, `query_params`),
+`path`, `headers` (case-insensitive), `body` (raw bytes), `timestamp`;
+`respond_with(status, body, headers)`, `respond_with_delay(ms, status, body, headers)`,
+`respond_with_timeout()`.
+
+`MockHttpClient` also carries `captured_requests`, `await_request(timeout)`,
+`await_connection_attempt(timeout)`, `reset()`, and the queue family
+(`queue_response`, `queue_responses`, `queue_timeout`, `queue_delayed_response`,
+`queue_response_for_host`, `queue_response_for_url`). Handlers are reassignable.
+
+A response body given as a dict or list is encoded to match what the client asked for,
+so specs that exercise the binary protocol need no special handling. Pass `bytes` to
+control the encoding yourself, alongside an explicit `Content-Type`.
+
+## ably-python traits that catch translations out
+
+- **The binary protocol is the default.** `use_binary_protocol` defaults to `True`, so
+  requests carry `Accept: application/x-msgpack` and bodies are msgpack. Decode request
+  bodies with `msgpack.unpackb` unless the spec sets `use_binary_protocol=False`.
+- **5xx and CloudFront responses retry across every host.** A handler that always answers
+  500 is called once per host, not once. Count requests accordingly, or branch on a counter.
+- **`client.request()` requires `version`.** It is not optional.
+- **`client.time()` returns milliseconds as a number**, not a datetime. The spec requirement
+  admits "a DateTime or timestamp", so this is idiomatic, not a deviation.
+- **`AblyException` carries `code`, `status_code` and `message`**, and `@catch_all` wraps
+  several public methods, so a transport error surfaces as `AblyException` with code 50000.
+- **A response with no `Content-Type` raises** `AblyException` with code 40013 from
+  `Response.to_native()`. Set `Content-Type` on any body passed as `bytes` or `str` that
+  the client is meant to decode.
+
+## Traps found while deriving the REST unit specs
+
+- **`/time` returns an array.** Several specs stub it as `{"time": N}`; the endpoint
+  and `time.md` both use `[N]`, and `AblyRest.time()` indexes it. Stub `[N]`.
+- **A single queued response is consumed by the first host.** A 5xx or CloudFront
+  response sends the client to the next fallback, so queue one response per host
+  (`queue_responses(3, ...)`) or answer from a handler.
+- **`msgpack.packb(..., use_bin_type=True)`** is required for a payload that must
+  arrive as msgpack `bin` rather than `str`. The mock's automatic encoding uses
+  `use_bin_type=False`, so encode such a body yourself with an explicit
+  `Content-Type`.
+- **`auth_url` requests go through the injected transport.**
+  `Auth.token_request_from_auth_url` uses the client's own HTTP layer, so a spec driving
+  `auth_url` is observed through the mock like any other request.
+- **Anything `PaginatedResult` paginates over needs a `Content-Type`**, since its
+  response processor decodes the body through `Response.to_native()`. A native dict or
+  list body gets one automatically.
+- **The mock enforces the client's read timeout**, so `respond_with_delay` beyond
+  `http_request_timeout` raises rather than arriving late.
+- **Several client options are missing entirely** — `max_message_size` and
+  `log_handler` among them — and raise `TypeError` rather than being ignored. Check
+  `ably/types/options.py` first.
+
+## Timers
+
+Time reaches a client through `ably.util.clock.Clock`, and `rest_client(mock_http, clock=...)`
+replaces it. A stand-in supplies `now_ms()`, `monotonic_ms()` and `timer(timeout_ms, callback)`,
+and is where a spec's `enable_fake_timers()` / `ADVANCE_TIME(ms)` goes. Where a spec shortens an
+interval through a client option instead (`fallback_retry_timeout=100`), follow the spec. The
+global pytest timeout is 30 seconds, so keep any real wait well under it.
+
+## Deviations
+
+Diagnose per the decision tree in `writing-derived-tests.md`, then apply one of:
+
+- **Env-gated skip**, for non-compliance expected to be fixed:
+  ```python
+  from test.uts.helpers.deviations import deviation
+
+  @deviation
+  async def test_rsa7b_client_id_from_token_details():
+      ...
+  ```
+  Reproduce with `RUN_DEVIATIONS=1 uv run --extra crypto pytest -k rsa7b`.
+- **Adapted assertion**, preferred where the behaviour is stable: assert what the SDK does,
+  with the spec's expectation in a comment above.
+- **Spec-error fail-fast**, only where the spec contradicts the features spec:
+  `pytest.fail('UTS spec error <point> - fix the spec first; see deviations.md')`.
+
+Never write a test that passes under either behaviour.
+
+Record every one in `test/uts/deviations.md` under its heading, keeping all four headings
+present and in order: UTS Spec Errors, Failing Tests, Adapted Tests, Mock Infrastructure
+Limitations. Each entry needs the spec point, what the spec says, what the SDK does, root
+cause where known, which tests are affected, and status.
+
+A differently spelled API is not a deviation. Record only wrong behaviour.
+
+## Checks
+
+```bash
+uv run ruff check ably/ test/
+uv run --extra crypto pytest test/uts -q
+```
+
+Both must pass. Line length is 115.

ably/http/http.py

@@ -135,7 +135,7 @@ def __init__(self, ably, options):
 
     @staticmethod
     def __create_client(options):
-        test_options = getattr(options, 'test_options', None)
+        test_options = getattr(options, '_test_options', None)
         if test_options is not None and test_options.http_transport is not None:
             return httpx.AsyncClient(transport=test_options.http_transport)
         return httpx.AsyncClient(http2=True)

ably/types/options.py

@@ -34,7 +34,7 @@ def __init__(self, client_id=None, log_level=0, tls=True, rest_host=None, realti
                  idempotent_rest_publishing=None, loop=None, auto_connect=True,
                  suspended_retry_timeout=None, connectivity_check_url=None,
                  channel_retry_timeout=Defaults.channel_retry_timeout, add_request_ids=False,
-                 vcdiff_decoder: VCDiffDecoder = None, transport_params=None, test_options=None,
+                 vcdiff_decoder: VCDiffDecoder = None, transport_params=None, _test_options=None,
                  **kwargs):
 
         super().__init__(**kwargs)
@@ -130,7 +130,7 @@ def __init__(self, client_id=None, log_level=0, tls=True, rest_host=None, realti
         self.__add_request_ids = add_request_ids
         self.__vcdiff_decoder = vcdiff_decoder
         self.__transport_params = transport_params or {}
-        self.__test_options = test_options
+        self.__test_options = _test_options
         self.__hosts = self.__get_hosts()
 
     @property
@@ -309,7 +309,7 @@ def transport_params(self):
         return self.__transport_params
 
     @property
-    def test_options(self):
+    def _test_options(self):
         return self.__test_options
 
     def __get_hosts(self):

ably/util/clock.py

@@ -35,7 +35,7 @@ def select_clock(options) -> Clock:
     drive time-dependent behaviour without waiting for it. Clients which supply
     none get `Clock`.
     """
-    test_options = getattr(options, 'test_options', None)
+    test_options = getattr(options, '_test_options', None)
     if test_options is not None and test_options.clock is not None:
         return test_options.clock
     return Clock()

test/unit/clock_test.py

@@ -56,7 +56,7 @@ async def handle_async_request(self, request):
 
 
 def test_auth_timestamps_from_the_injected_clock():
-    ably = AblyRest(key='name:secret', test_options=TestOptions(clock=FakeClock(1_500_000_000_000)))
+    ably = AblyRest(key='name:secret', _test_options=TestOptions(clock=FakeClock(1_500_000_000_000)))
     assert ably.auth._timestamp() == 1_500_000_000_000
 
 
@@ -71,7 +71,7 @@ async def test_the_cached_fallback_host_expires_on_the_clock():
     clock = FakeClock()
     transport = RecordingTransport()
     ably = AblyRest(token='foo', fallback_retry_timeout=2000,
-                    test_options=TestOptions(http_transport=transport, clock=clock))
+                    _test_options=TestOptions(http_transport=transport, clock=clock))
     primary = ably.options.get_host()
     transport.refusing = (primary,)
 
@@ -96,7 +96,7 @@ async def test_every_host_is_tried_while_the_retry_budget_lasts():
     # A second of clock time per reading, against the default 15 second budget
     clock = FakeClock(step_ms=1000)
     transport = RecordingTransport()
-    ably = AblyRest(token='foo', test_options=TestOptions(http_transport=transport, clock=clock))
+    ably = AblyRest(token='foo', _test_options=TestOptions(http_transport=transport, clock=clock))
     hosts = ably.http.get_hosts()
     transport.refusing = tuple(hosts)
 
@@ -113,7 +113,7 @@ async def test_retrying_stops_once_the_retry_budget_is_spent():
     clock = FakeClock(step_ms=1000)
     transport = RecordingTransport()
     ably = AblyRest(token='foo', http_max_retry_duration=0.5,
-                    test_options=TestOptions(http_transport=transport, clock=clock))
+                    _test_options=TestOptions(http_transport=transport, clock=clock))
     transport.refusing = tuple(ably.http.get_hosts())
 
     with pytest.raises(AblyException):

test/unit/http_test.py

@@ -36,7 +36,7 @@ async def handle_async_request(self, request):
 
 async def test_http_sends_requests_through_an_injected_transport():
     transport = RecordingTransport(lambda request: httpx.Response(200, json=[1500000000000]))
-    ably = AblyRest(token="foo", test_options=TestOptions(http_transport=transport))
+    ably = AblyRest(token="foo", _test_options=TestOptions(http_transport=transport))
 
     server_time = await ably.time()
 
@@ -52,7 +52,7 @@ def refuse(request):
         raise httpx.ConnectError("connection refused", request=request)
 
     transport = RecordingTransport(refuse)
-    ably = AblyRest(token="foo", test_options=TestOptions(http_transport=transport))
+    ably = AblyRest(token="foo", _test_options=TestOptions(http_transport=transport))
 
     with pytest.raises(AblyException):
         await ably.time()
@@ -75,7 +75,7 @@ def respond(request):
 
     transport = RecordingTransport(respond)
     ably = AblyRest(auth_url='https://auth.example.com/token',
-                    test_options=TestOptions(http_transport=transport))
+                    _test_options=TestOptions(http_transport=transport))
 
     await ably.auth.authorize()
 

test/uts/README.md

@@ -0,0 +1,45 @@
+# Universal Test Specifications
+
+Tests here are derived from the pseudocode specifications in the
+[ably/specification](https://github.com/ably/specification) repository under `uts/`.
+They are mechanical translations: each one names the spec point it covers and
+carries a `# UTS: <id>` comment identifying the specification it came from.
+
+Read `uts/docs/writing-derived-tests.md` in the specification repository before
+adding or changing tests here, alongside `.claude/skills/uts-to-python/SKILL.md`,
+which covers what is particular to this SDK. Record anything that departs from a
+specification in [deviations.md](deviations.md), which also covers how the
+specifications are adopted here and why.
+
+## Layout
+
+```
+helpers/     shared infrastructure the specifications assume
+rest/        specifications under uts/rest
+realtime/    specifications under uts/realtime
+```
+
+Unit tests serve every request from a mock and reach no network. Integration
+tests run against a sandbox app.
+
+## Installing the mock
+
+The specifications express mock installation as a global `install_mock(mock_http)`.
+Here a mock is passed to the client it serves:
+
+```python
+mock_http = MockHttpClient(
+    on_connection_attempt=lambda conn: conn.respond_with_success(),
+    on_request=lambda req: req.respond_with(200, {'result': 'ok'}),
+)
+ably = AblyRest(key=key, _test_options=TestOptions(http_transport=mock_http.as_transport()))
+```
+
+The client builds its HTTP client once, so construct the mock first. Teardown is
+`await ably.close()`, which stands in for `uninstall_mock()`.
+
+## Running
+
+```
+uv run --extra crypto pytest test/uts
+```

test/uts/conftest.py

@@ -0,0 +1,10 @@
+import pytest
+
+from test.uts.helpers.client import close_open_clients
+
+
+@pytest.fixture(autouse=True)
+async def close_clients():
+    """Closes the clients a test built, whether or not its assertions held."""
+    yield
+    await close_open_clients()