Skip to content

feat(extract): remove the autoparse fallback and add domain preparation - #54

Open
jonathan-zenrows wants to merge 5 commits into
mainfrom
feat/extract-no-default-fallback
Open

jonathan-zenrows wants to merge 5 commits into
mainfrom
feat/extract-no-default-fallback

Conversation

@jonathan-zenrows

@jonathan-zenrows jonathan-zenrows commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

What changes for callers

extract() and extract_async() no longer retry with autoparse. A 403 with code: "REQS007" and the legacy 402 with code: "AUTH010" come back to the caller as is, in one request. The automatic retry hid the prepare step and charged for an autoparse call that often returned unusable data. Autoparse is deprecated.

fallback_to_autoparse stays in both signatures, so existing calls keep working, but it is deprecated and has no effect. Passing True emits a DeprecationWarning that points at prepare_domain().

A 403 REQS007 problem body can now carry a reason. Check it when present; older responses may not carry it:

  • domain_not_prepared: prepare the domain with client.prepare_domain(), poll client.prepared_domain_status() until status is "ready", then retry.
  • page_not_recognised: the domain is served, but this page isn't one Extract extracts (a home, search, login or category page). Try a product, article or listing page, or retry once in a few minutes. Preparing again changes nothing.
  • domain_not_enabled: preparing doesn't help. Use css_extractor, or contact support.
response = client.extract(url)
if response.status_code == 403 and response.json().get("code") == "REQS007":
    reason = response.json().get("reason")  # None on older responses

New: prepare_domain(domain, example_urls) and prepared_domain_status(domain) (plus _async versions) call POST /v1/extract/prepared-domains and GET /v1/extract/prepared-domains/{domain} and return the parsed status. Errors raise PreparedDomainError.

Details

  • The fallback code path is deleted; no option re-enables it.
  • prepare_domain() / prepared_domain_status() send the key in X-API-Key, never poll, and skip the client's retries adapter, so a 429 or 503 comes back on the first answer. An empty or whitespace-only domain raises ValueError before any request. PreparedDomainError carries status_code, code, detail, reason, retry_after and the whole problem body.
  • The README and docstrings describe the three reason cases and mark the autoparse fetch parameter as deprecated.

This ships as feat commits (minor version) through release-please.

Tests

  • REQS007 and AUTH010 are returned as is, in one request, with and without fallback_to_autoparse=True (sync and async); passing True emits a DeprecationWarning attributed to the caller.
  • The reason is readable from the response body.
  • The prepare-domain methods against a stubbed HTTP server: request shape and auth header, parsed status, domain escaping, empty domains, no polling, no retries with retries=1, every documented error answer.

make check (ty, ruff check, ruff format --check), make test (256 passed) and the generated-models check pass locally. A production smoke test returned 403 REQS007 in one request with defaults and with fallback_to_autoparse=True, plus the warning.

extract() and extract_async() no longer retry with autoparse when Extract
answers 403 REQS007 (the domain isn't prepared for Extract yet) or the
legacy 402 AUTH010. Both error responses are now returned to the caller
as is, without a second request.

The fallback stays available as an opt-in: pass fallback_to_autoparse=True
to retry once with deprecated autoparse. To serve a domain that isn't
prepared yet, prepare it with POST /v1/extract/prepared-domains.

The README describes REQS007 and the opt-in fallback, and marks the
autoparse fetch parameter as deprecated.
ZenRowsClient.prepare_domain(domain, example_urls) submits a domain for
Extract with POST /v1/extract/prepared-domains, and
prepared_domain_status(domain) reads its status with
GET /v1/extract/prepared-domains/{domain}. Both send the API key in the
X-API-Key header and return the parsed status object. Neither waits for
the preparation: callers poll the status themselves.

An error answer raises the new PreparedDomainError, which carries the
status code, the problem code (invalid_argument, not_found, in_progress,
out_of_allowance, overloaded, disabled), detail, reason, Retry-After and
the whole problem body.

The extract() docstring and the README point REQS007 at both methods.
…n calls

prepare_domain() and prepared_domain_status() now raise ValueError for
an empty or whitespace-only domain before sending anything. An empty
domain used to request /v1/extract/prepared-domains/, which can resolve
to the list endpoint and come back looking like a status.

Both calls no longer go through the client's retry adapter, so a 429
out_of_allowance or 503 overloaded is returned on the first answer. GET
used to be retried on 429/503 when retries > 0 while POST was not; the
Go and Node SDKs skip client retries here too.

Add prepare_domain_async() and prepared_domain_status_async(), run on
the client's thread pool like the other _async wrappers.
@jonathan-zenrows jonathan-zenrows changed the title feat(extract): stop falling back to autoparse on REQS007 by default feat(extract): make the autoparse fallback opt-in and add domain preparation Oct 15, 2026
@jonathan-zenrows
jonathan-zenrows marked this pull request as ready for review October 16, 2026 06:25
jonathan-zenrows and others added 2 commits October 16, 2026 08:25
… reason

extract() and extract_async() no longer retry with autoparse on any
answer. REQS007 and AUTH010 come back as is, in one request.
fallback_to_autoparse stays in both signatures so existing calls keep
working, but it has no effect: passing True emits a DeprecationWarning
that points at prepare_domain().

The docstrings and the README describe the reason field of a 403
REQS007 problem body and its three cases: domain_not_prepared (prepare
the domain, then poll its status), page_not_recognised (try a product,
article or listing page) and domain_not_enabled (use css_extractor or
contact support). Older responses may not carry a reason.
@jonathan-zenrows jonathan-zenrows changed the title feat(extract): make the autoparse fallback opt-in and add domain preparation feat(extract): remove the autoparse fallback and add domain preparation Oct 16, 2026
@jonathan-zenrows
jonathan-zenrows requested a review from a team October 16, 2026 11:26

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant