Skip to content

fix(integrations): validate firecrawl, context.dev, apify, bright data, browser use, reducto, pulse and extend - #10180

Merged
waleedlatif1 merged 6 commits into
stagingfrom
fix/validate-scraping-docai
Nov 18, 2026
Merged

waleedlatif1 merged 6 commits into
stagingfrom
fix/validate-scraping-docai

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Validates the Firecrawl, Context.dev, Apify, Bright Data, TinyFish, Stagehand, Browser Use, Reducto, Pulse and Extend integrations against each vendor's official API spec (Firecrawl v2, Context.dev, Apify, Browser Use v2, Reducto and TinyFish OpenAPI specs, the Pulse and Extend API references, the Bright Data Scraper API reference, and the Stagehand SDK types), and fixes what was wrong.

  • Every tool's endpoint, method, params, body, response parsing and outputs was checked, plus the internal operations behind Stagehand, Browser Use, Reducto, Pulse and Extend. Claims that could not be verified were left unchanged.
  • Memory-load safety: polling loops (Firecrawl crawl, batch scrape, extract and agent, Apify async run, Bright Data discover, Browser Use) now read every vendor response with a byte cap and stop when the run is cancelled, instead of polling until the deadline. Firecrawl's job status cap is the 64 MiB workflow-value ceiling (MAX_DURABLE_LARGE_VALUE_BYTES), so no result that completed before fails now. Pulse's large-result download is capped at the 10 MiB internal-tool response limit that already applied to it.
  • No subBlock, operation, tool, param or output id was renamed or removed, and no OAuth scope was added. Hosted-key support is unchanged. firecrawl_enrich and apps/sim/enrichments/ are out of scope and untouched.
  • Saved workflows send the same requests as before, except for the bug fixes listed below.
  • TinyFish and Stagehand needed no change.
  • Left unchanged because they could not be verified:
    • TinyFish sends proxy_config.type: "tetra", which the current spec does not list (proxy_config is additionalProperties: false). Every unauthenticated probe stops at the API key check, so the server's behavior is unknown.
    • Context.dev Identify Transaction (/brand/transaction_identifier) is no longer in Context.dev's spec, so it still sends timeoutMS.
    • Firecrawl Extract reads creditsUsed, while the spec documents only tokensUsed for extract jobs. Hosted-key billing for Extract depends on it.
    • Firecrawl crawl and batch scrape return the first page (up to 10MB) when a finished job's result is larger. Following next would put the whole result in memory. The Crawl Status and Batch Scrape Status operations still return next.

Release notes

Firecrawl

  • Scrape now returns rawHtml, summary, links and screenshot when you request those formats. Before, only markdown, HTML and metadata came back.
  • Extract now fails right away when its job is cancelled. Before, it waited until the polling deadline.
  • Crawl, Batch Scrape, Extract and Agent stop polling when the workflow run is cancelled. Crawl, Batch Scrape and Agent also ask Firecrawl to cancel the job, so it stops using credits. Firecrawl has no cancel endpoint for Extract, so a cancelled Extract job keeps running on Firecrawl's side until it finishes.
  • A job status response is read up to 64 MiB, the largest value a workflow output can carry. Larger responses fail with a size error. As before, only the first page of a finished job's results is returned (Firecrawl pages results at 10MB). The Crawl Status and Batch Scrape Status operations return next for later pages.

Context.dev

  • The Timeout (ms) field works again on every operation. Context.dev renamed it to timeoutOpts.milliseconds. Before, Extract and Search failed when a timeout was set, and the other operations ignored it.
  • Because the timeout is now enforced, a request that runs past it fails with Context.dev's 408 error. Context.dev accepts 1,000 to 300,000 ms; some endpoints reject a lower value with a 400. A blank Timeout field sends no timeout. Identify Transaction is unchanged.

Apify

  • Run Actor (Async) stops polling when the workflow run is cancelled. If the dataset is too large to return, it now fails with a message to lower the item limit or page through Get Dataset Items, instead of loading it all into memory.

Bright Data

  • Snapshot Status, Download Snapshot and Cancel Snapshot now reject an empty snapshot ID, ., .., or an ID containing /, with a clear error. Before, the request went to the wrong URL.
  • Discover stops polling when the workflow run is cancelled.

Browser Use

  • New models: GPT-5.5, Gemini 3.1 Pro Preview, Gemini 3.5 Flash, Claude Sonnet 5, Claude Opus 4.7, Claude Opus 4.8 and Claude Opus 5.
  • GPT-4o, GPT-4o Mini and Claude Sonnet 4.6 are hidden for new selections because Browser Use no longer accepts them. Saved blocks keep their value.
  • The System Prompt Extension limit now reads 10,000 characters, matching Browser Use.

Reducto

  • Specific Pages now parses only the pages you list. Before, 1, 5 parsed (and billed) pages 1 through 5.
  • Specific Pages works when the block is used as an Agent tool. Before, the model's page list failed with .trim is not a function.

Pulse

  • Documents of 70 pages or more, and other large results, now return their content. Before, Pulse sent back a download link that was dropped, so markdown came back empty. A full result over 10 MiB fails with a "too large" error asking for fewer pages. The API key is sent to the download link only over HTTPS on Pulse's own host.
  • New extraction_id output, for use with Pulse's split and schema steps.
  • Chunk Size works when the block is used as an Agent tool. Before, the model's number failed with .trim is not a function.

Extend

  • Page count and credits used are now filled in. Before, they were always empty.
  • A file Extend could not parse now fails with Extend's reason (for example CORRUPT_FILE). Before, the block succeeded with no content.

@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Nov 18, 2026 9:07pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking issues were found.

Summary

This PR fixes request options, document results, and cancellation across the scraping and document integrations.

  • Firecrawl jobs stop polling when a workflow run is cancelled.
  • Context.dev requests now send timeouts in the vendor’s expected format.
  • Apify runs stop polling when cancelled and reject oversized dataset results.
  • Bright Data discovery stops on cancellation, and snapshot IDs stay within one URL path segment.
  • Reducto parses only the pages a workflow selects.
  • Pulse returns large extraction results and keeps their extraction ID.
  • Extend reports parse failures and fills in document counts and credits.
  • Firecrawl Scrape returns every requested format in its result.
  • Browser Use offers newer model choices and hides three older choices.
  • Browser Use response reads now have byte limits.

No new actionable issues found. The earlier shared-mock and optional-job-ID findings are addressed.

Reviews (6) · Last reviewed commit: "fix(firecrawl): cancel only a job Firecr..." · Reviewed by Greptile

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 53 files

Confidence score: 4/5

  • In crawl.ts, cancellation during interruptibleSleep can still allow the next Firecrawl poll to start. Check the signal after the sleep in all four polling loops.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/tools/firecrawl/crawl.ts">

<violation number="1" location="apps/sim/tools/firecrawl/crawl.ts:293">
P2: `interruptibleSleep` resolves early when cancelled, but this loop does not re-check the signal before starting the next poll. Add `signal?.throwIfAborted()` immediately after the sleep in all four Firecrawl polling loops.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/blocks/blocks/reducto.ts Resolved
Comment thread apps/sim/tools/brightdata/discover.ts Resolved
Comment thread apps/sim/tools/firecrawl/crawl.ts Resolved
Comment thread apps/sim/tools/firecrawl/scrape.ts Resolved
@waleedlatif1
waleedlatif1 force-pushed the fix/validate-scraping-docai branch from 1580947 to 96afe2d Compare November 18, 2026 18:48
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 53 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 54 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 54 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1
waleedlatif1 force-pushed the fix/validate-scraping-docai branch from 8cadcce to 543b648 Compare November 18, 2026 20:36
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

Comment thread apps/sim/tools/firecrawl/crawl.ts Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 56 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1
waleedlatif1 force-pushed the fix/validate-scraping-docai branch from 543b648 to bdad368 Compare November 18, 2026 21:02
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 56 files

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/tools/firecrawl/crawl.ts Resolved
@waleedlatif1
waleedlatif1 merged commit 0a5db24 into staging Nov 18, 2026
84 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/validate-scraping-docai branch November 18, 2026 21:21

This branch was successfully deployed

1 active deployment
Preview — bdad3688 Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant