Credit Limits and Resumable Crawling Now Available via Takumi API

You can now specify a credit limit when running crawls (blackbox-crawl workflow) via the Takumi API. You can also resume a completed crawl with additional credits to continue where it left off — just like the Additional Crawl feature in the web console.

Setting a Credit Limit

When dispatching a crawl workflow, use the input.credit_limit parameter to cap credit consumption. Once the limit is reached, the crawl exits and outputs all features discovered so far.

const { workflow_run_id } = await fetch(
  `${TAKUMI_API}/v1/o/${TAKUMI_ORG}/workflows/blackbox-crawl/dispatch`,
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      input: {
        language: "english",
        target_urls: ["https://app.example/"],

        // Credit limit
        credit_limit: 20,
      },
    }),
  },
).then((r) => r.json());

Resuming a Crawl with Additional Credits

To crawl more broadly or explore specific features in greater depth, you can resume a completed crawl. The resumed crawl inherits all previously discovered features and endpoints, and any newly found ones are added. You can also provide additional instructions to focus the crawl on specific areas.

const { workflow_run_id: crawl_workflow_run_id2 } = await fetch(
  `${TAKUMI_API}/v1/o/${TAKUMI_ORG}/workflows/blackbox-crawl/dispatch`,
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      input: {
        language: "english",
        target_urls: ["https://app.example/"],

        // Resume from a previous crawl
        resume: {
          workflow_id: "blackbox-crawl",
          workflow_run_id: crawl_workflow_run_id, // workflow_run_id of the previous crawl
        },

        // Credit limit (optional)
        credit_limit: 20,

        // Additional instructions (optional)
        additional_instructions:
          "Focus on crawling the APIs (/api/v1/users/...) used by the page accessible at /page/abc",
      },
      notification: { webhook_endpoint_ids: [WEBHOOK_ID] },
    }),
  },
).then((r) => r.json());

Getting Started

See the API documentation for details.