Domain-Scoped Site Crawler for Markdown Datasets

The columns you get

• Linkurl
• Name or titletitle
• Depthdepth
• Word countwordCount
• Http status codehttpStatusCode

From the published output schema, checked 2026-11-13.

What it costs

• 100 rows in one run$0.50
• 1,000 rows in one run$1.39
• 10,000 rows in one run$10.30

Free-plan prices: $0.99 per 1,000 results plus a $0.40 start fee (4 GB run) per run, plus Apify platform usage at $0.20 per compute unit. The Free plan includes $5 a month (Apify pricing).

What you fill in

• Start urls (required)
  One or more URLs to start crawling from. The crawler follows links found on these pages and stays on the domain of the first start URL.
• Max crawl depth
  How many links deep to follow from the start URLs. 0 means only the start URLs are crawled.
• Max crawl pages
  Maximum number of pages (and files) to store. The crawl stops when this limit is reached.
• Use sitemaps
  Look for sitemaps (from robots.txt and /sitemap.xml) and enqueue the URLs they list. Finds pages that are not linked from the start URLs.
• Use llms txt
  Look for /llms.txt at the start URL domains and enqueue the files listed there (common on documentation sites).
• Respect robots txt file
  Skip discovered pages that the site's robots.txt disallows. Start URLs are always crawled.
• Crawl only pages changed since
  Optional date (YYYY-MM-DD or full ISO date-time). Crawls only pages that may have changed since then and outputs a delta. Leave empty to crawl everything.
• Include link globs
  Only crawl discovered URLs matching these glob patterns (e.g. https://example.com/docs/*). Leave empty to allow everything on the domain.
• Exclude link globs
  Skip discovered URLs matching any of these glob patterns (e.g. *login*, *.pdf).
• Include link regexps
  Only crawl discovered URLs matching at least one of these regular expressions. Combined with glob patterns (both must pass when both are set).