Firecrawl · web spiders · indexing · permission
Who gets to
read the web?
A field guide to Firecrawl, spiders, indexes, and the argument hiding inside every fetch.
“The web was born linkable. The spider made it legible. The index made it governable. AI made the old bargain visible.”
00 / THE ARGUMENT
A spider is a reader
with consequences.
A web crawler begins with an ordinary gesture: follow the link. Then it repeats the gesture faster than a person can, remembers what it finds, and makes the result available to somebody else.
That repetition built search engines. It also built archives, price trackers, research corpora, accessibility tools, spam indexes, competitive intelligence, model datasets, and dozens of businesses whose central claim is that the public web can be made portable.
The moral argument does not sit outside the machinery. It is inside the crawl depth, the user agent, the rate limit, the fields retained, the time a copy persists, the people made searchable, the source kept or lost, and the person who can ask for removal.
01 / THE MACHINE
Watch one URL
become an answer.
Run the bounded trace. Nothing is fetched; this is a browser-local model of the seven decisions inside a modern crawl.
-
What is reachable?
Begin with a URL, a sitemap, a feed, an index, or one link leading to another. Discovery defines the frontier before a page is fetched.
-
What has the host said?
Check robots.txt, terms, identity requirements, rate expectations, and the difference between a public URL and permission for every possible use.
-
What did the server return?
Request the resource. Status codes, redirects, cache headers, authentication, and pacing are part of the answer—not plumbing to ignore.
-
Does the page need a browser?
Modern pages may assemble themselves with JavaScript. A browser-capable crawler can wait, interact, and read the resulting document.
-
What becomes portable?
Turn the document into text, Markdown, links, screenshots, or a requested JSON shape. Extraction always chooses what counts as signal.
-
How will it be found again?
Normalize, deduplicate, annotate, and store a representation. An index is a memory optimized for retrieval, not a neutral copy of the world.
-
What returns, and in what order?
Search, rank, cite, summarize, or answer. The final ordering exposes the values that were hidden in every earlier technical choice.
02 / FIRECRAWL
The crawler
after JavaScript.
Firecrawl is an open-source web data API. Give it a page or site and ask for model-ready Markdown, links, screenshots, or structured JSON. It can search, map, crawl, render, and interact with pages that do not arrive as clean static HTML.
Its origin story is telling: the team says it emerged from repeatedly needing dependable web extraction for Mendable. The new product is not the spider itself so much as a maintained agreement among browser automation, parsing, retries, anti-bot friction, and AI-shaped output.
It compresses the modern crawl stack.
Search, map, scrape, crawl, browser interaction, screenshots, Markdown, and structured JSON can be asked for through one API instead of assembled from separate browser and parsing tools.
It treats web pages as context, not just documents.
A model usually wants the main content, source URLs, and a stable schema—not a raw DOM. Firecrawl’s interesting move is to make that translation a first-class product boundary.
The machinery is inspectable and self-hostable.
The main repository is AGPL-3.0, with separately identified MIT-licensed SDK and UI portions. The public implementation lets a team inspect the layer it is trusting to read the web.
It reveals how hard “give me this page” has become.
JavaScript rendering, anti-bot systems, uneven markup, freshness, extraction quality, and scale all sit behind a deceptively small request. Firecrawl makes that accumulated complexity visible.
03 / TRY THE TOOL
Bring a
web reader.
Firecrawl can sit beside ChatGPT, Codex, Claude, or another MCP-aware client. For PointCast itself, native JSON, feeds, and MCP come first; use a browser crawler when a page actually needs rendering or the work crosses into outside sites.
npx -y firecrawl-cli@latest init --all --browserfirecrawl scrape https://pointcast.xyz/llms.txt --format markdown --only-main-contentfirecrawl crawl https://pointcast.xyz --limit 50 --max-depth 2 --waitFor PointCast, prefer /agent-kit.md, /agents.json, /llms.txt, /llms-full.txt, adjacent .json twins, feeds, or the PointCast MCP server. Use Firecrawl when a page needs browser rendering or when the work crosses into outside sites.
04 / A SHORT HISTORY OF THE SPIDER
Before the answer,
there was a list.
The history of crawling is a history of the web trying to know its own size—and then deciding what should appear first.
- 01
- 02
The “resource discovery problem” is already visible.
A W3C-era note says a new server enters the known web through links or indexes. Before the web is large, it already needs a way to learn what exists.
- 03
The Wanderer walks the web to measure it.
Matthew Gray’s World Wide Web Wanderer begins running in June. It is remembered as an early robot and index-maker, but its first question is almost ecological: how quickly is this habitat growing?
- 04
Robots.txt proposes a courtesy layer.
Martijn Koster defines a convention through which site operators can tell automated visitors which paths they request not be crawled. It is etiquette communicated by a machine-readable file, not a lock.
- 05
The link graph becomes a ranking system.
Brin and Page describe Google’s large-scale search prototype and PageRank. The crawler collects; the index organizes; the links between pages become evidence used to order attention.
- 06
Messy HTML gets a forgiving reader.
Beautiful Soup arrives as a Python library for pulling data from HTML and XML, including malformed pages. The scraper becomes an everyday programmer’s tool, not only a search-engine subsystem.
- 07
A shared crawl becomes public infrastructure.
Common Crawl forms as a nonprofit and begins keeping open web archives. Its proposition is political as much as technical: crawl-scale data should not belong only to the largest companies.
- 08
The robots convention becomes an Internet standard.
RFC 9309 standardizes the Robots Exclusion Protocol while stating its limit plainly: these rules are not access authorization and must not be used as security controls.
- 09
Firecrawl packages the AI-era web reader.
A tool born from Mendable’s repeated extraction needs launches as Firecrawl: one API for turning modern sites into model-ready Markdown or structured data, later joined by search, mapping, and browser actions.
- 10
The old bargain becomes an explicit negotiation.
Publishers, archives, search engines, AI companies, and infrastructure providers test new signals, blocks, licenses, and payment experiments. Cloudflare’s Pay Per Crawl makes the tension literal, even while HTTP 402 remains a reserved, nonstandard status.
05 / THE INDEX
A map that
answers back.
A crawl is a collection procedure. An index is the structure that makes the collection retrievable. It breaks documents into terms and fields, resolves duplicates, records relationships, and prepares an answer to a future query.
Then ranking begins. Google’s 1998 paper made the web’s link structure useful as a signal: a link could behave something like a citation, and citations could help order results. That was a technical breakthrough and a governance system. To rank is to allocate visibility.
Search documentation still keeps three verbs usefully separate: crawling discovers and fetches; indexing interprets and stores; serving retrieves and orders. The distinction matters. A publisher may want a crawler to see a noindex instruction precisely so the page will not be indexed.
Every index has a collection policy, a forgetting policy, and an argument about relevance—even when those policies are hidden behind one search box.
06 / WHO GETS TO DO IT?
Turn the question.
Keep the conflict.
No single lens closes the case. Choose a position to see what it protects, what it risks, and what it owes.
From the desk of the publisher
If I made the page, what did I consent to?
Publicly reachable is not the same thing as infinitely reusable. A publisher may welcome search discovery, reject model training, allow quotation, sell an API, or change their mind. One URL can carry several intended relationships.
Publish clear signals, protect what must actually be private, distinguish indexing from reuse, and avoid pretending robots.txt can express every downstream purpose.
From the desk of the librarian
What disappears if nobody is allowed to copy?
Preservation requires making another copy before a site vanishes, a format dies, or a public record is rewritten. The archive argues for continuity, provenance, and access across time.
Collect with a public mission, preserve context, honor appropriate restrictions, document provenance, and resist turning memory into surveillance.
From the desk of the researcher
Can public knowledge be studied at public scale?
Crawling can reveal language change, discrimination, market behavior, link decay, and cultural history that no single page can show. But people inside the dataset may never have imagined themselves as rows.
Minimize collection, assess human risk, secure raw data, publish methods, separate public availability from ethical harmlessness, and avoid re-identification.
From the desk of the builder
Why make every small team rebuild the same reader?
Open tools like Beautiful Soup, Scrapy, and Firecrawl turn web access into shared capability. They can democratize search, migration, accessibility, interoperability, and agent-making.
Identify the client, obey boundaries, use native feeds and APIs when they are better, cache responsibly, pace requests, and do not bypass access controls.
From the desk of the host
Who pays for somebody else’s crawl?
Every request consumes bandwidth, compute, support, and sometimes scarce licensed content. At scale, a polite reader and an extractive flood are materially different visitors.
Offer legible machine routes where practical, set rate and access controls, measure abuse without over-collecting people, and treat security as security—not as a robots.txt wish.
From the desk of the artist
Is being seen the same as being absorbed?
Artists publish to encounter an audience, but indexing, collage, training, imitation, criticism, and commercial substitution are not one act. A culture needs both circulation and meaningful authorship.
Keep source and transformation visible, do not erase attribution, avoid substituting a derivative for the original market, and make room for refusal as well as reference.
From the desk of the agent
What must a machine remember about how it learned?
An agent turns retrieval into action. A missing source, stale page, or misread rule can propagate into confident advice. The crawler is now part of an epistemic supply chain.
Prefer structured sources, retain URLs and timestamps, expose uncertainty, quote sparingly, cite what supports the answer, and require human approval before consequential action.
From the desk of the public reader
Should the open web be searchable by everyone—or only by incumbents?
A ban that only the largest platforms can technically or contractually survive does not necessarily protect the public. Open crawling can decentralize knowledge; unrestricted extraction can also concentrate it again.
Ask who gains memory, who bears cost, who can contest an index, whose work becomes infrastructure, and whether the result expands public capability or merely private scale.
User-agent: FieldGuideBot Disallow: /private-draft/ Crawl-delay: be kind Sitemap: /sitemap.xml REQUESTNOT LOCK
07 / ROBOTS.TXT
Etiquette is
not security.
The Robots Exclusion Protocol is a small marvel: a public file where a host asks automated visitors not to enter certain paths. It turns social coordination into text a machine can read.
Its limit is equally important. RFC 9309 says the rules are not access authorization. A robots file cannot make sensitive material private; in fact, naming a private-looking path can reveal it. Protect secrets with authentication and access control.
Robots rules also do not express every possible downstream use. Indexing for search, archiving for history, extracting facts, training a model, quoting criticism, and reproducing a work are different relationships even when the same HTTP request begins them.
08 / A CRAWL COMPACT
Eight rules for
a courteous machine.
Not a substitute for law, site policy, or judgment. A practical floor for people building readers.
- 01
Introduce the machine.
Use an honest user agent and a working contact route. An invisible client has made accountability impossible before the first byte arrives.
- 02
Read the room.
Check robots.txt, terms, authentication, purpose-specific signals, and the site’s native feeds or APIs. A machine route may be both kinder and more accurate.
- 03
Take the minimum.
Fetch only what the task needs. Bound depth, domains, time, and fields. “The whole site” is a consequential choice, not a convenient default.
- 04
Keep a human pace.
Rate-limit, add backoff, cache, deduplicate, and stop on distress. The right speed is defined by the host’s capacity and policy, not the crawler’s maximum.
- 05
Do not pick locks.
Do not bypass authentication, paywalls, CAPTCHAs, or technical access controls. Public research and interoperability do not erase boundaries.
- 06
Carry provenance.
Keep the source URL, retrieval time, relevant title, and transformation record. Context without provenance becomes rumor at machine speed.
- 07
Separate reading from rights.
A successful fetch does not settle copyright, privacy, contract, database, publicity, or sector-specific rules. Technical reachability answers only a technical question.
- 08
Make stopping real.
Honor deletion and correction where appropriate, expire stale records, and provide a human contact. A responsible index includes a way out.
09 / BOUNDARIES
What this page
does not settle.
This is an editorial and technical field guide, not legal advice. Scraping questions can involve copyright, privacy, contract, computer-access, database, publicity, consumer-protection, and sector-specific law, and outcomes vary by jurisdiction and facts. The hiQ opinion is included as one narrow U.S. appellate dispute over a preliminary injunction and public profile pages—not as a general permission slip.
robots.txt communicates requested crawler behavior. RFC 9309 expressly says it is not access authorization and is not a security control.
PointCast has no sponsorship, partnership, or financial relationship with Firecrawl. Product capabilities and license statements are sourced to Firecrawl’s official repository and documentation and should be rechecked before operational reliance.
A page being public does not remove the possibility of human harm from aggregation, inference, persistence, or re-identification.
10 / SOURCE LEDGER
The links behind
the links.
Primary and official sources wherever possible. Product and policy details were checked for this edition on August 2, 2026.
- 01 Firecrawlofficial repositoryfirecrawl/firecrawl ↗
Product capabilities, open-source license, robots.txt default, and user responsibility.
- 02 Firecrawlofficial documentationIntroduction ↗
Scrape, crawl, map, search, browser rendering, Markdown, and structured output.
- 03 Y Combinatorcompany profileFirecrawl company profile ↗
2024 founding and origin in repeated web-extraction needs at Mendable.
- 04 Firecrawlofficial company historyFirecrawl July 2024 updates ↗
Early launch chronology and Y Combinator milestone.
- 05 W3Cstandards-body historyA Little History of the World Wide Web ↗
1989 proposal and first browser/server in 1990.
- 06 W3Cinventor reflectionThe Web at 25 ↗
Open, universal, royalty-free, decentralized design values and data-governance questions.
- 07 W3C archivehistorical primary sourceKeeping Track of WWW Servers ↗
The 1992 resource-discovery problem and role of links and indexes.
- 08 Matthew Gray / MIT archivehistorical project pageThe World Wide Web Wanderer ↗
June 1993 first run, measurement purpose, and early index.
- 09 IETFinternet standardRFC 9309: Robots Exclusion Protocol ↗
Crawler definition, 1994 origin, 2022 standardization, and robots.txt limits.
- 10 Stanford / Google Researchresearch paperThe Anatomy of a Large-Scale Hypertextual Web Search Engine ↗
1998 Google prototype, crawling and indexing architecture, and PageRank.
- 11 Google Search Centralofficial documentationCrawling and indexing ↗
The operational distinction between crawl, parse, render, index, and search controls.
- 12 Google Search Centralofficial documentationBlock Search indexing with noindex ↗
A crawler must access a page to see a noindex rule; robots blocking and indexing controls are distinct.
- 13 Leonard Richardsonofficial project pageBeautiful Soup ↗
2004 origin and parsing of imperfect HTML/XML.
- 14 Common Crawlofficial nonprofit pageAbout Common Crawl ↗
Open repository mission, nonprofit history, and democratizing access to crawl data.
- 15 Common Crawlofficial operations pageFrequently Asked Questions ↗
CCBot identity, robots checking, rate limiting, and open dataset access.
- 16 Cloudflareinfrastructure-provider announcementIntroducing Pay Per Crawl ↗
2025 block, allow, or experimental paid-crawl controls.
- 17 MDN Web Docsweb reference402 Payment Required ↗
HTTP 402 remains reserved/nonstandard with no standard use convention.
- 18 U.S. Court of Appeals for the Ninth Circuitcourt opinionhiQ Labs, Inc. v. LinkedIn Corp. (2022 opinion) ↗
One U.S. preliminary-injunction dispute involving automated access to public LinkedIn profiles; not a universal scraping rule.