Firecrawl

Overview

Firecrawl is a web-scraping and crawling API that turns any URL, or a whole site, into clean markdown or structured JSON, ready to feed an LLM, a RAG pipeline, or an AI agent. It renders JavaScript with a real headless browser, follows links to crawl entire sites, and hands back parsed content over a simple HTTP API. This is an API service with no web dashboard to log into: you drive it from your own code, an agent framework, or the official SDKs, pointing them at this instance's URL.

Features

• /scrape a single URL to markdown, HTML, or a structured extract
• /crawl a whole site, following links, with per-job status
• /map a site's URLs quickly without fetching every page
• Real headless-Chromium rendering for JavaScript-heavy pages
• Optional LLM-powered /extract (structured data from a schema or prompt) once you add an OpenAI-compatible API key
• Optional web /search once you point it at a SearXNG endpoint
• malmo provisions and manages the cache Firecrawl needs, so there is nothing to wire up for the core scraping API

Good to know

• LLM and search features are off until you configure them. /extract and other LLM-backed extraction need an OpenAI-compatible API key, and /search needs a SearXNG endpoint. Set them on the app's page after install. Core scrape, crawl, and map work without either.
• This is a heavy app. It runs a Node API, a headless-Chromium browser service, a message broker, and a bundled Postgres alongside malmo's managed cache. That is several gigabytes to download and a few gigabytes of memory to run comfortably. Give it a box with room to spare, and large crawls want more.
• Firecrawl has no login of its own, so keep it private. malmo already keeps each app owner-only by default: the box login sits in front, so only you can reach this API. Leave it that way unless you specifically want an open, unauthenticated scraping endpoint, since anyone who can reach it can use it to fetch arbitrary URLs on your behalf.