Firecrawl: Your Go-To Tool for AI-Powered Web Content Extraction

firecrawl/firecrawl

Think about all the times you've needed to get content from a website to feed into a language model. Maybe you're building a chatbot that needs to answer questions based on a knowledge base, or an application that summarizes blog posts. Traditionally, this process involves a lot of manual work

Handling messy HTML
Websites are full of tags, scripts, and ads that you don't need. You'd have to write custom parsers to clean this up.

Dealing with different layouts
Every website is structured differently, so a parser you wrote for one site might not work on another.

Extracting relevant content
It's a challenge to figure out what's the "main content" of a page versus sidebar links or footers.

Firecrawl automates all of this. It's built specifically to handle the complexities of the web, so you can focus on building your core application. It saves you a ton of time and effort by turning a complex problem into a simple API call.

Getting up and running with Firecrawl is pretty straightforward. You'll use its API, which you can access in a few different ways. The most common is through a client library.

First, you'll need an API key. Head over to the Firecrawl website and sign up to get your key. This key authenticates your requests.

You can use the official Python or JavaScript SDKs. They make interacting with the API much easier.

Python

pip install firecrawl-py

JavaScript/TypeScript

npm install firecrawl-js

Once installed, you can start making requests. The main functions you'll use are crawl_url and scrape_url.

scrape_url
Use this for a single webpage. It's perfect for when you need to get the content from a specific URL.

crawl_url
Use this for an entire website. It follows internal links to collect content from multiple pages. This is super powerful for building a knowledge base from a whole site.

Let's look at a simple example of how to use the Python SDK to scrape a single URL and get the content back as markdown.

from firecrawl import FirecrawlApp
import os

# Initialize the FirecrawlApp with your API key
# Make sure to set your FIRECRAWL_API_KEY environment variable!
app = FirecrawlApp(api_key=os.environ.get('FIRECRAWL_API_KEY'))

# Define the URL you want to scrape
url_to_scrape = 'https://docs.firecrawl.dev/api-reference/scraping-a-url'

# Scrape the URL
try:
    scraped_data = app.scrape_url(url_to_scrape, {
        # You can add options here, like turning markdown off
        # 'markdown': False 
    })
    
    # The result is a dictionary. The content is under the 'content' key.
    markdown_content = scraped_data.get('content')
    print("Successfully scraped the page! Here's the markdown content:")
    print("---")
    print(markdown_content)
    print("---")

except Exception as e:
    print(f"An error occurred: {e}")

In this example, the scraped_data object will contain a clean markdown representation of the webpage's main content, ready to be passed to an LLM for summarization, question-answering, or any other task.

Here's the same example using the JavaScript SDK.

import { FirecrawlApp } from 'firecrawl-js';

// Initialize the FirecrawlApp with your API key
const app = new FirecrawlApp({ apiKey: process.env.FIRECLAW_API_KEY });

const urlToScrape = 'https://docs.firecrawl.dev/api-reference/scraping-a-url';

async function scrapePage() {
  try {
    const scrapedData = await app.scrapeUrl(urlToScrape, {
      // Options can be passed here as well
    });
    
    console.log("Successfully scraped the page! Here's the markdown content:");
    console.log("---");
    console.log(scrapedData.content);
    console.log("---");

  } catch (error) {
    console.error("An error occurred:", error);
  }
}

scrapePage();