Simplifying Web Scraping with Firecrawl API

mendableai/firecrawl

As a software engineer, you'll often encounter situations where you need to get data from a website, but the site's structure is messy and inconsistent. Maybe you're building a chatbot that needs to answer questions based on your company's documentation, or you're creating a data analysis tool that needs to process blog posts from a specific industry. Traditional web scraping can be a pain. You have to deal with complex HTML, JavaScript-rendered content, and constantly changing website layouts.

This is where mendableai/firecrawl comes in. It simplifies this whole process by

Handling the Complexity
It intelligently scrapes websites, handles JavaScript rendering, and navigates through pages to get all the content you need. This saves you from writing and maintaining a bunch of custom scraping logic.

Providing Clean Output
It automatically converts the website content into a clean markdown format or structured data. Markdown is great because it removes all the junk like ads and navigation menus, leaving you with just the core text and headings. This is perfect for feeding into an LLM.

Working with a Simple API
You don't have to worry about running complex software or setting up a bunch of dependencies. You just make an API call with the website URL, and it gives you the data back.

The easiest way to use this tool is through its API. Here's a basic rundown of the steps

Get an API Key
First, you'll need to sign up for an account on their website to get your unique API key. This key authenticates your requests.

Make a Simple API Call
You can use any programming language you're comfortable with (Python, JavaScript, etc.) to make a POST request to their API endpoint. You'll need to include the URL of the website you want to scrape and your API key in the request headers.

The API offers two main modes

scrape
This is for a single URL. It's fast and perfect if you only need the content from one page.

crawl
This is for an entire website. It follows all the links within a specified domain to scrape multiple pages. This is the one you'd use for something like a company's entire documentation site.

Let's look at a simple example using Python and the requests library. This shows how to scrape a single page and print the resulting markdown.

import requests
import json

# Replace with your actual API key
API_KEY = "YOUR_API_KEY"

# The URL you want to scrape
url_to_scrape = "https://www.mendable.ai/"

# API endpoint for scraping a single URL
scrape_url = "https://api.firecrawl.dev/v0/scrape"

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
}

data = {
    "url": url_to_scrape
}

try:
    response = requests.post(scrape_url, headers=headers, data=json.dumps(data))
    response.raise_for_status()  # Raise an exception for bad status codes
    
    # The API returns a dictionary with 'success' and 'data' keys
    result = response.json()
    
    if result["success"]:
        # The markdown content is in the 'content' field
        markdown_content = result["data"]["content"]
        print("--- Scraped Markdown Content ---")
        print(markdown_content)
    else:
        print("API call failed:", result.get("error", "No error message provided"))

except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")

And here's an example for crawling a whole website. The process is very similar, but you use the /crawl endpoint and you can optionally set a maximum depth for the crawl.

import requests
import json

# Replace with your actual API key
API_KEY = "YOUR_API_KEY"

# The domain you want to crawl
domain_to_crawl = "https://www.mendable.ai/"

# API endpoint for crawling a whole website
crawl_url = "https://api.firecrawl.dev/v0/crawl"

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
}

data = {
    "url": domain_to_crawl
}

try:
    response = requests.post(crawl_url, headers=headers, data=json.dumps(data))
    response.raise_for_status()

    result = response.json()

    if result["success"]:
        # The result of a crawl is an array of pages
        pages = result["data"]["pages"]
        print(f"--- Successfully crawled {len(pages)} pages ---")
        # You can now iterate through the pages to get the content
        for page in pages:
            print(f"URL: {page['loc']}")
            # You can access the markdown content with page['content']
            # print(page['content']) 
    else:
        print("API call failed:", result.get("error", "No error message provided"))

except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")