Beyond OCR: Boosting RAG Systems with ByteDance's Dolphin Model

bytedance/Dolphin

The ByteDance Dolphin model is a powerful, multimodal document image parsing model. In simple terms, it's designed to read and understand structured content from document images (like scans or PDFs that have been converted to images), including complex elements such as text paragraphs, figures, tables, and mathematical formulas.

It tackles the challenge with a clever "analyze-then-parse" two-stage paradigm

Analyze (Layout Analysis)
First, it analyzes the entire document page to understand the layout, generating a sequence of layout elements (like "Title," "Paragraph," "Table," "Figure") in the correct reading order. This preserves the structural relationships.

Parse (Content Extraction)
Second, it takes those layout elements as "anchors" and uses element-specific prompts to efficiently parse the content of each element in parallel. This focused, parallel parsing is what makes it highly accurate and fast, even for complex documents.

For engineers, Dolphin is a game-changer in any project that involves digitizing or extracting structured data from visual documents.

Since Dolphin is open-source and available on GitHub and Hugging Face, the introduction is quite straightforward.

You'll generally need

Python (usually 3.8 or later)

pip

A GPU for practical inference speed, although CPU inference may be possible for testing.

git and git-lfs (for cloning the repository and downloading the large model files).

The standard way involves cloning the repository and installing dependencies.

# 1. Clone the repository
git lfs install # Make sure git-lfs is installed for large model files
git clone https://github.com/ByteDance/Dolphin.git
cd Dolphin

# 2. Install Python dependencies
pip install -r requirements.txt

# 3. Download the pre-trained model weights
# You can use the Hugging Face CLI for this:
pip install huggingface_hub
huggingface-cli download ByteDance/Dolphin --local-dir ./hf_model

The official repository typically provides several ways to run the model. Here's an example using the Hugging Face framework for a page-level parsing task, which is common for full document processing. This script will take a document image or PDF page and output structured data.

Assuming you have an image file named document_page.jpeg in a demo directory

import os
import argparse
from transformers import AutoProcessor, AutoModelForVision2Text
from PIL import Image

# --- Setup Paths (Replace with your actual paths) ---
MODEL_PATH = "./hf_model"
INPUT_PATH = "./demo/document_page.jpeg"
SAVE_DIR = "./results"

# Ensure the output directory exists
os.makedirs(SAVE_DIR, exist_ok=True)

print("Loading model and processor...")
# Load the model and processor from the local directory
processor = AutoProcessor.from_pretrained(MODEL_PATH)
model = AutoModelForVision2Text.from_pretrained(MODEL_PATH)
print("Model loaded successfully.")

# Load the input image
try:
    image = Image.open(INPUT_PATH).convert("RGB")
except FileNotFoundError:
    print(f"Error: Input file not found at {INPUT_PATH}")
    exit()

# Define the Prompt for Page-Level Parsing
# The specific prompt directs the model on the task.
# This is a key feature of the "Heterogeneous Anchor Prompting" strategy.
page_parsing_prompt = "Generate the structured content of this document page in JSON and Markdown format."

# --- Preprocessing ---
# Prepare the input for the model
inputs = processor(images=image, text=page_parsing_prompt, return_tensors="pt")

# --- Inference ---
print("Running inference...")
# Generate the output sequence
outputs = model.generate(
    input_ids=inputs.input_ids.to(model.device),
    pixel_values=inputs.pixel_values.to(model.device),
    attention_mask=inputs.attention_mask.to(model.device),
    max_length=4096, # Adjust max length based on document complexity
    pad_token_id=processor.tokenizer.eos_token_id,
    eos_token_id=processor.tokenizer.eos_token_id,
)

# Decode the generated tokens
parsed_content = processor.tokenizer.decode(outputs[0], skip_special_tokens=True)

# --- Postprocessing and Saving ---
output_filename = os.path.join(SAVE_DIR, "parsed_output.txt")
with open(output_filename, "w", encoding="utf-8") as f:
    f.write(parsed_content)

print(f" Parsing complete! Output saved to: {output_filename}")
# You would then load the JSON/Markdown from the file for downstream tasks.