Haystack: Your Toolkit for RAG and Conversational AI

deepset-ai/haystack

Imagine you're building a complex application that needs to interact with large amounts of text data. You want to do things like

Question Answering
Given a document, find the specific answer to a user's question.

Semantic Search
Instead of just matching keywords, find documents that are conceptually similar to the user's query.

Chatbots & Conversational Agents
Create a bot that can have a natural, multi-turn conversation with a user, drawing from your private data.

Retrieval-Augmented Generation (RAG)
This is a really powerful technique where you retrieve relevant information from your documents before generating a response. This helps your model provide more accurate and relevant answers, reducing "hallucinations."

Haystack is an "AI orchestration framework." Think of it as a set of LEGO blocks for building these kinds of applications. It provides pre-built components for all the different parts you need

Models
Connect to various LLMs (Large Language Models) like OpenAI's models, Hugging Face models, and more.

Vector Databases
Store and search your documents efficiently using vector embeddings. Haystack supports popular options like ChromaDB, Pinecone, and Weaviate.

Document Converters
Easily read and process different file types like PDFs, text files, and DOCX files.

Pre-processing Tools
Clean and prepare your text data for the models.

The genius of Haystack is that it allows you to connect these components together into "Pipelines" or "Agents." This makes your application modular, scalable, and easy to maintain. You can swap out a model, change your vector database, or add a new component without rewriting your entire application.

Getting started with Haystack is straightforward. Here's the typical workflow

Installation
You can install Haystack with pip. Start with the basic package, and then add extras for the components you plan to use (e.g., specific vector databases or LLMs).

pip install "farm-haystack[qdrant, openai]"

Indexing Your Data
Before you can answer questions, you need to load your data and store it in a way that's searchable. This involves

Loading Documents
Reading your files (e.g., from a folder).

Preprocessing
Cleaning the text and splitting it into smaller chunks.

Creating Embeddings
Using an embedding model to convert the text chunks into numerical vectors.

Writing to a Vector Database
Storing these vectors and the original text in a database.

Building a Pipeline
Define the flow of information. A common RAG pipeline looks like this

Retriever
Finds the most relevant documents for a given query.

PromptNode or Generator
Takes the retrieved documents and the user's query, and generates a final answer.

Running the Pipeline
Execute your pipeline with a user's query and get the result!

Let's build a small, practical example. We'll set up a simple RAG pipeline that can answer questions about some text.

Pre-requisites

You'll need an OpenAI API key.

We'll use Qdrant as our vector database, which is easy to run locally.

Step 1
Installation

pip install "farm-haystack[qdrant, openai]"

Step 2
Indexing Documents

import os
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
from haystack.components.embedders.sentence_transformers import SentenceTransformersDocumentEmbedder
from haystack.components.preprocessors import DocumentSplitter
from haystack import Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.writers import DocumentWriter
from haystack.components.routers import FileTypeRouter
from haystack.components.converters import PyPDFToTextConverter

# A simple text document
documents = [
    {"content": "Haystack is an open-source framework for building LLM applications. It's a great tool for RAG.", "meta": {"name": "doc1"}},
    {"content": "You can use Haystack to build chatbots, semantic search engines, and more.", "meta": {"name": "doc2"}},
    {"content": "It supports many LLMs and vector databases, making it very flexible.", "meta": {"name": "doc3"}},
]

document_store = InMemoryDocumentStore()

# Create a simple indexing pipeline
indexing_pipeline = Pipeline()
indexing_pipeline.add_component("splitter", DocumentSplitter(split_by="word", split_length=10))
indexing_pipeline.add_component("embedder", SentenceTransformersDocumentEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"))
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))

indexing_pipeline.connect("splitter.documents", "embedder.documents")
indexing_pipeline.connect("embedder.documents", "writer.documents")

# Run the pipeline to index the documents
indexing_pipeline.run({"splitter": {"documents": documents}})

print("Documents indexed successfully!")

Step 3
Building and Running the RAG Pipeline

Now, let's create the RAG pipeline.

from haystack import Pipeline
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders.prompt_builder import PromptBuilder
from haystack.components.embedders.sentence_transformers import SentenceTransformersTextEmbedder
from haystack.utils import Secret

# Make sure you have your OpenAI API key set as an environment variable
# export OPENAI_API_KEY="your-api-key"

# Define the RAG pipeline
query_pipeline = Pipeline()
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"))
query_pipeline.add_component("retriever", InMemoryEmbeddingRetriever(document_store=document_store))
query_pipeline.add_component("prompt_builder", PromptBuilder(template="""
    Answer the question based on the provided context.
    Context: {{ context }}
    Question: {{ query }}
    Answer:
"""))
query_pipeline.add_component("llm", OpenAIGenerator(model="gpt-3.5-turbo", api_key=Secret.from_env_var("OPENAI_API_KEY")))

query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
query_pipeline.connect("retriever.documents", "prompt_builder.context")
query_pipeline.connect("prompt_builder.prompt", "llm.prompt")

# Run the query pipeline
question = "What is Haystack useful for?"
result = query_pipeline.run({"text_embedder": {"text": question}, "prompt_builder": {"query": question}})

print(result['llm']['replies'][0])

Modularity
You can easily swap out components. Don't like Qdrant? Switch to ChromaDB. Want to try a different LLM? Just change the Generator component.

Production-Readiness
Haystack is designed for production use. It handles things like asynchronous processing, scaling, and error handling.

Clear Abstractions
The concepts of Pipelines, Agents, and Components make the architecture easy to understand and debug.

Flexibility
It's not a black box. You have full control over the flow of your data, from retrieval to generation.

Community & Support
It has a very active community and great documentation, which is crucial when you're building complex systems.