Building Live Data-Aware LLM Apps: An Engineer's Perspective

pathwaycom/llm-app

For a software engineer, this project saves a ton of time and complexity. Instead of building the entire data pipeline from scratch, you get a pre-built solution that handles the heavy lifting.

Real-time Data Sync
Imagine building a chatbot for an e-commerce site. Customer data, product inventory, and prices are always changing. This tool can automatically synchronize your application with these live data sources (like a PostgreSQL database or a Kafka stream), ensuring your chatbot's responses are always accurate.

Simplified RAG (Retrieval-Augmented Generation)
RAG is a technique where an LLM retrieves information from an external knowledge base to provide more accurate and context-aware responses.

This project makes implementing RAG much simpler by providing templates that connect your LLM to sources like SharePoint or Google Drive. This is crucial for building enterprise search or internal knowledge bots.

Docker-Friendly
The templates are designed to be run in Docker containers. This is a huge win for development and deployment. It means you can easily get the application running on your local machine, and then deploy it to any cloud environment (like AWS, Azure, or Google Cloud) with minimal configuration changes. It makes the "it works on my machine" problem a thing of the past.

Unified API
It provides a consistent way to connect to diverse data sources. Whether you're pulling from an S3 bucket, a Google Drive folder, or a live API, the process is streamlined, so you don't have to write custom connectors for each one.

Getting up and running with pathwaycom/llm-app is straightforward, thanks to its Docker-friendly design. Here's a general guide.

Clone the Repository
First, you'll need to get the code. Open your terminal and run

git clone https://github.com/pathwaycom/llm-app.git

Choose Your Template
Navigate into the cloned directory. You'll find a few folders, each representing a different application template (e.g., templates/rag-api, templates/enterprise-search). Pick the one that fits your use case.

cd llm-app/templates/rag-api

Configure Your Environment
The templates use environment variables to configure things like your API keys (for OpenAI or another LLM provider) and your data source connections. You'll typically find a .env.template file that you can copy and fill out.

cp .env.template .env

Then, open the .env file and add your keys and data source details.

Run with Docker
This is the best part. With Docker installed, you can launch the entire application with a single command.

docker-compose up --build

Docker will handle building the images and starting the necessary services (like the application server and the data synchronization engine).

Let's walk through a simple example of how to build a RAG-based chatbot that can answer questions based on a set of PDF documents in a Google Drive folder.

Set up your .env file

You would edit your .env file in the rag-api template directory to include your Google Drive and OpenAI credentials.

# .env file

OPENAI_API_KEY="your_openai_key_here"

# Google Drive credentials
GOOGLE_DRIVE_TOKEN_FILE="token.json"
GOOGLE_DRIVE_FOLDER_ID="your_google_drive_folder_id_here"

Configure the application (optional, as some templates are ready to go)

The rag-api template is often pre-configured to process documents. Under the hood, it uses Pathway to create a live data pipeline. The code would look something like this

import pathway as pw
from pathway.llm import pathway_chat_api
from pathway.llm.vector_store import VectorStoreServer
from pathway.llm.vector_store_server import VectorStoreServerConfig

# Define the data source (Google Drive in this case)
google_drive_data = pw.io.google.drive.read(
    folder_id="your_google_drive_folder_id_here",
    refresh_interval=60  # Check for new files every 60 seconds
)

# Process the data (chunking, embedding, etc.)
processed_data = pathway_chat_api.process_documents(
    data=google_drive_data,
    api_key="your_openai_key_here"
)

# Create a vector store for search
vector_store = VectorStoreServer(
    data=processed_data,
    config=VectorStoreServerConfig()
)

# Expose the search endpoint
vector_store.run_server(host="0.0.0.0", port=8080)

This code is an example of what's happening internally. You typically don't need to write it from scratch; you just configure the provided template. The key here is the pw.io.google.drive.read() function, which creates a live data stream. If a new PDF is added to your Google Drive folder, Pathway detects it, processes it, and updates the RAG system automatically.

Run and Interact

Once your Docker containers are running, the RAG service will be available on a local port. You can then interact with it using a simple API call.

# Example API call with curl
curl -X POST http://localhost:8080/v1/query \
-H "Content-Type: application/json" \
-d '{
  "query": "What is the policy for remote work?"
}'

The application will take your query, search for relevant information in the documents, and use that context to generate a response with the LLM.