TEN-framework: Simplifying Real-Time Voice AI Agents

TEN-framework/ten-framework

The TEN-framework is a powerful tool for engineers who want to build real-time conversational voice AI agents. It simplifies the complex process of creating these applications by providing a pre-built structure. This means you don't have to start from scratch, saving a huge amount of development time. It's particularly useful for

Prototyping
Quickly build and test new ideas for AI voice agents.

Reduced Complexity
It handles the difficult parts like real-time audio processing, synchronization, and integrating different AI models (like speech-to-text, LLMs, and text-to-speech).

Modular Design
The framework's modular nature allows you to easily swap out components. For example, if you want to use a different speech-to-text service, you can do it without overhauling your entire application.

Focus on Core Logic
You can focus on building the unique business logic of your application instead of worrying about low-level infrastructure.

Getting the framework up and running is pretty straightforward. You'll need to have Python and pip installed. The best way to install it is using pip.

First, you need to clone the repository from GitHub

git clone https://github.com/TEN-framework/ten-framework.git
cd ten-framework

Next, you can install the necessary packages. It's a good practice to use a virtual environment to manage dependencies.

python -m venv venv
source venv/bin/activate  # On Windows, use `venv\Scripts\activate`
pip install -r requirements.txt

After installation, you'll need to configure your API keys for the different services you want to use, such as OpenAI, ElevenLabs, etc. You can do this by setting them as environment variables.

export OPENAI_API_KEY="your-api-key"
export ELEVENLABS_API_KEY="your-api-key"

Here’s a simple example of how you can use the TEN-framework to create a basic conversational AI agent. This code will set up a listener for a wake word, and when detected, it will start a conversation.

import asyncio
from ten_framework.core.pipeline import Pipeline
from ten_framework.components.input.mic import MicInput
from ten_framework.components.stt.whisper import WhisperSTT
from ten_framework.components.llm.openai import OpenAILLM
from ten_framework.components.tts.elevenlabs import ElevenLabsTTS
from ten_framework.components.output.speaker import SpeakerOutput

async def main():
    # Set up the pipeline components
    mic_input = MicInput(
        wake_word="hey framework",
        wake_word_threshold=0.5
    )
    stt = WhisperSTT()
    llm = OpenAILLM(
        model_name="gpt-4o",
        system_prompt="You are a helpful conversational AI assistant."
    )
    tts = ElevenLabsTTS()
    speaker_output = SpeakerOutput()

    # Build the pipeline
    pipeline = Pipeline(
        input_component=mic_input,
        stt_component=stt,
        llm_component=llm,
        tts_component=tts,
        output_component=speaker_output
    )

    # Start the conversation loop
    print("Agent is listening...")
    await pipeline.run()

if __name__ == "__main__":
    asyncio.run(main())

Imports
We import the necessary classes from the framework, such as Pipeline, and the specific components for input, STT (Speech-to-Text), LLM (Large Language Model), TTS (Text-to-Speech), and output.

Component Initialization
We create instances of each component. For example, MicInput is set up to listen for the wake word "hey framework."

Pipeline Assembly
The Pipeline class acts as the orchestrator. We pass the initialized components to it in the correct order
input -> stt -> llm -> tts -> output.

Running the Pipeline
The pipeline.run() method starts the main loop. The framework takes care of passing the data (audio, text, etc.) between the components in real-time.