Daft Explained: The Python/Rust Distributed Engine for ML Engineers

Eventual-Inc/Daft

At its core, Daft is a distributed query engine that's built for modern data science and machine learning workflows. Think of it as a powerful, distributed version of Pandas, but designed from the ground up to handle not just massive tabular data but also unstructured data like images, video, and audio. It's built with Python for its user-friendly interface and Rust for its high-performance, low-level execution engine.

From an engineering perspective, Daft tackles some of the most common pain points when working with large datasets

Simplifies Distributed Computing
Daft abstracts away the complexity of distributed processing. You can write your data transformation logic using a familiar, high-level API—similar to Pandas or PySpark—and Daft automatically figures out how to parallelize and execute it efficiently across a cluster. This means less time on infrastructure and more time on business logic.

Built for Modern AI Workloads
Unlike traditional data engines, Daft is "modality-aware." It has native support for working with and transforming different types of data (image, audio, text) directly within your data pipelines. This is a massive time-saver for anyone building ML models or data-intensive applications that go beyond simple CSV files.

Performance and Efficiency
The use of Rust in the execution engine means Daft is incredibly fast. It can handle petabyte-scale datasets and complex computations with efficiency that a pure-Python solution couldn't match. This is crucial for keeping development and production cycles fast.

Interoperability
Daft is designed to fit seamlessly into the existing Python data ecosystem. It can easily integrate with popular libraries like PyTorch, Hugging Face, and others, allowing you to load data directly from your Daft DataFrame into your models.

In short, Daft lets you focus on the what (your data transformations) and not the how (the distributed infrastructure), all while delivering excellent performance and reliability.

Getting Daft up and running is very straightforward. It's available on PyPI, so you can install it with pip.

Start by installing Daft in your Python environment

pip install daft

If you want to read data from cloud storage like S3 or GCS, you'll need to install the optional io dependencies

pip install daft[io]

The workflow is very similar to what you'd see with Pandas. You import the daft library, load your data, and then chain together operations. Daft uses lazy execution, so no computation happens until you explicitly ask for the result with a command like .collect() or .show().

Let's say you have a large Parquet file and you want to filter out some rows.

import daft
from daft import col

# Create a Daft DataFrame from a Parquet file (local or cloud path)
df = daft.read_parquet("s3://your-bucket/large_dataset.parquet")

# Filter the DataFrame for rows where the 'sales' column is greater than 100
# and add a new 'is_high_sales' column
filtered_df = df.where(col("sales") > 100).with_column("is_high_sales", col("sales") > 500)

# Display the first few rows (this triggers computation)
filtered_df.show()

# To get the data into a Pandas DataFrame for further analysis
result_df = filtered_df.collect().to_pandas()
print(result_df)

Notice how the syntax is clean and readable, just like with other data manipulation libraries.

This is where Daft truly shines. Let's imagine you have a large dataset of image URLs and you want to resize each image and save the result.

import daft
from daft import col
import daft.types as daft_types

# Create a Daft DataFrame from a list of image URLs
df = daft.from_pydict({"image_url": ["s3://images/dog.jpg", "s3://images/cat.jpg", "s3://images/bird.jpg"]})

# Read the images from the URLs into a new column, resize them, and save them to a new location.
# Note that Daft can handle the I/O and transformations in a distributed way.
processed_df = (
    df
    # Read image data from the URLs
    .with_column("image_bytes", daft_types.image(col("image_url")))
    # Resize the images to 256x256
    .with_column("resized_bytes", daft_types.image(col("image_bytes")).resize(256, 256))
    # Write the results to a new directory as a Parquet file
    .write_parquet("s3://processed-images-output")
)

# This line will trigger the execution of the entire pipeline
processed_df.collect()

In this example, Daft takes care of all the heavy lifting
fetching data from cloud storage, performing the image resizing, and writing the results back, all in a parallel and fault-tolerant way.