Milvus: A Vector Database for Scalable ANN Search

milvus-io/milvus

A vector database, like Milvus, is a specialized database designed to store, index, and manage massive collections of vectors. Vectors are numerical representations of objects, such as images, text, audio, or video, that capture their semantic meaning. A vector database excels at Approximate Nearest Neighbor (ANN) search, which quickly finds the vectors most similar to a given query vector.

For a software engineer, a vector database is extremely useful for building applications that rely on semantic search or similarity matching. Imagine you're building a system where users need to find similar items based on content, not just keywords. A traditional database with keyword-based search is inefficient for this. This is where a vector database shines.

Image and Video Search
Ever wondered how "search by image" works on Google? The image is converted into a vector, and the system finds similar image vectors in a database. You can build a similar feature for e-commerce sites or personal photo libraries.

Recommendation Systems
To recommend products or content, you can represent user preferences and items as vectors. Then, you can find items with vectors similar to a user's preference vector.

Natural Language Processing (NLP)
For tasks like semantic search or question-answering, text is converted into vectors (called embeddings). You can then query the database to find paragraphs or documents with similar meanings, even if they don't share keywords. This is the core of many modern chatbots and knowledge bases.

Audio Recognition
Similar to images and text, audio snippets can be converted into vectors for tasks like music identification (think Shazam).

The easiest way to start with Milvus is by running it with Docker. It's a single command and gets you a fully functional Milvus instance.

First, make sure you have Docker and Docker Compose installed. Then, use this command to get the Milvus Docker Compose file and start it up.

wget https://github.com/milvus-io/milvus/releases/download/v2.3.0/milvus-standalone-docker-compose.yml -O docker-compose.yml
sudo docker compose up -d

This command downloads the Milvus configuration and starts the Milvus service in the background.

Milvus provides several SDKs for different programming languages. Since you mentioned Go, let's look at the Go SDK. Install it with go get.

go get github.com/milvus-io/milvus-sdk-go/v2

Let's write a simple Go program to connect to Milvus, create a collection (like a table in a relational database), and perform a search.

Create a new Go module and file

mkdir milvus_example && cd milvus_example
go mod init milvus_example
touch main.go

This code snippet shows how to

Connect to your Milvus instance.

Define a schema for a new collection.

Create the collection.

Insert some data (vectors) into it.

Perform a vector search.

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/milvus-io/milvus-sdk-go/v2/client"
	"github.com/milvus-io/milvus-sdk-go/v2/entity"
)

const (
	milvusHost = "localhost:19530"
	collectionName = "go_example_collection"
	vectorDim      = 8
)

func main() {
	// Connect to Milvus
	ctx := context.Background()
	c, err := client.NewClient(ctx, client.Config{
		Address: milvusHost,
	})
	if err != nil {
		log.Fatal("Failed to connect to Milvus:", err.Error())
	}
	defer c.Close()

	// 1. Create a Collection
	schema := &entity.Collection{
		Name: collectionName,
		Schema: &entity.Schema{
			CollectionName: collectionName,
			Fields: []*entity.Field{
				{Name: "book_id", DataType: entity.FieldTypeInt64, IsPrimaryKey: true, AutoID: false},
				{Name: "word_count", DataType: entity.FieldTypeInt64},
				{Name: "book_intro", DataType: entity.FieldTypeFloatVector, Dim: vectorDim},
			},
		},
		Description: "A simple collection for Go example",
	}

	if err := c.CreateCollection(ctx, schema, client.WithConsistencyLevel(entity.ConsistencyLevelStrong)); err != nil {
		log.Fatalf("Failed to create collection: %s", err.Error())
	}
	fmt.Println("Collection created successfully.")

	// 2. Insert data
	bookIDs := []int64{1, 2, 3, 4}
	wordCounts := []int64{100, 200, 300, 400}
	vectors := [][]float32{
		{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8},
		{1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8},
		{2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8},
		{3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8},
	}
	
	insertRows := client.WithColumns(
		entity.NewColumnInt64("book_id", bookIDs...),
		entity.NewColumnInt64("word_count", wordCounts...),
		entity.NewColumnFloatVector("book_intro", vectorDim, vectors...),
	)

	_, err = c.Insert(ctx, collectionName, "", insertRows)
	if err != nil {
		log.Fatalf("Failed to insert data: %s", err.Error())
	}
	fmt.Println("Data inserted successfully.")

	// 3. Perform a search
	searchVector := []float32{1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8}
	
	sp, _ := entity.NewIndexAnnSearchParam(5) // Top 5 results
	
	searchResult, err := c.Search(
		ctx, collectionName,
		[]string{}, // Partitions to search, empty means all
		"",         // Aql expression to filter data (e.g., "book_id > 2")
		[]string{"book_id", "word_count"}, // Fields to retrieve
		[]entity.Vector{entity.FloatVector(searchVector)},
		"book_intro", // The vector field name
		entity.L2,    // Metric type: Euclidean distance (L2)
		5,            // Top k, i.e., the number of nearest neighbors to return
		sp,
	)
	if err != nil {
		log.Fatalf("Failed to search: %s", err.Error())
	}
	
	// Print search results
	for _, result := range searchResult {
		for i, id := range result.IDs {
			fmt.Printf("Search result ID: %v\n", id.AsInt64())
			fmt.Printf("Distance: %.4f\n", result.Scores[i])
		}
	}
}