Unlocking Image Segmentation: My Take on SAM


Unlocking Image Segmentation: My Take on SAM

facebookresearch/segment-anything

2025-07-24

Segment Anything Model (SAM) is a powerful new AI model developed by Meta (formerly Facebook Research) that can "cut out" any object in any image. Think of it like a super-smart, automated version of the "magic wand" tool in Photoshop, but infinitely more capable and versatile. It's trained on a massive dataset of over 11 million images and 1.1 billion masks, making it incredibly robust.

From my perspective as a software engineer, SAM is a game-changer because it allows for zero-shot image segmentation. This means I can segment objects in images that the model has never seen before, without needing to fine-tune it or provide specific training examples for those objects. This opens up a ton of possibilities!

I see several immediate applications where SAM can be incredibly helpful

Automated Image Editing and Processing
Imagine quickly isolating subjects from backgrounds for e-commerce product photos, or creating masks for video effects. This could save me hours of manual work.

Data Annotation
If I'm building a new AI model that requires segmented data, SAM can significantly speed up the annotation process. I can use it to generate initial masks, which then only need minor corrections.

Object Recognition and Tracking
SAM can provide precise masks for objects, which can then be used as input for other computer vision tasks like tracking objects in real-time video streams.

Augmented Reality (AR) Applications
For AR, accurately segmenting real-world objects is crucial for seamlessly overlaying virtual content. SAM can make this much more efficient.

Medical Imaging Analysis
While I'd need to be cautious about clinical use, SAM could potentially assist in segmenting anatomical structures or anomalies in medical images, speeding up analysis for researchers.

Robotics and Autonomous Systems
For robots to interact with their environment, they need to understand where objects are. SAM can provide this fine-grained spatial information.

The good news is that Meta has made SAM very accessible. Here's how I'd go about implementing it

The core of it all is the facebookresearch/segment-anything GitHub repository. This is where I'll find

Inference Code
The Python code needed to run the SAM model on my own images.

Pre-trained Model Checkpoints
These are the actual trained weights of the SAM model. They're quite large, but I'll need to download them to use the model.

Example Notebooks
These Jupyter notebooks are incredibly helpful for getting started, as they show me exactly how to load the model, input images, and get segmentation masks.

First, I'd clone the repository and install the necessary libraries. I'd typically do this in a virtual environment to keep my project dependencies organized.

git clone https://github.com/facebookresearch/segment-anything.git
cd segment-anything
pip install -e .

I'll also need torch and torchvision. Depending on my hardware (GPU vs. CPU), the installation command might vary. For GPU support (which is highly recommended for performance), I'd use something like

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 # For CUDA 11.8

I'll need to choose a model checkpoint to download. The repository provides links for different versions (e.g., vit_h, vit_l, vit_b), representing different model sizes and performance characteristics. The larger models are generally more accurate but slower. I'd usually start with vit_h for the best performance.

# Example: Download the largest model (ViT-H)
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

I'd save this .pth file in a convenient location, perhaps within my project directory.

Let's look at a basic Python example of how I would use SAM to segment an object in an image. This example assumes I've already installed the necessary libraries and downloaded a model checkpoint (e.g., sam_vit_h_4b8939.pth).

import torch
import torchvision
import cv2
import numpy as np
import matplotlib.pyplot as plt
from segment_anything import sam_model_registry, SamPredictor

# --- Configuration ---
MODEL_TYPE = "vit_h"  # Or "vit_l", "vit_b"
CHECKPOINT_PATH = "sam_vit_h_4b8939.pth" # Path to your downloaded model checkpoint
IMAGE_PATH = "your_image.jpg" # Replace with the path to your image

# --- Helper function to display results ---
def show_mask(mask, ax, random_color=False):
    if random_color:
        color = np.concatenate([np.random.random(3), np.array([0.6])], axis=0)
    else:
        color = np.array([30/255, 144/255, 255/255, 0.6])
    h, w = mask.shape[-2:]
    mask_image = mask.reshape(h, w, 1) * color.reshape(1, 1, -1)
    ax.imshow(mask_image)

def show_points(coords, labels, ax, marker_size=375):
    pos_points = coords[labels==1]
    neg_points = coords[labels==0]
    ax.scatter(pos_points[:, 0], pos_points[:, 1], color='green', marker='*', s=marker_size, edgecolor='white', linewidth=1.25)
    ax.scatter(neg_points[:, 0], neg_points[:, 1], color='red', marker='x', s=marker_size, edgecolor='white', linewidth=1.25)

def show_box(box, ax):
    x0, y0, w, h = box[0], box[1], box[2]-box[0], box[3]-box[1]
    ax.add_patch(plt.Rectangle((x0, y0), w, h, edgecolor='green', facecolor=(0,0,0,0), lw=2))

# --- Main Segmentation Logic ---
if __name__ == "__main__":
    # 1. Load the image
    image = cv2.imread(IMAGE_PATH)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # SAM expects RGB

    # 2. Initialize SAM
    sam = sam_model_registry[MODEL_TYPE](checkpoint=CHECKPOINT_PATH)
    sam.to(device="cuda" if torch.cuda.is_available() else "cpu") # Use GPU if available
    predictor = SamPredictor(sam)

    # 3. Set the image for the predictor
    predictor.set_image(image)

    # 4. Define your prompts (e.g., a single point, a bounding box)
    # Example: A single foreground point
    input_point = np.array([[500, 375]]) # X, Y coordinate (adjust based on your image)
    input_label = np.array([1]) # 1 for foreground, 0 for background

    # Example: A bounding box around an object (x_min, y_min, x_max, y_max)
    # input_box = np.array([425, 600, 700, 875]) # Uncomment and adjust for bounding box

    # 5. Predict masks
    # Use points:
    masks, scores, logits = predictor.predict(
        point_coords=input_point,
        point_labels=input_label,
        multimask_output=True, # Set to True to get multiple possible masks
    )

    # Or use a bounding box:
    # masks, scores, logits = predictor.predict(
    #    box=input_box,
    #    multimask_output=False, # Often False for boxes if you expect one main object
    # )

    # 6. Visualize the results
    plt.figure(figsize=(10,10))
    plt.imshow(image)
    for i, (mask, score) in enumerate(zip(masks, scores)):
        print(f"Mask {i+1}, Score: {score:.3f}")
        show_mask(mask, plt.gca(), random_color=True)
    show_points(input_point, input_label, plt.gca()) # If using points
    # show_box(input_box, plt.gca()) # If using bounding box
    plt.axis('off')
    plt.show()

    # Optional: Save one of the masks
    # best_mask = masks[np.argmax(scores)] # Get the mask with the highest confidence score
    # cv2.imwrite("output_mask.png", (best_mask * 255).astype(np.uint8))

Imports
I import necessary libraries like torch, cv2 (for image loading), numpy (for array manipulation), and matplotlib (for visualization). The core SAM components are imported from segment_anything.

Configuration
I define the MODEL_TYPE, CHECKPOINT_PATH, and IMAGE_PATH. Remember to change IMAGE_PATH to your actual image!

Helper Functions
show_mask, show_points, and show_box are utility functions provided in the SAM examples to visualize the segmentation results. They help me see what the model has segmented.

Load Image
The image is loaded using OpenCV (cv2) and converted from BGR to RGB, as SAM expects RGB format.

Initialize SAM

sam_model_registry[MODEL_TYPE](checkpoint=CHECKPOINT_PATH) loads the specific SAM model based on the type (vit_h, vit_l, vit_b) and the path to its pre-trained weights.

sam.to(device="cuda" if torch.cuda.is_available() else "cpu") moves the model to the GPU if one is available, which significantly speeds up inference. Otherwise, it runs on the CPU.

SamPredictor(sam) creates a predictor object, which is what I'll use to actually run the segmentation.

Set Image
predictor.set_image(image) is a crucial step. It pre-processes the image internally for SAM, making subsequent predictions faster. I only need to call this once per image.

Define Prompts
This is where I tell SAM what to segment. SAM is designed to be promptable. I can provide

Points
An array of (x, y) coordinates and corresponding labels (1 for foreground, 0 for background). I can click on an object to segment it.

Bounding Boxes
An array defining a rectangular region (x_min, y_min, x_max, y_max) around an object.

Masks
(More advanced) I can even give SAM a partial mask, and it will refine it. I've included examples for both points and bounding boxes. I'd typically choose one or the other based on my use case.

Predict Masks
predictor.predict(...) takes my prompts and runs the SAM model to generate segmentation masks, confidence scores, and raw logits.

multimask_output=True (when using points) tells SAM to return multiple possible masks if it's unsure, allowing me to choose the best one.

Visualize Results
The code then uses matplotlib to display the original image with the generated masks overlaid, along with the input points or box. This is vital for me to verify the segmentation quality.

Optional
Save Mask
I've added a commented-out line showing how to save one of the generated masks as a new image.

GPU is Your Friend
If I plan to do any serious work with SAM, I'd definitely make sure I have access to a GPU. CPU inference can be very slow, especially for larger images or models.

Experiment with Prompts
Play around with different input points and bounding boxes. See how SAM reacts. Sometimes a slightly different prompt can yield better results.

Explore multimask_output
When using points, try multimask_output=True to see if SAM offers alternative segmentation options. The scores array helps me decide which mask is most confident.

Web Demos
Check out some of the online SAM demos to get a feel for its interactive capabilities.

Beyond Basic Inference
The repository also contains information on how to fine-tune SAM on custom datasets, although that's a more advanced topic.


facebookresearch/segment-anything