Beyond ConstraintRegistry: Alternative Approaches for Constrained Probability Distributions in PyTorch

Purpose

• Transformations, on the other hand, are mathematical operations that map unconstrained values (typically real numbers) to the constrained space.
• Probability distributions often have parameters that reside within a specific range or adhere to certain rules. These are called constraints.
• This function serves as a registry mechanism within PyTorch to associate constraints with their corresponding transformations.

ConstraintRegistry Objects

• PyTorch provides two global ConstraintRegistry objects:
  • biject_to: This registry guarantees bijectivity (one-to-one and onto mapping) between the unconstrained and constrained spaces. The returned transforms have the property .bijective = True and implement the .log_abs_det_jacobian() method, which calculates the absolute determinant of the Jacobian of the transformation.
  • transform_to: This registry offers more flexibility but does not necessarily guarantee bijectivity. Transforms from this registry are suitable for unconstrained optimization algorithms where bijectivity might not be crucial.

Function Usage

import torch.distributions as dist
from torch.distributions.constraint_registry import ConstraintRegistry

# Example with the biject_to registry for bijective transformation
registry = ConstraintRegistry.instance('biject_to')
transform = registry.register(dist.constraints.positive, dist.transforms.ExpTransform())

# Example with the transform_to registry for non-bijective transformation
registry = ConstraintRegistry.instance('transform_to')
transform = registry.register(dist.constraints.simplex, SomeCustomTransform())

1. • torch.distributions provides access to probability distributions and their constraints.
   • ConstraintRegistry from torch.distributions.constraint_registry offers the registration functionality.

2. Choose the appropriate registry
   

   • ConstraintRegistry.instance('biject_to') is used for bijective transformations.
   • ConstraintRegistry.instance('transform_to') is employed for non-bijective transformations.

3. Register the constraint-transform pair
   

   • registry.register(constraint, transform) links the specified constraint (e.g., dist.constraints.positive for positive values) with the corresponding transformation (e.g., dist.transforms.ExpTransform() for exponential mapping).

Key Points

• The choice of registry (biject_to or transform_to) depends on whether you require bijectivity or not in your application.
• By registering constraints and transforms, PyTorch simplifies the process of working with constrained parameters in probability distributions.

---

Example 1: Bijective Transformation (Normal Distribution with Positive Output)

import torch
import torch.distributions as dist
from torch.distributions.constraint_registry import ConstraintRegistry

# Define a normal distribution with a positive constraint
base_dist = dist.Normal(torch.zeros(1), torch.ones(1))

# Registry for bijective transformations (ensures invertibility)
registry = ConstraintRegistry.instance('biject_to')

# Register positive constraint with exponential transformation
transform = registry.register(dist.constraints.positive, dist.transforms.ExpTransform())

# Apply transformation to create a new distribution with positive output
positive_normal = dist.TransformedDistribution(base_dist, transform)

# Sample from the positive normal distribution
sample = positive_normal.sample()
print(sample)  # Output: positive value (e.g., tensor(0.3456))

This example constructs a normal distribution with a positive constraint using the biject_to registry. The ExpTransform ensures that the output of the distribution is always positive.

Example 2: Non-Bijective Transformation (Beta Distribution with Custom Scaling)

import torch
import torch.distributions as dist
from torch.distributions.constraint_registry import ConstraintRegistry

class CustomScalingTransform(dist.transforms.Transform):
    def __init__(self, scale):
        super().__init__()
        self.scale = scale

    def __call__(self, x):
        return x * self.scale

    def inv(self, y):
        # This inverse might not be well-defined for all y values (non-bijective)
        return y / self.scale

# Define a beta distribution
base_dist = dist.Beta(torch.tensor(2.0), torch.tensor(3.0))

# Registry for general transformations (no bijectivity guarantee)
registry = ConstraintRegistry.instance('transform_to')

# Register simplex constraint (0 <= x <= 1) with custom scaling
transform = registry.register(dist.constraints.simplex, CustomScalingTransform(5.0))

# Apply transformation to create a new distribution with scaled output
scaled_beta = dist.TransformedDistribution(base_dist, transform)

# Sample from the scaled beta distribution
sample = scaled_beta.sample()
print(sample)  # Output: scaled value between 0 and 5 (e.g., tensor(3.1234))

This example creates a beta distribution and applies a custom scaling transformation using the transform_to registry. The CustomScalingTransform scales the output of the beta distribution, but it might not be invertible (non-bijective) for all values.

---

Manual Transformation

• However, it requires you to handle the invertibility (if needed) and Jacobian calculations manually.
• This approach offers more control over the transformation logic.
• You can directly apply the transformation to the distribution's parameters.

import torch
import torch.distributions as dist

# Define a normal distribution
base_dist = dist.Normal(torch.zeros(1), torch.ones(1))

# Define exponential transformation function
def exp_transform(x):
    return torch.exp(x)

# Apply transformation to parameters manually
transformed_params = exp_transform(base_dist.loc)

# Create a new distribution with transformed parameters
positive_normal = dist.Normal(transformed_params, base_dist.scale)

# Sample from the positive normal distribution
sample = positive_normal.sample()

Custom TransformedDistribution Class

• This approach allows for better encapsulation and reusability of the transformation logic.
• Implement the transformation logic and Jacobian calculation within the subclass.
• Create a subclass of dist.TransformedDistribution.

import torch
import torch.distributions as dist

class PositiveNormal(dist.TransformedDistribution):
    def __init__(self, loc, scale):
        base_dist = dist.Normal(loc, scale)
        transform = dist.transforms.ExpTransform()
        super().__init__(base_dist, transform)

    def log_abs_det_jacobian(self, x):
        # Implement Jacobian calculation for exponential transformation
        return torch.ones_like(x)

# Create a positive normal distribution
positive_normal = PositiveNormal(torch.zeros(1), torch.ones(1))

# Sample from the positive normal distribution
sample = positive_normal.sample()

Third-Party Libraries

• Libraries like scipy.stats (if interfacing with NumPy) or jax.distributions (for JAX framework) offer similar functionalities for handling constraints and transformations.

• If you already use other libraries, leveraging their existing capabilities can be beneficial.
• For complex transformations or reusable components, a custom TransformedDistribution class is preferred.
• For simple transformations, manual implementation might suffice.