Exploring torch.distributions.wishart.Wishart.log_prob() for Probability Distributions

Purpose

• Used in statistical modeling and Bayesian inference, especially when dealing with covariance matrices.
• Calculates the logarithm of the probability density function (log-pdf) for a sample matrix under the Wishart distribution.

Wishart Distribution

• Often employed as a conjugate prior for the covariance matrix in a multivariate Gaussian distribution.
• Represents the probability distribution of positive definite matrices.

Parameters

• covariance_matrix (Tensor): The scale matrix of the Wishart distribution (defaults to the identity matrix).
• df (float or Tensor): Degrees of freedom parameter (greater than the matrix dimension minus 1). Controls the shape of the distribution.
• value (Tensor): A square matrix representing the sample you want to compute the log-pdf for.

Functionality

1. • Ensures value is a square matrix.
   • Verifies df is a valid real number greater than the matrix dimension minus 1.

2. Log-pdf Calculation
   

   • Uses a combination of matrix determinant calculations, Cholesky decomposition (for efficiency), and mathematical formulas specific to the Wishart distribution.
   • Involves terms like the determinant of value, trace of the product of value and the inverse of covariance_matrix, and the gamma function.

3. Result
   

   • Returns a Tensor containing the log-pdf value(s) for each sample matrix in value.
   • Can be used for likelihood calculations, Bayesian inference, or loss functions.

Code Snippet (Illustrative)

import torch
from torch.distributions import Wishart

# Example data
data = torch.randn(2, 3, 3)  # Batch of 2 positive definite matrices

# Define Wishart distribution
df = 5  # Degrees of freedom
scale = torch.eye(3)  # Identity matrix as scale matrix (optional)
wishart = Wishart(df, scale)

# Calculate log-probabilities
log_prob = wishart.log_prob(data)
print(log_prob)  # Output: Tensor with log-pdf values for each matrix

Additional Notes

• Singular samples often lead to -inf log-pdf values. You might need to handle these cases appropriately in your application.
• In some cases, the sampling algorithm based on Bartlett decomposition might return singular matrix samples. The Wishart class attempts to correct these for a certain number of tries (controlled by the max_try_correction argument in rsample).
• The underlying implementation might leverage optimized C++ code for efficiency. Refer to PyTorch's source code for details.

---

Using Cholesky Decomposition for Efficiency

import torch
from torch.distributions import Wishart
from torch import linalg

# Sample data
data = torch.randn(2, 3, 3)  # Batch of 2 positive definite matrices

# Define Wishart distribution (specifying Cholesky factor)
df = 5
scale_tril = linalg.cholesky(torch.eye(3))  # Cholesky factor of identity matrix
wishart = Wishart(df, scale_tril=scale_tril)

# Calculate log-probabilities (using Cholesky decomposition internally)
log_prob = wishart.log_prob(data)
print(log_prob)

In this example, we directly provide the Cholesky factor of the scale matrix (scale_tril) for potentially faster computations as the Wishart distribution internally relies on Cholesky decomposition.

Handling Singular Samples (Potential Errors)

import torch
from torch.distributions import Wishart
import warnings

# Sample data with potential singular matrices
data = torch.randn(5, 3, 3)  # Batch of 5 (might include singular ones)

# Define Wishart distribution
df = 4
wishart = Wishart(df)

def handle_log_prob(data):
  # Try-except block to catch singular samples
  try:
    log_prob = wishart.log_prob(data)
  except RuntimeError as e:
    if "singular matrix" in str(e):
      warnings.warn("Encountered singular sample, returning -inf")
      log_prob = torch.full_like(data[:, 0, 0], float('-inf'))
    else:
      raise e
  return log_prob

# Calculate log-probabilities with error handling
log_prob = handle_log_prob(data)
print(log_prob)

This example demonstrates handling potential singular samples returned by the Wishart distribution's internal sampling algorithm. It uses a try-except block with a custom warning for singular samples and replaces their log-pdf values with -inf. This approach avoids potential errors during calculations involving singular matrices. However, note that singular samples might not be realistic in all scenarios.

Using log_prob in Loss Function

import torch
from torch import nn
from torch.distributions import Wishart

# Sample data and target covariance matrix
data = torch.randn(10, 3, 3)
target_cov = torch.eye(3) * 2

# Define Wishart distribution
df = 5
wishart = Wishart(df)

# Negative log-likelihood loss function
class NLLoss(nn.Module):
  def __init__(self, wishart):
    super().__init__()
    self.wishart = wishart

  def forward(self, data, target_cov):
    log_prob = self.wishart.log_prob(data)
    loss = -log_prob.mean()  # Take mean for batch loss
    return loss

# Create loss function instance
loss_fn = NLLoss(wishart)

# Calculate negative log-likelihood loss
loss = loss_fn(data, target_cov)
print(loss)

This example showcases using log_prob within a custom loss function. The NLLoss class calculates the negative log-likelihood loss (NLL) for a batch of samples based on the Wishart distribution and the target covariance matrix. This loss function can be used in optimization algorithms for learning parameters or fitting models.

---

Manual Calculation

• If you're comfortable with the mathematical formulas involved, you can implement the Wishart log-pdf calculation yourself. This requires knowledge of matrix determinants, the trace operator, the gamma function, and specific terms related to the Wishart distribution. While less convenient, it provides complete control over the implementation.

Third-Party Libraries

Alternative Distributions

• Depending on your application, consider alternative distributions that model positive definite matrices. If you don't strictly need the Wishart distribution, explore:
  • Inverse-Wishart Distribution
    The conjugate prior for the covariance matrix in a multivariate Gaussian distribution. It represents the probability of inverse covariance matrices. You can use its log-pdf function (InverseWishart.log_prob()) in PyTorch, but the interpretation might differ from the Wishart distribution.
  • Matrix Normal Distribution
    Represents the probability distribution of matrices. While not specifically limited to positive definite matrices, it can be used to model covariance matrices in certain cases. However, calculating the log-pdf for this distribution might be more complex.

Approximations

• If computational efficiency is a concern, explore approximations for the Wishart log-pdf. This might involve using simpler distributions or numerical methods depending on your specific needs. However, keep in mind the accuracy trade-offs involved in approximations.

Choosing the Best Approach

The best alternative depends on your specific requirements. Consider factors like:

• Computational Efficiency
  Do you need a highly optimized solution, or is a less efficient approach acceptable?
• Accuracy
  How important is the accuracy of the log-pdf calculation?
• Integration
  How easy is it to integrate the alternative with your PyTorch workflow?
• Control
  Do you need complete control over the implementation, or are you comfortable relying on a built-in function?