One Dimension at a Time: Exploring ndim in pandas Extensions

Extensions in pandas

• Inherit from the base class pandas.api.extensions.ExtensionArray.
• Examples include categorical data, time series data, etc.
• Provide functionality for data types beyond standard NumPy data types.

ndim property

• This enforces a restriction that ExtensionArrays must be one-dimensional (flat).
• In pandas.api.extensions.ExtensionArray, ndim is set to 1.
• Stands for "number of dimensions."

Reasoning behind the limitation

• For multi-dimensional data, pandas recommends using nested ExtensionArrays or combining them with NumPy arrays within a DataFrame or Series.
• Supporting multi-dimensional ExtensionArrays would add significant complexity due to the need to manage these structures across dimensions.
• ExtensionArrays often have more complex underlying data structures compared to NumPy arrays.

Key points

• This simplifies their implementation and avoids challenges with managing complex data structures in higher dimensions.
• ExtensionArrays are limited to one dimension.
• pandas.api.extensions.ExtensionArray.ndim is always 1.

• For complex multi-dimensional data, consider using nested ExtensionArrays or combining them with NumPy arrays within pandas data structures.
• While ExtensionArrays themselves are 1D, the DataFrames or Series that contain them can have higher dimensions.

---

import pandas as pd

# Create a categorical ExtensionArray
categories = ["Red", "Green", "Blue"]
data = pd.Categorical(["Red", "Green", "Blue"], categories=categories)

# Check the number of dimensions (ndim)
print(data.ndim)  # Output: 1

# Trying to create a 2D ExtensionArray (will raise an error)
# two_dim_data = pd.Categorical([["Red", "Green"], ["Blue", "Red"]], categories=categories)

In this example:

1. We import pandas as pd.
2. We define categories for a categorical ExtensionArray.
3. We create a pandas Categorical object (data) with the defined categories.
4. We print the ndim property of data, which will be 1 (one dimension).
5. We uncomment the commented section to try creating a 2D Categorical object (which will raise an error because ExtensionArrays are limited to 1D).

---

1. • If you're curious about the data itself within the ExtensionArray, you might be able to access its internal representation and check its dimensionality. This can vary depending on the specific ExtensionArray type.
   • For example, a categorical ExtensionArray might store data as codes and categories in separate arrays. You could potentially access these arrays and check their dimensions (though this is not recommended for general use as it's implementation-specific).

2. Checking Dimensionality of Containing Structure
   

   • Remember that ExtensionArrays are used within pandas DataFrames or Series. These DataFrames and Series can have higher dimensions.
   • You can use data.shape (where data is your DataFrame or Series) to check the overall dimensionality of the data structure containing the ExtensionArray.

3. Reshaping Data
   

   • If you need to work with multi-dimensional data, consider reshaping your data before using ExtensionArrays. You can:
     • Create nested ExtensionArrays within a DataFrame or Series.
     • Combine ExtensionArrays with NumPy arrays within the same DataFrame or Series.

Remember
Directly manipulating the ndim property of an ExtensionArray isn't possible or meaningful since it's enforced as 1 for simplicity and consistency. Choose an approach based on whether you want to know the dimensionality of:

• How to handle multi-dimensional data using ExtensionArrays (consider reshaping or nested structures).
• The overall structure containing the ExtensionArray (DataFrame or Series).
• The internal data within the ExtensionArray (implementation-specific).