Skip to content

Commit 4804fe6

Browse files
FrancescAltedFrancescAlted
committed
Add conversion functions to/from numpy dtypes
1 parent e5e9f80 commit 4804fe6

5 files changed

Lines changed: 720 additions & 0 deletions

File tree

‎doc/numpy-integration.md‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
# NumPy Integration for MiniExpr
2+
3+
This document explains how to integrate miniexpr with NumPy arrays in Python bindings.
4+
5+
## Overview
6+
7+
The file `miniexpr_numpy.h` provides conversion functions between miniexpr's dtype enum and NumPy's type number system, enabling seamless integration with NumPy arrays.
8+
9+
## Why Not Use NumPy Type Numbers Directly?
10+
11+
While it might seem convenient to use NumPy type numbers directly in miniexpr, there are good reasons not to:
12+
13+
1. **ME_AUTO Conflict**: miniexpr needs `ME_AUTO = 0` for automatic type inference, but NumPy uses `0` for bool
14+
2. **Sparse Numbers**: NumPy has gaps (9, 10, 13) for unsupported types, which would waste memory
15+
3. **Performance**: Dense enum values are faster for switch statements and array indexing
16+
4. **Internal Efficiency**: The 13×13 type promotion table requires dense indexing
17+
18+
## API Functions
19+
20+
### `me_dtype_from_numpy(int numpy_type_num)`
21+
22+
Converts a NumPy `dtype.num` value to a miniexpr dtype.
23+
24+
```c
25+
// In your Python C extension
26+
PyArrayObject *array = ...; // Your NumPy array
27+
int numpy_num = PyArray_TYPE(array);
28+
me_dtype dtype = me_dtype_from_numpy(numpy_num);
29+
if (dtype < 0) {
30+
// Unsupported type - error message already printed to stderr
31+
PyErr_SetString(PyExc_TypeError, "Unsupported NumPy dtype");
32+
return NULL;
33+
}
34+
```
35+
36+
**Returns:**
37+
- The corresponding `me_dtype` value (0-13)
38+
- `-1` for unsupported NumPy types (with error message to stderr)
39+
40+
### `me_dtype_to_numpy(me_dtype dtype)`
41+
42+
Converts a miniexpr dtype to a NumPy type number.
43+
44+
```c
45+
me_expr *expr = me_compile(...);
46+
int numpy_num = me_dtype_to_numpy(expr->dtype);
47+
// Use numpy_num to create output array
48+
```
49+
50+
**Returns:**
51+
- NumPy type number (0-15)
52+
- `-1` for `ME_AUTO`
53+
54+
### `me_numpy_type_supported(int numpy_type_num)`
55+
56+
Checks if a NumPy type is supported by miniexpr.
57+
58+
```c
59+
if (!me_numpy_type_supported(PyArray_TYPE(array))) {
60+
PyErr_SetString(PyExc_TypeError, "Unsupported NumPy dtype");
61+
return NULL;
62+
}
63+
```
64+
65+
### `me_numpy_type_name(int numpy_type_num)`
66+
67+
Returns a human-readable name for error messages.
68+
69+
```c
70+
const char *name = me_numpy_type_name(numpy_num);
71+
fprintf(stderr, "Unsupported type: %s\n", name);
72+
```
73+
74+
## Type Mapping
75+
76+
| NumPy Type | dtype.num | ME dtype | Supported |
77+
|--------------|-----------|-----------------|-----------|
78+
| bool | 0 | ME_BOOL | ✓ |
79+
| int8 | 1 | ME_INT8 | ✓ |
80+
| uint8 | 2 | ME_UINT8 | ✓ |
81+
| int16 | 3 | ME_INT16 | ✓ |
82+
| uint16 | 4 | ME_UINT16 | ✓ |
83+
| int32 | 5 | ME_INT32 | ✓ |
84+
| uint32 | 6 | ME_UINT32 | ✓ |
85+
| int64 | 7 | ME_INT64 | ✓ |
86+
| uint64 | 8 | ME_UINT64 | ✓ |
87+
| float16 | 9 | - | ✗ |
88+
| longdouble | 10 | - | ✗ |
89+
| float32 | 11 | ME_FLOAT32 | ✓ |
90+
| float64 | 12 | ME_FLOAT64 | ✓ |
91+
| clongdouble | 13 | - | ✗ |
92+
| complex64 | 14 | ME_COMPLEX64 | ✓ |
93+
| complex128 | 15 | ME_COMPLEX128 | ✓ |
94+
95+
## Usage Examples
96+
97+
### Using Cython
98+
99+
```cython
100+
# mymodule.pyx
101+
cimport numpy as np
102+
from libc.stdlib cimport malloc, free
103+
104+
cdef extern from "miniexpr_numpy.h":
105+
int me_dtype_from_numpy(int numpy_type_num)
106+
int me_dtype_to_numpy(int me_dtype)
107+
108+
cdef extern from "miniexpr.h":
109+
ctypedef struct me_expr:
110+
pass
111+
112+
me_expr* me_compile(const char *expr, ...)
113+
void me_eval(me_expr *expr)
114+
void me_free(me_expr *expr)
115+
116+
def evaluate_expression(str expression, np.ndarray[double] a, np.ndarray[double] b):
117+
cdef np.ndarray[double] result = np.zeros_like(a)
118+
119+
# Convert NumPy dtype to miniexpr dtype
120+
cdef int me_dtype = me_dtype_from_numpy(np.NPY_FLOAT64)
121+
122+
# Setup and compile (simplified)
123+
cdef me_expr *expr = me_compile(expression.encode(), ...)
124+
125+
# Evaluate
126+
me_eval(expr)
127+
128+
# Cleanup
129+
me_free(expr)
130+
131+
return result
132+
```
133+
134+
### Using ctypes
135+
136+
```python
137+
import ctypes
138+
import numpy as np
139+
140+
# Load library
141+
miniexpr = ctypes.CDLL('./libminiexpr.so')
142+
143+
# Setup function signatures
144+
miniexpr.me_dtype_from_numpy.argtypes = [ctypes.c_int]
145+
miniexpr.me_dtype_from_numpy.restype = ctypes.c_int
146+
147+
# Use it
148+
a = np.array([1.0, 2.0, 3.0], dtype=np.float64)
149+
me_dtype = miniexpr.me_dtype_from_numpy(a.dtype.num)
150+
```
151+
152+
### Pure Python (for reference)
153+
154+
If you can't use C extensions, you can implement the mapping in pure Python:
155+
156+
```python
157+
NUMPY_TO_ME = {
158+
0: 1, # bool -> ME_BOOL
159+
1: 2, # int8 -> ME_INT8
160+
7: 5, # int64 -> ME_INT64
161+
12: 11, # float64 -> ME_FLOAT64
162+
# ... etc
163+
}
164+
165+
me_dtype = NUMPY_TO_ME[array.dtype.num]
166+
```
167+
168+
## Performance Notes
169+
170+
- Conversion overhead: ~2 CPU cycles (array lookup)
171+
- Negligible compared to expression evaluation
172+
- No runtime penalty for using conversion functions
173+
174+
## Testing
175+
176+
Run the test suite to verify conversions:
177+
178+
```bash
179+
make build/test_numpy_conversion
180+
./build/test_numpy_conversion
181+
```
182+
183+
All 27 tests should pass.
184+
185+
## See Also
186+
187+
- `doc/numpy_integration_example.py` - Complete example code
188+
- `tests/test_numpy_conversion.c` - Comprehensive test suite
189+
- NumPy dtype documentation: https://numpy.org/doc/stable/reference/arrays.dtypes.html

‎doc/numpy_integration_example.py‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
"""
2+
Example Python bindings for miniexpr using NumPy conversion functions
3+
4+
This demonstrates how to use miniexpr_numpy.h in Python bindings
5+
to seamlessly integrate with NumPy arrays.
6+
"""
7+
8+
# Example using ctypes (simple approach)
9+
# For production, use Cython, pybind11, or cffi
10+
11+
import ctypes
12+
import numpy as np
13+
from ctypes import c_void_p, c_int, c_char_p, POINTER
14+
15+
# Load the miniexpr library
16+
# miniexpr = ctypes.CDLL('./libminiexpr.so')
17+
18+
# Example type conversion in Python
19+
NUMPY_TO_ME_DTYPE = {
20+
0: 1, # bool -> ME_BOOL
21+
1: 2, # int8 -> ME_INT8
22+
2: 6, # uint8 -> ME_UINT8
23+
3: 3, # int16 -> ME_INT16
24+
4: 7, # uint16 -> ME_UINT16
25+
5: 4, # int32 -> ME_INT32
26+
6: 8, # uint32 -> ME_UINT32
27+
7: 5, # int64 -> ME_INT64
28+
8: 9, # uint64 -> ME_UINT64
29+
11: 10, # float32 -> ME_FLOAT32
30+
12: 11, # float64 -> ME_FLOAT64
31+
14: 12, # complex64 -> ME_COMPLEX64
32+
15: 13, # complex128 -> ME_COMPLEX128
33+
}
34+
35+
ME_DTYPE_TO_NUMPY = {v: k for k, v in NUMPY_TO_ME_DTYPE.items()}
36+
37+
def numpy_dtype_to_me(dtype):
38+
"""Convert NumPy dtype to miniexpr dtype code"""
39+
numpy_num = dtype.num
40+
if numpy_num not in NUMPY_TO_ME_DTYPE:
41+
raise ValueError(f"Unsupported NumPy dtype: {dtype}")
42+
return NUMPY_TO_ME_DTYPE[numpy_num]
43+
44+
def me_dtype_to_numpy(me_dtype_code):
45+
"""Convert miniexpr dtype code to NumPy dtype"""
46+
if me_dtype_code not in ME_DTYPE_TO_NUMPY:
47+
raise ValueError(f"Invalid miniexpr dtype: {me_dtype_code}")
48+
numpy_num = ME_DTYPE_TO_NUMPY[me_dtype_code]
49+
return np.dtype(f'i{numpy_num}') # This is simplified
50+
51+
# Example usage with NumPy arrays
52+
def example_usage():
53+
"""
54+
Example of how miniexpr would be used with NumPy arrays
55+
"""
56+
# Create NumPy arrays
57+
a = np.array([1, 2, 3, 4, 5], dtype=np.int64)
58+
b = np.array([10, 20, 30, 40, 50], dtype=np.int64)
59+
result = np.zeros(5, dtype=np.float64)
60+
61+
print("NumPy Integration Example")
62+
print("=" * 50)
63+
print(f"Array a: dtype={a.dtype}, dtype.num={a.dtype.num}")
64+
print(f"Array b: dtype={b.dtype}, dtype.num={b.dtype.num}")
65+
print(f"Result: dtype={result.dtype}, dtype.num={result.dtype.num}")
66+
print()
67+
68+
# Convert dtypes
69+
a_me_dtype = numpy_dtype_to_me(a.dtype)
70+
b_me_dtype = numpy_dtype_to_me(b.dtype)
71+
result_me_dtype = numpy_dtype_to_me(result.dtype)
72+
73+
print(f"Converted to miniexpr:")
74+
print(f" a: ME dtype = {a_me_dtype}")
75+
print(f" b: ME dtype = {b_me_dtype}")
76+
print(f" result: ME dtype = {result_me_dtype}")
77+
print()
78+
79+
# In real bindings, you would call:
80+
# expr = miniexpr.me_compile(
81+
# "sqrt(a*a + b*b)",
82+
# variables=[
83+
# {"name": "a", "dtype": a_me_dtype, "address": a.ctypes.data},
84+
# {"name": "b", "dtype": b_me_dtype, "address": b.ctypes.data}
85+
# ],
86+
# output=result.ctypes.data,
87+
# nitems=len(result),
88+
# dtype=result_me_dtype
89+
# )
90+
# miniexpr.me_eval(expr)
91+
92+
print("In real usage:")
93+
print(" expr = me_compile('sqrt(a*a + b*b)', variables, ...)")
94+
print(" me_eval(expr)")
95+
print(" # result array now contains computed values")
96+
97+
if __name__ == "__main__":
98+
example_usage()
99+
100+
print()
101+
print("=" * 50)
102+
print("Python Binding Patterns:")
103+
print("=" * 50)
104+
print()
105+
print("1. Using C conversion functions (recommended):")
106+
print(" - Include miniexpr_numpy.h in your C extension")
107+
print(" - Use me_dtype_from_numpy(array.dtype.num)")
108+
print(" - Use me_dtype_to_numpy(expr->dtype)")
109+
print()
110+
print("2. Using Python dict mapping (shown above):")
111+
print(" - Pure Python, no C compilation needed")
112+
print(" - Good for prototyping")
113+
print()
114+
print("3. For Cython:")
115+
print(" cdef extern from 'miniexpr_numpy.h':")
116+
print(" int me_dtype_from_numpy(int)")
117+
print(" int me_dtype_to_numpy(int)")

0 commit comments

Comments
 (0)