Table of Contents
Contents
- Table of Contents
- Introduction
- Getting Started with ctypes
- NumPy arrays' ctypes property
- NumPy's ndpointer with ctypes argtypes
- Dynamic allocation through callbacks
- More useful code frags
- Heterogeneous Types Example
- Fibonacci example (using NumPy arrays, C and Scons)
- Pertinent Mailing List Threads
- Documentation
Introduction
ctypes is an advanced Foreign Function Interface package for Python 2.3 and higher. It is included in the standard library for Python 2.5.
ctypes allows to call functions exposed from DLLs/shared libraries and has extensive facilities to create, access and manipulate simple and complicated C data types in Python - in other words: wrap libraries in pure Python. It is even possible to implement C callback functions in pure Python.
ctypes also includes a code generator tool chain which allows automatic creation of library wrappers from C header files. ctypes works on Windows, Mac OS X, Linux, Solaris, FreeBSD, OpenBSD and other systems.
Ensure that you have at least ctypes version 1.0.1 or later.
Other possibilities to call or run C code in python include: SWIG, Cython, Weave, etc
Getting Started with ctypes
The ctypes tutorial and the ctypes documentation for Python provide extensive information on getting started with ctypes.
Assuming you've built a library called foo.dll or libfoo.so containing a function called bar that takes a pointer to a buffer of doubles and an int as arguments and returns an int, the following code should get you up and running. The following sections cover some possible build scripts, C code and Python code.
If you would like to build your DLL/shared library with distutils, take a look at the SharedLibrary distutils extension included with OOF2. This should probably be included in numpy.distutils at some point.
Nmake Makefile (Windows)
Run nmake inside the Visual Studio Command Prompt to build with the following file.
You should be able to build the DLL with any version of the Visual Studio compiler regardless of the compiler used to compile Python. Keep in mind that you shouldn't allocate/deallocate memory across different debug/release and single-threaded/multi-threaded runtimes or operate on FILE*s from different runtimes.
CXX = cl.exe
LINK = link.exe
CPPFLAGS = -D_WIN32 -D_USRDLL -DFOO_DLL -DFOO_EXPORTS
CXXFLAGSALL = -nologo -EHsc -GS -W3 -Wp64 $(CPPFLAGS)
CXXFLAGSDBG = -MDd -Od -Z7 -RTCcsu
CXXFLAGSOPT = -MD -O2
#CXXFLAGS = $(CXXFLAGSALL) $(CXXFLAGSDBG)
CXXFLAGS = $(CXXFLAGSALL) $(CXXFLAGSOPT)
LINKFLAGSALL = /nologo /DLL
LINKFLAGSDBG = /DEBUG
LINKFLAGSOPT =
#LINKFLAGS = $(LINKFLAGSALL) $(LINKFLAGSDBG)
LINKFLAGS = $(LINKFLAGSALL) $(LINKFLAGSOPT)
all: foo.dll
foo.dll: foo.obj
$(LINK) $(LINKFLAGS) foo.obj /OUT:foo.dll
svm.obj: svm.cpp svm.h
$(CXX) $(CXXFLAGS) -c foo.cpp
clean:
-erase /Q *.obj *.dll *.exp *.lib
SConstruct (GCC)
You can use the following file with SCons to build a shared library.
foo.cpp
1 #include <stdio.h>
2
3 #ifdef FOO_DLL
4 #ifdef FOO_EXPORTS
5 #define FOO_API __declspec(dllexport)
6 #else
7 #define FOO_API __declspec(dllimport)
8 #endif /* FOO_EXPORTS */
9 #else
10 #define FOO_API extern /* XXX confirm this */
11 #endif /* FOO_DLL */
12
13 #ifdef __cplusplus
14 extern "C" {
15 #endif
16
17 extern FOO_API int bar(double* data, int len) {
18 int i;
19 printf("data = %p\n", (void*) data);
20 for (i = 0; i < len; i++) {
21 printf("data[%d] = %f\n", i, data[i]);
22 }
23 printf("len = %d\n", len);
24 return len + 1;
25 }
26
27 #ifdef __cplusplus
28 }
29 #endif
30
When building the DLL for foo on Windows, define FOO_DLL and FOO_EXPORTS (this is what you want to do when building a DLL for use with ctypes). When linking against the DLL, define FOO_DLL. When linking against a static library that contains foo, or when including foo in an executable, don't define anything.
If you're unclear about what extern "C" is for, read section 3 of the C++ dlopen mini HOWTO. This allows you to write function wrappers with C linkage on top of a bunch of C++ classes so that you can use them with ctypes. Alternatively, you might prefer to write C code.
foo.py
NumPy arrays' ctypes property
A ctypes property was recently added to NumPy arrays:
In [18]: x = N.random.randn(2,3,4) In [19]: x.ctypes.data Out[19]: c_void_p(14394256) In [21]: x.ctypes.data_as(ctypes.POINTER(c_double)) In [24]: x.ctypes.shape Out[24]: <ctypes._endian.c_long_Array_3 object at 0x00DEF2B0> In [25]: x.ctypes.shape[:3] Out[25]: [2, 3, 4] In [26]: x.ctypes.strides Out[26]: <ctypes._endian.c_long_Array_3 object at 0x00DEF300> In [27]: x.ctypes.strides[:3] Out[27]: [96, 32, 8]
In general, a C function might take a pointer to the array's data, an integer indicating the number of array dimensions, (pass the value of the ndim property here) and two int pointers to the shapes and stride information.
If your C function assumes contiguous storage, you might want to wrap it with a Python function that calls NumPy's ascontiguousarray function on all the input arrays.
NumPy's ndpointer with ctypes argtypes
Starting with ctypes 0.9.9.9, any class implementing the from_param method can be used in the argtypes list of a function. Before ctypes calls a C function, it uses the argtypes list to check each parameter.
Using NumPy's ndpointer function, some very useful argtypes classes can be constructed, for example:
1 from numpy.ctypeslib import ndpointer
2 arg1 = ndpointer(dtype='<f4')
3 arg2 = ndpointer(ndim=2)
4 arg3 = ndpointer(shape=(10,10))
5 arg4 = ndpointer(flags='CONTIGUOUS,ALIGNED')
6 # or any combination of the above
7 arg5 = ndpointer(dtype='>i4', flags='CONTIGUOUS')
8 func.argtypes = [arg1,arg2,arg3,arg4,arg5]
Now, if an argument doesn't meet the requirements, a TypeError is raised. This allows one to make sure that arrays passed to the C function is in a form that the function can handle.
See also the mailing list thread on ctypes and ndpointer.
Dynamic allocation through callbacks
ctypes supports the idea of callbacks, allowing C code to call back into Python through a function pointer. This is possible because ctypes releases the Python Global Interpreter Lock (GIL) before calling the C function.
We can use this feature to allocate NumPy arrays if and when we need a buffer for C code to operate on. This could avoid having to copy data in certain cases. You also don't have to worry about freeing the C data after you're done with it. By allocating your buffers as NumPy arrays, the Python garbage collector can take care of this.
Python code:
This isn't the prettiest way to define the allocator (I'm also not sure if c_long is the right return type), but there are a few bugs in ctypes that seem to make this the only way at present. Eventually, we'd like to write the allocator like this (but it doesn't work yet):
The following also seems to cause problems:
Possible failures include a SystemError exception being raised, the interpreter crashing or the interpreter hanging. Check these mailing list threads for more details:
Time for an example. The C code for the example:
1 #ifndef CSPKREC_H
2 #define CSPKREC_H
3 #ifdef FOO_DLL
4 #ifdef FOO_EXPORTS
5 #define FOO_API __declspec(dllexport)
6 #else
7 #define FOO_API __declspec(dllimport)
8 #endif
9 #else
10 #define FOO_API
11 #endif
12 #endif
13 #include <stdio.h>
14 #ifdef __cplusplus
15 extern "C" {
16 #endif
17
18 typedef void*(*allocator_t)(int, int*);
19
20 extern FOO_API void foo(allocator_t allocator) {
21 int dim = 2;
22 int shape[] = {2, 3};
23 float* data = NULL;
24 int i, j;
25 printf("foo calling allocator\n");
26 data = (float*) allocator(dim, shape);
27 printf("allocator returned in foo\n");
28 printf("data = 0x%p\n", data);
29 for (i = 0; i < shape[0]; i++) {
30 for (j = 0; j < shape[1]; j++) {
31 *data++ = (i + 1) * (j + 1);
32 }
33 }
34 }
35
36 #ifdef __cplusplus
37 }
38 #endif
39
Check the The Function Pointer Tutorials if you're new to function pointers in C or C++. And the Python code:
1 from ctypes import *
2 import numpy as N
3
4 allocated_arrays = []
5 def allocate(dim, shape):
6 print 'allocate called'
7 x = N.zeros(shape[:dim], 'f4')
8 allocated_arrays.append(x)
9 ptr = x.ctypes.data_as(c_void_p).value
10 print hex(ptr)
11 print 'allocate returning'
12 return ptr
13
14 lib = cdll['callback.dll']
15 lib.foo.restype = None
16 ALLOCATOR = CFUNCTYPE(c_long, c_int, POINTER(c_int))
17 lib.foo.argtypes = [ALLOCATOR]
18
19 print 'calling foo'
20 lib.foo(ALLOCATOR(allocate))
21 print 'foo returned'
22
23 print allocated_arrays[0]
The allocate function creates a new NumPy array and puts it in a list so that we keep a reference to it after the callback function returns. Expected output:
calling foo foo calling allocator allocate called 0xaf5778 allocate returning allocator returned in foo data = 0x00AF5778 foo returned [[ 1. 2. 3.] [ 2. 4. 6.]]
Here's another idea for an Allocator class to manage this kind of thing. In addition to dimension and shape, this allocator function takes a char indicating what type of array to allocate. You can get these typecodes from the ndarrayobject.h header, in the NPY_TYPECHAR enum.
1 from ctypes import *
2 import numpy as N
3
4 class Allocator:
5 CFUNCTYPE = CFUNCTYPE(c_long, c_int, POINTER(c_int), c_char)
6
7 def __init__(self):
8 self.allocated_arrays = []
9
10 def __call__(self, dims, shape, dtype):
11 x = N.empty(shape[:dims], N.dtype(dtype))
12 self.allocated_arrays.append(x)
13 return x.ctypes.data_as(c_void_p).value
14
15 def getcfunc(self):
16 return self.CFUNCTYPE(self)
17 cfunc = property(getcfunc)
Use it like this in Python:
Corresponding C code:
None of the allocators presented above are thread safe. If you have multiple Python threads calling the C code that invokes your callbacks, you will have to do something a bit smarter.
More useful code frags
Suppose you have a C function like the following, which operates on a pointer-to-pointers data structure.
You can create the necessary structure from an existing 2-D NumPy array using the following code:
f4ptr*len(x) creates a ctypes array type that is just large enough to contain a pointer to every row of the array.
Heterogeneous Types Example
Here's a simple example when using heterogeneous dtypes (record arrays).
But, be warned that NumPy recarrays and corresponding structs in C may not be congruent.
Also structs are not standardized across platforms ...In other words, be aware of padding issues!
sample.c
SConstruct
sample.py
1 import numpy as N
2 import ctypes as C
3
4 dat = [[1126877361,'sunny'], [1126877371,'rain'], [1126877385,'damn nasty'], [1126877387,'sunny']]
5
6 dat_dtype = N.dtype([('timestamp','i4'),('desc','|S12')])
7 arr = N.rec.fromrecords(dat,dtype=dat_dtype)
8
9 _sample = N.ctypeslib.load_library('libsample','.')
10 _sample.print_weather.restype = None
11 _sample.print_weather.argtypes = [N.ctypeslib.ndpointer(dat_dtype, flags='aligned, contiguous'), C.c_int]
12
13
14 def print_weather(x):
15 _sample.print_weather(x, x.size)
16
17
18
19 if __name__=='__main__':
20 print_weather(arr)
Fibonacci example (using NumPy arrays, C and Scons)
The following was tested and works on Windows (using MinGW) and GNU/Linux 32-bit OSs (last tested 13-08-2009). Copy all three files to the same directory.
The C code (this calculates the Fibonacci number recursively):
1 /*
2 Filename: fibonacci.c
3 To be used with fibonacci.py, as an imported library. Use Scons to compile,
4 simply type 'scons' in the same directory as this file (see www.scons.org).
5 */
6
7 /* Function prototypes */
8 int fib(int a);
9 void fibseries(int *a, int elements, int *series);
10 void fibmatrix(int *a, int rows, int columns, int *matrix);
11
12 int fib(int a)
13 {
14 if (a <= 0) /* Error -- wrong input will return -1. */
15 return -1;
16 else if (a==1)
17 return 0;
18 else if ((a==2)||(a==3))
19 return 1;
20 else
21 return fib(a - 2) + fib(a - 1);
22 }
23
24 void fibseries(int *a, int elements, int *series)
25 {
26 int i;
27 for (i=0; i < elements; i++)
28 {
29 series[i] = fib(a[i]);
30 }
31 }
32
33 void fibmatrix(int *a, int rows, int columns, int *matrix)
34 {
35 int i, j;
36 for (i=0; i<rows; i++)
37 for (j=0; j<columns; j++)
38 {
39 matrix[i * columns + j] = fib(a[i * columns + j]);
40 }
41 }
The Python code:
1 """
2 Filename: fibonacci.py
3 Demonstrates the use of ctypes with three functions:
4
5 (1) fib(a)
6 (2) fibseries(b)
7 (3) fibmatrix(c)
8 """
9
10 import numpy as nm
11 import ctypes as ct
12
13 # Load the library as _libfibonacci.
14 # Why the underscore (_) in front of _libfibonacci below?
15 # To mimimise namespace pollution -- see PEP 8 (www.python.org).
16 _libfibonacci = nm.ctypeslib.load_library('libfibonacci', '.')
17
18 _libfibonacci.fib.argtypes = [ct.c_int] # Declare arg type, same below.
19 _libfibonacci.fib.restype = ct.c_int # Declare result type, same below.
20
21 _libfibonacci.fibseries.argtypes = [nm.ctypeslib.ndpointer(dtype = nm.int),\
22 ct.c_int,\
23 nm.ctypeslib.ndpointer(dtype = nm.int)]
24 _libfibonacci.fibseries.restype = ct.c_void_p
25
26 _libfibonacci.fibmatrix.argtypes = [nm.ctypeslib.ndpointer(dtype = nm.int),\
27 ct.c_int, ct.c_int,\
28 nm.ctypeslib.ndpointer(dtype = nm.int)]
29 _libfibonacci.fibmatrix.restype = ct.c_void_p
30
31 def fib(a):
32 """Compute the n'th Fibonacci number.
33
34 ARGUMENT(S):
35 An integer.
36
37 RESULT(S):
38 The n'th Fibonacci number.
39
40 EXAMPLE(S):
41 >>> fib(8)
42 13
43 >>> fib(23)
44 17711
45 >>> fib(0)
46 -1
47 """
48 return _libfibonacci.fib(int(a))
49
50 def fibseries(b):
51 """Compute an array containing the n'th Fibonacci number of each entry.
52
53 ARGUMENT(S):
54 A list or NumPy array (dim = 1) of integers.
55
56 RESULT(S):
57 NumPy array containing the n'th Fibonacci number of each entry.
58
59 EXAMPLE(S):
60 >>> fibseries([1,2,3,4,5,6,7,8])
61 array([ 0, 1, 1, 2, 3, 5, 8, 13])
62 >>> fibseries(range(1,12))
63 array([ 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55])
64 """
65 b = nm.asarray(b, dtype=nm.intc)
66 result = nm.empty(len(b), dtype=nm.intc)
67 _libfibonacci.fibseries(b, len(b), result)
68 return result
69
70 def fibmatrix(c):
71 """Compute a matrix containing the n'th Fibonacci number of each entry.
72
73 ARGUMENT(S):
74 A nested list or NumPy array (dim = 2) of integers.
75
76 RESULT(S):
77 NumPy array containing the n'th Fibonacci number of each entry.
78
79 EXAMPLE(S):
80 >>> from numpy import array
81 >>> fibmatrix([[3,4],[5,6]])
82 array([[1, 2],
83 [3, 5]])
84 >>> fibmatrix(array([[1,2,3],[4,5,6],[7,8,9]]))
85 array([[ 0, 1, 1],
86 [ 2, 3, 5],
87 [ 8, 13, 21]])
88 """
89 tmp = nm.asarray(c)
90 rows, cols = tmp.shape
91 c = tmp.astype(nm.intc)
92 result = nm.empty(c.shape, dtype=nm.intc)
93 _libfibonacci.fibmatrix(c, rows, cols, result)
94 return result
Here's the SConstruct file contents (filename: SConstruct):
In Python interpreter (or whatever you use), do:
>>> import fibonacci as fb >>> fb.fib(8) 13 >>> fb.fibseries([5,13,2,6] array([ 3, 144, 1, 5])
etc.
Pertinent Mailing List Threads
Some useful threads on the ctypes-users mailing list:
Thomas Heller's answers are particularly insightful.