Table of Contents
Contents
Introduction
These are simple NumPy and SWIG examples which use the numpy.i interface file. There is also a MinGW section for people who may want to use these in a Win32 environment. The following code is C, rather than C++.
The information contained here was first made available by Bill Spotz in his article numpy.i: a SWIG Interface File for NumPy, and the NumPy SVN which can be checked out using the following command:
svn co http://scipy.org/svn/numpy/trunk numpy
The NumPy+SWIG manual is available here: http://scipy.org/svn/numpy/trunk/doc/swig/doc/numpy_swig.pdf
The numpy.i file can be downloaded from the SVN: http://scipy.org/svn/numpy/trunk/doc/swig/numpy.i
and the pyfragments.swg file, hich is also needed, is available from http://scipy.org/svn/numpy/trunk/doc/swig/pyfragments.swg
These two files (and others) are also available in the numpy source tarball: http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103
Initial setup
gcc and SWIG
Check that both gcc and SWIG are available (paths known):
swig -version
and
gcc -v
Both should output some text...
Modifying the pyfragments.swg file (MinGW only)
This is from my own tests, running SWIG Version 1.3.36 and gcc version 3.4.5 (mingw-vista special r3). I had to remove the 'static' statements from the source, otherwise your SWIGed sources won't compile. There are only two 'static' statements in the file, both will need removing. Here is my modified version: pyfragments.swg
Compilation and testing
A setup.py file specific to each module must be written first. I based mine on the reference setup.py available in http://scipy.org/svn/numpy/trunk/doc/swig/test/ with added automatic handling of swig.
On a un*x like system, the command-line is:
python setup.py build
In a Win32 environment (either cygwin or cmd), the setup command-line is (for use with MinGW):
python setup.py build --compiler=mingw32
The command handles both the SWIG process (generation of wrapper C and Python code) and gcc compilation. The resulting module (a pyd file) is built in the build\lib.XXX directory (e.g. for a Python 2.5 install and on a Win32 machine, the build\lib.win32-2.5 directory).
A simple ARGOUT_ARRAY1 example
This is a re-implementation of the range function. The module is called ezrange. One thing to remember with ARGOUT_ARRAY1 is that the dimension of the array must be passed from Python.
From Bill Spotz's article: The python user does not pass these arrays in, they simply get returned. For the case where a dimension is specified, the python user must provide that dimension as an argument.
This is useful for functions like numpy.arange(N), for which the size of the returned array is known in advance and passed to the C function.
For functions that follow array_out = function(array_in) where the size of array_out is not known in advance and depends on memory allocated in C, see the example given in Cookbook/SWIG_Memory_Deallocation.
The C source (ezrange.c and ezrange.h)
Here is the ezrange.h file:
1 void range(int *rangevec, int n);
Here is the ezrange.c file:
The interface file (ezrange.i)
Here is the ezrange.i file.
Don't forget that you will also need the numpy.i file in the same directory.
Setup file (setup.py)
This is my setup.py file:
1 #! /usr/bin/env python
2
3 # System imports
4 from distutils.core import *
5 from distutils import sysconfig
6
7 # Third-party modules - we depend on numpy for everything
8 import numpy
9
10 # Obtain the numpy include directory. This logic works across numpy versions.
11 try:
12 numpy_include = numpy.get_include()
13 except AttributeError:
14 numpy_include = numpy.get_numpy_include()
15
16 # ezrange extension module
17 _ezrange = Extension("_ezrange",
18 ["ezrange.i","ezrange.c"],
19 include_dirs = [numpy_include],
20 )
21
22 # ezrange setup
23 setup( name = "range function",
24 description = "range takes an integer and returns an n element int array where each element is equal to its index",
25 author = "Egor Zindy",
26 version = "1.0",
27 ext_modules = [_ezrange]
28 )
Compiling the module
The setup command-line is:
python setup.py build
or
python setup.py build --compiler=mingw32
depending on your environment.
Testing the module
If everything goes according to plan, there should be a _ezrange.pyd file available in the build\lib.XXX directory. You will need to copy the file in the directory where the ezrange.py file is (generated by swig), in which case, the following will work (in python):
>>> import ezrange >>> ezrange.range(10) array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
A simple INPLACE_ARRAY1 example
This example doubles the elements of the 1-D array passed to it. The operation is done in-place, which means that the array passed to the function is changed.
The C source (inplace.c and inplace.h)
Here is the inplace.h file:
1 void inplace(double *invec, int n);
Here is the inplace.c file:
The interface file (inplace.i)
Here is the inplace.i interface file:
Setup file (setup.py)
This is my setup.py file:
1 #! /usr/bin/env python
2
3 # System imports
4 from distutils.core import *
5 from distutils import sysconfig
6
7 # Third-party modules - we depend on numpy for everything
8 import numpy
9
10 # Obtain the numpy include directory. This logic works across numpy versions.
11 try:
12 numpy_include = numpy.get_include()
13 except AttributeError:
14 numpy_include = numpy.get_numpy_include()
15
16 # inplace extension module
17 _inplace = Extension("_inplace",
18 ["inplace.i","inplace.c"],
19 include_dirs = [numpy_include],
20 )
21
22 # NumyTypemapTests setup
23 setup( name = "inplace function",
24 description = "inplace takes a double array and doubles each of its elements in-place.",
25
26 author = "Egor Zindy",
27 version = "1.0",
28 ext_modules = [_inplace]
29 )
Compiling the module
The setup command-line is:
python setup.py build
or
python setup.py build --compiler=mingw32
depending on your environment.
Testing the module
If everything goes according to plan, there should be a _inplace.pyd file available in the build\lib.XXX directory. You will need to copy the file in the directory where the inplace.py file is (generated by swig), in which case, the following will work (in python):
>>> import numpy >>> import inplace >>> a = numpy.array([1,2,3],'d') >>> inplace.inplace(a) >>> a array([2., 4., 6.])
A simple ARGOUTVIEW_ARRAY1 example
Big fat multiple warnings
Please note, Bill Spotz advises against the use of argout_view arrays, unless absolutely necessary:
Argoutview arrays are for when your C code provides you with a view of its internal data and does not require any memory to be allocated by the user. This can be dangerous. There is almost no way to guarantee that the internal data from the C code will remain in existence for the entire lifetime of the NumPy array that encapsulates it. If the user destroys the object that provides the view of the data before destroying the NumPy array, then using that array my result in bad memory references or segmentation faults. Nevertheless, there are situations, working with large data sets, where you simply have no other choice.
Python does not take care of memory de-allocation, as stated here by Travis Oliphant: http://blog.enthought.com/?p=62
The tricky part, however, is memory management. How does the memory get deallocated? The suggestions have always been something similar to “make sure the memory doesn’t get deallocated before the NumPy array disappears.” This is nice advice, but not generally helpful as it basically just tells you to create a memory leak.
Memory deallocation is also difficult to handle automatically as there is no easy way to do module "finalization". There is a Py_InitModule() function, but nothing to handle deletion/destruction/finalization (this will be addressed in Python 3000 as stated in PEP3121. In my example, I use the python module atexit but there must be a better way.
Having said all that, if you have no other choice, here is an example that uses ARGOUTVIEW_ARRAY1. As usual, comments welcome!
The module declares a block of memory and a couple of functions:
- ezview.set_ones() sets all the elements (doubles) in the memory block to one and returns a numpy array that is a VIEW of the memory block.
- ezview.get_view() simply returns a view of the memory block.
- ezview.finalize() takes care of the memory deallocation (this is the weak part of this example).
The C source (ezview.c and ezview.h)
Here is the ezview.h file:
1 void set_ones(double *array, int n);
Here is the ezview.c file:
The interface file (ezview.i)
Here is the ezview.i interface file:
1 %module ezview
2
3 %{
4 #define SWIG_FILE_WITH_INIT
5 #include "ezview.h"
6
7 double *my_array = NULL;
8 int my_n = 10;
9
10 void __call_at_begining()
11 {
12 printf("__call_at_begining...\n");
13 my_array = (double *)malloc(my_n*sizeof(double));
14 }
15
16 void __call_at_end(void)
17 {
18 printf("__call_at_end...\n");
19 if (my_array != NULL)
20 free(my_array);
21 }
22 %}
23
24 %include "numpy.i"
25
26 %init %{
27 import_array();
28 __call_at_begining();
29 %}
30
31 %apply (double** ARGOUTVIEW_ARRAY1, int *DIM1) {(double** vec, int* n)}
32
33 %include "ezview.h"
34 %rename (set_ones) my_set_ones;
35
36 %inline %{
37 void finalize(void){
38 __call_at_end();
39 }
40
41 void get_view(double **vec, int* n) {
42 *vec = my_array;
43 *n = my_n;
44 }
45
46 void my_set_ones(double **vec, int* n) {
47 set_ones(my_array,my_n);
48 *vec = my_array;
49 *n = my_n;
50 }
51 %}
Don't forget that you will also need the numpy.i file in the same directory.
Setup file (setup.py)
This is my setup.py file:
1 #! /usr/bin/env python
2
3 # System imports
4 from distutils.core import *
5 from distutils import sysconfig
6
7 # Third-party modules - we depend on numpy for everything
8 import numpy
9
10 # Obtain the numpy include directory. This logic works across numpy versions.
11 try:
12 numpy_include = numpy.get_include()
13 except AttributeError:
14 numpy_include = numpy.get_numpy_include()
15
16 # view extension module
17 _ezview = Extension("_ezview",
18 ["ezview.i","ezview.c"],
19 include_dirs = [numpy_include],
20 )
21
22 # NumyTypemapTests setup
23 setup( name = "ezview module",
24 description = "ezview provides 3 functions: set_ones(), get_view() and finalize(). set_ones() and get_view() provide a view on a memory block allocated in C, finalize() takes care of the memory deallocation.",
25 author = "Egor Zindy",
26 version = "1.0",
27 ext_modules = [_ezview]
28 )
Compiling the module
The setup command-line is:
python setup.py build
or
python setup.py build --compiler=mingw32
depending on your environment.
Testing the module
If everything goes according to plan, there should be a _ezview.pyd file available in the build\lib.XXX directory. You will need to copy the file in the directory where the ezview.py file is (generated by swig), in which case, the following will work (in python):
The test code test_ezview.py follows:
1 import atexit
2 import numpy
3 print "first message is from __call_at_begining()"
4 import ezview
5
6 #There is no easy way to finalize the module (see PEP3121)
7 atexit.register(ezview.finalize)
8
9 a = ezview.set_ones()
10 print "\ncalling ezview.set_ones() - now the memory block is all ones.\nReturned array (a view on the allocated memory block) is:"
11 print a
12
13 print "\nwe're setting the array using a[:]=arange(a.shape[0])\nThis changes the content of the allocated memory block:"
14 a[:] = numpy.arange(a.shape[0])
15 print a
16
17 print "\nwe're now deleting the array - this only deletes the view,\nnot the allocated memory!"
18 del a
19
20 print "\nlet's get a new view on the allocated memory, should STILL contain [0,1,2,3...]"
21 b = ezview.get_view()
22 print b
23
24 print "\nnext message from __call_at_end() - finalize() registered via module atexit"
Launch test_ezview.py and the following will hopefully happen:
~> python test_ezview.py first message is from __call_at_begining() __call_at_begining... calling ezview.set_ones() - now the memory block is all ones. Returned array (a view on the allocated memory block) is: [ 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.] we re setting the array using a[:]=arange(a.shape[0]) This changes the content of the allocated memory block: [ 0. 1. 2. 3. 4. 5. 6. 7. 8. 9.] we re now deleting the array - this only deletes the view, not the allocated memory! let s get a new view on the allocated memory, should STILL contain [0,1,2,3...] [ 0. 1. 2. 3. 4. 5. 6. 7. 8. 9.] next message from __call_at_end() - finalize() registered via module atexit __call_at_end...
Error handling using errno and python exceptions
I have been testing this for a few months now and this is the best I've come-up with. If anyone knows of a better way, please let me know.
From the opengroup website, the lvalue errno is used by many functions to return error values. The idea is that the global variable errno is set to 0 before a function is called (in swig parlance: $action), and checked afterwards. If errno is non-zero, a python exception with a meaningful message is generated, depending on the value of errno.
The following example comprises two examples: First example uses errno when checking whether an array index is valid. Second example uses errno to notify the user of a malloc() problem.
The C source (ezerr.c and ezerr.h)
Here is the ezerr.h file:
Here is the ezerr.c file:
1 #include <stdlib.h>
2 #include <errno.h>
3
4 #include "ezerr.h"
5
6 //return the array element defined by index
7 int val(int *array, int n, int index)
8 {
9 int value=0;
10
11 if (index < 0 || index >=n)
12 {
13 errno = EPERM;
14 goto end;
15 }
16
17 value = array[index];
18
19 end:
20 return value;
21 }
22
23 //allocate (and free) a char array of size n
24 void alloc(int n)
25 {
26 char *array;
27
28 array = (char *)malloc(n*sizeof(char));
29 if (array == NULL)
30 {
31 errno = ENOMEM;
32 goto end;
33 }
34
35 //don't keep the memory allocated...
36 free(array);
37
38 end:
39 return;
40 }
The interface file (ezerr.i)
Here is the ezerr.i interface file:
1 %module ezerr
2 %{
3 #include <errno.h>
4 #include "ezerr.h"
5
6 #define SWIG_FILE_WITH_INIT
7 %}
8
9 %include "numpy.i"
10
11 %init %{
12 import_array();
13 %}
14
15 %exception
16 {
17 errno = 0;
18 $action
19
20 if (errno != 0)
21 {
22 switch(errno)
23 {
24 case EPERM:
25 PyErr_Format(PyExc_IndexError, "Index out of range");
26 break;
27 case ENOMEM:
28 PyErr_Format(PyExc_MemoryError, "Failed malloc()");
29 break;
30 default:
31 PyErr_Format(PyExc_Exception, "Unknown exception");
32 }
33 SWIG_fail;
34 }
35 }
36
37 %apply (int* IN_ARRAY1, int DIM1) {(int *array, int n)}
38
39 %include "ezerr.h"
Note the SWIG_fail, which is a macro for goto fail in case there is any other cleanup code to execute (thanks Bill!).
Don't forget that you will also need the numpy.i file in the same directory.
Setup file (setup.py)
This is my setup.py file:
1 #! /usr/bin/env python
2
3 # System imports
4 from distutils.core import *
5 from distutils import sysconfig
6
7 # Third-party modules - we depend on numpy for everything
8 import numpy
9
10 # Obtain the numpy include directory. This logic works across numpy versions.
11 try:
12 numpy_include = numpy.get_include()
13 except AttributeError:
14 numpy_include = numpy.get_numpy_include()
15
16 # err extension module
17 ezerr = Extension("_ezerr",
18 ["ezerr.i","ezerr.c"],
19 include_dirs = [numpy_include],
20
21 extra_compile_args = ["--verbose"]
22 )
23
24 # NumyTypemapTests setup
25 setup( name = "err test",
26 description = "A simple test to demonstrate the use of errno and python exceptions",
27 author = "Egor Zindy",
28 version = "1.0",
29 ext_modules = [ezerr]
30 )
Compiling the module
The setup command-line is:
python setup.py build
or
python setup.py build --compiler=mingw32
depending on your environment.
Testing the module
If everything goes according to plan, there should be a _ezerr.pyd file available in the build\lib.XXX directory. You will need to copy the file in the directory where the ezerr.py file is (generated by swig), in which case, the following will work (in python):
The test code test_err.py follows:
1 import traceback,sys
2 import numpy
3 import ezerr
4
5 print "\n--- testing ezerr.val() ---"
6 a = numpy.arange(10)
7 indexes = [5,20,-1]
8
9 for i in indexes:
10 try:
11 value = ezerr.val(a,i)
12 except:
13 print ">> failed for index=%d" % i
14 traceback.print_exc(file=sys.stdout)
15 else:
16 print "using ezerr.val() a[%d]=%d - should be %d" % (i,value,a[i])
17
18 print "\n--- testing ezerr.alloc() ---"
19 amounts = [1,-1] #1 byte, -1 byte
20
21 for n in amounts:
22 try:
23 ezerr.alloc(n)
24 except:
25 print ">> could not allocate %d bytes" % n
26 traceback.print_exc(file=sys.stdout)
27 else:
28 print "allocated (and deallocated) %d bytes" % n
Launch test_err.py and the following will hopefully happen:
~> python test_err.py
--- testing ezerr.val() ---
using ezerr.val() a[5]=5 - should be 5
>> failed for index=20
Traceback (most recent call last):
File "test_err.py", line 11, in <module>
value = ezerr.val(a,i)
IndexError: Index out of range
>> failed for index=-1
Traceback (most recent call last):
File "test_err.py", line 11, in <module>
value = ezerr.val(a,i)
IndexError: Index out of range
--- testing ezerr.alloc() ---
allocated (and deallocated) 1 bytes
>> could not allocate -1 bytes
Traceback (most recent call last):
File "test_err.py", line 23, in <module>
ezerr.alloc(n)
MemoryError: Failed malloc()
Dot product example (from Bill Spotz's article)
The last example given in Bill Spotz's artice is for a dot product function. Here is a fleshed-out version.
The C source (dot.c and dot.h)
Here is the dot.h file:
1 double dot(int len, double* vec1, double* vec2);
Here is the dot.c file:
The interface files (dot.i and numpy.i)
Here is the complete dot.i file:
1 %module dot
2
3 %{
4 #define SWIG_FILE_WITH_INIT
5 #include "dot.h"
6 %}
7
8 %include "numpy.i"
9
10 %init %{
11 import_array();
12 %}
13
14 %apply (int DIM1, double* IN_ARRAY1) {(int len1, double* vec1), (int len2, double* vec2)}
15
16
17 %include "dot.h"
18 %rename (dot) my_dot;
19
20 %inline %{
21 double my_dot(int len1, double* vec1, int len2, double* vec2) {
22 if (len1 != len2) {
23 PyErr_Format(PyExc_ValueError, "Arrays of lengths (%d,%d) given", len1, len2);
24 return 0.0;
25 }
26 return dot(len1, vec1, vec2);
27 }
28 %}
Setup file (setup.py)
This is the setup.py file:
1 #! /usr/bin/env python
2
3 # System imports
4 from distutils.core import *
5 from distutils import sysconfig
6
7 # Third-party modules - we depend on numpy for everything
8 import numpy
9
10 # Obtain the numpy include directory. This logic works across numpy versions.
11 try:
12 numpy_include = numpy.get_include()
13 except AttributeError:
14 numpy_include = numpy.get_numpy_include()
15
16 # dot extension module
17 _dot = Extension("_dot",
18 ["dot.i","dot.c"],
19 include_dirs = [numpy_include],
20 )
21
22 # dot setup
23 setup( name = "Dot product",
24 description = "Function that performs a dot product (numpy.i: a SWIG Interface File for NumPy)",
25 author = "Egor Zindy (based on the setup.py file available in the numpy tree)",
26 version = "1.0",
27 ext_modules = [_dot]
28 )
Compiling the module
The setup command-line is:
python setup.py build
or
python setup.py build --compiler=mingw32
depending on your environment.
Testing
If everything goes according to plan, there should be a _dot.pyd file available in the build\lib.XXX directory. You will need to copy the file in the directory where the dot.py file is (generated by swig), in which case, the following will work (in python):
>>> import dot >>> dot.dot([1,2,3],[1,2,3]) 14.0
Conclusion
That's all folks (for now)! As usual, comments welcome!
TODO: Code clean-up and moving the examples over to the !SciPy/!NumPy repository?
Regards, Egor