Development environment quickstart guide (Ubuntu)¶
This quickstart guide will cover:
setting up and maintaining a development environment, including installing compilers and SciPy build dependencies;
creating a personal fork of the SciPy repository on GitHub;
using git to manage a local repository with development branches;
performing an in-place build of SciPy; and
creating a virtual environment that adds this development version of SciPy to the Python path
in Ubuntu. (Tested on 16.04, 18.04, and 20.04). Users running Windows can follow these instructions after setting up Windows Subsystem for Linux or an Amazon EC2 instance with Ubuntu 20.04. However, the instructions for setting up a development environment with Docker may be more reliable.
This guide does not present the only way to set up a development environment; there are many valid choices of Python distribution, C/Fortran compiler, and installation options. The steps here can often be adapted for other choices, but we cannot provide documentation tailored for them.
This guide assumes that you are starting without an existing Python 3 installation. If you already have Python 3, you might want to uninstall it first to avoid ambiguity over which Python version is being used at the command line.
Download, install, and test the latest release of the Anaconda Distribution of Python. In addition to the latest version of Python 3, the Anaconda Distribution includes dozens of the most popular Python packages for scientific computing, the
condapackage manager, and tools for managing virtual environments.
If you’re installing using the terminal, be sure to follow the “Next Steps” listed after the installer finishes. You might also need to restart your terminal window or enter
source ~/.bashrcfor all the changes to take effect.
(Optional) In a terminal window, enter
This shows a list of all the Python packages that came with the Anaconda Distribution of Python. Note the latest released version of SciPy is among them; this is not the development version you are going to build and will be able to modify.
Ideally, we’d like to have both versions, and we’d like to be able to switch between the two as needed. Virtual environments can do just that. With a few keystrokes in the terminal or even the click of an icon, we can enable or disable our development version. Let’s set that up.
condais not a recognized command, try restarting your terminal. If it is still not recognized, please see “Should I add Anaconda to the macOS or Linux PATH?” in the Anaconda FAQ.
conda config --env --add channels conda-forgeto tell Anaconda the source we want for our packages. Then enter
conda create --name scipydev python=3.8 numpy pybind11 cython pythran pytest gfortran_linux-64 gxx_linux-64 sphinx pydata-sphinx-theme sphinx-panels matplotlib mypy git.
condato create a virtual environment named
scipydev(or another name that you prefer) with several packages.
numpy pybind11 cython pythranare four packages that SciPy depends on.
gfortran_linux-64 gxx_linux-64are compilers used to build SciPy’s Fortran, C, and C++ source code.
pytestis needed for running the test suite.
matplotlibare required to render the SciPy documentation.
mypyis a static type checker for Python. Consider using it.
gitis a version control system used to download and manage the SciPy source code.
Note that we’re installing SciPy’s build dependencies and some other software, but not SciPy itself.
conda createan empty virtual environment first, then
conda installthe packages, but creating the virtual environment with all the packages you need is preferable to installing packages individually because it makes it easier for
condato solve the package dependencies optimally.
You’re still in the base environment. Activate your new virtual environment by entering
conda activate scipydev.
If you’re working with an old version of
conda, you might need to type
source activate scipydevinstead (see here). Note that you’ll need to have this virtual environment active whenever you want to work with the development version of SciPy.
Browse to your fork. Your fork will have a URL like https://github.com/mdhaber/scipy, except with your GitHub username in place of “mdhaber”.
Click the big, green “Clone or download” button, and copy the “.git” URL to the clipboard. The URL will be the same as your fork’s URL, except it will end in “.git”.
Create a folder for the SciPy source code in a convenient place on your computer. Navigate to it in the terminal.
Enter the command
git clonefollowed by your fork’s .git URL. Note that this creates in the terminal’s working directory a
scipyfolder containing the SciPy source code.
In the terminal, navigate into the
scipyroot directory (e.g.
Initialize git submodules:
git submodule update --init.
Do an in-place build: enter
python3 setup.py build_ext --inplace.
This will compile the C, C++, and Fortran code that comes with SciPy. We installed
setup.pyis a script in the root directory of SciPy, which is why you have to be in the SciPy root directory to call it.
build_extis a command defined in
--inplaceis an option we’ll use to ensure that the compiling happens in the SciPy directory you already have rather than the default location for Python packages. By building in-place, you avoid having to re-build SciPy before you can test changes to the Python code.
Test the build: enter
python3 runtests.py -v.
runtests.pyis another script in the SciPy root directory. It runs a suite of tests that make sure SciPy is working as it should, and
--verboseoption to show all the test output. If the tests are successful, you now have a working development build of SciPy! You could stop here, but you would only be able to use this development build when the Python working directory is the SciPy root directory.
conda develop ., where
.refers to the present directory.
This will allow us to
importthe development version of SciPy in Python regardless of Python’s working directory.
In a new terminal window, test your setup. If you activate your virtual environment (e.g.
conda activate scipydev) and run Python code that imports from SciPy, any changes you make to the SciPy code should be reflected when the code runs. After deactivating the virtual environment (
conda deactivate), Python imports from the version of SciPy installed by Anaconda. You can also check which version of SciPy you’re using by executing in Python:
import scipy print(scipy.__version__)
If you have successfully imported a development version of SciPy, the word
devwill appear in the output, e.g.: