Introspecting build steps#

When you have an issue with a particular Python extension module or other build target, there are a number of ways to figure out what the build system is doing exactly. Beyond looking at the meson.build content for the target of interest, these include:

Reading the generated build.ninja file in the build directory,
Using meson introspect to learn more about build options, dependencies and flags used for the target,
Reading <build-dir>/meson-info/*.json for details on discovered dependencies, where Meson plans to install files to, etc.

These things are all available after the configure stage of the build (i.e., meson setup) has run. It is typically more effective to look at this information, rather than running the build and reading the full build log.

The `ninja.build` file#

As an example, let’s say we are interested in scipy.linalg._decomp_update. From scipy/linalg/meson.build we learn that this extension is written in templated Cython code, and there are no special compilation flags used nor include directories beyond the numpy one. So the next step is to look at build.ninja. Open that file in an editor and search for _decomp_update. You will find this set of generic and target-specific rules that apply (note, comments in this code block are not present in build.ninja but only added in this doc section to explain what is happening):

# These rules are usually not needed to understand the problem, but can be looked up at the top of the file:
rule c_COMPILER
 command = /home/username/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc $ARGS -MD -MQ $out -MF $DEPFILE -o $out -c $in
 deps = gcc
 depfile = $DEPFILE_UNQUOTED
 description = Compiling C object $out

rule c_LINKER
 command = /home/username/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc $ARGS -o $out $in $LINK_ARGS
 description = Linking target $out

# step 1: `.pyx.in` to `.pyx` code generation with Tempita
build scipy/linalg/_decomp_update.pyx: CUSTOM_COMMAND ../scipy/linalg/_decomp_update.pyx.in | ../scipy/_build_utils/tempita.py /home/username/anaconda3/envs/scipy-dev/bin/python3.10
 COMMAND = /home/username/anaconda3/envs/scipy-dev/bin/python3.10 ../scipy/_build_utils/tempita.py ../scipy/linalg/_decomp_update.pyx.in -o scipy/linalg
 description = Generating$ scipy/linalg/_decomp_update$ with$ a$ custom$ command

# step 2: `.pyx` to `.c` compilation with Cython
build scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so.p/_decomp_update.c: CUSTOM_COMMAND scipy/linalg/_decomp_update.pyx | /home/username/code/scipy/scipy/_build_utils/cythoner.py scipy/__init__.py scipy/linalg/__init__.py scipy/linalg/cython_blas.pyx
 DESC = Generating$ 'scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so.p/_decomp_update.c'.
 COMMAND = /home/username/anaconda3/envs/scipy-dev/bin/python3.10 /home/username/code/scipy/scipy/_build_utils/cythoner.py scipy/linalg/_decomp_update.pyx scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so.p/_decomp_update.c

# step 3: use C compiler to go from `.c` to object file (`.o`)
build scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so.p/meson-generated__decomp_update.c.o: c_COMPILER scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so.p/_decomp_update.c
 DEPFILE = scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so.p/meson-generated__decomp_update.c.o.d
 DEPFILE_UNQUOTED = scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so.p/meson-generated__decomp_update.c.o.d
 ARGS = -Iscipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so.p -Iscipy/linalg -I../scipy/linalg -I/home/username/anaconda3/envs/scipy-dev/lib/python3.10/site-packages/numpy/core/include -I/home/username/anaconda3/envs/scipy-dev/include/python3.10 -fvisibility=hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c99 -O2 -g -Wno-unused-but-set-variable -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -fPIC -Wno-cpp

# step 4: generate a symbol file (uses `meson --internal symbolextractor`); you can safely ignore this step
build scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so.p/_decomp_update.cpython-310-x86_64-linux-gnu.so.symbols: SHSYM scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so
 IMPLIB = scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so

# step 5: link the `.o` file to obtain the file extension module (`.so`)
build scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so: c_LINKER scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so.p/meson-generated__decomp_update.c.o | /home/username/anaconda3/envs/scipy-dev/x86_64-conda-linux-gnu/sysroot/lib64/libm-2.12.so /home/username/anaconda3/envs/scipy-dev/x86_64-conda-linux-gnu/sysroot/usr/lib64/libm.a
 LINK_ARGS = -L/home/username/anaconda3/envs/scipy-dev/lib -Wl,--as-needed -Wl,--allow-shlib-undefined -shared -fPIC -Wl,--start-group -lm -Wl,--end-group -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/home/username/anaconda3/envs/scipy-dev/lib -Wl,-rpath-link,/home/username/anaconda3/envs/scipy-dev/lib

Using `meson introspect`#

If we want to look at _decomp_update from another perspective, we can use (for example) meson introspect --targets -i <build-dir> > targets.json to generate readable JSON. Searching that generated file for our target of interest shows:

{
    "name": "_decomp_update",
    "id": "b4ac6f0@@_decomp_update@cus",
    "type": "custom",
    "defined_in": "/home/username/code/scipy/scipy/linalg/meson.build",
    "filename": [
        "/home/username/code/scipy/build/scipy/linalg/_decomp_update.pyx"
    ],
    "build_by_default": false,
    "target_sources": [
        {
            "language": "unknown",
            "compiler": [
                "/home/username/anaconda3/envs/scipy-dev/bin/python3.10",
                "/home/username/code/scipy/scipy/_build_utils/tempita.py",
                "@INPUT@",
                "-o",
                "@OUTDIR@"
            ],
            "parameters": [],
            "sources": [
                "/home/username/code/scipy/scipy/linalg/_decomp_update.pyx.in"
            ],
            "generated_sources": []
        }
    ],
    "extra_files": [],
    "subproject": null,
    "installed": false
},
{
    "name": "_decomp_update.cpython-310-x86_64-linux-gnu",
    "id": "b4ac6f0@@_decomp_update.cpython-310-x86_64-linux-gnu@sha",
    "type": "shared module",
    "defined_in": "/home/username/code/scipy/scipy/linalg/meson.build",
    "filename": [
        "/home/username/code/scipy/build/scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so"
    ],
    "build_by_default": true,
    "target_sources": [
        {
            "language": "c",
            "compiler": [
                "/home/username/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc"
            ],
            "parameters": [
                "-I/home/username/code/scipy/build/scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so.p",
                "-I/home/username/code/scipy/build/scipy/linalg",
                "-I/home/username/code/scipy/scipy/linalg",
                "-I/home/username/anaconda3/envs/scipy-dev/lib/python3.10/site-packages/numpy/core/include",
                "-I/home/username/anaconda3/envs/scipy-dev/include/python3.10",
                "-fvisibility=hidden",
                "-fdiagnostics-color=always",
                "-D_FILE_OFFSET_BITS=64",
                "-Wall",
                "-Winvalid-pch",
                "-std=c99",
                "-O2",
                "-g",
                "-Wno-unused-but-set-variable",
                "-Wno-unused-function",
                "-Wno-conversion",
                "-Wno-misleading-indentation",
                "-fPIC",
                "-Wno-cpp"
            ],
            "sources": [],
            "generated_sources": [
                "/home/username/code/scipy/build/scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so.p/_decomp_update.c"
            ]
        }
    ],
    "extra_files": [],
    "subproject": null,
    "installed": true,
    "install_filename": [
        "/home/username/code/scipy/build-install/lib/python3.10/site-packages/scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so"
    ]
},

This tells us a lot of things, like which include directories will be used, where the Cython-generated C code can be found, and what compile flags are used. meson introspect --help has good documentation on the full range of capabilities and how to use them.

`meson-info` JSON files#

There are a number of different JSON files in <build-dir>/meson-info/. These have descriptive names, hinting at their content. For example, where the final _decomp_update extension gets installed to is described in intro-install_plan.json (note, these files aren’t prety-printed, running them through a JSON formatter helps):

"/home/username/code/scipy/build/scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so":{
   "destination":"{py_platlib}/scipy/linalg/_decomp_update.cpython-310-x86_64-linux-gnu.so",
   "tag":"runtime"
},

We may also be interested in knowing what dependencies were detected by the configure stage of the build. So we look in intro-dependencies.json:

[
   {
      "name":"python",
      "version":"3.10",
      "compile_args":[
         "-I/home/username/anaconda3/envs/scipy-dev/include/python3.10"
      ],
      "link_args":[

      ]
   },
   {
      "name":"openblas",
      "version":"0.3.20",
      "compile_args":[
         "-I/home/username/anaconda3/envs/scipy-dev/include"
      ],
      "link_args":[
         "/home/username/anaconda3/envs/scipy-dev/lib/libopenblas.so"
      ]
   },
   {
      "name":"threads",
      "version":"unknown",
      "compile_args":[
         "-pthread"
      ],
      "link_args":[
         "-pthread"
      ]
   }
]

This tells us that we have three dependencies that were found. Note: numpy and a few other build-time dependencies are missing here because we do not (yet) search for those with the builtin dependency() Meson command.

Introspecting build steps#

The ninja.build file#

Using meson introspect#

meson-info JSON files#

The `ninja.build` file#

Using `meson introspect`#

`meson-info` JSON files#