Skip to content

AGENTS.md

Latest commit

 

History

History
212 lines (171 loc) · 12.4 KB

AGENTS.md

File metadata and controls

212 lines (171 loc) · 12.4 KB
Raw

AGENTS.md

This file provides guidance to AI coding agents (e.g., Claude Code) when working with code in this repository.

What this package is

suitesparse-graphblas is a low-level Python CFFI binding around the C library SuiteSparse:GraphBLAS. It exposes the raw ffi and lib symbols and, on top of them, a functional API in the suitesparse_graphblas.api package (matrix, vector, scalar, plus binaryop, monoid, semiring, descriptor, iterator, …). matrix, vector, and scalar (also re-exported as suitesparse_graphblas.matrix etc. for backward compatibility) are the main entry point for users of this library directly — they are module-level functions operating on opaque CFFI handles, e.g. A = matrix.matrix_new(lib.GrB_BOOL, 3, 3); matrix.set_bool(A, True, 2, 2). Higher-level syntax wrappers (python-graphblas and pygraphblas) build on top of this same package for users who want a more Pythonic, OO-style interface.

The currently targeted SuiteSparse:GraphBLAS version is pinned in GB_VERSION.txt.

Building from source

Building requires a working SuiteSparse:GraphBLAS C library on the system. Point at it via GraphBLAS_ROOT (must contain include/GraphBLAS.h and lib/):

export GraphBLAS_ROOT="/path/to/graphblas"   # or `$(brew --prefix suitesparse)` on macOS
pip install -e . --no-deps                    # editable dev install

If GraphBLAS_ROOT is unset, the build falls back to:

  • C:\GraphBLAS on Windows
  • /usr/local if /usr/local/include/suitesparse exists (the path used by suitesparse.sh)
  • sys.prefix (works for conda-installed graphblas)

The CI build uses conda-forge graphblas=$(cat GB_VERSION.txt). To build SuiteSparse from source instead, run bash suitesparse.sh refs/tags/$(cat GB_VERSION.txt).0. That script also honors SUITESPARSE_FAST_BUILD / SUITESPARSE_FASTEST_BUILD env vars to disable many type specializations for much faster local builds.

Tests, lint, and other commands

Testing requires the compiled CFFI extension and the SuiteSparse:GraphBLAS C library, so tests should be run inside the Docker container. Build the image once (this compiles GraphBLAS from source and takes several minutes), then run tests against it:

# Build the test image (uses the psg stage which includes pytest)
docker build --target psg \
    --build-arg SUITESPARSE=v$(cat GB_VERSION.txt) \
    --build-arg VERSION=99.0.0.0 \
    -t psg-test .

# Run the full test suite (unit tests + all doctests)
docker run --rm -w /tmp psg-test pytest --pyargs suitesparse_graphblas --doctest-modules -v

# Run only the unit tests (no doctests)
docker run --rm -w /tmp psg-test pytest --pyargs suitesparse_graphblas.tests -v

# Run a single test file
docker run --rm -w /tmp psg-test pytest --pyargs suitesparse_graphblas.tests.test_scalar -v

# Run by test name substring
docker run --rm -w /tmp psg-test pytest --pyargs suitesparse_graphblas --doctest-modules -k test_print_jit_config -v

# Run linters/formatters (locally, no C library needed)
pre-commit run --all-files

Rebuild the Docker image after making changes — the ADD . /psg layer picks up the current working tree. The SuiteSparse compilation layer is cached so rebuilds are fast.

conftest.py calls suitesparse_graphblas.initialize() once per session — GrB_init may only be called once per process, so test_initialize.py is run as a separate process in CI:

docker run --rm -w /tmp psg-test python3 -m suitesparse_graphblas.tests.test_initialize

Coverage runs in CI use CYTHON_COVERAGE=true so the Cython utils.pyx extension is recompiled with line tracing.

Architecture: how the binding is generated and assembled

There are two layers of generated code, and understanding them is essential before touching anything related to types, defines, or the FFI surface.

1. Header generation — suitesparse_graphblas/create_headers.py

This script regenerates suitesparse_graphblas.h, suitesparse_graphblas_no_complex.h, and source.c from an upstream GraphBLAS.h. It:

  • Copies GraphBLAS.h from the install, runs the C preprocessor (using pycparser's fake_libc_include), and parses the result with pycparser.
  • Emits a cleaned-up header that cffi can cdef(), plus a complex-free variant for platforms (notably MSVC) where _Complex types don't work.
  • Manually tracks DEFINES, CHAR_DEFINES, IGNORE_DEFINES, and DEPRECATED sets. When updating to a new SuiteSparse:GraphBLAS version, these lists are the things most likely to need editing. New macros, new deprecations, or removed symbols all flow through here.
  • CI runs this script and git diff --exit-code to fail the build if the committed headers drift from upstream. Re-running it locally and committing the result is the standard fix.

2. CFFI compilation — build_graphblas_cffi.py

ffibuilder calls set_source() with source.c and cdef() with suitesparse_graphblas.h to produce the compiled extension suitesparse_graphblas._graphblas (which exposes ffi and lib). On Windows it instead emits a _graphblas.c file and runs a textual patch (float _Complex → _Fcomplex, double _Complex → _Dcomplex, -DGxB_HAVE_COMPLEX_MSVC) because cffi cannot represent MSVC's complex types — see get_extension() for the patching logic. setup.py chooses between the cffi-driven and Extension-driven paths based on build_graphblas_cffi.is_win.

setup.py also cythonizes any *.pyx under suitesparse_graphblas/ (currently utils.pyx). When Cython is unavailable, it falls back to checked-in *.c files; the build will refuse to proceed if any are missing.

3. Python layer (suitesparse_graphblas/__init__.py and friends)

The package re-exports ffi/lib from the compiled extension and adds only thin helpers:

  • initialize(blocking=False, memory_manager="numpy") — must be called exactly once before any GraphBLAS calls. The numpy memory manager routes allocation through PyDataMem_NEW/FREE (defined in utils.pyx::call_gxb_init) so buffers can be claimed zero-copy by NumPy and tracked by tracemalloc.
  • check_status(obj, info) — central error handler. Maps GrB_Info codes to exception classes in exceptions.py and pulls the human-readable message via the type-specific *_error() function, looked up by cdata cname in _error_func_lookup.
  • vararg(val) — a workaround for variadic GraphBLAS calls on osx-arm64 and ppc64le where ARM64 calling conventions force variadic args onto the stack. Prefer the non-variadic typed variants (e.g. GxB_Matrix_Option_get_INT32) when they exist.
  • libget(name) — fallback that retries a GrB_* lookup as GxB_* when SuiteSparse moves a symbol between standard and extension namespaces.
  • burble — context manager / global toggle for GxB_BURBLE diagnostic output.
  • api/matrix.py, api/vector.py, api/scalar.py (and the other api/ modules) — the functional API of the package, and the primary user-facing surface for code that uses suitesparse-graphblas directly. __init__.py re-exports matrix, vector, scalar, and iterator at the bottom of the file (to avoid a circular import) so suitesparse_graphblas.matrix keeps working. Each module follows the same convention: <module>.<module>_new(...) (e.g. matrix.matrix_new) returns an ffi.gc-managed cdata handle (GrB_Matrix*, GrB_Vector*, or GxB_Scalar*) and every other function takes that handle as its first argument and routes errors through check_status. The design is deliberately functional rather than class-based so the same handles can be passed through higher-level wrappers without object-identity friction. When adding features to this package, this is the layer where new user-facing helpers belong.
  • api/io/serialize.py, api/io/binary.py — supporting I/O helpers (compressed serialize / deserialize, binary format read/write), also importable as suitesparse_graphblas.io. api/matrix.py and api/vector.py already re-export serialize / deserialize from api/io/serialize.py so callers can reach them as matrix.serialize(A) etc.

4. utils.pyx and free-threading

The Cython module exists to (a) call GxB_init with NumPy's allocators by casting the cffi function pointer through uintptr_t into a real C function pointer, and (b) move buffers between GraphBLAS and NumPy without copying. Its module docstring is the reference for (b).

Memory that changes owner must be freed by the allocator that allocated it. GraphBLAS frees with the allocator of the buffer's arena (arena 0 is NumPy's PyDataMem_* allocator for memory_manager="numpy", libc's for "c"; GxB_arena_init adds more). NumPy frees with the array's memory handler (NEP 49), or with libc free if it has none. Don't assume any two of these agree: with NumPy ≥ 2.5, NumPy's allocator is PyMem_RawMalloc, which is mimalloc on free-threaded Python ≥ 3.15 and gains debug hooks under python -X dev. So, per arena, utils.pyx asks GraphBLAS for the arena's functions and uses NumPy's default handler if they are NumPy's, else its own handler that calls them (suitesparse_graphblas_arena<k>):

  • GraphBLAS → NumPy: claim_buffer / claim_buffer_2d attach that handler.
  • NumPy → GraphBLAS: use give_buffer (a context manager) around the GraphBLAS call that takes ownership (GxB_*_pack_*, GxB_*_load with GrB_DEFAULT + arena, ...). It hands over the array itself only if can_unclaim_buffer (owns its data, writeable, and its handler is the arena's), otherwise a copy in empty(...) memory, and it commits only if GraphBLAS set the pointer to NULL. Never hand GraphBLAS libc malloc memory or arrays of unknown provenance; api/io/binary.py::binread shows the pattern.
  • arena defaults to the global data arena. GraphBLAS itself uses the data arena of the Context engaged on the calling thread, which can't be queried, so code that must be right under any Context names the arena: serialize_* call GxB_*_serialize_arena, and binread uses the data arena of the matrix it just created. GxB_*_unpack_* and GxB_*_export_* move the object to that arena before handing out its arrays, so the arena an object was made in says nothing about them.

tests/test_memory.py checks all of this (both managers, a libc arena as an explicit, global, and Context arena, threads, binread) in a subprocess under PYTHONMALLOC=debug, which turns a mismatch into an abort on any Python. Its buffers are over 1024 bytes on purpose: NumPy caches smaller freed blocks, which hides a wrong free. conftest.py also makes NumPy warn, failing the test, whenever it frees an array that owns its data but has no memory handler; that catches a missing handler even where the allocators happen to agree.

When claiming a buffer from GxB_*_unload, use the arena in the returned handling (handling - GrB_DEFAULT), and never claim one with handling >= GxB_IS_READONLY: whoever loaded it read-only still owns it.

The file is marked freethreading_compatible=True. The package does nothing special for free-threading itself — correctness depends on SuiteSparse:GraphBLAS being thread-safe, which it is required to be.

Style and tooling

  • Black, isort, flake8 (config in .flake8, line length 100, double quotes), pyupgrade (--py311-plus), autoflake, shellcheck — all wired through pre-commit. Run pre-commit run --all-files before pushing.
  • pre-commit also lints GitHub Actions workflows (actionlint, zizmor) and scans dependencies for known vulnerabilities (pysentry). actionlint runs shellcheck on run: scripts only when shellcheck is on PATH — it is on the CI runner, so quote $(...) and "$GITHUB_ENV" even if it passes locally. Don't expand ${{ ... }} contexts inside run: scripts (zizmor's template-injection audit); pass them through env: instead.
  • pre-commit also blocks direct commits to main.
  • pytest is strict (see [tool.pytest.ini_options]): warnings are errors, markers must be registered, and xfails must fail.
  • Python ≥ 3.11. NumPy ≥ 2.0 is required at build time (CFFI extension), ≥ 1.24 at runtime.
  • Generated headers (suitesparse_graphblas.h, suitesparse_graphblas_no_complex.h, source.c) are checked in and must be regenerated via create_headers.py rather than hand-edited. CI enforces this.