O slideshow foi denunciado.
Seu SlideShare está sendo baixado. ×

PyCon2022 - Building Python Extensions

Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Próximos SlideShares
What's new in Python 3.11
What's new in Python 3.11
Carregando em…3
×

Confira estes a seguir

1 de 52 Anúncio

PyCon2022 - Building Python Extensions

Baixar para ler offline

Talk at PyCon2022 over building binary packages for Python. Covers an overview and an in-depth look into pybind11 for binding, scikit-build for creating the build, and build & cibuildwheel for making the binaries that can be distributed on PyPI.

Talk at PyCon2022 over building binary packages for Python. Covers an overview and an in-depth look into pybind11 for binding, scikit-build for creating the build, and build & cibuildwheel for making the binaries that can be distributed on PyPI.

Anúncio
Anúncio

Mais Conteúdo rRelacionado

Mais de Henry Schreiner (20)

Mais recentes (20)

Anúncio

PyCon2022 - Building Python Extensions

  1. 1. Building Binary Packages Henry Schreiner April 29, 2022
  2. 2. Is Python fast? 2 Python: 4 minutes Python -> NumPy, SciPy High level language + libraries: Easy exploration of di ff erent algorithms Compiled! From The counter-intuitive rise of Python in scienti fi c computing: Problem: projection of ~1B cells https://cerfacs.fr/coop/fortran-vs-python Fortran: 6 hours and 30 minutes
  3. 3. Python has great performance 3 As long as you have a library that has your desired algorithm But what do you do if you don’t?
  4. 4. Solution 0: dump Python 4 Other languages can o ff er native performance (C, Rust, etc) But Python is easy to learn, fast to write, and has a massive ecosystem We can split code in to “driver” code that takes virtually no time to execute and “performance-critical” code that takes most of the execution time Python Not Python
  5. 5. Solution 1: Numba 5 Pros As fast or faster than any other possible solution Will target the exact hardware used Supports parallelization, GPUs, and more Can be ahead-of-time compiled Cons Takes time to JIT (though pretty fast) (Somewhat) heavy dependency Expansive but still limited feature set - numeric heavy Slow to support new Python, NumPy, etc. JIT compiler for Python functions using LLVM(Lite) @numba.vectorize def f(x): return x*2 @numba.jit def f(x): return x*2
  6. 6. Solution 2: Make Python Faster 6 This is in not the original goal of Python, but recently is actually being worked on. PyPy is a JIT version of Python Pyjion is a JIT for regular Python 3.10+ Facebook has Cinder, a fast fork of CPython 3.8 Dropbox has Pyston, a fast version of Python And CPython itself is getting faster now! But these projects generally don’t make heavy numeric work faster. Why? Python was designed to support the solution we’ll see on the next page. NumPy, etc
  7. 7. Solution 3: (Pre)compile 7 This is in the original design of Python - it’s how CPython works Write code using CPython API Compile extension Build wheels Write compiled lib with interface Setuptools Scikit-build cibuildwheel pybind11 Cython SWIG ctypes cf f multibuild This is why faster Python doesn’t make scienti fi c codes much faster - they are already compiled! mesonpy maturin-action
  8. 8. Python What is a binary extension? 8 User code Python module Compiled + Python User code Binary extension Can be any language: C, C++, Rust, Cython, MyPyC, … Easy: Lots of examples, similar to user code Portable: Usually works anywhere High level: can express complex relationships easily Complex: limited examples, multilingual Speci fi c: Compiled per architecture / Python version Low level: can achieve high performance Calls Calls
  9. 9. Pure / Binary wheels 9 mypy-0.950-py3-none-any.whl Pure wheel. Any impl Any ABI Any platform Should not contain compiled extensions. You can force wheels with
 --only-binary=:all: / PIP_ONLY_BINARY=:all: mypy-0.950.tar.gz SDist. Not a wheel. Contains raw source code. Requires the build backend (setup.py, pyproject.toml, etc) mypy-0.950-cp310-cp310-win_amd64.whl Compiled wheel. CPython 3.10 CPython 3.10 Windows 64-bit Pip selects the most speci fi c wheel This is the fastest one!
  10. 10. But why compile? 10 Speed High level Python uses low level compiled libraries for performance! NumPy Pandas MyPy (optional) PyTorch Tornado pydantic uvloop twisted websockets pyyaml markupsafe psutil matplotlib TensorFlow Code reuse There are lots of libraries out there in other languages. No need to rewrite! Check for yourself with pipx run pypi-command-line wheels <package>
  11. 11. Disclamers 11 This will be biased. I’ll often focus on stu ff I am part of. But I also liked these things enough to join them. And I know those things best. We will look at lots of “good practices”, maybe “best”, but certainly not “only”.
  12. 12. But it’s hard? 12 Let’s divide the problem into three stages: Binding / Coding Generic ctypes cf fi C++ Boost.Python pybind11 nanobind Python MyPyC Numba (AOC mode) Other Cython (custom lang) SWIG (multi-lang) Build System Setuptools + custom code Scikit-build MesonPy Enscons Hatch-mypyc No native support in fl it, hatchling, poetry, etc, but some have plugins / custom code examples. Wheel building cibuildwheel multibuild maturin-action (Rust) Intentional omissions! Numba (JIT Python) CPPYY (JIT C++) PyPy (JIT Python) Pyjion (JIT Python) We will select one tool for each job for this talk - and sometimes mention some of the others. The goal is not to show you the “right” way, but how simple it can/should be, and what you should not settle for.
  13. 13. Bindings / Coding 13 Access to (maybe existing) compiled code C, C++, Rust, Go, etc. From-scratch fast code May prefer something that looks like Python ✅ Can reuse existing work ✅ Can use across languages ✅ Can leverage strong language support ❌ Must know at least two languages ✅ Less to learn (but not general) ✅ Can make extension optional
  14. 14. A fork in the road 14 (not a pun on UNIX forks) A fork in Python extension C interface library float square(float x) { return x*x; } from ctypes import cdll, c_float lib = cdll.LoadLibrary('./simple.so') lib.square.argtypes = (c_float,) lib.square.restype = c_float lib.square(2.0) static PyObject* square_wrapper(PyObject* self, PyObject* args) { float input, result; if (!PyArg_ParseTuple(args, "f", &input)) { return NULL; } result = square(input); return PyFloat_FromDouble(result); } static PyMethodDef pysimple_methods[] = { { "square", square_wrapper, METH_VARARGS, "Square function" }, { NULL, NULL, 0, NULL } }; static struct PyModuleDef pysimple_module = { PyModuleDef_HEAD_INIT, "pysimple", NULL, -1, pysimple_methods }; PyMODINIT_FUNC PyInit_pysimple(void) { return PyModule_Create(&pysimple_module); }
  15. 15. Accessing shared libraries 15 CTypes CFFI Builtin Simple (best wrapped) No protection against mistakes Third-party library Reads C headers Good option for PyPy
  16. 16. Not just a JIT 16 Numba Numba supports AOT (ahead of time) compilation, too! Limited support, anyway. from numba.pycc import CC cc = CC('my_module') @cc.export('square', 'f8(f8)') def square(a): return a ** 2 from distutils.core import setup from source_module import cc setup(..., ext_modules=[cc.distutils_extension()]) import my_module my_module.square(1.414)
  17. 17. Fast Python 17 MyPyC Piggy-backs on modern Python typing - used to make MyPy 5x faster! import time def fib(n: int) -> int: if n <= 1: return n else: return fib(n - 2) + fib(n - 1) t0 = time.time() fib(32) print(time.time() - t0) python fib.py mypyc fib.py python -c "import fib" # 10x faster! For comparison, Numba JIT is 35x faster
  18. 18. Fast not-quite-Python 18 Cython # cython: language_level=3 import time import cython @cython.ccall def fib(n: cython.int) -> cython.int: if n <= 1: return n else: return fib(n - 2) + fib(n - 1) t0 = time.time() fib(32) print(time.time() - t0) Python-like language, transpiles to C/C++ ✅ Fast (arrays too) (with the proper directives) ✅ Can also bind C & C++ (verbose) ✴ Many ways to do things ❌ May need oldest-supported-numpy when building pipx run --spec cython cythonize fib.py clang $(python3-config --cflags --ldflags) -shared -undefined dynamic_lookup fib.c -o fib$(python3-config —extension-suffix) # macOS python -c "import fib" # 9x faster
  19. 19. Binding: 19 Header-only pure C++11 CPython/PyPy interface Trivial to add to a project No special build requirements No dependencies No precompile phase Not a new language Think of it like the missing C++ API for CPython Designed for one purpose!
  20. 20. Example of usage 20 #include <pybind11/pybind11.h> int add(int i, int j) { return i + j; } PYBIND11_MODULE(example, m) { m.def("add", &add); } Standard include Normal C++ to bind Create a Python module Signature statically inferred Docs and parameter names optional g++ -shared -fPIC example.cpp $(pipx run pybind11 --includes) -o example$(python3-config --extension-suffix) Complete, working, no-install example (linux version)!
  21. 21. Many great features 21 Simple enough for tiny projects 659K code mentions of pybind11 on GitHub Powerful enough for huge projects SciPy, PyTorch, dozens of Google projects Small binaries prioritized Perfect for WebAssembly with Pyodide Powerful object lifetime controls py::keep_alive, std::shared_ptr, and more NumPy support without NumPy headers No need to lock minimum NumPy at build Supports interop in both directions Can even be used to embed Python in C++ Most STL containers and features supported Including C++17 additions, like std::variant (or boost) Vectorize methods or functions py::vectorize, even on a lambda function Trampoline classes and multiple inheritance Complex C++ is supported Complete control of the Python representation Special methods, inheritance, pickling, and more Bu ff er protocol and array classes Includes Eigen support too Cross-extension ABI One extension can return a type wrapped in another one
  22. 22. Nanobind 22 C++17+ & Python 3.8+ only Similar API to pybind11 Intentionally more limited than pybind11 Focus on small, e ffi cient bindings Some ideas can be backported to pybind11 https://github.com/wjakob/nanobind
  23. 23. Example project 23 Let’s work through a complete example! Every line of code needed. Period. All platforms. Everything. Let’s bind a tiny bit of CLI11 (argument parser library) from cli11 import App app = App("hello") app.add_flag("--this") app.parse(["--this"]) assert app["--this"] == 1 # Should work print(app) app["--that"] # -> KeyError
  24. 24. src/cli11.cpp 24 #include <pybind11/pybind11.h> #include <pybind11/stl.h> #include <CLI/CLI.hpp> namespace py = pybind11; using namespace pybind11::literals; PYBIND11_MODULE(cli11, m) { py::register_exception<CLI::OptionNotFound>(m, "OptionNotFound", PyExc_KeyError); py::class_<CLI::App>(m, "App") .def(py::init<std::string, std::string>(), "app_description"_a = "", "app_name"_a = "") .def("add_flag", [](CLI::App &self, std::string name) { self.add_flag(name); }) .def("__getitem__", py::overload_cast<std::string>(&CLI::App::count, py::const_)) .def("parse", [](CLI::App& self, std::vector<const std::string&> vals){self.parse(vals);}) .def("__str__", [](const CLI::App &app) { return app.help(); }) ; }
  25. 25. src/cli11/__init__.py 25 from .cli11 import App __version__ = "1.2.3"
  26. 26. tests/test_works.py 26 import pytest from cli11 import App def test_simple(): app = App("hello") app.add_flag("--this") with pytest.raises(KeyError): app["--that"] assert app["--this"] == 0 app.parse(["--this"]) assert app["--this"] == 1 assert "--this" in str(app)
  27. 27. Build systems 27 There are some great pure Python build systems today with PEP 621! But your choices for building binaries is limited! Setuptools/distutils No C++ std support No multithreaded builds No partial builds No compiler features (etc) Native Cython support numpy.distutils mypyc.build numba.pycc.CC().distutils_extension pybind11.setup_helpers setuptools_rust setuptools_golang Extensions Enscons Early PEP 517 adopter (Uses distutls/setuptools internally) MesonPy New PEP 621 meson adaptor Maturin Rust PEP 621 builder From scratch Scikit-build Currently a wrapper around setup Wrapper Planned move in this direction!
  28. 28. Pybind11 and setuptools 28 setup.py Setuptools Helpers can be used for easy setuptools support Proper C++ fl ags & pybind11 headers from pybind11.setup_helpers import Pybind11Extension, build_ext module = Pybind11Extension( "python_example", ["src/main.cpp"], cxx_std=11, ) setup( ..., ext_modules=ext_modules, cmdclass={"build_ext": build_ext}, # Optional! ) Optional parallel compiler utility included
  29. 29. Scikit-build 29 Scikit-build is a CMake-setuptools adaptor from KitWare, the makers of CMake First introduced as PyCMake at SciPy 2014 and renamed in 2016 Includes CMake for Python and ninja for Python Pybind11 has a scikit-build example! pybind/cmake_example is one of the most popular examples of combining CMake and Python on GitHub Updated now with lots of fi xes! But this duplicates the code for everyone Adding Apple Silicon support will now have to be done on every project that copied the example, etc. Two new maintainers recently joined the project One from pybind11/cibuildwheel/build (me), one from cibuildwheel/manylinux (Matthieu Darbois)! cmake package for Python manylinux archs • musllinux • Apple Silicon • cibuildwheel • nox Revamped ninja for Python too OS’s and archs • cibuildwheel • nox If you need a refresher on CMake, I wrote a book for that: https://cliutils.gitlab.io/modern-cmake/
  30. 30. Scikit-build plans 30 Develop scikit-build-core PEP 517 builder, setuptools/distutils free Compatibility layer for scikit-build Limited public API helps Proper setuptools extension And Hatch, Poetry, etc. Generalize, perhaps? PEP 621 direct build Best for many cases? Add extension discovery mechanism Easy integration with pybind11, other Python packages! Possible support in CMake itself # pyproject.toml 
 requires = ["pybind11", …] 
 
 # CMakeLists.txt 
 find_package(pybind11 CONFIG REQUIRED) https://iscinumpy.dev/post/scikit-build-proposal/
  31. 31. CMakeLists.txt 31 cmake_minimum_required(VERSION 3.14...3.22) project(cli11 VERSION "0.1") include(FetchContent) FetchContent_Declare( pybind11 URL https://github.com/pybind/pybind11/archive/refs/tags/v2.9.2.tar.gz URL_HASH SHA256=6bd528c4dbe2276635dc787b6b1f2e5316cf6b49ee3e150264e455a0d68d19c1) FetchContent_MakeAvailable(pybind11) FetchContent_Declare( cli11 URL https://github.com/CLIUtils/CLI11/archive/refs/tags/v2.2.0.tar.gz URL_HASH SHA256=d60440dc4d43255f872d174e416705f56ba40589f6eb07727f76376fb8378fd6) FetchContent_MakeAvailable(cli11) pybind11_add_module(cli11 MODULE src/cli11.cpp) target_link_libraries(cli11 PRIVATE CLI11::CLI11) target_compile_features(cli11 PUBLIC cxx_std_14) install(TARGETS cli11 DESTINATION .)
  32. 32. setup.py 32 from skbuild import setup setup( name="cli11", version="1.2.3", description="a minimal example package (with pybind11)", author="Henry Schreiner", license="MIT", packages=["cli11"], package_dir={"": "src"}, cmake_install_dir="src/cli11", python_requires=">=3.7", extras_require={"test": ["pytest"]}, )
  33. 33. pyproject.toml 33 [build-system] requires = ["setuptools", "scikit-build>=0.14", "cmake", "ninja"] build-backend = "setuptools.build_meta" # More later!
  34. 34. Redistributables 34 PyPI Wheels for all common platforms Conda-forge Mostly automated, just propose a recipe manylinux/musllinux images Linux Controlled docker images MacOS Target version important Windows Easiest platform for Python python.org Python required Any Python will do (NuGet, etc) Arm / Universal2 cross compile Multiple architectures 32-bit still important, fl edgling ARM Auditwheel Delocate Delvewheel (newish)
  35. 35. cibuildwheel 🎡 35 Supports all major CI providers GitHub Action provided too! Can run locally A ffi liated (shared maintainer) with manylinux Close collaboration with PyPy devs Joined the PyPA in 2021 Used by matplotlib, mypy, scikit-learn, and more Supports: Targeting macOS 10.9+ Apple Silicon cross-compiling 3.8+ All variants of manylinux (including emulation) musllinux PyPy 3.7-3.9 Repairing and testing wheels Reproducible pinned defaults (can unpin) New in cibuildwheel 2 Python 2 & 3.5 removed, 3.10 added Pre-release Python support pyproject.toml support Optional pypa/build support All pybind examples include cibuildwheel! New in cibuildwheel 2.1-2.4 Local Windows & MacOS runs TOML overrides array manylinux2014 default musllinux Environment variable passthrough Experimental Windows ARM New in 2.5, released today! Stable ABI support Build from SDist tomllib on Python 3.11 (host)
  36. 36. cibuildwheel tips 36 Build wheels locally pipx run cibuildwheel --platform linux version: 2 updates: - package-ecosystem: "github-actions" directory: "/" schedule: interval: "daily" ignore: - dependency-name: "actions/*" update-types: - version-update:semver-minor - version-update:semver-patch Keep the GitHub Action up-to-date! .github/dependabot.yml
  37. 37. pyproject.toml 37 [build-system] requires = ["setuptools", "scikit-build>=0.14", "cmake", "ninja"] build-backend = "setuptools.build_meta" [tool.cibuildwheel] test-extras = "test" test-command = "pytest {project}/tests" [tool.cibuildwheel.macos] archs = ["auto", "universal2"] test-skip = ["*universal2:arm64"] [tool.pytest.ini_options] minversion = "6.0" testpaths = ["tests"]
  38. 38. .github/workflows/cd.yml (1) 38 on: workflow_dispatch: release: types: - published jobs: make_sdist: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - run: pipx run build --sdist - uses: actions/upload-artifact@v3 with: path: dist/*.tar.gz
  39. 39. .github/workflows/cd.yml (2) 39 build_wheels: name: Wheel on ${{ matrix.os }} runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: os: [ubuntu-20.04, windows-2022, macos-11] steps: - uses: actions/checkout@v3 with: fetch-depth: 0 submodules: true - uses: pypa/cibuildwheel@v2.4.0 - uses: actions/upload-artifact@v3 with: path: wheelhouse/*.whl
  40. 40. .github/workflows/cd.yml (3) 40 upload_all: needs: [build_wheels, make_sdist] runs-on: ubuntu-latest if: github.event_name == 'release' && github.event.action == 'published' steps: - uses: actions/download-artifact@v3 with: name: artifact path: dist - uses: pypa/gh-action-pypi-publish@v1.5.0 with: password: ${{ secrets.pypi_password }} repository_url: https://test.pypi.org/legacy/
  41. 41. Produced fi les: 41 cli11-1.2.3-cp310-cp310-macosx_10_9_universal2.whl cli11-1.2.3-cp38-cp38-win_amd64.whl cli11-1.2.3-cp310-cp310-macosx_10_9_x86_64.whl cli11-1.2.3-cp39-cp39-macosx_10_9_universal2.whl cli11-1.2.3-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl cli11-1.2.3-cp39-cp39-macosx_10_9_x86_64.whl cli11-1.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl cli11-1.2.3-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl cli11-1.2.3-cp310-cp310-musllinux_1_1_i686.whl cli11-1.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl cli11-1.2.3-cp310-cp310-musllinux_1_1_x86_64.whl cli11-1.2.3-cp39-cp39-musllinux_1_1_i686.whl cli11-1.2.3-cp310-cp310-win32.whl cli11-1.2.3-cp39-cp39-musllinux_1_1_x86_64.whl cli11-1.2.3-cp310-cp310-win_amd64.whl cli11-1.2.3-cp39-cp39-win32.whl cli11-1.2.3-cp37-cp37m-macosx_10_9_x86_64.whl cli11-1.2.3-cp39-cp39-win_amd64.whl cli11-1.2.3-cp37-cp37m-manylinux_2_17_i686.manylinux2014_i686.whl cli11-1.2.3-pp37-pypy37_pp73-macosx_10_9_x86_64.whl cli11-1.2.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl cli11-1.2.3-pp37-pypy37_pp73-manylinux_2_17_i686.manylinux2014_i686.whl cli11-1.2.3-cp37-cp37m-musllinux_1_1_i686.whl cli11-1.2.3-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl cli11-1.2.3-cp37-cp37m-musllinux_1_1_x86_64.whl cli11-1.2.3-pp37-pypy37_pp73-win_amd64.whl cli11-1.2.3-cp37-cp37m-win32.whl cli11-1.2.3-pp38-pypy38_pp73-macosx_10_9_x86_64.whl cli11-1.2.3-cp37-cp37m-win_amd64.whl cli11-1.2.3-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl cli11-1.2.3-cp38-cp38-macosx_10_9_universal2.whl cli11-1.2.3-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl cli11-1.2.3-cp38-cp38-macosx_10_9_x86_64.whl cli11-1.2.3-pp38-pypy38_pp73-win_amd64.whl cli11-1.2.3-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl cli11-1.2.3-pp39-pypy39_pp73-macosx_10_9_x86_64.whl cli11-1.2.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl cli11-1.2.3-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl cli11-1.2.3-cp38-cp38-musllinux_1_1_i686.whl cli11-1.2.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl cli11-1.2.3-cp38-cp38-musllinux_1_1_x86_64.whl cli11-1.2.3-pp39-pypy39_pp73-win_amd64.whl cli11-1.2.3-cp38-cp38-win32.whl cli11-1.2.3.tar.gz
  42. 42. Think outside the box 42 clang-format-wheel Scikit-Build Runs LLVM’s CMake build cibuildwheel Builds python-independent binary wheels 1-2 MB binaries on PyPI No “binding”, only entrypoint! - repo: https://github.com/pre-commit/mirrors-clang-format rev: "v14.0.1" hooks: - id: clang-format types_or: [c++, c, cuda] pipx run clang-format Use with pre-commit, even on pre-commit.ci!
  43. 43. NEW Quickstart: scikit-hep/cookie 43 You can make a project following the Scikit-HEP developer guidelines quickly with cookiecutter pipx run cookiecutter gh:scikit-hep/cookie Choose pybind11 or skbuild from the eleven backend choices! Generation tested with Nox in GitHub Actions Pyodide powered in-browser repo-review tool See https://scikit-hep.org/developer & https://github.com/scikit-hep/cookie (Not HEP speci fi c, except for the defaults) Also maturin (Rust)!
  44. 44. Key Takeaways 44 A great place for modern packaging advice: https://scikit-hep.org/developer My blog with useful links at the top: https://iscinumpy.dev Great examples: https://github.com/pybind/python_example https://github.com/pybind/scikit_build_example https://github.com/pybind/cmake_example https://github.com/scikit-build/scikit-build-sample-projects Example source code: https://github.com/henryiii/pybind11_skbuild_cibuildwheel_example This work was partially supported by the National Science Foundation under Cooperative Agreement OAC-1836650.
  45. 45. My Projects 45 Plumbum • POVM • PyTest GHA annotate-failures https://iscinumpy.dev https://scikit-hep.org https://iris-hep.org C++ & Python Building Python Packages Scikit-HEP: Other Other C++ Scikit-HEP: Histograms pybind11 (python_example, cmake_example, scikit_build_example) • Conda-Forge ROOT cibuildwheel • build • scikit-build (cmake, ninja, sample-projects) • Scikit-HEP/cookie boost-histogram • Hist • UHI • uproot-browser Vector • Particle • DecayLanguage • repo-review Other Python Jekyll-Indico Other Ruby CLI11 • GooFit Modern CMake • CMake Workshop Computational Physics Class Python CPU, GPU, Compiled minicourses Level Up Your Python My books and workshops
  46. 46. Backup / extras
  47. 47. A few features I’ve helped with 47 py::kw_only and py::pos_only Support Python 3 keyword only arguments Support Python 3.8 position only arguments All from any version of Python (C API) py::prepend Add to the beginning of the overload chain py::type Access and manipulate the type https://iscinumpy.gitlab.io/post/pybind11-2-6-0/ Checks can be run from nox Easier for new developers to contribute Large clang-tidy cleanups Readability, better performance, modernization CMake Integration with standard CMake features CMake 3.4+ required Support for newer features (3.18.2+ best) FindPython optionally supported CUDA as a language supported and tested Python discovery can be deactivated Portable con fi g fi le (now included in PyPI package) Helper functions added Check importable libraries easily New modular target design Follows best practices given in https://cliutils.gitlab.io/modern-cmake
  48. 48. New CI 48 Massive rewrite of the CI in GHA, with 60+ jobs covering far more than ever before Special thanks to Google for funding extra parallel jobs! Jobs: Windows, macOS and Linux, Python 2.7-3.9 (3.10 now) & PyPI GCC (4.8 and several newer), Clang (8 versions), MSVC 2015-2019 ICC, NVCC (CUDA 11), and NVHPC (PGI) 20.9 MinGW CentOS 7 and 8 C++: 11, 14, 17, and 20 Debug build Valgrind build Clang-tidy build CMake con fi gure check, 3.4, 3.7, 3.8, and 3.18 Packaging tests verifying every fi le in the wheel/SDist Newly supported compilers! Discovered and fi xed bug in CPython 3.9.0
  49. 49. A family of projects 49 pybind/python_example: A setuptools project using setuptools helpers and binary wheel builds, conda recipe, and more. pybind/scikit_build_example: A CMake project using scikit-build (new for 2.6). pybind/cmake_example: A CMake project using manual setuptools integration. Major cleanup for 2.6, now includes Apple Silicon support and more. pybind/pybind11_mkdoc: A documentation parser using LLVM. Support for conda, pip, and cibuildwheel work fl ows on GitHub Actions Support for Apple Silicon cross-compile Support for PyPy wheels Dependabot GHA updates Template repositories now And several more that I’ve not helped with
  50. 50. Snippets from boost-histogram 50 Exporting C++ addition to Python .def(py::self += py::self) Custom equality, with cast .def("__eq__", [](const histogram_t& self, const py::object& other) { try { return self == py::cast<histogram_t>(other); } catch(const py::cast_error&) { return false; } } )
  51. 51. Snippets from boost-histogram 51 Static properties Custom repr .def_property_readonly_static( "_storage_type", [](py::object) { return py::type::of<typename histogram_t::storage_type>(); } ) .def("__repr__", [](py::object self) { const A& item = py::cast<const A&>(self); std::ostringstream out; out << x; return py::str("{0.__class__.__name__}({1})").format(self, out.str()); } )
  52. 52. Snippets from boost-histogram 52 Static constructor with automatic vectorization (ufuct-like) Direct access to structured memory in NumPy .def_static("_make", py::vectorize([](const double& a, const double& b) { return weighted_sum(a, b); })) PYBIND11_NUMPY_DTYPE(weighted_mean, sum_of_weights, sum_of_weights_squared, value, _sum_of_weighted_deltas_squared);

×