Pyodide 0.21 is a major release focused on improvements to the systems for building and loading packages. We made it much easier to build and use binary wheels that are not included in our distribution. We added support for Rust packages and made major progress in simplifying support for CMake packages. We also added 34 new packages including a large number of popular packages, for example bitarray, opencv-python, shapely, and xgboost.

We updated the compiler from Emscripten v2.0.27 to Emscripten v3.1.14 which includes major improvements including many fixes to the Python core test suite contributed by us and by Christian Heimes. There were also many minor improvements and bug fixes to the Pyodide Python package.

We separated the Pyodide test system into a pytest plugin called pytest-pyodide and released it as a Python package. This may help people testing dedicated Pyodide applications.

We have also done a lot of work outside of the Pyodide repository, on Rust, Emscripten, PyO3, Numpy, and on other Python packages. We hope to continue with more community work on Python packages in the future.

For the complete list of changes see the changelog.

Building binary wheels for Pyodide

In Pyodide 0.21.0 we have added experimental support for building binary packages as wheels for Pyodide “out of tree”, similarly to how standard Python wheels are built.

Pyodide has so far used a monolithic collection of package recipes inspired by Conda to build binary extensions. Using a collection of package recipes has significant advantages, allowing us to maintain patches and test all of the packages together. However, it also has several downsides:

  1. It requires all packages to be maintained in a monorepo.

  2. It doesn’t easily allow packages to support Pyodide in their CI.

  3. There is no convenient way for someone to use a private package with a binary extension with Pyodide.

  4. There is no good way to have several different versions of the same package. For instance, this would be very helpful for using Pyodide to make interactive documentation for old versions of a package.

We can build most packages out of tree, though this feature is still a bit experimental. We will continue using the collection of package recipes for our in-tree builds. Our goal is to add Pyodide to the CI of key packages, for instance as a proof of concept we have added Pyodide to the Numpy CI in a draft PR.

For example, a simple Github Action to build a wheel for use with Pyodide could look like:

runs-on: ubuntu-latest
  steps:
  - uses: actions/checkout@v3
  - uses: actions/setup-python@v4
     with:
       python-version: 3.10.2
  - uses: mymindstorm/setup-emsdk@v11
     with:
       version: 3.1.14
  - run: pip install pyodide-build==0.21.0
  - run: pyodide build

Note that it is not necessary to check out the Pyodide repository, to use a docker image, or anything complicated like that. Testing the package is still complicated. We are actively working on making the testing similar to testing normal Python packages.

In order to limit CI time, we only run a small fragment of each package’s tests in Pyodide’s CI. However, the package’s CI can run all of its tests in Pyodide. This should help tremendously with locating and fixing subtle Pyodide-only bugs.

Improvements to package loading

We have also updated micropip to be able to load binary wheels (which must have a filename ending in -cp310-cp310-emscripten_3_1_14_wasm32.whl) and to be able to generate custom lock files. We also added the capability of specifying custom lock files in loadPyodide. Custom lock files feature may help significantly with loading time for certain applications and can also be useful for people who want to add extra packages to their repls to be loaded with loadPackagesFromImports.

Rust and CMake support

Pyodide can now build Rust packages! In fact, PyO3 packages just work with almost no extra configuration. This blog post has a breakdown of the technical issues that needed to be fixed for this. The main Rust/Python integration tools PyO3 setuptools-rust and maturin have added Pyodide to their test suites.

We also made progress with CMake support. The current status is that we know how to build most CMake packages but it requires specialized knowledge. Hopefully future work will make this simpler.

Acknowledgements

Thanks to everyone who contributed code to this release and to all users who reported issues and provided feedback. Thanks to the PyO3 developers Messense and David Hewitt, to the Rust developers and Sam Clegg who reviewed PRs to improve the Rust Emscripten target. Thanks to Samuel Colvin for adding Pyodide to the pydantic CI and providing helpful feedback. Thanks also to Christian Heimes for continuing to work on improving core Python Emscripten support.

The following people commited to Pyodide in this release:

Alexey Ignatiev, Andrey Smelter, andrzej, Antonio Cuni, Ben Jeffery, Brian Benjamin Maranville, David Lechner, dragoncoder047, echorand (Amit Saha), Filipe, Frank, Gyeongjae Choi, Hanno Rein, haoran1062, Henry Schreiner, Hood Chatham, Jason Grout, jmdyck, Jo Bovy, John Wason, josephrocca, Kyle Cutler, Lester Fan, Liumeo, lukemarsden, Mario Gersbach, Michael Droettboom, Michael Gilbert, Michael Neil, Nicholas Bollweg, pysathq, Ricardo Prins, Rob Gries, Roman Yurchak, Ryan May, Ryan Russell, stonebig, Szymswiat, Tobias Megies, Vic Kumar, Victor, Wei Ji, Will Lachance