Pyodide 0.23.0 release

Pyodide 0.23 is a major release focused on updating CPython version to 3.11. It also brings several new features, enhancements, and bug fixes. This release includes updates to the Python standard library, deployment and build system enhancements, and new CLI commands. We also added several new packages to Pyodide, including fastparquet, pynacl and binary wheels for mypy.

Note that there are several API deprecations; please refer to the changelog for details.

Updating to Python 3.11.2

Pyodide 0.23 uses CPython 3.11.2 which in addition to many feature and enhancements is also the first version to officially support the WebAssembly as a PEP11 Tier 3 platform.

For year Pyodide has been distributing a custom build of CPython with many patches. Now that we can rely on a mostly unmodified CPython, it has become significantly easier to update CPython versions which is very positive for the project’s sustainability. We are very grateful for the CPython maintainers ¹ who made this possible.

Python 3.11 is estimated to be between 10-60% faster than Python 3.10 for native execution, and we see comparable improvements with the WebAssembly runtime on our benchmark suite.

Load time and size optimizations

Reducing the load time and size of a minimal web page using Pyodide has been a continuous effort. Several techniques have been implemented aiming to reduce download size,

• tests and some standard library modules are unvendored when Pyodide is initialized with loadPyodide({..., fullStdLib=False}), the default currently, and need to be explicitly loaded. In this release we additionally unvendored _pydecimal and pydoc_data modules.
• Python standard libraries are now vendored in a zipfile, using zipimport. This would make it easier to adjust the contents of this file if needed, particularly that loadPyodide can now load it from a custom location using the stdLibURL parameter.

To some extent, however, these optimizations have been offset by the growth of the Python standard library, and the enabling of additional functionality over time.

As can be seen in the following graph, there are size increases for each major Python version, with an overall, slowly decreasing trend for the download size,

In this release we have also added a py-compiled build for the Python standard library and packages. This leads to faster initialization times, however because the size of .pyc files increased in Python 3.11 (cpython#99554) the overall download size is larger:

(these benchmarks were done with a 100 MBps connection and browser cache disabled)

Similarly load time improvements x1.2 - x2.0 were observed for larger packages such as pandas, sympy or scikit-learn (pyodide#3269)

You can use to py-compiled builds by either setting,

https://cdn.jsdelivr.net/pyodide/v0.23.0/pyc/

as the indexURL or by appending ?build=pyc to the REPL URL. Pyodide CLI now also has the command pyodide py-compile to py-compile Python wheels to for a given Python version.

A known limitation of py-compiled builds is that the error traceback no longer contain code snippets, only the line numbers.

Javascript API and Foreign Function Interface

The Python / Javascript type conversion API in Pyodide upon which many downstream projects build, is considered fairly stable and we do not expect major changes to its API in the future. Nevertheless, this release includes several improvements of the FFI.

In this release async Python iterators get proxied to async iterable JavaScript objects. For instance, the following code now works,

for await (let x of async_iterator_pyproxy) {
  // ...
}

previously it would have failed with a TypeError.

Another longstanding issue is that Python does not allow reserved key words as attributes, which is an issue when calling JavaScript object methods. For instance Array.from method in JavaScript, cannot be called from Python as it would produce a SyntaxError.To handle this, if an attribute to a JsProxy consists of a Python reserved word followed by one or more underscores, we now remove a single underscore from the end of the attribute. For instance, Array.from_ would access from on the underlying JavaScript object, whereas o.from__ accesses the from_ attribute.

Class inheritance of PyProxy was also improved,

• Methods for checking PyProxy capabilities (e.g., supportsHas, isCallable) are now deprecated, in favor of using e.g., instanceof pyodide.ffi.PyCallable instead
• A JsProxy of a JavaScript error object can be directly thrown as Python exceptions. Previously Pyodide automatically wrapped them in a JsException but that is no longer needed – now JsException inherits from both JsProxy and Exception
• Checking whether an object is an instance of a PyProxy now only recognizes a PyProxy generated from the same Python interpreter. This means that creating multiple interpreters and importing a PyProxy from one into another no longer causes a fatal error

There were also some deprecations that you can find in the changelog and in the deprecation timeline

The build system and CLI

In this release we have finished deprecating all pyodide-build based commands, in favor of switching to the top level pyodide CLI,

$ pyodide --help
 Usage: pyodide [OPTIONS] COMMAND [ARGS]...                                                
                                                                                           
 A command line interface for Pyodide.                                                     
 Other CLI subcommands are registered via the plugin system by installing Pyodide          
 compatible packages (e.g. pyodide-build).                                                 
                                                                                           
╭─ Registered by: auditwheel_emscripten ──────────────────────────────────────────────────╮
│ auditwheel  Auditwheel-like tool for emscripten wheels and shared libraries.            │
╰─────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Registered by: pyodide-build ──────────────────────────────────────────────────────────╮
│ build          Use pypa/build to build a Python package from source, pypi or url.       │
│ build-recipes  Build packages using yaml recipes and create repodata.json               │
│ config         Manage config variables used in pyodide                                  │
│ py-compile     Compile .py files to .pyc in a wheel, a zip file, or a folder with       │
│                wheels or zip files.                                                     │
│ skeleton       Add a new package build recipe or update an existing recipe              │
│ venv           Create a Pyodide virtual environment                                     │
╰─────────────────────────────────────────────────────────────────────────────────────────╯

In particular, pyodide build-recipes had many improvements, such as nicer progress logging with Rich, auto-detection of the number of CPUs for parallel builds, and the --no-deps option to skip dependency resolution.

As to pyodide build command for building wheels out-of-tree, it can now build a list of packages in a requirements.txt file via the -r <requirements.txt> parameter. It can also recursively build package dependencies with the --build-dependencies option and allows better build reproducibility with the --output-lockfile parameter.

Experimental support for SDL2

There has been a long-standing request to add support for graphics applications to Pyodide. In this release, we have added support for SDL-based packages relying on the functionality Emscripten provides. This is still an experimental feature and there are known issues, but we aim to resolve them in the next release to make it more stable.

If you’d like to try it out, in this release we’ve added support for Pyxel, a popular Python game library. You can try running a game made with Pyxel in Pyodide here.

This feature was made possible with the help of the Pyxel maintainers, so we would like to thank them for their help.

Acknowledgements

Thanks to everyone who contributed code to this release and all users who reported issues and provided feedback.

The following people commited to Pyodide in this release:

Alexey Ignatiev, Andrea Giammarchi, Arpit, Christian Clauss, Deepak Cherian, Eli Lamb, Feodor Fitsner, Gyeongjae Choi, Hood Chatham, Jeff Glass, Jo Bovy, Joe Marshall, josephrocca, Loïc Estève, martinRenou, messense, Nicholas Bollweg, Roman Yurchak, TheOnlyWayUp, Victor Blomqvist, Ye Joo Park