I’m a huge fan of using hatch to manage my Python projects. It lets me define my projects
with a single
pyproject.toml file and define all of my extra environments and scripts in the
same place (i.e testing / linting / docs). One big thing it’s
missing though is an integration with
What is hatch-pip-compile?
hatch-pip-compile is a hatch plugin that connects your hatch-managed virtual environment to a lockfile managed with pip-compile. The plugin detects whether your environment or lockfile is out to date and automatically syncs them when needed - and it’s fast!
For all my projects I found myself regenerating manual lock files using complex
pip-compile shell commands wired up to hatch to get a reproducible environments
across machines. Every time I added or updated a package, I had to manually regenerate
the lockfile. This was a huge pain and I wanted something that would automatically
keep my lockfile up to date. The functionality is useful for both Python libraries and
One of my favorite parts of the tool is that the lockfiles aren’t in some proprietary
format (no offense poetry, PDM, pipenv) so if your project has a contributor who
isn’t interested in learning how to use hatch they can simply
pip install -r <lockfile> and get going quickly.
How does it work?
hatch has an amazing
environment plugin system that allows you to extend the functionality of your
hatch environments. The plugin itself is a subclass of hatch’s
VirtualEnvironment class and
overrides some methods to add the lockfile functionality:
There is quite a bit more to the plugin, but the above methods are the
core of the implementation. Behind the scenes the plugin installs pip-tools into the
virtual environment, gathers the dependencies from the
and pipes them into
pip-compile to generate a lockfile.
The plugin writes its own header on the lockfile to ensure that it can
identify when a lockfile is out of sync with the environment and update it.
The plugin also supports the linking of multiple environments together to
ensure that dependencies are synced to the same versions across environments
(such as between your
Installation and usage of the plugin is as simple adding the following to your
requires = [
type = "pip-compile"
Learnings from the project
Lockfiles are hard
Platform / Python Version Specific Requirements
Lockfiles are a great way to ensure that your project’s dependencies are reproducible.
In the case of pip-compile, it generates a
pip compatible file that
can be used to install your project’s dependencies.
The problem is that the lockfiles can not always be portable across Python versions and
operating systems. For example, if you generate a lockfile on a Mac using
3.9, it may not be compatible with a Linux machine running Python
is because libraries can have requirements that depend on a specific Python version or
Let’s take the example of keyring which is a library that provides cross-platform support for storing secure secrets (it’s also a dependency of hatch):
importlib_metadata >= 4.11.4; python_version < "3.12"
importlib_resources; python_version < "3.9"
We can see some of the requirements are platform specific, and some are Python version
specific. This means that if you generate a lockfile using
pip-compile on a Mac using
3.9, it will not see the
SecretStorage requirement and will not include it
on the resulting lockfile. This is a problem if you try to install the lockfile on a
Linux machine running Python
3.8, and you’re using the
--no-deps flag to
install your dependencies. If you’re not using the
--no-deps flag, then
will identify that those dependencies are required for your platform / Python
version and will install them.
This behavior is why the plugin uses
pip instead of pip-sync to install the lockfile
by default and doesn’t use the
--no-deps flag. This behavior is of course configurable,
but it’s important to understand why it’s the default.
Let’s take another example from the pandas library, a well-known data analysis / manipulation
package. As of this writing, the current supported versions of Python are
3.12. If you’re publishing a Python library that depends on pandas, you’ll likely
want to support all of these versions. However, there is not currently a version of pandas that
supports all of these versions of Python and has a binary available for all of these versions
too. You can sometimes build a package from source if you have the right build dependencies,
but this is not always possible and usually not desirable.
This means that if you generate a lockfile using
pip-compile on a Mac using Python
3.9, it will
likely pin you on the current version of pandas (
2.1.4) which does not have a binary available
3.8. If you pinned your project to
pandas==2.0.3 (the last version with a binary available
3.8), then you would not be able to install your project on a Mac using Python
there is no binary available for that version of Python. This means that you’ll probably end up
with something like this in your dependencies:
pandas~=2.0.3; python_version < '3.9'
pandas~=2.1.4; python_version > '3.8'
numpy~=1.24.4; python_version < '3.9'
numpy~=1.26.1; python_version > '3.8'
This will allow you to install your project on all versions of Python, but it will not be
reproducible across machines. This is because the
pip-compile will choose whichever version
of pandas is available for its Python interpreter and exclude the other version.
The plugin’s ability to support matrix environments is a great way to get around this.
When a matrix environment is also a
pip-compile environment, the plugin will generate a
separate lockfile for each Python version.