Research

Notes on Python editable installs

This page is managed by StJohn Piano.

If you have any questions, comments, or suggestions - please contact StJohn Piano on Tela:
tela.app/id/stjohn_piano/7c51a6

Situation: You want to build a feature that will require changes to multiple Python packages that are in a dependency chain

It will be easiest to edit all the packages simultaneously.

This can be accomplished by using "editable installs" - the parent package will install a link file in its virtualenv that points to your local copy of the child package.

Any changes that you make in the local copy of a dependency package will be immediately reflected within the virtualenv of the parent package.

You can then work on all the packages at the same time, using a branch on each package repo with the same name e.g. feature/20240101_feature_name, until the feature is ready. Then you can go through the merge process for each package repo.

Demo

I have developed two simple Python packages:

https://github.com/sj-piano/python3-editable-installs-1

https://github.com/sj-piano/python3-editable-installs-2

python3-editable-installs-1 imports python3-editable-installs-2.

python3-editable-installs-2 has been released on Github, so there's a tag that can be used for specifying the package version.

I assume that you've aliased python to python3.

Open a terminal.

mkdir editable && cd editable

git clone git@github.com:sj-piano/python3-editable-installs-1.git

git clone git@github.com:sj-piano/python3-editable-installs-2.git

cd python3-editable-installs-1

python -m venv .venv

. .venv/bin/activate

pip install -r requirements.txt

Here's the contents of requirements.txt:

# PyPI imports
pytest==8.0.0

# Github imports
python3-editable-installs-2 @ git+https://github.com/sj-piano/python3-editable-installs-2@v0.1.1

Now if I run pip freeze, I see this output:

(.venv) stjohn@horizon python3-editable-installs-1 % pip freeze
iniconfig==2.0.0
packaging==23.2
pluggy==1.4.0
pytest==8.0.0
python3-editable-installs-2 @ git+https://github.com/sj-piano/python3-editable-installs-2@acbdb57c1744354c147deff3175b7e0a08e18ddf

Note: Although I specified a tag for python3-editable-installs-2 in requirements.txt, the tag is mapped to a commit hash, which is then used.

Ok, now that python3-editable-installs-1 has been set up, we can change python3-editable-installs-2 to be an editable install of the local copy of the package.

pip install --no-deps --editable ../python3-editable-installs-2

We don't need to uninstall it first. The new pip install command will overwrite the existing package.

We use the --no-deps option to prevent pip from changing any dependencies of the target package. This becomes useful in the situation where we are editing 3 packages simultaneously, for example if python3-editable-installs-2 had another child package called python3-editable-installs-3.

Now, run the test (stored in test/test_baz.py):

pytest test

This should succeed.

Now, in the local copy of python3-editable-installs-2, open the file editable_installs_2/foo.py and change the value of x from 1 to 2.

Now re-run the test command pytest test. It should fail.

Conclusion

What's happened here ? Well, you've changed the local copy of a dependency, and this change has been reflected immediately in the parent package that imports it.

This means that you can work on both packages simultaneously.

I recommend creating a new branch in each package repo, with the same name e.g. feature/20240101_feature_name. When the feature is ready, go through the merge process for each package repo.

This approach can be extended to 3 or more packages in a chain. Be sure to use the --no-deps option when making each editable install.

Extra

The PyCharm editor can't follow the editable install links. It will apply a red underline to the name of the editable package in the code. However, it can still run the tests.

Troubleshooting

If you get this type of error: ImportError: cannot import name '<foo>' from 'python-editable-installs2' (unknown location)
Then look at your PYTHONPATH variable ( echo $PYTHONPATH ), and try adding the path to the package manually ( export PYTHONPATH=$PYTHONPATH:/path/to/package )

If you're using PyCharm, then you'll have to open the Project Settings and manually add a new Content Root that points at the package. You'll then have to remove it again if you want to switch back. Note: Don't change the PYTHONPATH variable.