devpi

Project URL

devpi

Introduction

devpi lets you host one or more Python package indexes locally. The following diagram and examples demonstrate how pip will work after following the instructions on this page.

flowchart TB

  subgraph internet ["Internet"]
    pypi-pub[("PyPI<br>(Python Package Index)")]
  end

  subgraph intranet ["Intranet"]
    internal-art[("python-local<br>(Artifactory repo)")]
  end

  subgraph devpi ["devpi"]
    pypi-devpi[\"pypi<br>(pass-through)"/]
    internal-devpi[("python-local<br>(cache)")]
    local[("local")]

    pypi-devpi & internal-devpi --> local
  end

  pypi-pub --> pypi-devpi
  internal-art --> internal-devpi

  local --> pip

Two examples of how this will work:

• pip install -- numpy

  1. pip searches devpi's local index for a package named numpy.

  2. devpi doesn't find numpy in the local devpi index, so devpi searches the pypi and python-local devpi indexes.

  3. The pypi devpi index searches the PyPI index on the Internet. numpy is found but is not cached locally.

  4. The python-local devpi index searches the python-local Artifactory index within the intranet. numpy is not found.

     • Even if the intranet is not available, the result is the same: numpy is not found.

  5. The numpy found via the pypi devpi index has a greater version string than the numpy found via the python-local devpi index (vacuously true since the python-local devpi index could not find any version of numpy), so the numpy found via the pypi devpi index is served to the local devpi index.

  6. devpi's local index serves numpy to pip.

• pip install -- private-package

  1. pip searches devpi's local index for a package named private-package.

  2. devpi doesn't find private-package in the local devpi index, so devpi searches the pypi and python-local devpi indexes.

  3. The pypi devpi index searches the the PyPI index on the Internet. private-package is not found.

  4. The python-local devpi index searches the python-local Artifactory index within the intranet.

     • If the intranet is available, the latest version of private-package is found and is cached locally.

     • If the intranet is not available, a cached copy of the latest version of private-package previously downloaded is found.

       • If private-package has never been downloaded before, then private-package is not found and we stop here, with pip reporting that private-package could not be found.

  5. The private-package found via the python-local devpi index has a greater version string than the private-package found via the pypi devpi index (vacuously true since the pypi devpi index could not find any version of private-package), so the private-package found via the python-local devpi index is served to the local devpi index.

  6. devpi's local index serves private-package to pip.

Please note that if both PyPI and the python-local Artifactory repo both have a package with the same name, then devpi will fetch the package with the greatest version string.

For this reason, internal Python package names should have an org-specific prefix such as foo. to reduce the likelihood of name collisions with public Python package names. Please see my Python library template for a project template that enforces such a prefix.

Installation instructions

First, install Pixi. Please note that Pixi installation and configuration instructions are outside of the scope of this document.

Create some directories we'll need:

mkdir -p -- \
  "${HOME}/.devpi" \
  "${HOME}/.taskfile/devpi" \
  "${HOME}/Library/LaunchAgents"

Place the Pixi manifest file in the ~/.taskfile/devpi directory:

pixi.toml

# vim: set ft=toml :


[project]
name = "devpi"
channels = ["conda-forge"]
platforms = [
  "osx-64",
  "osx-arm64",
]


[dependencies]
devpi-client = "*"
devpi-server = "*"

Create an isolated environment in which we'll install devpi, then initialize devpi:

pushd -q -- "${HOME}/.taskfile/devpi"
pixi update
pixi install
pixi run -- devpi-init \
  --role standalone \
  --root-passwd root \
  --serverdir "${HOME}/.devpi/server" \
  --storage sqlite
popd -q

Now let's create a launchd service that will make it easy to automatically start and stop devpi. Add this devpi launchd service definition to the ~/Library/LaunchAgents directory, editing usernames and pathnames as needed:

net.devpi.server.plist

<?xml version="1.0" encoding="UTF-8"?>
<!-- vim: set ft=xml : -->
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Disabled</key>
    <false/>
    <key>EnvironmentVariables</key>
    <dict>
        <key>PATH</key>
        <string>/Users/manselmi/.prefix/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
        <key>TZ</key>
        <string>UTC</string>
    </dict>
    <key>KeepAlive</key>
    <dict>
        <key>SuccessfulExit</key>
        <false/>
    </dict>
    <key>Label</key>
    <string>net.devpi.server</string>
    <key>ProgramArguments</key>
    <array>
    <array>
        <string>/Users/manselmi/.devpi/devpi-server</string>
        <string>/Users/manselmi/.taskfile/devpi/.pixi/envs/default/bin/devpi-server</string>
        <string>--port</string>
        <string>3141</string>
        <string>--serverdir</string>
        <string>/Users/manselmi/.devpi/server</string>
    </array>
    </array>
</dict>
</plist>

If you would like to learn more about launchd, please see Creating Launch Daemons and Agents.

Second, let's create the ~/.devpi/devpi-server file invoked by the launchd service, editing usernames and pathnames as needed:

devpi-server

#!/Users/manselmi/.taskfile/devpi/.pixi/envs/default/bin/python
# vim: set ft=python :

import signal
import subprocess
import sys


def main():
    process = None

    def handler(signum, frame):
        if process is not None:
            process.send_signal(signal.SIGINT)

    signal.signal(signal.SIGINT, handler)
    signal.signal(signal.SIGTERM, handler)

    process = subprocess.Popen(sys.argv[1:], start_new_session=True)
    process.wait()

    sys.exit(process.returncode)


if __name__ == '__main__':
    main()

The launchd service we created will run when loaded, so let's load the service:

launchctl bootstrap "gui/$(id -u)/" ~/Library/LaunchAgents/net.devpi.server.plist

Please note that upon future logins, the service will automatically be loaded and hence automatically started.

Now let's confirm that devpi is up and running. Navigate your browser to http://localhost:3141. If you see a devpi page, then so far so good.

Configuration instructions

In this section we'll configure devpi to behave as described in the introduction, and then we'll configure pip to use devpi.

These commands configure devpi. I suggest running these specific commands one at a time.

pushd -q -- "${HOME}/.taskfile/devpi"
pixi shell
unset DEVPI_INDEX
devpi use --always-set-cfg no http://localhost:3141/  # this might complain; no worries
devpi login --password root root
devpi index pypi mirror_use_external_urls=True  # don't cache PyPI packages
devpi index -c python-local \
  mirror_url='https://artifactory.example.com/artifactory/api/pypi/python-local/simple/' \
  title='Intranet: python-local' \
  type=mirror \
  volatile=False
devpi index -c local \
  bases='root/python-local,root/pypi' \
  title='Local: personal index'
devpi use --venv - root/local
exit
popd -q

Note

If you would like to cache all packages (including those from PyPI), run the following command before exiting the Pixi environment shell:

devpi index pypi mirror_use_external_urls=False

However, be aware that devpi will consume more disk space.

Configure pip, distutils, buildout etc to use devpi:

~/Library/Application Support/pip/pip.conf

[global]
index-url = http://localhost:3141/root/local/+simple/
trusted-host = localhost

[search]
index = http://localhost:3141/root/local/

~/.pydistutils.cfg

[easy_install]
index_url = http://localhost:3141/root/local/+simple/

~/.buildout/default.cfg

[buildout]
index = http://localhost:3141/root/local/+simple/

Validation

Let's confirm that pip and devpi are working as expected.

First, let's try downloading numpy:

rm -f -- numpy-*.whl
python -m pip --no-cache-dir download --no-deps --prefer-binary -- numpy
# Looking in indexes: http://localhost:3141/root/local/+simple/
# Collecting numpy
#   Downloading http://localhost:3141/root/pypi/%2Bf/afd/5ced4e5a96dac/numpy-1.26.1-cp312-cp312-macosx_11_0_arm64.whl (13.7 MB)
#      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.7/13.7 MB 14.8 MB/s eta 0:00:00
# Saved ./numpy-1.26.1-cp312-cp312-macosx_11_0_arm64.whl
# Successfully downloaded numpy

Finally, let's try downloading private-package while connected to the intranet:

rm -f -- private-package-*.tar.gz
python -m pip --no-cache-dir download --no-deps -- private-package
# Looking in indexes: http://localhost:3141/root/local/+simple/
# Collecting private-package
#   Downloading http://localhost:3141/root/python-local/%2Bf/77b/253cc0ae627fb/private-package-2.0.0.tar.gz (208 kB)
#      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 208.8/208.8 kB 675.9 MB/s eta 0:00:00
#   Preparing metadata (setup.py) ... done
# Saved ./private-package-2.0.0.tar.gz
# Successfully downloaded private-package

We're seeing a fast download speed because my devpi instance already has the latest version of private-package in its cache. devpi checked with Artifactory and determined that I already had the latest version cached, so devpi served the cached copy… very quickly.

Now let's try while not connected to the intranet.

rm -f -- private-package-*.tar.gz
python -m pip --no-cache-dir download --no-deps -- private-package
# Looking in indexes: http://localhost:3141/root/local/+simple/
# Collecting private-package
#   Downloading http://localhost:3141/root/python-local/%2Bf/77b/253cc0ae627fb/private-package-2.0.0.tar.gz (208 kB)
#      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 208.8/208.8 kB 573.3 MB/s eta 0:00:00
#   Preparing metadata (setup.py) ... done
# Saved ./private-package-2.0.0.tar.gz
# Successfully downloaded private-package

Same result.

Rancher Desktop

A nice bonus is that we can also instruct pip within a container to leverage devpi. This makes it easier to work with Python within a container regardless of whether or not we're connected to the intranet.

Ensure you're not connected to the intranet before running the following command:

nerdctl container run \
  --env PIP_INDEX_URL=http://host.lima.internal:3141/root/local/+simple/ \
  --env PIP_TRUSTED_HOST=host.lima.internal \
  --rm \
  -- \
  docker.io/library/python:3.12.0-bookworm \
  pip download --no-deps -- private-package numpy
# Looking in indexes: http://host.lima.internal:3141/root/local/+simple/
# Collecting private-package
#   Downloading http://host.lima.internal:3141/root/python-local/%2Bf/77b/253cc0ae627fb/private-package-2.0.0.tar.gz (208 kB)
#      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 208.8/208.8 kB 19.1 MB/s eta 0:00:00
#   Preparing metadata (setup.py): started
#   Preparing metadata (setup.py): finished with status 'done'
# Collecting numpy
#   Downloading http://host.lima.internal:3141/root/pypi/%2Bf/a03/fb25610ef560a/numpy-1.26.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (13.9 MB)
#      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.9/13.9 MB 15.4 MB/s eta 0:00:00
# Saved /private-package-2.0.0.tar.gz
# Saved /numpy-1.26.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
# Successfully downloaded private-package numpy

Maintenance

To stop the devpi service:

launchctl bootout "gui/$(id -u)/net.devpi.server"

To start the devpi service (if not already running):

launchctl bootstrap "gui/$(id -u)/" ~/Library/LaunchAgents/net.devpi.server.plist

To upgrade devpi:

launchctl bootout "gui/$(id -u)/net.devpi.server"
pushd -q -- "${HOME}/.taskfile/devpi"
pixi update
pixi install
popd -q
launchctl bootstrap "gui/$(id -u)/" ~/Library/LaunchAgents/net.devpi.server.plist

To uninstall devpi, run:

launchctl bootout "gui/$(id -u)/net.devpi.server"
rm -r -- \
  "${HOME}/.buildout" \
  "${HOME}/.devpi" \
  "${HOME}/.pydistutils.cfg" \
  "${HOME}/Library/Application Support/pip/pip.conf" \
  "${HOME}/Library/LaunchAgents/net.devpi.server.plist"
pushd -q -- "${HOME}/.taskfile/devpi"
pixi clean
popd -q