Dependency not installed when installing namespace package

[penguinpee]

I have a namespace package, which upon installation, needs to have the top level __init__.py available. That __init__.py contains all the classes the modules depend on and is provided by another package. So, I put the dependency in install_requires. However, the required package is not installed by pip.

Here is the required structure:

site-packages
└───pymunin
    ├───__init__.py
    ├───plugins
    │   ├───__init__.py
    │   ├───apachestats.py
    │   └───asteriskstats.py
    └───sysinfo
        ├───__init__.py
        |───apache.py
        └───asterisk.py

The namespace package PyMuninPlugins provides the plugins and sysinfo namespaces. The __init__.py in pymunin is provided by PyMunin, which is defined as dependency in setup.cfg of PyMuninPlugins.

Both packages are available on testpypi. I used the command as described in the docs to install from there:

pip install --index-url https://test.pypi.org/simple/ PyMuninPlugins

But the dependency is not installed. Any suggestions?

[abravalheri]

Hi @penguinpee, thank you very much for bringing this issue up.

I am a bit confused, maybe I am not understanding what is going on exactly, you would mind providing a bit more details regarding the following doubt?

• I don't understand why are you referring to the packages and subpackages you have as "namespaces".
  Normally when we refer to namespaces in Python we are talking about directories without a __init__.py. Or would it be the case that you are referring to "legacy namespaces" created with pkg_resources or pkginfo?

Please note that if you have a __init__.py file and don't use either pkg_resources or pkginfo to declare namespaces, there is a chance that your packages get overwritten once you install a new package...
This is how Python works... If you want to have multiple distributions installing packages to the same location, the recommendation is to not use __init__.py at all (or use the legacy approach). Please note that all the distributions have to follow the same rules (e.g. it is very likely that removing __init__.py files from PyMuninPlugins will not work if PyMunin still has them).

Note: I had a look on your setup.cfg file and it seems that you have misspelled packages as pakages. Maybe that is also part of the problem?

[penguinpee]

Thank you for spotting the typo @abravalheri 👏 Strange that build didn't stumble on that. I'm pretty sure it complained before when I made a type in one of the declarations. Anyway, it doesn't make a difference. The dependency is still not installed.

I refer to the package PyMuninPlugins as a namespace package, since it installs in the pymunin namespace, but it doesn't provide that namespace. The pymunin directory is empty from the PyMuninPlugins perspective. I was following the Setuptools documentation on finding namespace packages.

The required __init__.py file is installed in the pymunin namespace when PyMunin is installed. I just noticed the setup.cfg of PyMunin contains the same typo. I'll fix that as well and try again. But since the packages install correctly, I doubt it will fix the dependency issue.

[abravalheri]

I tried the following:

rm -rf /tmp/PyMuninPlugins
git clone --branch pymunin_plugins_proposal https://github.com/penguinpee/PyMunin /tmp/PyMuninPlugins
cd /tmp/PyMuninPlugins
virtualenv .venv
.venv/bin/python -m pip install -U pip build
.venv/bin/python -m build
# ...
.venv/bin/python -m pip install dist/*.whl
unzip -c dist/*.whl  PyMuninPlugins-3.0.2.dist-info/METADATA | grep -i 'requires-dist'
# ... nothing there ...

Then it came to me: it seems that in your setup.cfg file you are placing install_requires in the [metadata] section... but it should be in the [options] section... Have you seen this wrong in one of our docs? (If that is the case, please let me know and I will fix it).

I moved the option to the correct section and then I tried it again:

.venv/bin/python -m build
# ...
.venv/bin/python -m pip install dist/*.whl
unzip -c dist/*.whl  PyMuninPlugins-3.0.2.dist-info/METADATA | grep -i 'requires-dist'
# Requires-Dist: PyMunin (>=3)
.venv/bin/python -m pip install dist/*.whl
# ERROR: Could not find a version that satisfies the requirement PyMunin>=3 (from pymuninplugins)  ...

At least now it tries to download PyMunin.

[abravalheri]

Setuptools is (slowly) migrating towards the pyproject.toml file.

For this new type of file there are stronger validations in place that will (hopefully) help users to debug incorrect settings.

You can try removing setup.cfg and adding the following to your pyproject.toml file:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "PyMuninPlugins"
version = "3.0.2"
description = "PyMunin Multigraph Plugins"
keywords = ["munin", "monitoring", "plugins"]
authors = [{name = "Ali Onur Uyar", email = "[email protected]"}]
maintainers = [{name = "Sandro", email = "[email protected]"}]
readme = "README.md"
license = {text = "GPLv3"}
dependencies = ["PyMunin >=3"]
requires-python = ">=3.7"
classifiers = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
    "Operating System :: OS Independent",
    "Programming Language :: Python :: 3",
    "Topic :: Utilities",
    "Topic :: System :: Monitoring",
]

[project.urls]
Homepage = "https://aouyar.github.io/PyMunin"
"Source Code" = "https://github.com/aouyar/PyMuninPlugins"
"Bug Tracker" = "https://github.com/aouyar/PyMuninPlugins/issues"

(You are going to see a warning about it being experimental, but that will change soon, the only experimental part will be the [tool.setuptools] table, which most of the times users don't need to specify - see #3347).

If you want to try pyproject.toml, please just have in mind the following:

1. When writing setup.py please don't assume sys.path includes the current working directory
2. Try not use the [tool.setuptools] section.

[abravalheri]

If even after fixing the configuration you still have problems due to the limitations we previously discussed (i.e. the way Python and pip work), I suggest giving it a try to entry-points, that is the standard approach in the community for handling plugins...

[penguinpee]

Then it came to me: it seems that in your setup.cfg file you are placing install_requires in the [metadata] section... but it should be in the [options] section... Have you seen this wrong in one of our docs? (If that is the case, please let me know and I will fix it).

Looks like it was a fault of my own making. I reviewed the documentation I used and the examples have it in the [options] section. We live and learn...

I moved the option to the correct section and then I tried it again:

.venv/bin/python -m build
# ...
.venv/bin/python -m pip install dist/*.whl
unzip -c dist/*.whl  PyMuninPlugins-3.0.2.dist-info/METADATA | grep -i 'requires-dist'
# Requires-Dist: PyMunin (>=3)
.venv/bin/python -m pip install dist/*.whl
# ERROR: Could not find a version that satisfies the requirement PyMunin>=3 (from pymuninplugins)  ...

At least now it tries to download PyMunin.

Thanks a lot. 👏 I'll fix it and ponder the use of namespace packages some more.

[penguinpee]

Setuptools is (slowly) migrating towards the pyproject.toml file.

For this new type of file there are stronger validations in place that will (hopefully) help users to debug incorrect settings.

Yes, that would definitely be an improvement. I take it there is now lint tool available for setup.cfg. It's quite daunting that build just swallows whatever you throw at it. I know, I'm exaggerating 😜

You can try removing setup.cfg and adding the following to your pyproject.toml file:

I'll look into it. Seeing that it is mentioned as experimental in the documentation, made me not consider it. But knowing it has validation makes me reconsider. Maybe the documentation can be updated, stating that it will be the future standard. Calling it beta is less off-putting than calling it experimental.

If you want to try pyproject.toml, please just have in mind the following:

1. When writing setup.py please don't assume sys.path includes the current working directory
2. Try not use the [tool.setuptools] section.

Noted.

If even after fixing the configuration you still have problems due to the limitations we previously discussed (i.e. the way Python and pip work), I suggest giving it a try to entry-points, that is the standard approach in the community for handling plugins...

I'm aware of entry-points (console_scripts). That is how the plugins (scripts in fact) were installed previously. There's a downside to that approach in case of Munin plugins. The scripts are wrappers around the Python modules. That's okay for the usual command line scripts with built-in help and maybe a man page. But Munin plugins use inline documentation/help that is parsed by perldoc wrapped in munindoc. From a user's perspective, that documentation is quite often essential for configuring the plugins correctly. It gets lost using the console_scripts approach. I'm working on copying the plugin modules/scripts as is to the desired location. That requires some renaming (dropping the .py extension) and adding shebangs. But I suppose I can leverage on the work done previously.

Thanks again for your help @abravalheri! Now I get to mark my own reply as answer since I messed up posting as a reply to your first answer. 😕 🎆 🥇