find_namespace_packages vs. include_package_data

[emlys]

Much thanks to @abravalheri and the other maintainers for all the help in these discussions around namespace packages and PEP 420. I'm hoping to get clarification on another angle of this issue.

I understand that any directory can be a python package, whether or not it contains any python code. Given that, there is a behavior of setup packages that seems counterintuitive to me. Considering these four cases:

test/
|- setup.py
|- src/
   |- case_1/    # A directory that contains python files only
   |  |- foo.py
   |- case_2/    # A directory that contains python and non-python files
   |  |- foo.py
   |  |- bar.txt
   |- case_3/    # A directory that contains non-python files only
   |  |- bar.txt
   |- case_4/    # A directory that is empty

In all four cases the directory is a package according to python's import mechanism. And all four directories are returned by find_namespace_packages. This makes sense to me.

>>> from src import case_1, case_2, case_3, case_4
>>> find_namespace_packages('src')
['case_2', 'case_3', 'case_4', 'case_1']

Then the following behavior seems contradictory. If the contents of setup.py are

from setuptools import setup, find_namespace_packages
setup(
    packages=find_namespace_packages('src'),
    package_dir={'': 'src'})

then after running python -m build --wheel, the contents of build/lib are

lib/
|- case_1/
|  |- foo.py
|- case_2/
   |- foo.py

• case_3 and case_4 were excluded even though they were included in packages
• bar.txt was excluded from case_2

I think I would expect that

• If a directory doesn't need any python files to be a package, then it should not need any python files to be included in a distribution
• If a directory is included in a distribution, its contents should also be included

Therefore I'd expect the non-python files to be included in the distribution by default. I was surprised I needed to specify them with include_package_data and package_data/MANIFEST.in/setuptools_scm. I am wondering is my understanding correct or is this behavior expected?

[abravalheri]

Hi @emlys thank you very much for bringing this point up for discussion. Please see my notes bellow:

If a directory is included in a distribution, its contents should also be included.

I think I disagree with you here. My opinion is the following:

• If the configuration given to setuptools implies that some files should not be added to the final package, these files should not be added (regardless if the directory is a listed package or not).

If we were to add all the files regardless of the configuration, then we would see a bunch of transient files being included by accident (e.g. *.bkp, *~, *.swp, *.un~, *.bak used by text editors or *.pyc cache files)...

Please note that configurations via setup.cfg and setup.py imply that the default value for include_package_data is False. But configurations via pyproject.toml flip this default value to True... The idea here is that most of the times the users do want to add files automatically...

If a directory doesn't need any python files to be a package, then it should not need any python files to be included in a distribution

This one depends on the opinion that a person has about the previous topic... Is it worthy to add empty directories to .whl files? .whl files are simply zips, and empty directories are usually not added to zips (and it is also not trivial)... What would be the tradeoff "gained utility" x "implementation cost" of adding empty dirs?

Given these "boundary conditions", I think the current behaviour is reasonable.

Please not that it is really trivial to get setuptools to automatically scan the project folder and find the packages/files for you if your package is simple enough. For example, if you don't need any C/C++/Cython extension and stick to a 100% pyproject.toml configuration (no setup.cfg) like the following:

[build-system]
requires = ["setuptools>=63", "setuptools-scm[toml]>=7.0.3"]
build-backend = "setuptools.build_meta"

[project]
name = "myproj"
dependencies = [...]
...
# see https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html

most of the times setuptools will get things right. If your project evolves to be more complex, than maybe you have to tweak some things... I also recommend using a src-layout, because it makes the scanning trivial and reduces the chances of errors that require changes in configs.