56

I use setuptools. In setup.cfg, I can define

[options.package_data] myModule = '*.csv' 

to make sure that data will be installed for my users.

Can I achive the same with pyproject.toml instead?

7 Answers 7

54

Starting in setuptools version 61.0.0 there is beta support for this feature. See the documentation https://setuptools.pypa.io/en/stable/userguide/datafiles.html

The syntax is

[tool.setuptools.package-data] myModule = ["*.csv"] 

Update 2024-01-12: This is no longer in beta. A handful of different mechanisms can enable for data inclusion. One mechanism is to ask for automatic discovery based on git. It should suffice to set the setuptools version and enable the -scm plugin. Now all files tracked by git will be considered data files. The relevant section is...

[build-system] requires = [ "setuptools>=60", "setuptools-scm>=8.0"] 

I produced a complete example and posted on github. https://github.com/el-hult/mypkg

Read the full documentation for more options.

Sign up to request clarification or add additional context in comments.

7 Comments

If you have both setup.cfg and pyproject.toml, adding the above in pyproject.toml seems not to work.
You may need: find . -name '*.egg-info' -exec rm -r {} + && rm -Rf build
Note: if your project is namespaced, the syntax is "path.to.myModule" = [...] i.e. you need quotes around the key
to include the whole folder, for example "data" folder located in my-project/my-package/data you do my-package = ["data/**/*"]
Google brought me here. After looking into the link, the most straight forward way to go about this seems to be adding in pyproject.toml: [tool.setuptools.package-data], new line, mypkg = ["path/to/my/csv/*.csv", "path/to/my/html/*.html"]. Hope this helps someone.
|
11

Assuming a project dir structure like

project_root_directory ├── pyproject.toml └── src └── mypkg ├── __init__.py ├── data1.rst ├── data2.rst ├── data1.txt └── data2.txt 

simply add the following to your pyproject.toml

[tool.setuptools.packages.find] where = ["src"] [tool.setuptools.package-data] mypkg = ["*.txt", "*.rst"] 

More info can be found in the docs @ https://setuptools.pypa.io/en/latest/userguide/datafiles.html

3 Comments

Indeed. That is also what my answer indicate, and you provide the same link as I do.
Ah not sure how i missed that! wanted to add a quick example anyway for anyone searching for a quick solution :)
This answer is better than stackoverflow.com/a/73062301/567059 because it showed the need for [tool.setuptools.packages.find].
6

if I understand your concern, you are using setuptools as a building and distributing system and you want to move some configs from setup.[py,cfg] namely package_data to pyproject.toml, if so you have to use an other tool to build and distribute your package e.g poetry as stated in @Romibuzi's answer because it's not possible unless the setuptools' team plan a new major release to include a full support of pyproject.toml and then no need for extra/standalone config setup.cfg file.

some references:

Update

as per v61.0.0, setuptools brings a partial support (still in beta version) for this feature.

refer to @LudvigH's answer. (thank you btw)

3 Comments

so is it possible or not?
As this is the chosen answer, please update it based on LudvigH answer. Setuptools now works.
I think now if you use the pyproject.toml and build using the build pypi package, all should work. pip install build, then python -m build --wheel
6

As of now, if you are using pyproject.toml and setuptool you do not need to do anything. Just add some extra files in the same directory as your Python code.

But note that it will only add them if they are checked into source control, and you are using setuptools-scm (the example tells you to so you probably are).

For example this is my pyproject.toml

[build-system] requires = ["setuptools", "setuptools-scm"] build-backend = "setuptools.build_meta" [project] name = "foo" authors = [ {name = "...", email = "..."}, ] description = "..." readme = "README.md" requires-python = ">=3.9" dependencies = [ "pydantic", "jinja2" ] version = "1.0.1" [project.scripts] foo = "foo:main" 

And I have these files:

. ├── pyproject.toml ├── pyrightconfig.json ├── README.md ├── setup.py └── src └── foo ├── command.py ├── index.html ├── __init__.py ├── render.py └── triage.py 

And when I run python3 -m build it does add index.html to the wheels.

Also setup.py is just for backwards compatibility. You don't actually need it.

1 Comment

Even after reading the instructions, I was still missing setuptools-scm. Pay close attention to step #1 and #2 under include_package_data
4

In addition to the other existing answers, one can also simply use a MANIFEST.in file.

See https://packaging.python.org/en/latest/guides/using-manifest-in/

1 Comment

This is a good complementary point. It often serves a different purpose than package_data, but can still be very useful.
2

If you're using Hatchling, per the docs, you can also do this via:

[tool.hatch.build] include = [ "myModule/**/*.csv", "**/*.py", ] exclude = [ "tests/**", ] 

Comments

0

One of the things I like is to be able to make a change and then double-check it before(!) I commit it.

For this issue, one thing you can do is open the src/mymod.egg-info directory. In it, there is a SOURCES.txt file and in that file your data files should show up:

 LICENSE.txt MANIFEST.in README.md pyproject.toml setup.cfg src/mymod/__init__.py src/mymod/some_cool_script1.py src/mymod/some_data.json <=== a data file! src/mymod/some_script2.py <etc.> 

So do this:

  • modify pyproject.toml to get the data you want.
  • DO NOT change the version!
  • run your process to build the egg. I use do_publish_setup.py:
import setuptools setuptools.setup() 

run it

python do_publish_setup.py -q sdist 
  • run twine to check it:
twine check dist/* 
  • if there's no issues, then the egg-info should have been created and you can double-check the SOURCES.txt file for everything you expect.

This process would be very nice to put into the setuptools doc... somewhere.

PS Here's the relevant pyproject.toml content:

[tool.setuptools.packages.find] where = ['src'] exclude = ['.gitignore'] [tool.setuptools] include-package-data = true <== not sure if this is 100% needed [tool.setuptools.package-data] mymod=['*.json'] <== not sure if mymod should use quotes 

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.