0

I have a directory structure as below.

. ├── package ├── app │ ├── __init__.py │ ├── file1.py │ └── file2.py ├── master_sql_folder │ │ │ │──sql_folder_1 │ │ ├── world.sql │ │ ├── earth.sql │ │ │ └──sql_folder_2 │ ├── planet.sql │ ├── sun.sql │ └── wrapper_scripts │ ├── wrapper.py │ └── __init__.py ├── setup.py

I am trying to include all the sql files that exist in sub folders of master_sql_folder ( world.sql, earth.sql, planet.sql, sun.sql ) to setuptools while packaging as egg,I cannot use sql_folder_1 and sql_folder_2 in the path as new folders can be added in future under master_sql_folder and I need the code to read them too.I have tried adding the following lines to my setup.py but its not including the sql files in the build.

package_data={'master_sql_folder':['*']} packages=['app', 'wrapper_scripts']

I appreciate your help in advance.

Improve this question
4
  • Possible duplicate of this question. Commented Sep 30, 2020 at 8:37
  • The mentioned link did help me to use this data_files=[('master_sql_folder/sql_folder_1', ['master_sql_folder/sql_folder_1/world.sql'])] and this works But I cannot use sql_folder_1 in the path, Is there a way that can pick the files from all the subfolders like data_files=[('master_sql_folder', ['master_sql_folder/*/*.sql'])]. It results in error :can't copy 'master_sql_folder**.sql': doesn't exist or not a regular file Commented Sep 30, 2020 at 9:21
  • 1
    If I remember correctly, each file has to be named individually. Or, write a function that collects the files and builds the list of files for you. Commented Sep 30, 2020 at 9:36
  • 1
    Thanks @S3DEV for your valuable suggestions. Commented Sep 30, 2020 at 11:09

2 Answers 2

Reset to default
1

I tried a few iterations of this without success.

In the end what worked was the official solution: https://python-packaging.readthedocs.io/en/latest/non-code-files.html

Often packages will need to depend on files which are not .py files: e.g. images, data tables, documentation, etc. Those files need special treatment in order for setuptools to handle them correctly. The mechanism that provides this is the MANIFEST.in file. This is relatively quite simple: MANIFEST.in is really just a list of relative file paths specifying files or globs to include.: include README.rst include docs/*.txt include funniest/data.json In order for these files to be copied at install time to the package’s folder inside site-packages, you’ll need to supply include_package_data=True to the setup() function.

super easy to do, literally just include MANIFEST.in in the same dir as your setup.py, and as per the documentation, you’ll need to supply include_package_data=True to the setup() function.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

Based on @S3DEV 's suggestion and accepted answer here How to add package data recursively in Python setup.py?. I was able to get a solution.

#sql_file_picker.py -- script to feed files to setup.py

import glob class FilePicker: def __init__(self): pass def sql_file_picker(self): sql_files = [] directories = glob.glob('master_sql_folder\\**\\') for directory in directories: files = glob.glob(directory + '*.sql') if len(files) != 0: sql_files.append((directory, files)) return sql_files

in setup.py

from wrapper_scripts.sql_file_picker import FilePicker from setuptools import setup setup( name='XXXXXXXXX', version='X.X.X', packages=['app', 'wrapper_scripts'], url='', license='', author='', author_email='', description='XXXXXXXXXXXXX', data_files=FilePicker().sql_file_picker() )
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.