Skip to content

Commit 4039af5

Browse files
jferardjferard
committed
First commit
0 parents commit 4039af5

File tree

11 files changed

+1188
-0
lines changed

11 files changed

+1188
-0
lines changed

‎.gitignore‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
*.pyc
2+
*.csv
3+
*.db
4+
*.egg-info
5+
6+
# dirs
7+
__pycache__
8+
dist

‎LICENSE.txt‎

Lines changed: 674 additions & 0 deletions
Large diffs are not rendered by default.

‎MANIFEST.in‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Include the license file
2+
include LICENSE.txt
3+
4+
# Include examples
5+
include examples *.dbf *.py *.sql

‎README.rst‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
===========================
2+
sqliteondbf - SQLite on DBF
3+
===========================
4+
5+
Copyright (C) J. Férard 2018
6+
7+
SQLite on DBF is a very simple tool that allows to get a sqlite3 connection or to process a SQL script (sqlite flavour) on a set of dBase tables (dbf files).
8+
Under GPL v.3
9+
10+
Notes:
11+
12+
* A part of this tool is inspired by https://github.com/olemb/dbfread/blob/master/examples/dbf2sqlite by Ole Martin Bjørndalen / UiT The Arctic University of Norway (under MIT licence)
13+
* The example files are adapted from https://www.census.gov/data/tables/2016/econ/stc/2016-annual.html (I didn't find a copyright, but this is fair use I believe)
14+
15+
-------
16+
Example
17+
-------
18+
As a script processor
19+
=====================
20+
In the ``sqliteondbf`` directory:
21+
22+
.. code:: bash
23+
24+
python sqliteondbf -v examples/example.sql
25+
26+
It's also possible to execute an inline command (see below for the ``$convert`` command):
27+
28+
.. code:: bash
29+
30+
python sqliteondbf.py -e "$convert 'examples' example.db 'utf-8'"
31+
32+
This will convert all the dbf files in the ``examples`` directory and subdirectories into a sqlite3 databas names ``example.db``.
33+
34+
As a module
35+
===========
36+
37+
In a python script (see examples/examples.py):
38+
39+
.. code:: python
40+
41+
import sqliteondbf
42+
43+
logging.basicConfig(level=logging.INFO)
44+
connection = sqliteondbf.connect("path/to/dbf/dir")
45+
46+
# now use the sqlite3 connection as usual
47+
48+
----------
49+
The script
50+
----------
51+
There is a mandatory blank line between instructions, but no semi colon is needed.
52+
53+
SQL instructions
54+
================
55+
Usual SQL (sqlite flavour) instructions are simply executed on the current connection.
56+
57+
Special instructions
58+
====================
59+
There are four special instructions that begins with a ``$`` sign: ``connect``, ``convert``, ``export``, ``def``.
60+
61+
``connect``
62+
-----------
63+
To use a set of dbf files, type:
64+
65+
.. code:: sql
66+
67+
$connect dbf path/to/files/ [encoding]
68+
69+
The current connection is set to an in-memory database which contains all dbf tables.
70+
71+
To use an existing sqlite database a source, type:
72+
73+
.. code:: sql
74+
75+
$connect sqlite path/to/sqlite.db
76+
77+
The current connection is set to a slite database. This is equivalent to ``sqlite3.connect("path/to/sqlite.db")`` in a python script.
78+
79+
``convert``
80+
-----------
81+
Similar to connect, but for saving the sqlite database
82+
83+
.. code:: sql
84+
85+
$convert path/to/files/ path/to/sqlite.db [encoding]
86+
87+
The current connection to the database is set to the new sqlite database.
88+
89+
``export``
90+
----------
91+
Save the result of the last select to a csv file:
92+
93+
.. code:: sql
94+
95+
$export file.csv
96+
97+
``def``
98+
-------
99+
To use a custom python function in the script:
100+
101+
.. code:: sql
102+
103+
$def func(args):
104+
...
105+
return ret
106+
107+
TODO
108+
====
109+
* A ``script`` instruction that stores the session
110+
* A ``print`` instruction that prints the result of the last query
111+
* A ``dump`` instruction to dump the in memory database
112+
* An ``aggregate`` to create aggregate functions

‎examples/2016-stc-detailed.dbf‎

30.1 KB
Binary file not shown.

‎examples/example.py‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# coding: utf-8
2+
""" sqliteondbf - SQLite on DBF
3+
Copyright (C) 2016-2018 J. Férard <https://github.com/jferard>
4+
This file is part of sqliteondbf.
5+
sqliteondbf is free software: you can redistribute it and/or modify
6+
it under the terms of the GNU General Public License as published by
7+
the Free Software Foundation, either version 3 of the License, or
8+
(at your option) any later version.
9+
sqliteondbf is distributed in the hope that it will be useful,
10+
but WITHOUT ANY WARRANTY; without even the implied warranty of
11+
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12+
GNU General Public License for more details.
13+
You should have received a copy of the GNU General Public License
14+
along with this program. If not, see <http://www.gnu.org/licenses/>.
15+
"""
16+
17+
# Note: The example files are adapted from https://www.census.gov/data/tables/2016/econ/stc/2016-annual.html (I didn't find a copyright, but this is fair use I believe)
18+
19+
import logging
20+
import sys
21+
22+
sys.path.insert(0, ".")
23+
import sqliteondbf
24+
25+
logging.basicConfig(level=logging.INFO)
26+
connection = sqliteondbf.connect(".")
27+
28+
cursor = connection.cursor()
29+
cursor.execute("""SELECT
30+
SUM(amount)
31+
FROM "2016-stc-detailed"
32+
WHERE state_code = '0' AND item_code='T00' """)
33+
amount = next(cursor)
34+
35+
print ("Total US: {}".format(amount[0]))
36+
37+
cursor.execute("""SELECT
38+
group_concat(item_name),
39+
SUM(amount)
40+
FROM "2016-stc-detailed" d
41+
LEFT JOIN item i
42+
ON d.item_code = i.item_code
43+
WHERE state_code = '0' AND d.item_code<>'T00' AND d.item_code NOT LIKE 'TA%' """)
44+
amount = next(cursor)
45+
46+
print ("Total US ({}): {}".format(*amount))
47+
48+
cursor.execute("""SELECT
49+
group_concat(item_name),
50+
SUM(amount)
51+
FROM "2016-stc-detailed" d
52+
LEFT JOIN item i
53+
ON d.item_code = i.item_code
54+
WHERE state_code = '0' AND d.item_code IN ('T01', 'TA1', 'TA3', 'TA4', 'TA5') """)
55+
amount = next(cursor)
56+
57+
print ("Total US ({}): {}".format(*amount))
58+
59+
cursor.execute("""SELECT
60+
group_concat(s.state_name),
61+
SUM(amount)
62+
FROM "2016-stc-detailed" d
63+
LEFT JOIN state s
64+
ON d.state_code = s.state_code
65+
WHERE d.state_code <> '0' AND item_code='T00' """)
66+
amount = next(cursor)
67+
68+
print ("Total States ({}): {}".format(*amount))

‎examples/example.sql‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
/* sqliteondbf - SQLite on DBF
2+
Copyright (C) 2016-2018 J. Férard <https://github.com/jferard>
3+
This file is part of sqliteondbf.
4+
sqliteondbf is free software: you can redistribute it and/or modify
5+
it under the terms of the GNU General Public License as published by
6+
the Free Software Foundation, either version 3 of the License, or
7+
(at your option) any later version.
8+
sqliteondbf is distributed in the hope that it will be useful,
9+
but WITHOUT ANY WARRANTY; without even the implied warranty of
10+
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11+
GNU General Public License for more details.
12+
You should have received a copy of the GNU General Public License
13+
along with this program. If not, see <http://www.gnu.org/licenses/>.
14+
*/
15+
16+
-- Note: The example files are adapted from https://www.census.gov/data/tables/2016/econ/stc/2016-annual.html (I didn't find a copyright, but this is fair use I believe)
17+
18+
$connect dbf "." "utf-8"
19+
20+
$def my_uppercase(v):
21+
return v.upper()
22+
23+
DROP VIEW IF EXISTS detailed
24+
25+
CREATE VIEW detailed AS
26+
SELECT
27+
d.state_code,
28+
my_uppercase(s.state_name) as state_name,
29+
d.survey_yea,
30+
d.item_code,
31+
my_uppercase(i.item_name) as item_name,
32+
d.amount
33+
FROM
34+
"2016-stc-detailed" d
35+
LEFT JOIN state s
36+
ON d.state_code = s.state_code
37+
LEFT JOIN item i
38+
ON d.item_code = i.item_code
39+
40+
SELECT * FROM detailed
41+
ORDER BY CAST(state_code as INT), CAST(item_code as INT)
42+
43+
$export "detailed.csv"
44+
45+
SELECT
46+
state_code,
47+
my_uppercase(state_name),
48+
survey_yea,
49+
SUM(amount) as amount
50+
FROM detailed
51+
GROUP BY state_code, my_uppercase(state_name), survey_yea
52+
ORDER BY SUM(amount) DESC
53+
54+
$export "by_state.csv"
55+
56+
DROP VIEW IF EXISTS detailed

‎examples/item.dbf‎

1.43 KB
Binary file not shown.

‎examples/state.dbf‎

982 Bytes
Binary file not shown.

‎setup.py‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# coding: utf-8
2+
""" sqliteondbf - SQLite on DBF
3+
Copyright (C) 2016-2018 J. Férard <https://github.com/jferard>
4+
This file is part of sqliteondbf.
5+
sqliteondbf is free software: you can redistribute it and/or modify
6+
it under the terms of the GNU General Public License as published by
7+
the Free Software Foundation, either version 3 of the License, or
8+
(at your option) any later version.
9+
sqliteondbf is distributed in the hope that it will be useful,
10+
but WITHOUT ANY WARRANTY; without even the implied warranty of
11+
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12+
GNU General Public License for more details.
13+
You should have received a copy of the GNU General Public License
14+
along with this program. If not, see <http://www.gnu.org/licenses/>.
15+
"""
16+
from setuptools import setup
17+
from os import path
18+
19+
here = path.abspath(path.dirname(__file__))
20+
21+
# Get the long description from the README file
22+
with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
23+
long_description = f.read()
24+
25+
setup(
26+
name='sqliteondbf',
27+
version='0.0.1',
28+
description='SQLite on DBF is a very simple tool that allows to get a sqlite3 connection or to process a SQL script (sqlite flavour) on a set of dBase tables (dbf files)',
29+
long_description=long_description,
30+
url='https://github.com/jferard/sqliteondbf',
31+
author='Julien Férard',
32+
33+
classifiers=[
34+
'Development Status :: 3 - Alpha',
35+
'Intended Audience :: Developers',
36+
'Topic :: Database :: Front-Ends',
37+
'License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)',
38+
'Programming Language :: Python :: 3',
39+
],
40+
41+
keywords='sqlite dbf converter',
42+
install_requires=['dbfread>=2.0.7'],
43+
entry_points={
44+
'console_scripts': [
45+
'sqliteondbf=sqliteondbf:main',
46+
],
47+
},
48+
)

0 commit comments

Comments
 (0)