0

I am trying to use Python's parquet module to read in some Parquet files written from a local MapR instance.

The command I used to output these parquet files is: 

df.sqlContext.sql("SQL HERE").write.format("parquet").option("mergeSchema", "true").save("/path/to/parquet/test.parquet")

This is what the file looks like on my Linux host:

drwxr-xr-x 2 mapr mapr 403 Oct 5 13:56 igayfvpwrs.parquet

Unfortunately, when I use the Python here (https://pypi.python.org/pypi/parquet) - I receive the following exception:

IOError: [Errno 21] Is a directory: '/mnt/mapr/saw/user/mapr/igayfvpwrs.parquet'

Any idea? These files work great in MapR.

EDIT 2:

I was able to figure it out. Since the original .parquet "file" is a directory, just loop through the directory with glob for all the inner .parquet files - the original code for Python-parquet works in there. 

for filename in glob.glob("/mnt/mapr/saw/user/mapr/{0}.parquet/*.parquet".format(tempTableID)): with open(filename) as foo: for row in parquet.DictReader(foo, columns=["column"]): print(json.dumps(row))

EDIT: Here is what is inside the parquet file:

-rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 _common_metadata -rwxr-xr-x 1 mapr mapr 2.4K Oct 5 13:58 _metadata -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00000-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00001-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00002-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00003-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00004-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00005-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00006-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00007-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00008-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00009-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00010-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00011-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00012-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00013-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00014-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00015-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00016-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00017-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00018-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00019-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00020-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00021-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00022-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00023-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00024-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00025-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00026-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00027-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00028-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00029-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00030-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00031-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00032-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00033-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00034-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00035-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00036-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00037-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00038-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00039-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00040-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00041-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00042-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00043-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00044-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00045-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00046-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00047-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00048-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00049-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00050-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00051-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00052-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 1.2K Oct 5 13:58 part-r-00053-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00054-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00055-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00056-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00057-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00058-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00059-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00060-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00061-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00062-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00063-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00064-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00065-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00066-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00067-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00068-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00069-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00070-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00071-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00072-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00073-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00074-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00075-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00076-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00077-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00078-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00079-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00080-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00081-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00082-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00083-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00084-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00085-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00086-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00087-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00088-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00089-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00090-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00091-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00092-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00093-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00094-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00095-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00096-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00097-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00098-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00099-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00100-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00101-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00102-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00103-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00104-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 1.2K Oct 5 13:58 part-r-00105-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00106-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00107-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00108-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00109-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00110-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00111-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00112-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00113-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00114-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00115-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00116-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00117-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00118-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00119-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00120-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00121-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00122-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00123-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00124-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 1.2K Oct 5 13:58 part-r-00125-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00126-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00127-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00128-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00129-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00130-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00131-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00132-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00133-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00134-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00135-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00136-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00137-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00138-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00139-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00140-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00141-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00142-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00143-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00144-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00145-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00146-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00147-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00148-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00149-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00150-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00151-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00152-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00153-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00154-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00155-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00156-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00157-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00158-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00159-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00160-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00161-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00162-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00163-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00164-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00165-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00166-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00167-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00168-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00169-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00170-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00171-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00172-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00173-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00174-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00175-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00176-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00177-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00178-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00179-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00180-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00181-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00182-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00183-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00184-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00185-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00186-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00187-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00188-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00189-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00190-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00191-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00192-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00193-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00194-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00195-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00196-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00197-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00198-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00199-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00200-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00201-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00202-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00203-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00204-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00205-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00206-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00207-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00208-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00209-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00210-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00211-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00212-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00213-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00214-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00215-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00216-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00217-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00218-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00219-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00220-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00221-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00222-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00223-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00224-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00225-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00226-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00227-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00228-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00229-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00230-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00231-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00232-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00233-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00234-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00235-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00236-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00237-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00238-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00239-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00240-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00241-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00242-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00243-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00244-02f0b80b-2993-46f8-a191-f111c6db1dad.gz.parquet -rwxr-xr-x 1 mapr mapr 514 Oct 5 13:58 part-r-00245-02f0b80b-2993-46f8- [truncated] -rwxr-xr-x 1 mapr mapr 0 Oct 5 13:58 _SUCCESS
Improve this question

2 Answers 2

Reset to default
1

There is no issue with parquet on spark here. The DataFrameWriter writes to parquet format into a directory and partition the output according the number of partition of the DataFrame it is writing.

What you are getting is absolutely normal.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

Thank you! I am figuring this out. How do I open the entire parquet file in Parquet-Python as I would via Spark/Scala?
You can read it as a parquet directory files with sqlContext.read.load("path/to/parquet")
How big as your data ?
Depends. We have a few Peta Bytes in our cluster. I figured it out. See above.
Let's say that for good practice coalesce the DataFrame into 1 partition isn't the solution you want when you have petabytes because usually the resources of the master node are limited. And since you have that much data, the only solution is using plain spark on python to read the whole directory. I don't see another way to do it. You don't need parquet-python project nor pypi/parquet just spark
|
0

Based on the output of your ls command, itlooks like igayfvpwrs.parquet is actually a directory. Can you check for the data inside?

Improve this answer

3 Comments

It IS a directory, but on the hdfs Hue also sees all .parquet files as directories. See my update with what's inside the .parquet files.
I found this (tutorialspoint.com/spark_sql/spark_sql_parquet_files.htm).
did you try to open one of those files with python?

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.