Convert Parquet to CSV in python

Convert Parquet to CSV in python

You can convert a Parquet file to a CSV file in Python using the pyarrow library, which provides functions for working with Parquet files and data. Here's how you can do it:

  1. Install pyarrow: If you haven't already installed pyarrow, you can install it using pip:

    pip install pyarrow

  2. Convert Parquet to CSV: Here's an example code snippet to convert a Parquet file to a CSV file using pyarrow:

    import pyarrow.parquet as pq import pandas as pd # Specify the paths to the Parquet and CSV files parquet_file_path = 'input.parquet' csv_file_path = 'output.csv' # Read the Parquet file using pyarrow table = pq.read_table(parquet_file_path) # Convert the Parquet table to a pandas DataFrame df = table.to_pandas() # Save the DataFrame to a CSV file df.to_csv(csv_file_path, index=False)

    In this example, replace 'input.parquet' with the path to your Parquet file and 'output.csv' with the desired path for the CSV output file. The code reads the Parquet file using pyarrow, converts it to a pandas DataFrame, and then saves the DataFrame as a CSV file using to_csv().

Remember that converting from Parquet to CSV might lead to data type conversions and might not preserve some Parquet-specific features, such as nested structures or complex data types. Additionally, Parquet is a columnar storage format optimized for analytical queries, while CSV is a row-based format. So, the choice to convert depends on your specific use case and requirements.

Examples

  1. Convert Parquet to CSV in Python using PyArrow:

    • Description: Utilize PyArrow library to read Parquet file and save it as CSV.
    • Code:
      import pyarrow.parquet as pq # Convert Parquet to CSV in Python using PyArrow table = pq.read_table('input.parquet') table.to_pandas().to_csv('output.csv', index=False)

  2. Convert Parquet to CSV in Python using Pandas:

    • Description: Use Pandas library to read Parquet file and save it as CSV.
    • Code:
      import pandas as pd # Convert Parquet to CSV in Python using Pandas df = pd.read_parquet('input.parquet') df.to_csv('output.csv', index=False)

  3. Convert Parquet to CSV in Python with custom delimiter:

    • Description: Convert Parquet file to CSV with a custom delimiter (e.g., ';', '|').
    • Code:
      import pandas as pd # Convert Parquet to CSV in Python with custom delimiter df = pd.read_parquet('input.parquet') df.to_csv('output.csv', sep=';', index=False)

  4. Convert Parquet to CSV in Python with specific columns:

    • Description: Select specific columns from the Parquet file and save them as CSV.
    • Code:
      import pandas as pd # Convert Parquet to CSV in Python with specific columns df = pd.read_parquet('input.parquet', columns=['col1', 'col2']) df.to_csv('output.csv', index=False)

  5. Convert Parquet to CSV in Python with header and index labels:

    • Description: Include header and index labels while converting Parquet to CSV.
    • Code:
      import pandas as pd # Convert Parquet to CSV in Python with header and index labels df = pd.read_parquet('input.parquet') df.to_csv('output.csv', index_label='index', header=True)

  6. Convert Parquet to CSV in Python with specific encoding:

    • Description: Convert Parquet file to CSV with a specific encoding (e.g., 'utf-8', 'latin1').
    • Code:
      import pandas as pd # Convert Parquet to CSV in Python with specific encoding df = pd.read_parquet('input.parquet') df.to_csv('output.csv', index=False, encoding='utf-8')

  7. Convert Parquet to CSV in Python with chunking for large files:

    • Description: Handle large Parquet files by converting them to CSV in chunks.
    • Code:
      import pandas as pd # Convert Parquet to CSV in Python with chunking for large files chunk_size = 10000 parquet_reader = pd.read_parquet('input.parquet', chunksize=chunk_size) for i, chunk in enumerate(parquet_reader): chunk.to_csv(f'output_chunk_{i}.csv', index=False)

  8. Convert Parquet to CSV in Python with datetime format conversion:

    • Description: Convert datetime columns from Parquet file to a specific format while saving as CSV.
    • Code:
      import pandas as pd # Convert Parquet to CSV in Python with datetime format conversion df = pd.read_parquet('input.parquet') df['datetime_column'] = df['datetime_column'].dt.strftime('%Y-%m-%d %H:%M:%S') df.to_csv('output.csv', index=False)

  9. Convert Parquet to CSV in Python with compression:

    • Description: Save CSV file with compression (e.g., gzip, zip) after converting from Parquet.
    • Code:
      import pandas as pd # Convert Parquet to CSV in Python with compression df = pd.read_parquet('input.parquet') df.to_csv('output.csv.gz', index=False, compression='gzip')

  10. Convert Parquet to CSV in Python with null value handling:

    • Description: Handle null values (NaN or None) while converting Parquet file to CSV.
    • Code:
      import pandas as pd # Convert Parquet to CSV in Python with null value handling df = pd.read_parquet('input.parquet') df.fillna('', inplace=True) # Replace NaN or None with empty string df.to_csv('output.csv', index=False)

More Tags

afnetworking wkhttpcookiestore method-call dropwizard dplyr kotlin-android-extensions spring-mvc thickbox overlay sklearn-pandas

More Python Questions

More Other animals Calculators

More General chemistry Calculators

More Organic chemistry Calculators

More Stoichiometry Calculators