Early Development: This project is in active development. Format and APIs may change.

Persistent metadata caching for Parquet files.

PARX caches Parquet metadata in sidecar files (.parx) to eliminate repeated metadata fetches.

The problem: Parquet stores metadata at the end of files. Reading it requires 3 requests: HEAD for file size, GET_RANGE for footer length, GET_RANGE for footer. When 10 workers read the same files, that's 30 requests per file.

The solution: Cache metadata once in a .parx sidecar. All workers fetch the sidecar (1 request) instead of reading the footer (3 requests).

file.parquet (2.7 MB) file.parquet.parx (282 KB) 

Single-file format:

┌──────────────────────────────────────────┐ │ Header (16 bytes) │ │ - Magic: "PARX" │ │ - Version, Flags │ ├──────────────────────────────────────────┤ │ Footer Payload (variable, raw/compressed)│ │ - Raw Parquet footer bytes │ ├──────────────────────────────────────────┤ │ Page Index Payload (optional) │ │ - ColumnIndex + OffsetIndex │ ├──────────────────────────────────────────┤ │ Manifest (Protobuf) │ │ - Offsets, lengths, checksums │ │ - Source file size │ ├──────────────────────────────────────────┤ │ Trailer (12 bytes) │ │ - Manifest length, CRC32C │ │ - Magic: "PARX" │ └──────────────────────────────────────────┘ 

Bundle format (for directories):

┌──────────────────────────────────────────┐ │ Bundle Header (24 bytes) │ │ - Magic: "PRXB" │ │ - Version, Flags │ │ - Entry count │ ├──────────────────────────────────────────┤ │ Entry 0: Footer (+ optional page indexes)│ ├──────────────────────────────────────────┤ │ Entry 1: Footer (+ optional page indexes)│ ├──────────────────────────────────────────┤ │ ... (N entries) │ ├──────────────────────────────────────────┤ │ Bundle Manifest (Protobuf) │ │ - Path→Entry mapping │ ├──────────────────────────────────────────┤ │ Trailer (12 bytes) │ │ - Manifest length, CRC32C │ │ - Magic: "PRXB" │ └──────────────────────────────────────────┘ 

See FORMAT_SPEC.md for detailed byte-level layout. Bundle entries can optionally include page-index payloads using policy-controlled caps.

# Core library cd implementations/rust/parx cargo build --release # CLI tool (install to ~/.cargo/bin) cd implementations/rust/parx-cli cargo install --path . --locked # Benchmarks cd benchmarks/parx_benchmarks make all

If parx is not found after install, ensure ~/.cargo/bin is on your PATH.

# Build sidecar for single file parx build file.parquet # Verify sidecar parx verify file.parquet.parx # Inspect contents parx inspect file.parquet.parx # Bundle directory parx bundle build /data/events/ # Creates: /data/events/_parx_bundle.parx # Bundle directory with page indexes (optional, capped) parx bundle build /data/events/ \ --include-page-indexes \ --max-page-index-bytes-per-file 262144 \ --max-total-page-index-bytes 16777216 # Extract bundle parx bundle extract /data/events/_parx_bundle.parx --output /output/

For a full local CLI validation walkthrough (install + generated fixtures + end-to-end commands), see: docs/CLI_SMOKE_TEST.md

use parx_rs::{ParxReader, ParxWriter}; // Write: build from Parquet file directly let mut writer = ParxWriter::from_parquet_file("file.parquet")?; let parx_bytes = writer.finish(); std::fs::write("file.parquet.parx", parx_bytes)?; // Read: load cached footer from .parx sidecar let parx_data = std::fs::read("file.parquet.parx")?; let reader = ParxReader::open(&parx_data)?; let footer = reader.footer_bytes(); // Raw Parquet footer, ready to use

Local tests with 4 schema types (simple, medium, wide, nested):

Arrow async vs PARX:

• Requests: 3.0 → 1.0 per file (66.7% reduction)
• Latency: ~428 µs → ~208 µs (~2x faster)
• Bytes: ~25 KB → ~26 KB (~2% overhead)

Note: this benchmark measures metadata read path with prebuilt .parx sidecars; one-time sidecar creation is excluded.

Run benchmarks:

cd benchmarks/parx_benchmarks make arrow-vs-parx # Arrow vs PARX comparison make prefetch # Prefetch hint testing

Use it:

• Cloud storage (S3/GCS/Azure)
• Multiple processes reading same files
• Immutable or versioned files
• Parquet V2 with page indexes (including bundle mode with policy caps)

Skip it:

• Single-process work (in-memory cache is fine)
• Local SSD (minimal benefit)
• Delta/Iceberg/Hudi tables (built-in metadata)
• Frequently updated files

parx/ ├── implementations/rust/ │ ├── parx/ # Core library │ └── parx-cli/ # CLI tool ├── benchmarks/ │ └── parx_benchmarks/ # Performance tests ├── spec/ │ └── proto/ # Protobuf schema └── FORMAT_SPEC.md # Format specification 

# Unit tests cargo test # Integration tests cd implementations/rust/parx-cli cargo test --test cli_integration

Apache 2.0