I have a pair of NVMe Crucial P5 Plus partitioned with an mdadm RAID1 partition formatted as XFS for / and another ZFS RAIDZ1 partition for the remainder of the disk.
Performance of the XFS partition is:
Performance of the ZFS partition is:
I have already reasonably optimized the ZFS pool:
$ zfs get all web NAME PROPERTY VALUE SOURCE web type filesystem - web creation Wed Mar 19 17:17 2025 - web used 1.70M - web available 391G - web referenced 112K - web compressratio 1.00x - web mounted yes - web quota none default web reservation none default web recordsize 128K default web mountpoint /data/web local web sharenfs off default web checksum on default web compression lz4 local web atime off local web devices on default web exec on default web setuid on default web readonly off default web zoned off default web snapdir hidden default web aclmode discard default web aclinherit restricted default web createtxg 1 - web canmount on default web xattr on local web copies 1 default web version 5 - web utf8only on - web normalization formD - web casesensitivity sensitive - web vscan off default web nbmand off default web sharesmb off default web refquota none default web refreservation none default web guid 10056213854848620788 - web primarycache all default web secondarycache all default web usedbysnapshots 0B - web usedbydataset 112K - web usedbychildren 1.59M - web usedbyrefreservation 0B - web logbias latency default web objsetid 54 - web dedup off default web mlslabel none default web sync standard default web dnodesize legacy default web refcompressratio 1.00x - web written 112K - web logicalused 580K - web logicalreferenced 50K - web volmode default default web filesystem_limit none default web snapshot_limit none default web filesystem_count none default web snapshot_count none default web snapdev hidden default web acltype posix local web context none default web fscontext none default web defcontext none default web rootcontext none default web relatime off local web redundant_metadata all default web overlay on default web encryption off default web keylocation none default web keyformat none default web pbkdf2iters 0 default web special_small_blocks 0 default web prefetch all default web direct standard default web longname off default Namely, the following tuned options:
atime=off relatime=off xattr=sa compression=lz4 dedup=off I have tried toggling compression and checksum off, to no effect.
The system is running a Xeon W-1370 with 128GB of memory. The NVMe drives are running on native PCIe 3.0 m.2 ports on the motherboard. Performance is within the expected range for the XFS partition, but performance tanks with random 4K reads and writes. All filesystems are trimmed and disk usage is under 10%. These drives are very healthy at above 80%. I am running the latest Fedora 41.
Why is performance tanking with the ZFS partition?
This problem isn't limited to just this pool, although it is easier to demonstrate the discrepancy there, since this array also contains an XFS partition. Here is the performance as seen a ZFS RAIDZ1 mirror on a pair of SATA MX500s, configured to use the whole disk. Performance is abysmal at 4K.
