(Probably) Corrupted LUKS Header, Restoring Header does not work

I tried everything I was able to find online. Hours of research since yesterday ;( I found no one struggling with the errors I'm facing, except from GitLab (error code -1 instead of -4 I'm getting), Reddit or this mailing list from 2006. I might give unnecessary details, sorry!

I have this 5 TB WD drive where I already have dozens of files. Decided to build a small NAS from a Raspberry Pi 4. Problem was I wanted LUKS encryption, with BTRFS as the file system; the drive was at that time a 5 TB one-partition EXT4.

I split the drive in 2 partitions (on my main computer) (only 2.3 TB was occupied), creating a LUKS protected BTRFS partition half the drive size: moved everything to the encrypted BTRFS partition, deleting the EXT4 part, growing LUKS, opening the encryption and then grew the BTRFS partition to fill the entire drive, passphrase still worked for LUKS, for a very long time. I thought nothing could happen when I have the LUKS Header Backup. The 5 TB LUKS-BTRFS partition is only protected with a passphrase, no additional slots etc. configured. I was able to unlock the drive and mount it now maybe for 3 weeks without any hiccups and error codes on all my devices (Artix-Linux x86_64, Linuxmint, Debian Aarch64, Parted Magic).

The operating system I chose for the Pi 4 was Debian, not Raspbian OS, since it was lacking the Crypto API/Functionality in the kernel I guess needed for serpent-xts-plain64, my drive encryption cipher. The NAS solution I went with was OpenMediaVault. It does not support unlocking LUKS volumes etc. on its own, so I unlocked it via SSH, mounted the device from the Web UI, created an SMB share, was even able to connect and exchange files for a day.

The other day when I wake up I notice when I connect to the SMB share there are no files?! A quick lsblk made clear the drive was not mounted, and the encryption was already closed. Mounting it was now impossible, tried many distros/kernels, architectures (aarch64 & amd64), tried mounting using GParted on many systems, KDE's own disk mounter etc. but no, guess I'm stuck. Funny thing is I was able to change the passphrase using cryptsetup luksChangeKey /dev/sdd1, it happily accepted my password, then successfully changed it to something else (as far as I know when I restore the header the old password is valid). Like I said before I have the LUKS header backup available, it's the right file I know it, since I heard restoring the wrong header makes things more complicated.

I hope I don't have to reinvent the wheel to decrypt the drive, but if it's necessary, I'll do it :/

As far as I can remember I did luksFormat using this command, it was inside my .zshrc:

cryptsetup -v luksFormat /dev/sdd1 --use-random --verify-passphrase --key-size=512 --hash=whirlpool --cipher=serpent-xts-plain64 --pbkdf=argon2id --type luks2 

Here is the output of cryptsetup --debug --verbose luksOpen /dev/sdd1 crypt:

❯ sudo cryptsetup --debug --verbose luksOpen /dev/sdd1 crypt [sudo] password for user: # cryptsetup 2.4.2 processing "cryptsetup --debug --verbose luksOpen /dev/sdd1 crypt" # Running command open. # Locking memory. # Installing SIGINT/SIGTERM handler. # Unblocking interruption on signal. # Allocating context for crypt device /dev/sdd1. # Trying to open and read device /dev/sdd1 with direct-io. # Initialising device-mapper backend library. # Trying to load any crypt type from device /dev/sdd1. # Crypto backend (OpenSSL 1.1.1l 24 Aug 2021) initialized in cryptsetup library version 2.4.2. # Detected kernel Linux 5.15.8-zen1-1-zen x86_64. # Loading LUKS2 header (repair disabled). # Acquiring read lock for device /dev/sdd1. # Opening lock resource file /run/cryptsetup/L_8:49 # Verifying lock handle for /dev/sdd1. # Device /dev/sdd1 READ lock taken. # Trying to read primary LUKS2 header at offset 0x0. # Opening locked device /dev/sdd1 # Verifying locked device handle (bdev) # LUKS2 header version 2 of size 16384 bytes, checksum sha256. # Checksum:cd57d8cf3e5d6bd82e34925c05ac3f84114d564dc1535d443d6003847ede9c03 (on-disk) # Checksum:cd57d8cf3e5d6bd82e34925c05ac3f84114d564dc1535d443d6003847ede9c03 (in-memory) # Trying to read secondary LUKS2 header at offset 0x4000. # Reusing open ro fd on device /dev/sdd1 # LUKS2 header version 2 of size 16384 bytes, checksum sha256. # Checksum:1fa2c8c216bef143a6841c7e6d7b1e737b39a832e3e8067ce580b103673c67b6 (on-disk) # Checksum:1fa2c8c216bef143a6841c7e6d7b1e737b39a832e3e8067ce580b103673c67b6 (in-memory) # Device size 5000946236928, offset 16777216. # Device /dev/sdd1 READ lock released. # PBKDF argon2id, time_ms 2000 (iterations 0), max_memory_kb 1048576, parallel_threads 4. # Activating volume crypt using token (any type) -1. # dm version [ opencount flush ] [16384] (*1) # dm versions [ opencount flush ] [16384] (*1) # Detected dm-ioctl version 4.45.0. # Detected dm-crypt version 1.23.0. # Device-mapper backend running with UDEV support enabled. # dm status crypt [ opencount noflush ] [16384] (*1) No usable token is available. # Interactive passphrase entry requested. Enter passphrase for /dev/sdd1: # Activating volume crypt [keyslot -1] using passphrase. # dm versions [ opencount flush ] [16384] (*1) # dm status crypt [ opencount noflush ] [16384] (*1) # Keyslot 0 priority 1 != 2 (required), skipped. # Trying to open LUKS2 keyslot 0. # Running keyslot key derivation. # Reading keyslot area [0x47000]. # Acquiring read lock for device /dev/sdd1. # Opening lock resource file /run/cryptsetup/L_8:49 # Verifying lock handle for /dev/sdd1. # Device /dev/sdd1 READ lock taken. # Reusing open ro fd on device /dev/sdd1 # Device /dev/sdd1 READ lock released. # Verifying key from keyslot 0, digest 0. # Loading key (64 bytes, type logon) in thread keyring. # dm versions [ opencount flush ] [16384] (*1) # dm status crypt [ opencount noflush ] [16384] (*1) # Calculated device size is 9767440351 sectors (RW), offset 32768. # DM-UUID is CRYPT-LUKS2-355457dcd03343349b2121f41f3e0a5c-crypt # Udev cookie 0xd4de97d (semid 4) created # Udev cookie 0xd4de97d (semid 4) incremented to 1 # Udev cookie 0xd4de97d (semid 4) incremented to 2 # Udev cookie 0xd4de97d (semid 4) assigned to CREATE task(0) with flags DISABLE_LIBRARY_FALLBACK (0x20) # dm create crypt CRYPT-LUKS2-355457dcd03343349b2121f41f3e0a5c-crypt [ opencount flush ] [16384] (*1) # dm reload (254:3) [ opencount flush securedata ] [16384] (*1) device-mapper: reload ioctl on crypt (254:3) failed: Invalid argument # Udev cookie 0xd4de97d (semid 4) decremented to 1 # Udev cookie 0xd4de97d (semid 4) incremented to 2 # Udev cookie 0xd4de97d (semid 4) assigned to REMOVE task(2) with flags DISABLE_LIBRARY_FALLBACK (0x20) # dm remove crypt [ opencount flush securedata ] [16384] (*1) # Uevent not generated! Calling udev_complete internally to avoid process lock-up. # Udev cookie 0xd4de97d (semid 4) decremented to 1 # dm versions [ opencount flush ] [16384] (*1) # dm status crypt [ opencount noflush ] [16384] (*1) # Udev cookie 0xd4de97d (semid 4) decremented to 0 # Udev cookie 0xd4de97d (semid 4) waiting for zero # Udev cookie 0xd4de97d (semid 4) destroyed # Requesting keyring logon key for revoke and unlink. # Releasing crypt device /dev/sdd1 context. # Releasing device-mapper backend. # Closing read only fd for /dev/sdd1. # Unlocking memory. Command failed with code -4 (wrong device or file specified). 

The output of fdisk -l:

Disk /dev/sdd: 4.55 TiB, 5000947302400 bytes, 9767475200 sectors Disk model: My Passport 2627 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: 2505C284-7B8A-4EAE-90CB-950187A84D57 Device Start End Sectors Size Type /dev/sdd1 2048 9767475166 9767473119 4.5T Linux filesystem 

The luksDump, arrently needed too; output of cryptsetup luksDump /dev/sdd1:

❯ sudo cryptsetup luksDump /dev/sdd1 LUKS header information Version: 2 Epoch: 5 Metadata area: 16384 [bytes] Keyslots area: 16744448 [bytes] UUID: 355457dc-d033-4334-9b21-21f41f3e0a5c Label: (no label) Subsystem: (no subsystem) Flags: (no flags) Data segments: 0: crypt offset: 16777216 [bytes] length: (whole device) cipher: serpent-xts-plain64 sector: 4096 [bytes] Keyslots: 0: luks2 Key: 512 bits Priority: normal Cipher: serpent-xts-plain64 Cipher key: 512 bits PBKDF: argon2id Time cost: 5 Memory: 1048576 Threads: 4 Salt: 67 4b ad d5 89 b5 64 b7 b7 46 61 0f a4 9f cb be 52 90 11 99 8c c0 fb 81 be 6a d6 ac 58 f5 3c 12 AF stripes: 4000 AF hash: sha256 Area offset:290816 [bytes] Area length:258048 [bytes] Digest ID: 0 Tokens: Digests: 0: pbkdf2 Hash: whirlpool Iterations: 68985 Salt: d7 56 5e 8a d3 7c 7a 86 d3 fc b5 f8 d8 1e 6f 8d b3 fd 04 34 e7 08 ab 9a 33 92 2f 08 96 4b ff 74 Digest: ed 9c d5 5f 0e df b3 f3 5b 71 95 09 9d f0 a8 b5 9c a5 02 cb d0 1f f7 7b 52 d2 24 29 ee b2 7b 3f ed bc bd 1d f8 f7 bb 9f f7 c9 68 9b c9 be 86 66 8b 24 5a 3c b7 b2 3e 93 7e d0 42 7c 7e e1 6d ec 

S.M.A.R.T. values output using smartctl -a /dev/sdd:

❯ sudo smartctl -a /dev/sdd smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.8-zen1-1-zen] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Elements / My Passport (USB, AF) Device Model: WDC WD50NDZW-11MR8S1 Serial Number: WD-WXD1E995WRAF LU WWN Device Id: 5 0014ee 211f0443e Firmware Version: 02.01A02 User Capacity: 5,000,947,523,584 bytes [5.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 2.5 inches TRIM Command: Available, deterministic Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-3 (minor revision not indicated) SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Dec 17 16:02:40 2021 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 249) Self-test routine in progress... 90% of test remaining. Total time to complete Offline data collection: ( 2940) seconds. Offline data collection capabilities: (0x1b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. No Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 776) minutes. SCT capabilities: (0x30b5) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 2 3 Spin_Up_Time 0x0027 253 253 021 Pre-fail Always - 4808 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 825 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 1577 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 321 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 176 193 Load_Cycle_Count 0x0032 198 198 000 Old_age Always - 6431 194 Temperature_Celsius 0x0022 119 098 000 Old_age Always - 33 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 1 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] Selective Self-tests/Logging not supported 

And here's the DMESG output (simply dmesg, due to character limit couldn't post everything):

[ 46.940566] wlan0: associated [ 46.989890] wlan0: Limiting TX power to 23 (23 - 0) dBm as advertised by 5c:49:79:56:19:f7 [ 50.007552] usb 2-6: new SuperSpeed USB device number 2 using xhci_hcd [ 50.020426] usb 2-6: New USB device found, idVendor=1058, idProduct=2627, bcdDevice=40.08 [ 50.020439] usb 2-6: New USB device strings: Mfr=2, Product=3, SerialNumber=1 [ 50.020444] usb 2-6: Product: My Passport 2627 [ 50.020448] usb 2-6: Manufacturer: Western Digital [ 50.020452] usb 2-6: SerialNumber: 575844314539393557524146 [ 50.664550] usb-storage 2-6:1.0: USB Mass Storage device detected [ 50.665002] scsi host4: usb-storage 2-6:1.0 [ 50.665220] usbcore: registered new interface driver usb-storage [ 50.676478] usbcore: registered new interface driver uas [ 51.678278] scsi 4:0:0:0: Direct-Access WD My Passport 2627 4008 PQ: 0 ANSI: 6 [ 51.678667] scsi 4:0:0:1: Enclosure WD SES Device 4008 PQ: 0 ANSI: 6 [ 51.682041] sd 4:0:0:0: [sdd] Spinning up disk... [ 51.703600] scsi 4:0:0:1: Wrong diagnostic page; asked for 1 got 8 [ 51.703603] scsi 4:0:0:1: Failed to get diagnostic page 0x1 [ 51.703605] scsi 4:0:0:1: Failed to bind enclosure -19 [ 52.701886] ......ready [ 57.822064] sd 4:0:0:0: [sdd] Very big device. Trying to use READ CAPACITY(16). [ 57.822250] sd 4:0:0:0: [sdd] 9767475200 512-byte logical blocks: (5.00 TB/4.55 TiB) [ 57.822255] sd 4:0:0:0: [sdd] 4096-byte physical blocks [ 57.822540] sd 4:0:0:0: [sdd] Write Protect is off [ 57.822544] sd 4:0:0:0: [sdd] Mode Sense: 47 00 10 08 [ 57.823041] sd 4:0:0:0: [sdd] No Caching mode page found [ 57.823048] sd 4:0:0:0: [sdd] Assuming drive cache: write through [ 57.983930] sdd: sdd1 [ 57.985534] sd 4:0:0:0: [sdd] Attached SCSI disk [ 57.985680] ses 4:0:0:1: Attached Enclosure device [ 137.355239] nvidia-nvlink: Nvlink Core is being initialized, major device number 507 [ 137.355244] NVRM: The NVIDIA probe routine was not called for 1 device(s). [ 137.356116] NVRM: This can occur when a driver such as: NVRM: nouveau, rivafb, nvidiafb or rivatv NVRM: was loaded and obtained ownership of the NVIDIA device(s). [ 137.356117] NVRM: Try unloading the conflicting kernel module (and/or NVRM: reconfigure your kernel without the conflicting NVRM: driver(s)), then try loading the NVIDIA kernel module NVRM: again. [ 137.356118] NVRM: No NVIDIA devices probed. [ 137.356296] nvidia-nvlink: Unregistered the Nvlink Core, major device number 507 [ 317.920451] device-mapper: table: 254:3: crypt: Device size is not multiple of sector_size feature [ 317.920455] device-mapper: ioctl: error adding target to table [ 2685.464145] raid6: skip pq benchmark and using algorithm avx2x4 [ 2685.464148] raid6: using avx2x2 recovery algorithm [ 2685.468011] xor: automatically using best checksumming function avx [ 2685.528254] Btrfs loaded, crc32c=crc32c-intel, zoned=yes, fsverity=yes [ 2685.564424] JFS: nTxBlock = 8192, nTxLock = 65536 [ 2685.582407] NILFS version 2 loaded [ 2685.676402] SGI XFS with ACLs, security attributes, realtime, scrub, repair, quota, no debug enabled [ 2692.757592] sda: sda1 sda2 sda3 sda4 [ 2694.215474] sdd: sdd1 [ 2768.779512] device-mapper: table: 254:3: crypt: Device size is not multiple of sector_size feature [ 2768.779536] device-mapper: ioctl: error adding target to table [ 3123.484363] usb 2-6: USB disconnect, device number 2 [ 4886.654141] usb 2-6: new SuperSpeed USB device number 3 using xhci_hcd [ 4886.667772] usb 2-6: New USB device found, idVendor=1058, idProduct=2627, bcdDevice=40.08 [ 4886.667776] usb 2-6: New USB device strings: Mfr=2, Product=3, SerialNumber=1 [ 4886.667778] usb 2-6: Product: My Passport 2627 [ 4886.667779] usb 2-6: Manufacturer: Western Digital [ 4886.667780] usb 2-6: SerialNumber: 575844314539393557524146 [ 4886.669555] usb-storage 2-6:1.0: USB Mass Storage device detected [ 4886.669800] scsi host4: usb-storage 2-6:1.0 [ 4887.692812] scsi 4:0:0:0: Direct-Access WD My Passport 2627 4008 PQ: 0 ANSI: 6 [ 4887.693055] scsi 4:0:0:1: Enclosure WD SES Device 4008 PQ: 0 ANSI: 6 [ 4887.694634] ses 4:0:0:1: Attached Enclosure device [ 4887.695784] sd 4:0:0:0: [sdd] Spinning up disk... [ 4887.696087] ses 4:0:0:1: Wrong diagnostic page; asked for 1 got 8 [ 4887.696090] ses 4:0:0:1: Failed to get diagnostic page 0x1 [ 4887.696092] ses 4:0:0:1: Failed to bind enclosure -19 [ 4888.716288] ......ready [ 4893.836679] sd 4:0:0:0: [sdd] Very big device. Trying to use READ CAPACITY(16). [ 4893.836793] sd 4:0:0:0: [sdd] 9767475200 512-byte logical blocks: (5.00 TB/4.55 TiB) [ 4893.836795] sd 4:0:0:0: [sdd] 4096-byte physical blocks [ 4893.837071] sd 4:0:0:0: [sdd] Write Protect is off [ 4893.837072] sd 4:0:0:0: [sdd] Mode Sense: 47 00 10 08 [ 4893.837383] sd 4:0:0:0: [sdd] No Caching mode page found [ 4893.837385] sd 4:0:0:0: [sdd] Assuming drive cache: write through [ 4893.996397] sdd: sdd1 [ 4893.997502] sd 4:0:0:0: [sdd] Attached SCSI disk [ 4951.411265] device-mapper: table: 254:3: crypt: Device size is not multiple of sector_size feature [ 4951.411286] device-mapper: ioctl: error adding target to table