11

In what I now know feel was a stupid decision, I attempted to dual boot Windows and linux by using the windows installer, After booting into the windows installer I chose one of a 2 cloned hard drives about 500GB in size to erase, as they were cloned it wouldn't matter if I picked one over the other.

After doing so, the installer said it changed the partition table for one of the 500GB hard drives and then failed to install windows. Crashing with an error without copying files, or so it says, I am unsure if I can trust it when it says it failed to even begin installing.

So I booted into my linux install to check which drive it had overwritten and manually install it. What greeted me instead was one of my other drives, a 6TB dm-luks and btrfs drive, missing. Not only were both 500GB drives untouched, but the 6TB drive had seemingly a mess of partitions added to it. 6 partitions in the order of 499M, 99M, 499M, 100M, 499M, 100M.

As my drive is quite large and additionally slow, running hexdump -C /dev/sda |grep LUKS has so far produced this much, I will update when it finishes:

8d411ce0 e1 ad 4c 55 4b 53 c0 85 22 3d de 49 dd 44 fd 08 |..LUKS.."=.I.D..| e6449610 d5 cf 4a 86 9f cc 4c 55 4b 53 a9 a9 16 cc ba 1d |..J...LUKS......| 446ea9a70 b3 db a9 bf 8b 2e 41 4c 55 4b 53 ef f0 75 b0 18 |......ALUKS..u..| 4732c6040 e0 b3 bb ff 4c 55 4b 53 4c c2 5b 12 c6 41 fc d6 |....LUKSL.[..A..|

So far the only thing that has even touched the disk since this has happened is hexdump, I am hesitant to run testdisk as I have heard it overwrites data on the drive and it does not list luks as something it can search for.

I can see others have used hexdump to check for intact headers, however, I do not know what exactly I am looking for.

What can I do at this point to see if I can recover any bit of the header. Is there a way to run testdisk or another tool to look for luks headers in order to tell if they've been overwritten? Any way that allows me to know whether everything is FUBAR or not is equally as welcome as a way to recover my data.

EDIT

Running hexdump on the first bit of the drive without grep shows that there's at least some intact JSON, from 00005000 to 00005310 shows as much, I am even less sure what I am specifically looking for now to know if its still intact. It seems it overwrote data up until this exact string.

00005000 7b 22 6b 65 79 73 6c 6f 74 73 22 3a 7b 22 30 22 |{"keyslots":{"0"| 00005010 3a 7b 22 74 79 70 65 22 3a 22 6c 75 6b 73 32 22 |:{"type":"luks2"|

Cutting out the data in-between because it includes the salt, but the block ends in:

000052f0 22 2c 22 6b 65 79 73 6c 6f 74 73 5f 73 69 7a 65 |","keyslots_size| 00005300 22 3a 22 31 36 37 34 34 34 34 38 22 7d 7d 00 00 |":"16744448"}}..| 00005310 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| *

It it intact enough?

Improve this question
8
  • 2
    | grep LUKS does not help for overwritten headers. This might be your only chance: cryptsetup repair (only for LUKS 2) but you have to know the correct offset and find one of LUKS JSON metadata strings first. Damaged key material cannot be recovered. Unfortunately in a lot of these cases even a slightly damaged LUKS header simply means "game over". Commented Mar 30, 2023 at 10:47
  • 2
    Was the drive partitioned before? If not then the LUKS header would simply sit at offset 0 (and die as soon as anything creates a partition table) and you could check hexdump -C in full (no grep). If it was partitioned, was it a single partition? That would usually be offset 1 MiB these days. Scanning the entire drive for the offset is only useful if you had multiple partitions and LUKS was only one of them at an unknown offset. You can also use strings -t d -n 4 /dev/disk | less and see if any LUKS related strings pop up (aes-xts etc.) for LUKS 2 this would show JSON metadata strings Commented Mar 30, 2023 at 10:57
  • @frostschutz It was treated as a single device, I'm not sure which of the two it was, but you gave me the perfect command to run, because it gave me some hope. I can see an entire block of luks JSON! Im not sure if its fully intact, it looks like just before the block it might have gotten overwritten, How do I tell if its fully intact? It starts with this 00005000 7b 22 6b 65 79 73 6c 6f 74 73 22 3a 7b 22 30 22 |{"keyslots":{"0"| Commented Mar 30, 2023 at 20:22
  • 1
    then try and see if the repair method I linked works for you Commented Mar 30, 2023 at 20:33
  • 1
    Slight confusion here. The JSON you posted above should be the backup header. The key material on the other hand exists only once. Can't say more since I don't know what your header looks like. You need either primary or backup header to be intact. And then you also need intact key material (there is no backup of that). Commented Mar 30, 2023 at 22:33

1 Answer 1

Reset to default
19
+250

cryptsetup repair, Part Two — Full Header Recovery

In order to recover a partially overwritten LUKS2 header, you need at minimum two things: One, the key material of at least one of your keyslots. Two, the metadata describing how that key material is to be used (algorithm, iteration counts, salts, etc.).

The key material is roughly 256KB of random data, usually found at offset 32768 (0x8000) and beyond, however the exact offset and size has to be determined from metadata.

The metadata is a JSON string, usually found at offsets 4096 (0x1000) and 20480 (0x5000). LUKS2 maintains two identical copies of it (primary and secondary header). The key material itself exists only once.

If the partition table itself is also lost, you'll also have to determine the correct partition offset.

For demonstration purposes, this answer first creates, then damages, then recovers a LUKS2 partition in a sample disk.img file.

Create a sample disk.img file: (do not run this on your real drive!)

# truncate -s 128M disk.img # losetup --find --show --partscan disk.img /dev/loop0 # parted /dev/loop0 -- mklabel gpt # parted /dev/loop0 -- mkpart luks $((RANDOM%100))MiB 100% # cryptsetup luksFormat --type luks2 /dev/loop0p1 WARNING! ======== This will overwrite data on /dev/loop0p1 irrevocably. Are you sure? (Type 'yes' in capital letters): YES Enter passphrase for /dev/loop0p1: Verify passphrase: # cryptsetup open /dev/loop0p1 luks Enter passphrase for /dev/loop0p1: # mkfs.ext2 -L encrypted /dev/mapper/luks # blkid /dev/mapper/luks /dev/mapper/luks: LABEL="encrypted" […] TYPE="ext2"

A shiny new encrypted filesystem!

Damage the sample disk.img file: (do not run this on your real drive!)

# cryptsetup close luks # wipefs -a /dev/loop0p1 /dev/loop0p1: 6 bytes were erased at offset 0x00000000 (crypto_LUKS): 4c 55 4b 53 ba be /dev/loop0p1: 6 bytes were erased at offset 0x00004000 (crypto_LUKS): 53 4b 55 4c ba be # dd count=32 if=/dev/urandom of=/dev/loop0p1 32+0 records in 32+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.000334077 s, 49.0 MB/s # wipefs -a /dev/loop0 /dev/loop0: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54 /dev/loop0: 8 bytes were erased at offset 0x063ffe00 (gpt): 45 46 49 20 50 41 52 54 /dev/loop0: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa /dev/loop0: calling ioctl to re-read partition table: Success # losetup -d /dev/loop0

So this is a sample disk.img with a LUKS2 partition at an unknown offset, with a damaged header on it (magic bytes erased, partially overwritten, partition table wiped).

Metadata recovery:

Since the LUKS2 JSON string is plain ASCII, it can be found with strings, which will also show the offset:

# stdbuf -oL strings -n 64 -t d disk.img | grep '"keyslots":' 60837888 {"keyslots":{"0":{"type":"luks2","key_size":64,"af":{"type":"luks1","stripes":4000,"hash":"sha256"},"area":{"type":"raw","offset":"32768","size":"258048","encryption":"aes-xts-plain64","key_size":64},"kdf":{"type":"argon2id","time":13,"memory":1048576,"cpus":4,"salt":"R1z3arzSCjRb3STaCAnstIygkHCXf0CHf6kXl5yQj/E="}}},"tokens":{},"segments":{"0":{"type":"crypt","offset":"16777216","size":"dynamic","iv_tweak":"0","encryption":"aes-xts-plain64","sector_size":512}},"digests":{"0":{"type":"pbkdf2","keyslots":["0"],"segments":["0"],"hash":"sha256","iterations":324435,"salt":"0nSkpvmDJlvfkDaQteVVo6JdD/Oqt3vnndkZt1Qnd84=","digest":"lefQ21EaiuSdHFhSIFW3wDfMcRqG0HLCAO1bGI3SfvM="}},"config":{"json_size":"12288","keyslots_size":"16744448"}}

So here we have an intact JSON string at offset 60837888. Copy&Paste it into a header.json file. The file should start with { and end with }. You can use jq to make sure it's indeed a valid JSON string, and also to show it in a more human-readable form:

# jq < header.json { "keyslots": { "0": { "type": "luks2", […] } }

Partition recovery:

The offset of the JSON metadata within the LUKS2 header is usually either 4096 or 20480, depending on whether it's the primary or secondary header. You have to substract these values from the offset that strings found earlier.

Thus, the correct partition offset in this case could be either 60837888 - 4096 = 60833792 = 58.02MiB or 60837888 - 20480 = 60817408 = 58 MiB. Since the latter is MiB aligned, it's the more likely candidate for the correct partition offset.

If in doubt, try both.

Key material recovery:

According to the JSON metadata, this LUKS2 header has a single keyslot and its key material is to be found at "offset":"32768","size":"258048". Let's grab it with dd:

# partition=60817408 # offset=32768 # size=258048 # dd bs=1 skip=$((partition+offset)) count=$((size)) if=disk.img of=header.$((offset))

If there are multiple keyslots, repeat this process for each of them.

The key material is supposed to look like random data. To verify this, you could look at the whole thing with hexdump -C.

# hexdump -C header.32768 00000000 f1 3b 23 73 98 d7 8f e3 22 24 9a 9d 5a 2c a9 ae |.;#s...."$..Z,..| 00000010 95 82 3e c6 df e7 0e a0 f4 ba 54 6c 7f e9 fa f6 |..>.......Tl....| 00000020 b7 12 64 8d 7d a5 ca 4b c8 89 89 08 3e de 59 0d |..d.}..K....>.Y.| […] 0003efe0 b2 b3 bc cd de 60 17 a7 57 bb 1a 84 5a 15 68 95 |.....`..W...Z.h.| 0003eff0 7f 1f 07 ee ee d1 e8 a2 6c cf 5f 40 0b 73 00 0b |[email protected]..| 0003f000

Or you could try to compress it and see if the compressed result is any smaller:

# gzip < header.32768 > header.32768.gz # stat -c %s header.* 258048 258106

Random data usually can't be compressed at all, so if the gzipped version isn't any smaller (or even a few bytes larger), there's a good chance that the whole thing is random data.

True verification is only possible at the very end — when it accepts your passphrase, or not.

Full Header Recovery:

After you've collected the necessary ingredients above, you can attempt rebuilding a full header from them:

# truncate -s 16M luks.recovery # cryptsetup luksFormat --type luks2 luks.recovery # cryptsetup luksErase luks.recovery

Use cryptsetup produce a valid, albeit unusable header without keyslots. The purpose here is to obtain a file that is set up with all the correct magic bytes, UUIDs, etc. — unrelated to encryption, but it's what makes a LUKS header a LUKS header.

Now transplant your metadata onto it:

# printf "%s\0" "$(jq -c < header.json)" | dd conv=notrunc bs=1 seek=4096 of=luks.recovery # printf "%s\0" "$(jq -c < header.json)" | dd conv=notrunc bs=1 seek=20480 of=luks.recovery

As well as the key material:

# dd conv=notrunc bs=1 seek=32768 if=header.32768 of=luks.recovery

At this point, we'd finally be done, except:

# cryptsetup luksDump luks.recovery Device luks.recovery is not a valid LUKS device. # cryptsetup repair luks.recovery Device luks.recovery is not a valid LUKS device.

Checksum recovery:

Ehhh, what's wrong now? Add --debug to find out:

# cryptsetup luksDump --debug luks.recovery […] # LUKS2 header version 2 of size 16384 bytes, checksum sha256. # Checksum:5babf58f0f788911897989ff3d9a580de1c22db8869b3b08cd0d6d56906005cb (on-disk) # Checksum:b2ff5dd7b53978723402103ba914ed87ef2c5b5d9a9062d68363e4df38aebf6f (in-memory) […]

LUKS2 has a checksum for its primary and secondary headers. Since we modified the JSON metadata without updating the checksum, it's a mismatch. Thankfully, cryptsetup shows the expected value so we don't have to calculate it manually.

This checksum is part of the binary header, so you have to use xxd -r -p to convert it to binary:

# echo 5babf58f0f788911897989ff3d9a580de1c22db8869b3b08cd0d6d56906005cb | xxd -r -p | hexdump -C 00000000 5b ab f5 8f 0f 78 89 11 89 79 89 ff 3d 9a 58 0d |[....x...y..=.X.| 00000010 e1 c2 2d b8 86 9b 3b 08 cd 0d 6d 56 90 60 05 cb |..-...;...mV.`..| 00000020 # echo b2ff5dd7b53978723402103ba914ed87ef2c5b5d9a9062d68363e4df38aebf6f | xxd -r -p | hexdump -C 00000000 b2 ff 5d d7 b5 39 78 72 34 02 10 3b a9 14 ed 87 |..]..9xr4..;....| 00000010 ef 2c 5b 5d 9a 90 62 d6 83 63 e4 df 38 ae bf 6f |.,[]..b..c..8..o| 00000020

Replace the wrong on-disk checksum with the correct in-memory checksum:

# hexdump -C luks.recovery | grep '5b ab f5 8f 0f 78 89 11' 000001c0 5b ab f5 8f 0f 78 89 11 89 79 89 ff 3d 9a 58 0d |[....x...y..=.X.| # echo b2ff5dd7b53978723402103ba914ed87ef2c5b5d9a9062d68363e4df38aebf6f | xxd -r -p | dd conv=notrunc bs=1 seek=$((0x000001c0)) of=luks.recovery

And that should allow things to proceed.

# cryptsetup repair luks.recovery # cryptsetup luksDump luks.recovery LUKS header information Version: 2 […]

Wrapping things up:

# losetup --find --show --read-only --offset 60817408 disk.img /dev/loop0 # cryptsetup open --read-only --header luks.recovery /dev/loop0 luksrecovery Enter passphrase for /dev/loop0: # blkid /dev/mapper/luksrecovery /dev/mapper/luksrecovery: LABEL="encrypted" […] TYPE="ext2"

Done. Finally.

Improve this answer
4
  • 9
    An Incredibly intricate response to a seemingly hopeless situation, and it works flawlessly. This is much more than I could gleam from the dm-luks documentation myself, absolutely outstanding dive into it. My data is back, and headers finally backed up. Commented Apr 4, 2023 at 1:27
  • I got stuck... You make a single mention of multiple keyslots, but don't fully explain at further steps what to do, so I had to do some trial and error. I got to the end, cryptsetup open seemed to work, but blkid seemed to have no output. If it worked, I'd guess next step would be cryptsetup luksHeaderRestore /dev/DEVICE --header-backup-file /path/to/luks.recovery, but I was afraid to run that without mounting first, and couldn't figure out how to mount /dev/mapper/luksrecovery. Commented Nov 27, 2024 at 9:04
  • You my dear Sir are a SAINT. Had a NDE for a while there... Happy NEXT Year! Commented Dec 27, 2024 at 2:15
  • The key material description is inaccurate here. Its size should be key_size * af_stripes, so 64 * 4000 = 256000 for a 512bit/64byte key, 32 * 4000 = 128000 for a 256bit key, etc. cryptsetup then aligns this value to 4K boundaries, so the last 2048 bytes / 3072 bytes of reserved keymaterial area are actually unused. In very old LUKS headers, before luksFormat wiped the entire area with random data, it is normal to have some non-random bytes appear at the end. For LUKS2 headers this would only happen if it was converted from LUKS1. Might edit it later… Commented Feb 10 at 11:06

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.