TL;DR
I have an encrypted image and the cleartext version, and I'm almost certain it's an XOR cipher, but I can't figure out how to decrypt it. I have a few clues listed below.
- Ciphertext excerpt (20kB)
- Cleartext excerpt (20kB)
Detail
I'm trying to reverse engineer the firmware for a pro audio device, and I've got most of the way but I'm stuck on a step in the middle.
I'm pretty sure it's an XOR cipher (as there was one XOR decryption already to get to this point, and there is another one after). It seems to be applied in blocks of 0x1000 bytes, as the key seems to reset after this many bytes. The initial key seems to be 2 and it is incremented by 1 every 0x2000 bytes. At the start of each block, the current 'initial key' is XOR'd with 0x4002 to produce the real key. Thus the keys reset to these values at these offsets:
Offset | Key -------+-------------------- 0x0000 | 2 ^ 0x4002 = 0x4000 0x1000 | 2 ^ 0x4002 = 0x4000 0x2000 | 3 ^ 0x4002 = 0x4001 0x3000 | 3 ^ 0x4002 = 0x4001 0x4000 | 4 ^ 0x4002 = 0x4006 The cipher seems to work in units/words of 16-bits, little endian. After each word has been XOR'd with the key, the key is bit-shifted right by 1.
However this is where I have come unstuck. Sometimes, apparently at random, the key needs to be XOR'd by 0x4002. I cannot see the pattern as to when this happens. Initially it looks like it happens every four bytes (two words) but this falls apart at offset 0x22.
I tried XOR'ing the ciphertext and cleartext to produce a long XOR keyfile, and then tried to work out how to reproduce that keyfile. By XOR'ing my guess against the large keyfile, it should end up as 0x00 bytes if I am correct. However it often ends up as 0x02 0x40 bytes, and by writing code that detected these mistakes and adjusted the key accordingly, I was able to see where and when the key needs to be XOR'd by 0x4002 in order to decrypt the data correctly. I've been staring at it for a few hours now, but it just looks random to me, I can't spot any pattern!
For example, in the block starting at offset 0, the key needs to be XOR'd with 0x4002 at offsets 0, 4, 8, 12 (every four bytes) but then at offset 20 it drops back to every two bytes (20, 22, 24, -not 26-, 28, 2A, 2A, -not 2E-, etc.) I can't see the pattern as to when this extra XOR on the key should happen. It doesn't seem to be related to the previous value, the offset, the key value, etc. (of course it probably is I just can't see it.)
Background
The source data has come from a firmware file from the manufacturer, which I have decoded from MIDI SysEx events and decrypted with an XOR cipher. I know this cipher is correct as it produces a correct header and footer, along with block numbers every 256 bytes. After processing these I end up with a source image that is exactly the same size as the firmware dump taken direct from the device's ROM chip with a Minipro EEPROM reader.
The XOR cipher I am struggling with is the last step in going from the manufacturer's firmware file to what gets flashed into the ROM chip (and vice versa, as programming a custom ROM is my ultimate goal).
Note that what does get flashed into the ROM chip (what I am referring to here as the cleartext) is itself further XOR encrypted, however I have successfully decrypted that so it is no issue here (and this in-flash encryption is required as the bootloader expects it.)
If anyone is able to assist me in figuring out this algorithm, I will be implementing it and releasing it as open source, as part of a project I am working on.
Disassembly
Further to the question from @Igor Skochinsky, I haven't yet tried to disassemble the bootloader as I'm not sure how to go about it. Here's the bootloader code, which may contain a 10-byte header according to the Blackfin ADSP-BF531 datasheet.
To avoid confusion if anyone is able to disassemble the code, the XOR encryption I am asking about is only applied during the flash ROM programming stage. A block of data comes in as 7-bit data via the MIDI port (function number 0x34 "write flash block"), is decoded into 8-bit data, an unrelated XOR decrypt with the key TZ'04 is applied (the key is loaded from offset 0x2C84), the CRC is checked, the three-byte header is stripped and the remaining 256-byte block is (presumably) stored in RAM.
The manual says the device only acknowledges the write after the 16th 256-byte block, which is 4 kB of data. The XOR algorithm in question resets the key every 4 kB, so it would appear that the device collects the 256-byte blocks until it reaches 4 kB and applies the unknown XOR cipher at that point, after which it writes the resulting cleartext onto the flash chip (which is an SST39SF040, identified as chip ID 0xBFB7).
Unrelated to this (but to help guide any disassembly), the bootloader also loads data from the flash chip during a normal boot and then XOR decodes it with an 'application key'. This runtime XOR is solved as follows:
- Bootloader XOR key loaded from offset 0x3002, length 0x38.
- Bootloader ciphertext read from offset 0x303A, length 0x38.
- Bootloader XOR and ciphertext applied to produce the 'application key'.
- Flash memory is read from offset 0x4000, application key used for XOR, result stored in memory ready to execute.