I am trying to load a DLL in IDA and OllyDbg, but it says the file is compressed. I checked with a couple of tools and came to know that the file is compressed with PECompact. I think it also has reverse-engineer protection enabled because debuggers are not able to load it but other tools can disassemble it.
So, my question is how do I unpack the DLL and start debugging ?
There exist unpackers for PECompact.
Try unpecompact2, I think there are two versions. http://www.woodmann.com/collaborative/tools/index.php/UnPECompact2_(MadMickael_version) http://www.woodmann.com/collaborative/tools/index.php/UnPECompact2_(smola_version)
If those doesn't work, you can try generic unpacker such as RL!dePacker, linxerUnpacker, Quick Unpack
As I never worked with PECompact before I downloaded a few unpackmes and tested it. It looks like a very easy system to unpack.
The script PeCompact 2.xx - 3.xx OEP Finder by atom0s found the OEP correctly for every unpackme I tested. Please note that you will need to ignore a couple of exceptions (INT3 breaks and Memory access violations), but the anti-debugging will be defeated by the DBH instruction in ODBGScript.
For some of the unpackmes I found a simple API redirection was also present: each reference to the KERNEL32 API was redirected to a couple of instructions where the correct value was loaded in EAX followed by a JMP EAX instruction (see image):
For such cases, I wrote this small script to solve the redirection (You will need to set the hard-coded value to the address of the first wrong entry):
var oep var next mov oep, eip mov next, 00460974 // FIXME FIX: mov eip, [next] sto mov [next], eax add next, 4 cmp [next], 0 jne FIX mov eip, oep ret Once this second script is completed you can dump the file and fix the IAT as usual.
Also, it is not clear in your post if your problems are due to the fact that you are working on a DLL instead of an EXE file. If that's the case, you can either use the LOADDLL tool or modify the PE header to load the DLL as an EXE (Here is an example of how to do it on LordPE).
