First off, you will need a DOSBox version that is compiled with the built-in debugger. If you are using Windows and don't want to compile it yourself, you can grab the latest version of the DOSBox debugger from http://www.vogons.org/viewtopic.php?t=7323. The unpacking target of my choice is Omniscent, one of the most impressive 4kb DOS intros that exist. It uses a custom packer to compress the executable. I wanted to have the unpacked version of this intro and compare how the common DOS executable packer compress this file. This tutorial works for both
Start DOSBox but do not run your targeted executable yet. Enter
DEBUG followed by the name of your executable that you want to unpack and press
Return. It seems like the DOSBox Debugger has a bug that causes the window to not refresh itself after trapping into the program. Click into the Debugger window and press a key like Space to refresh the window. Your screen looks now like this:
The normal DOS executable unpacking stub first copies itself plus the packed data to a higher memory location. It then starts to unpack the data to the address space where to execution started. As a result, the memory location will now contain the unpacked code with the Original Entry Point (OEP) of the real program. After the unpacking is done, the stub jumps back to the address of the first executed instruction and starts the execution of the real program. This behaviour can be used as a quite easy method to unpack the packed executable now.
Enter the command
BP CS:IP to set a breakpoint at the current instruction. Execute the current instruction with the
F11 key and press
F5 to continue the normal execution. After a short time the debugger breaks again at the starting address
0100. Finally the execution stopped at OEP before the first instruction of the unpacked program executes.
Enter the command MEMDUMPBIN CS:IP 60000 and press Return. The unpacked program should have been saved into the current directory now. The last parameter of the command is the number of bytes to dump and can be adjusted for larger programs. Your debugger window should look like this now:
Rename the file
.exe depending on your program type. You can use a hex-editor to remove zero-bytes at the end of the program. Run the renamed executable to see if the unpacking worked. For my unpacking example, the DOS screen is now full of awesomeness:
This unpacking method should work on DOS packers such as 624 and UPX.