First off, you will need a debugger enabled version of DOSBox. If you are using Windows and don’t want to compile it yourself, you can grab the latest version of the DOSBox debugger from http://vogons.zetafleet.com/viewtopic.php?t=7323. The unpacking target of my choice is Omniscent, one of the most impressive 4kb DOS intros that exist. It uses a custom packer to compress the executable. I want to have the unpacked version of this intro to see how the common DOS executable packer compress this file. This tutorial works for both .com and .exe files.
Start the debugger enabled version of DOSBox but don’t run your targeted executable that you want to debug yet. Enter DEBUG followed by the name of your executable that you want to unpack and press Return. Click into the Debugger window and press a key like Space to make the window refresh, it seems like a bug to me that the window doesn’t refresh itself after trapping into the debugger. Your screen will look like this:
Usual DOS executable unpacking stubs will copy itself to a higher memory including the packed data and then start to unpack this data to the address space where to execution started. After it is done unpacking the stub will jump to the address of the first executed instruction, which was the first instruction of the unpacker stub mover and now hows become the first instruction of the unpacked executable. We can use this behaviour for a quite easy method to unpack the packed executable now.
Enter the command BP CS:IP to set a breakpoint at the current instruction. Press F11 to execute the current instruction, this step is needed to make sure that DOSBox internally sets the breakpoint in the right way. Press F5 to continue normal execution, after a short time the debugger should break again at our starting address. Now you are in the state right before the first instruction of the unpacked program is executed. Enter the command MEMDUMPBIN CS:IP 60000 and press Return. The unpacked program should have been saved into the current directory now, the last parameter of the command was the number of bytes to dump, you may have to adjust this value for bigger programs. Your debugger window should look like this now:
Rename the file MEMDUMP.BIN to .com or .exe depending on your program. You can use a hex-editor to remove bytes after the end of your program, in my case I know that the unpacked binary size is 4.782 byte. If I would not have known that I would have truncated the file before I see a first start of a memory area that contains a lot 0 bytes. Run the renamed executable to see if the unpacking has worked. In my case the DOS screen is full of awesomeness: