skip to Main Content

Unpack packed DOS binaries with DOSBox debugger

Set up the DOSBox debugger

First off, you will need a DOSBox version that is compiled with the built-in debugger. If you are using Windows and don’t want to compile it yourself, you can grab the latest version of the DOSBox debugger from The unpacking target of my choice is Omniscent, one of the most impressive 4kb DOS intros that exist. It uses a custom packer to compress the executable. I wanted to have the unpacked version of this intro and compare how the common DOS executable packer compress this file. This tutorial works for both .com and .exe files.

Start DOSBox but do not run your targeted executable yet. Enter DEBUG followed by the name of your executable that you want to unpack and press Return. It seems like the DOSBox Debugger has a bug that causes the window to not refresh itself after trapping into the program. Click into the Debugger window and press a key like Space to refresh the window. Your screen looks now like this:

DOSBox debugger 1DOSBox debugger 2

The normal DOS executable unpacking stub first copies itself plus the packed data to a higher memory location. It then starts to unpack the data to the address space where to execution started. As a result, the memory location will now contain the unpacked code with the Original Entry Point (OEP) of the real program. After the unpacking is done, the stub jumps back to the address of the first executed instruction and starts the execution of the real program. This behaviour can be used as a quite easy method to unpack the packed executable now.

Debug the target program

Enter the command BP CS:IP to set a breakpoint at the current instruction. Execute the current instruction with the F11 key and press F5 to continue the normal execution. After a short time the debugger breaks again at the starting address 0100. Finally the execution stopped at OEP before the first instruction of the unpacked program executes.

Dump the program at OEP

Enter the command MEMDUMPBIN CS:IP 60000 and press Return. The unpacked program should have been saved into the current directory now. The last parameter of the command is the number of bytes to dump and can be adjusted for larger programs. Your debugger window should look like this now:

DOSBox debugger 3

Rename the file MEMDUMP.BIN to .com or .exe depending on your program type. You can use a hex-editor to remove zero-bytes at the end of the program. Run the renamed executable to see if the unpacking worked. For my unpacking example, the DOS screen is now full of awesomeness:

DOSBox debugger 4

This unpacking method should work on DOS packers such as 624 and UPX.

29 byte DOS intro called fr29b

When going back down memory lane about my programming projects the first memory that comes up is developing software under DOS. The simple way of writing programs as .com files allowed for a lot of fun stuff like coding size competitions. Those executable files did not include any header and got executed as code starting from first byte of the file.

My favorite production of the old days is this 29 byte small intro that I developed 2001 using assembly language. It did fit into the 32 byte intro competition of the 0a000h demo party 2002 and won the first place.

org 100h                        ; tell nasm that this program will be loaded at 0x100 address
    mov     al, 13h
    int     10h                 ; set 320*200 graphics mode with 256 colors
    lds     bp, [bx]

    rdtsc                       ; read time stamp counter
    xchg    ax, di              ; use it as random offset to VGA buffer
    mov     cl, 8ch             ; use a 140*140 box

    mov     bl, 8ch

    inc     byte [bx+di]        ; increase color index in this box
    or      byte [bx+di], 80h   ; ensure upper bit is set
    dec     bx
    jnz     next_pixel
    add     di, 140h            ; add 320 = point to next line
    loop    next_line
    jmp     next_box

If you want to assemble it yourself into a .com file, download nasm, save the code as fr29b.asm and execute this command on a shell:

nasm -f bin -o fr29b.asm

To run DOS programs on a system that doesn’t include a DOS subsystem (Like Windows Version up to Windows XP 32 Bit did) you should get DOSBox. If you run Windows Vista or later, open “Control Panel\Default Programs\Associate a file type or protocol with a program”, scroll to .com file extension and then set the installed DOSBox binary as default program for this file type. Now you are able to execute .com files directly in an emulated and safe environment on your machine.

The algorithm behind the plasma effect is quite simple. At a random offset on the screen i increase the color index of a 140 pixel *140 pixel box that starts at this random offset. You can increase or decrease the size of the box and you will get a slightly different looking effect, but in my tests I preferred the one using 140. If the random offsets starts near a line end, drawing over the end of the line will draw on the beginning of the start of the next line. Looking at the default VGA palette you will notice that the first 16 (CGA compatible) colors don’t fade nicely into each other when we just increase the palette index in memory for those. To circumvent this problem i make sure that the highest bit of each pixel (which represents an index to the VGA palette) is always 1, so that increasing random pixel will just rotate the index to the palette in the range of 128 and 255.



If you are unfamiliar with the lds trick at third code line, let me explain it: The lds instruction will load a pointer into ds and the register specified as argument. On startup of a dos program the cpu register bx is 0, therefore the instruction “lds bp, [bx]” will load the 2 bytes at memory address 0x0000 into bp and the following 2 bytes at address 0x0002 into ds. The com file will get loaded at offset 0x0100 by the program loader, the 0x100 bytes in front of that memory contains the Program Segment Prefix.

Looking at the meaning of the first values you can see that bx will contain the opcode of the int 20h instruction and ds the memory size in paragraphs. The memory size in paragraphs is usually 0x9fff and only 1 number and therefor 16 byte off (1 paragraph represents 16 byte) of our desired 0xa000 address which represents the VGA memory segment.

Back To Top