nezumi-lab
February 9th, 2009, 13:17
once upon a time was MS-DOS and ancient debuggers like Turbo-Debugger, Soft-Ice and many others. and there were anti-debug tricks. one of them was based on self-overwritten REP STOS/MOVS instruction. it worked great against all existing debuggers, including CUP 386 ("ftp://ftp.elf.stuba.sk/pub/pc/pack/ucfcup34.zip") (exe unpacker with build-in CPU emulator).
I used this tricks for years. I would almost forget about if Silviocesare not posted “Anti-debugging prefetch tricks and single stepping through a rep stos/movs” ("http://silviocesare.wordpress.com/2009/02/02/anti-debugging-prefetch-tricks-and-single-stepping-through-a-rep-stosmovs/") article on his blog (very nice blog, btw).
I was interested: what’s about modern debuggers? what’s about emulators like BOCHS? what’s about IDA-Pro 5.4 with BOCHS-based debugger ("http://hexblog.com/2008/11/bochs_plugin_goes_alpha.html")? imagine, how surprised I was when I found out that IDA-Pro 5.3, Olly 1.10 and Soft-Ice not only can be detected this way, but also lost the control during step over tracing! debugged code just escapes out of the debugger!!! IDA-Pro 5.4 with BOCHS module fails to emulate the self-overwritten REP STOS/MOVS instruction (so it can be detected as well) and lost control on Step Over tracing. only Olly 2.00i recognizes attempts to espace and blocks them, however it can be detected the same way.
for testing reasons I wrote a simple program with self-overwritten REP STOSB command (see source bellow).
now, run the program without tracing. CPU pipelines REP STOSB and executes it until ECX > 0. REP STOSB modifies only data cache, while the instruction is executed on the pipeline and CPU does not recognizes modification of the code, so REP loop finishes with ECX = 0.
Olly 1.10/200i and Soft-Ice also fails to trace self-overwritten REP STOSB instruction. of course, if we trace the program with our hands, it’s easy to set a breakpoint _after_ it and run the code without tracing, but!!! many plug-ins use trace engine for their needs, so the trace engine should work fine and it’s possible to fix debuggers - just before executing REP STOS/MOVS we have to perform some checks and if the command overwrites itself we either set a breakpoint either emulate CPU behavior.
ok, run the program under BOCHS (IDA-Pro 5.4 support a special plug-in, allowing us to debug code on the fly). regardless of whether we trace program or not, the REP loop finishes with ECX = 1. well-known x86-emu plug-in gives us the same result, and this result is definitely wrong.
by the way, did I hear a question: how long CPU is executing overwritten REP STOS/MOVS command? there is no universal answer. it depends on CPU internal behavior and external evens like interrupts. when an interrupt is generated, CPU stops executing overwritten REP STOS/MOVS. a good way to create a pseudo-random generator! I’m going to write about it in the next post.
meanwhile, some CPU have a bug. they executes CLD commands _before_ overwritten REP STOS/MOVS will be stopped or finishes. as result, REP STOS/MOVS changes the direction and hits the memory not supposed to be written. I’m investigating this case now, will publish the result soon.
well, let’s return to our muttons. load the program into IDA-Pro 5.3/5.4 and perform Step Over tracing. move cursor to REP STOSB, press F8 and… the debugger lost the control!!! why? the answer is: to gain control back after REP STOSB command IDA-Pro sets a software breakpoint on the next command. we all know that a software breakpoint it’s just INT 03 (CCh) instruction and in our case this instruction is overwritten by REP STOSB. thus the breakpoint is wiped out and the process is executed until another breakpoint will be triggered. if there is no other breakpoints - the debugged code escapes out of the debugger!
what’s about Olly 1.10 and Soft-Ice?! fast check shows us like they do not lost the control and stop after REP STOSB. but… how they do it? well, they just use hardware breakpoints!!! and if all four hardware breakpoints are in use, the debuggers set a software breakpoint like IDA-Pro does, so they lost control as well. not good!
Olly 2.00i (didn’t check other versions) is the only debugger who is able to detect that the breakpoint was wiped out. if it happens we have a warning message. impressive! Olly 2.00 is a great debugger doubtless!!!
http://nezumi-lab.org/blog/?p=120
I used this tricks for years. I would almost forget about if Silviocesare not posted “Anti-debugging prefetch tricks and single stepping through a rep stos/movs” ("http://silviocesare.wordpress.com/2009/02/02/anti-debugging-prefetch-tricks-and-single-stepping-through-a-rep-stosmovs/") article on his blog (very nice blog, btw).
I was interested: what’s about modern debuggers? what’s about emulators like BOCHS? what’s about IDA-Pro 5.4 with BOCHS-based debugger ("http://hexblog.com/2008/11/bochs_plugin_goes_alpha.html")? imagine, how surprised I was when I found out that IDA-Pro 5.3, Olly 1.10 and Soft-Ice not only can be detected this way, but also lost the control during step over tracing! debugged code just escapes out of the debugger!!! IDA-Pro 5.4 with BOCHS module fails to emulate the self-overwritten REP STOS/MOVS instruction (so it can be detected as well) and lost control on Step Over tracing. only Olly 2.00i recognizes attempts to espace and blocks them, however it can be detected the same way.
for testing reasons I wrote a simple program with self-overwritten REP STOSB command (see source bellow).
stddownload ("http://nezumi-lab.org/ffh/rep_stos.zip") it, load into IDA-Pro 5.3 and start tracing REP STOSB instruction (F7 hot key). what do we see? REP STOSB changes NOP to INC EBX, overwrites four commands and overwrites itself (STOSB). since, during tracing CPU generates a single step exception every iteration, REP STOSB becomes REP INC EBX and as far as we all know, REP works only with string commands, so REP INC EBX is not executed and REP loop finishes with ECX = 1.
xor ebx, ebx
mov al, 43h ; // INC EBX
mov edi, offset end_of
mov ecx, 6
REP STOSB
NOP
NOP
NOP
end_of: NOP
cld
now, run the program without tracing. CPU pipelines REP STOSB and executes it until ECX > 0. REP STOSB modifies only data cache, while the instruction is executed on the pipeline and CPU does not recognizes modification of the code, so REP loop finishes with ECX = 0.
Olly 1.10/200i and Soft-Ice also fails to trace self-overwritten REP STOSB instruction. of course, if we trace the program with our hands, it’s easy to set a breakpoint _after_ it and run the code without tracing, but!!! many plug-ins use trace engine for their needs, so the trace engine should work fine and it’s possible to fix debuggers - just before executing REP STOS/MOVS we have to perform some checks and if the command overwrites itself we either set a breakpoint either emulate CPU behavior.
ok, run the program under BOCHS (IDA-Pro 5.4 support a special plug-in, allowing us to debug code on the fly). regardless of whether we trace program or not, the REP loop finishes with ECX = 1. well-known x86-emu plug-in gives us the same result, and this result is definitely wrong.
by the way, did I hear a question: how long CPU is executing overwritten REP STOS/MOVS command? there is no universal answer. it depends on CPU internal behavior and external evens like interrupts. when an interrupt is generated, CPU stops executing overwritten REP STOS/MOVS. a good way to create a pseudo-random generator! I’m going to write about it in the next post.
meanwhile, some CPU have a bug. they executes CLD commands _before_ overwritten REP STOS/MOVS will be stopped or finishes. as result, REP STOS/MOVS changes the direction and hits the memory not supposed to be written. I’m investigating this case now, will publish the result soon.
well, let’s return to our muttons. load the program into IDA-Pro 5.3/5.4 and perform Step Over tracing. move cursor to REP STOSB, press F8 and… the debugger lost the control!!! why? the answer is: to gain control back after REP STOSB command IDA-Pro sets a software breakpoint on the next command. we all know that a software breakpoint it’s just INT 03 (CCh) instruction and in our case this instruction is overwritten by REP STOSB. thus the breakpoint is wiped out and the process is executed until another breakpoint will be triggered. if there is no other breakpoints - the debugged code escapes out of the debugger!
what’s about Olly 1.10 and Soft-Ice?! fast check shows us like they do not lost the control and stop after REP STOSB. but… how they do it? well, they just use hardware breakpoints!!! and if all four hardware breakpoints are in use, the debuggers set a software breakpoint like IDA-Pro does, so they lost control as well. not good!
Olly 2.00i (didn’t check other versions) is the only debugger who is able to detect that the breakpoint was wiped out. if it happens we have a warning message. impressive! Olly 2.00 is a great debugger doubtless!!!
http://nezumi-lab.org/blog/?p=120