0

Alright, so I'm trying to learn reverse engineering and x64 assembly. As an example I wrote this little test program in C:

#include <stdio.h>

int square(int num) {
        return num * num;
}

int main() {
        int ans = square(5);
        printf("%d", ans);
}

This results in the following assembly code for the square function:

push   rbp
mov    rbp,rsp
mov    DWORD PTR [rbp-0x4],edi
mov    eax,DWORD PTR [rbp-0x4]
imul   eax,eax
pop    rbp
ret

This seems kind of weird to me, as edi is the function argument, which I believe is then stored on the stack and loaded back into eax, where it is then multiplied? Why not skip lines 3 and 4 and just imul edi, edi?
So I opened up radare2 and did just that, but now the program returns seemingly random numbers, I'm guessing memory addresses?

radare2 screenshot + patched binary running

Can someone explain to me why GCC uses this seemingly redundant register, and what I did wrong trying to patch the binary?

1 Answers1

2

You didn't enable optimizations, so you're looking at debug code. With -O3, the output is:

enter image description here

One note about your question, you need to assign to eax, because eax stores the return value in an x86 application.

Blindy
  • 65,249
  • 10
  • 91
  • 131
  • Right, that makes sense. And I found out what the problem was with my patch: `edi` is later overwritten by the other argument to printf: the pointer to the actual format string. The return value of square should be in `esi`. – Julia van der Kris May 19 '20 at 14:51
  • Which is fine, because `edi` is used to pass parameters. Keep in mind that `edi` and `esi` are the only registers that can run the `rep *sb` instructions, and so are good candidates for this. There are a whole bunch of conventions compilers use (almost) universally, if you really want to code assembly alongside the compiler you need to know them, or you'll generate invalid code. – Blindy May 19 '20 at 14:57
  • Actually programming in assembly is probably a bit ambitious for now, at the moment I'm mostly just trying to learn how to read assembly code generated by the compiler. But thanks for the help, I really appreciate it! – Julia van der Kris May 19 '20 at 15:05
  • 1
    Please don't post pictures of text. You can easily copy/paste text from Godbolt's asm output pane. – Peter Cordes May 19 '20 at 17:32
  • But without the color formatting. It's okay if you don't like pictures, feel free to add your own picture-free answer if it will improve everyone else's lives. – Blindy May 19 '20 at 17:36
  • Didn't see your reply at the time. When I copy text from Godbolt, I sometimes separate blocks with a blank line, or just comment that asm. That gives the same general value that color highlighting gave, but without being an image. – Peter Cordes Jan 03 '22 at 20:31