9

I am looking at some x86 code, which I believe was built using a Microsoft tool chain, and am trying to figure out the calling convention used during this call:

 push esi ; save ESI (it gets restored later) lea esi, [ebp-0xC] ; set param 1 for call to FOO call FOO test eax, eax ; test return value jz somelabel

The function FOO starts like this:

 FOO: mov edi, edi push ebx xor ebx, ebx push ebx ; null push esi ; pass ESI in as second param to upcoming call, which has been set by caller push ptr blah mov [esi+0x8], ebx mov [esi+0x4], ebx mov [esi], ebx call InterlockedCompareExchange ; known stdcall func which takes 3 params test eax, eax ...

as ESI is not initialized in the body of FOO, I have assumed it is passed in as a param by the caller.

What is this calling convention? It looks to be a variant of fastcall. Is there a name for this convention?

Improve this question
2
  • Isn't it a struct passed through the arguments ? The esi will be the base address and access to each field is passed through several offsets. No ? Commented Aug 22, 2013 at 15:47
  • yes the parameter is a pointer to a local variable (12 byte structure) of the caller, but what is the calling convention that passes this pointer as a parameter to FOO? Commented Aug 22, 2013 at 15:54

1 Answer 1

Reset to default
12

There is no "official" calling convention that works like that. What you're seeing is most likely the result of Link-time Code Generation, also known as LTO (Link-time optimization) or WPO (Whole program optimization).

When it is enabled, the optimization and code generation is done at link time, when the compiler has access to the code of whole program and all compile units, and can use this information for the more extreme optimizations.

From MSDN:

When /LTCG is used to link modules compiled by using /Og, /O1, /O2, or /Ox, the following optimizations are performed:

  • Cross-module inlining

  • Interprocedural register allocation (64-bit operating systems only)

  • Custom calling convention (x86 only)

  • Small TLS displacement (x86 only)

  • Stack double alignment (x86 only)

  • Improved memory disambiguation (better interference information for global variables and input parameters)

In the code snippet you quoted the compiler detected that the function FOO is not called from outside of the program, so it could customize the calling convention to something that uses register values already set up at the place of call, or otherwise improve register allocation. With heavily templated code you can even get several copies of often-used functions that accept arguments in different sets of registers and/or stack.

Improve this answer
3
  • 2
    What exactly is "stack double alignment"? Commented Aug 23, 2013 at 8:22
  • Don't know for sure, but maybe they mean avoiding having to re-align stack in every function that works with SSE data if they know it's already aligned in every place the function is called. Commented Aug 23, 2013 at 10:59
  • 1
    Not just SSE data, but any data that works better 8-byte aligned (e.g. double primitives). See this valuable post that appears to come from a Microsoft employee for a description of each LTCG optimization. codeproject.com/Articles/1660/… Commented Dec 12, 2013 at 0:25

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.