Calling conventions are something you generally don’t have to worry about as a programmer because the compiler usually takes care of everything for you. But when you start interacting with code outside of your control (such as shared libraries), calling conventions can become very important.
When you call a function, several things have to happen under the hood. Stack frames must be allocated so there is space for local variables, parameters must be passed in a way the callee can understand and control must be transferred to the new function. On return, any return values must be placed somewhere the caller can find them, and cleanup may need to happen as well. Some CPUs have a strict definition of how this work is performed, and whether the caller or the callee is responsible for performing it. However, the x86 architecture makes no such demands — it is up to the programmer to determine how to call methods. This has led to the rise of several different conventions to use when calling methods.
Two of the more common calling conventions are called stdcall and cdecl. The cdecl calling convention is one where parameters are pushed onto the stack in right to left order, return values are placed in the EAX register, and the caller is responsible for cleaning up the stack when the function returns. The stdcall calling convention is similar in that parameters are still passed right to left, and the return value is placed into EAX. However, the callee (the function being called) is responsible for cleaning up the stack when the function returns.
The differences in calling conventions is very important to understand because mismatches can be disastrous. If you have a callee that is cleaning up the stack, and a caller that is also cleaning up the stack, then you’ve stomped the stack by cleaning it up twice!
Compilers will generally pick a default calling convention for you automatically, depending on the language. C compilers will often use cdecl by default. C++ compilers will usually use thiscall for class instance methods. If you have an optimizing compiler, it will sometimes make methods with only one parameter into fastcall methods. All of this happens without the programmer intervening, and is why you’ve likely not had to worry about calling conventions before; they were something taken care of for you. However, when designing API boundaries (public functions and callback functions) for your frameworks, you should always be explicit about the calling convention you intend to use. This will reduce the likelihood of mismatches between the compiler you’ve compiled the library with, and the compiler the target audience happens to be using.
As a more concrete example, let’s say we’re designing a library with two functions and a callback. The target audience of this library are other C/C++ programmers, as well as some C# folks, all on Windows. Because of the target audience, we’re going to use C as the base language for all public APIs. Our library headers should look something like this:
extern "C" __declspec( dllexport ) void __cdecl DoSomethingImportant( int i, double d ); typedef int (__cdecl *SomeCallback)( const char *data ); extern "C" __declspec( dllexport ) int __cdecl DoSomethingElseImportant( const char *data, SomeCallback callback );
I realize the declarations look pretty foreboding, but there’s a reason for everything there. Breaking it down a bit:
- extern “C” is used to ensure that there is no name mangling that happens on the function declaration, so it comes out with external C-style linkage.
- __declspec( dllexport ) is used to ensure that the functions are exported into the shared library without requiring something more manual, like a definitions file. When a user imports the header file, that should actually read __declspec( dllimport ) instead of dllexport. Because of this, headers usually use a macro to determine which __declspec is being used. For building the library, it uses dllexport, and for consuming the library, it uses dllimport. I’ve elided such a macro for brevity. Note that this is a Microsoft-specific convention.
- Next comes the return type of the functions, which should be self-explanatory.
- We’re being explicit and saying that we require the cdecl calling convention. Nothing mandates that we pick cdecl over stdcall, it’s just a choice we have to make. For instance, all of the Win32 APIs use stdcall as the calling convention. There’s a historical reasoning behind why you’d use one over another, but you can effectively just pick one and use it. The syntax being used is Microsoft-specific, but other compilers have a similar syntax. For instance, in gcc, you’d use __attribute__((cdecl)).
- The rest is the function signature (name, parameters) which should also be self-explanatory
It’s not just on Windows that you have to worry about calling conventions in this way — it’s any x86 targeted machine, including OS X and Linux. Every compiler has their own way of specifying calling conventions, but they all tend to be similar in nature. The calling convention is specified after the return type, but before the function signature. When designing cross-platform libraries, you’ll likely make heavy use of macros to ensure that the function signatures will work with multiple compilers on multiple platforms instead of leaving everything naked like in the above example.
tl;dr: not being explicit about your calling conventions when writing a library can lead to stacking stomping crashes, and make it more difficult for others to consume your library.
