Lately ive created an API for a game which is no longer maintained by its developers which will be loaded using DLL-injection. It allows users to write plug-ins for that game. The DLL containing the Framework consists of pure managed code (C#) and is therefore loaded using the ICLRRuntimeHost::ExecuteInDefaultAppDomain method.
As its hard to debug an executable without having the source code of it ive started to create my own symbol database for functions and global objects. I planed to use that table in my stack trace to see more detailed information if i did something wrong (those who now throw in debuggers may know that the game had anti debugging mechanics implemented (as it was an online game back then) and i was to lazy to bypass them (sadly...)).
Since ages i use the function RtlCaptureStackBackTrace to easily create stack traces in unmanaged world. So i did the same in C# using DllImport. After setting framesToSkip to 0 i was pretty shocked that i only got 1 frame returned by the function which was actually RtlCaptureSBT itself. I tried several things and then used Marshal.GetLastWin32Error() to get more information on what im doing wrong. Interestingly it returned that no error happened (yes, SetLastError=true ;)). So the function said something is wrong and GetLastError said nothing is wrong, oh yeah!
So i launched IDA and stepped through the function in another .NET executable which wasn't injected to the game. On my (in this time) x86 machine what it did was calling RtlWalkFrameChain which then called RtlWalk32BitStack. That function loops through the stack frames and calls for the address of each of them the function RtlpIsPointerInDllRange from ntdll.dll which determines if a given pointer is in the accessible memory of any of the loaded modules. If it returns false RtlWalk32BitStack returns false thus RtlWalkFrameChain returns false which also causes RtlCaptureStackBackTrace to return false. But as none of these functions calls SetLastError it stays 0 also on failure.
Ok, so i started a debug run. First frame worked good and it was added to the list. Second frame was in a managed module which caused RtlpIsPointerInDllRange to return false. Now all aborted.
Well then i dug trough ntdll.dll searching references of the global variable used to indicate accessible ranges and found the function RtlpStkMarkDllRange. This function is only referenced by LdrpLoadOrMapDll which is used by LdrLoadDll (called by LoadLibrary). RtlpStkMarkDllRange is also not exported so other DLLs can not actually call it. So i searched for references to RtlpIsPointerInDLLRange (also not exported) and found that its only used by functions that perform stack walks.
So i got the feeling that somehow they forgot to include the call to RtlpStkMarkDllRange when a fully managed DLL is loaded because actually there is no reason why a .NET executable should be not valid for holding pointers to functions! Even more as StackWalk64 works as intended and returns all frames while RtlCaptureStackBackTrace isnt.
What do you think?