Message Values, Managed String Literals, and Obfuscating Code

Article
10/11/2019

C++ Q&A

Message Values, Managed String Literals, Obfuscating Code, and More

Paul DiLascia

Code download available at:CQA0407.exe(360 KB)

Anumber of C++ fans have expressed concern that too much C# has crept into my columns of late. I agree! My only defense is that as the Microsoft® .NET Framework has gained wider acceptance, more readers have been sending me questions about C#, and since C# is so similar to C++, well, I've sort of been answering them. It was never my intent to alienate C++ fans—lord knows, I'm one of them! However, by popular demand, starting this month the C++ column will focus more exclusively on C++, including managed extensions and old favorites like MFC. So bring on your C++ questions! I especially encourage you to submit questions about managed C++. What experiences have you had with it?

Q In your March 2004 column, you implemented your CMyOpenDlg initialization using a redefined WM_USER+1. I think this is a misuse of the WM_USER range in general (it is reserved for use by whomever does the RegisterClass), but also wrong inasmuch as WM_USER+1 is already the predefined dialog message DM_SETDEFID. Shouldn't you have used a different value for the message?

Jeff Partch

A You're perfectly right! WM_USER is reserved for whomever implements the window class—whether it be you, the friendly Redmondtonians, or aliens from Planet Gleepoid. Figure 1 shows the official breakdown of Windows® message values, which everyone should review at least once a decade. WM_USER through 0x7FFF is reserved for private window classes. You can think of this range as dedicated to messages that have meaning within the particular window class. For example, the status bar control uses WM_USER+1 for SB_SETTEXTA. And as you point out, dialogs use WM_USER+0 and WM_USER+1 for DM_GETDEFID and DM_SETDEFID. My use of WM_USER+1 in the March 2004 column conflicts with DM_GETDEFID.

Figure 1 Message Ranges in Windows

Range	Description
0 - (WM_USER-1)	Messages reserved for use by the system. Examples: WM_CREATE and WM_SCROLL.
WM_USER - 0x7FFF	Integer messages for use by private window classes. Examples: DM_GETDEFID (dialog), TB_ENABLEBUTTON (toolbar), and PBM_SETRANGE (progress bar).
WM_APP - 0xBFFF	Messages available for use by applications.
0xC000 - 0xFFFF	String messages for use by applications (RegisterWindowMessage).
> 0xFFFF	Reserved by Windows for future use.

Applications that want to define their own messages should use WM_APP. WM_APP is guaranteed not to conflict with the system (WM_CREATE, and so on) or class/control-specific messages like DM_GETDEFID. Windows defines WM_APP like so:

#if (WINVER >= 0x0400) #define WM_APP 0x8000 #endif

As every Windows geek knows, WINVER 0x0400 is Windows 95, Windows 98, and Windows NT®. So WM_APP is just under a decade old, which explains why I overlooked it—I'm not due for my next decennary message-range review until 2005!

But should I really have used WM_APP for CMyOpenDlg? CMyOpenDlg lives somewhere between Windows and the application. I'm writing it as an extension for other programmers to use in their apps. If a programmer is already using WM_APP and uses CMyOpenDlg, is there a conflict? It's unlikely in the case of the file open dialog—but what if I were writing a custom control like a toolbar or progress bar and needed to define my own messages? I could base them on WM_APP to avoid conflicting with the base control (but then risk conflicting with the app) or I could pick a number like WM_USER+400 that's well out of the range of the Windows control. Unfortunately, there's no right answer. It's a problem that's plagued control and class library designers ever since Windows was released. There's simply no perfect way to divide the message space that's guaranteed never to cause a conflict. You have to make a practical decision in each case. For CMyOpenDialog, I should have chosen something like WM_USER+10, which doesn't conflict with the DM_XXX messages. Using a registered message is probably overkill.

Speaking of registered messages, what's the difference between the WM_APP range and the registered message range (0xC000 - 0xFFFF), and when should you choose to use one over the other? If you're only talking to yourself—that is, if you plan to send the message only to windows that live within your own application, you can use the WM_APP range. Registered messages are for global messages that you want to send to other apps written by other programmers who know the name of the message. For example, if you're writing a conspiracy-management app that identifies itself to like-minded programs by sending a special message, you should use a registered message:

UINT WM_SAYHELLO = RegisterWindowMessage(_T("WmSayHello"));

Now other conspiracy apps can register using the special name "WmSayHello" and get the same value, so they can all communicate. The actual value WM_SAYHELLO won't be the same for each Windows session, but within a session it will have the same value for all apps that register it. And, of course, the value will always be in the range 0xC000 - 0xFFFF. Gosh—MSDN® Magazine has the sharpest readers!

Q I'm writing an app in managed C++ and have a question about string literals. I know I can create a string with _T("sample string") or S"Sample String". I've read that the S string literal is more efficient, but no one seems to know why or how much more efficient. What is the difference between the two types of literals? What exactly is a managed string literal, and why is it better?

Randy Goodwin

A The best way to understand the difference between managed string literals and ordinary C/C++ literals is to write some code that uses both and look at the compiled results. I wrote a simple managed C++ program, StrLit.cpp, that creates some Strings using both C++ and .NET-style literals. Figure 2 shows the code; Figure 3 and Figure 4 show Microsoft intermediate language (MSIL) code disassembled with ILDASM. As many of you probably know, the primary difference is that when you use a C++ literal, the compiler generates a call to the appropriate String constructor:

// String* c2 = new String(_T("This is a TCHAR string")); ldsflda valuetype $ArrayType$0x11197cc2 modopt([Microsoft.VisualC]Microsoft.VisualC.IsConstModifier) '?A0x1f1e2151.unnamed-global-0' newobj instance void [mscorlib]System.String::.ctor(int8*)

// String* c2 = new String(_T("This is a TCHAR string")); ldsflda valuetype $ArrayType$0x11197cc2 modopt([Microsoft.VisualC]Microsoft.VisualC.IsConstModifier) '?A0x1f1e2151.unnamed-global-0' newobj instance void [mscorlib]System.String::.ctor(int8*)

Figure 4 StrLit Disassembled

////////////////// // .asm file for strlit.cpp, edited to show highlights. // Produced with cl /FAs (generate assembly with source). // .method public static int32 modopt([mscorlib]System.Runtime.CompilerServices.CallConvCdecl) main() cil managed { .vtentry 1 : 1 // Code size 111 (0x6f) .maxstack 1 .locals init ([0] string c1, [1] string s3, [2] string s2, [3] string s1, [4] string c2) ••• // String* c1 = new String(_T("This is a TCHAR string")); IL_000b: ldsflda valuetype $ArrayType$0x11197cc2 modopt([Microsoft.VisualC]Microsoft.VisualC.IsConstModifier) '?A0x1f1e2151.unnamed-global-0' IL_0010: newobj instance void [mscorlib]System.String::.ctor(int8*) IL_0015: stloc.0 ••• // String* s1 = new String(S"This is a String literal"); IL_002e: ldstr "This is a String literal" IL_0033: stloc.3 ••• // Console::WriteLine(_T("Goodbye,")); IL_0052: ldsflda valuetype $ArrayType$0x51890b12 modopt([Microsoft.VisualC]Microsoft.VisualC.IsConstModifier) '?A0x1f1e2151.unnamed-global-2' IL_0057: newobj instance void [mscorlib]System.String::.ctor(int8*) IL_005c: call void [mscorlib]System.Console::WriteLine(string) // Console::WriteLine(S"World"); IL_0061: ldstr "World" IL_0066: call void [mscorlib]System.Console::WriteLine(string) ••• // end of method 'Global Functions'::main }

Figure 2 StrLit.cpp

//////////////////////////////////////////////////////////////// // MSDN Magazine -- July 2004 // If this code works, it was written by Paul DiLascia. // If not, I don't know who wrote it. // Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3. // To see the difference between using C++ string literals (TCHAR) and // .NET (managed) string literals, compile this program and then // disassemble the .exe with ILDASM. You can also compile with /FAs to // view the assembly code. // // cl /clr /FAs strlit.cpp // #include "tchar.h" #using <mscorlib.dll> using namespace System; int _tmain() { // First two TCHAR strings are the same. String* c1 = new String(_T("This is a TCHAR string")); Console::WriteLine(c1); String* c2 = new String(_T("This is a TCHAR string")); Console::WriteLine(c1); // First two String literals are the same, third is different. String* s1 = new String(S"This is a String literal"); Console::WriteLine(s1); String* s2 = new String(S"This is a String literal"); Console::WriteLine(s2); String* s3 = new String(S"This is a different String literal"); Console::WriteLine(s3); // What happens when you pass to managed functions? Console::WriteLine(_T("Goodbye,")); Console::WriteLine(S"World"); return 0; }

I'll explain the gobbledy-gook in a minute, but the basic idea is that the compiler loads the address of your C++ string literal, then generates a newobj instruction that passes it to the String constructor. In addition, the compiler creates a special Array type for your unmanaged literal—that's the strange-looking $ArrayType$0x11197cc2 which appears in the previous snippet and also in Figure 3, with a static instance named ?A0x1f1e2151.unnamed-global-0.

Figure 3 StrLit in the Disassembler

Figure 3** StrLit in the Disassembler **

If you spend much time looking at MSIL, you'll discover modopts all over the place. A full description of modopt is beyond the scope of this column, but the basic idea is that compilers can use modopt to supply additional information about types not understood by the common language runtime (CLR), but which other consumers of the assembly may understand—in this case, the info is that the global is const. The CLR itself has no notion of const, but other consumers of the assembly might. If the assembly were imported into C++, the C++ compiler would know the variable is const; whereas if the assembly were used by C#, the C# compiler would ignore modopt const since C# has no notion of const. As you can see, there's a lot of voodoo that goes on when you use native types in managed code.

When you use a managed literal using the S syntax, however, everything is simple:

// String* s1 = new String(S"This is a String literal"); ldstr "This is a String literal"

There's no function call, just an ldstr instruction. This instruction pushes a new object reference to a string literal stored in the program's metadata. This is true whether you use the literal to create a String object or pass it directly to some function that requires a string:

// Console::WriteLine(S"World"); ldstr "World" call void [mscorlib]System.Console::WriteLine(string)

Another way to see what's happening is to compile your program with /FAs to generate an assembly listing. (Yes, it works for MSIL too!) Figure 5 shows the edited results from compiling StrLit with /FAs. What's interesting here is that you can see that the compiler generates two strings ($SG1883 and $SG2266) for the TCHAR strings "This is a TCHAR string", even though they have the same content. However, there's only one managed string literal ($StringLiteralTok$70000001$) for S"This is a String literal", which is used twice. Most programmers call this process "uniquification," but for some reason the Redmondtonians call it "string interning." As the documentation states: "The Common Language Infrastructure (CLI) guarantees that the result of two ldstr instructions referring to two metadata tokens that have the same sequence of characters return precisely the same string object."

Figure 5 StrLit.asm

; Edited to show only highlights—Paul DiLascia ; Listing generated by Microsoft Compiler Version 13.10.3077 ; Generated by Visual C++ for Common Language Runtime .file "StrLit.cpp" .rdata $SG1883: .ascii "This is a TCHAR string\000" $SG2266: .ascii "This is a TCHAR string\000" .global .bss .local $StringLiteralTok$70000001$,4 .local $StringLiteralTok$70000001$,4 ••• .text .global ?main@@$$HYAHXZ ; main ••• ; String* c1 = new String(_T("This is a TCHAR string")); ldsflda $SG1883 newobj ??0String@System@@$$FQ$AAM@PAC@Z stloc.0 ; _c1$ ••• ; String* c2 = new String(_T("This is a TCHAR string")); ldsflda $SG2266 newobj ??0String@System@@$$FQ$AAM@PAC@Z ••• ; String* s1 = new String(S"This is a String literal"); ldstr $StringLiteralTok$70000001$ stloc.3 ; _s1$ ••• ; String* s2 = new String(S"This is a String literal"); ldstr $StringLiteralTok$70000001$ stloc.2 ; _s2$ ••• ret .end ?main@@$$HYAHXZ ; main text ENDS

The main point of all this is that managed literals are faster. In an attempt to find out how much faster, I wrote another managed C++ program, StrTime, that allocates a bunch of strings and reports how long it took. I used the ShowTime class from my June 2004 column:

_stprintf(msg, _T("Test 1: Allocate %d strings using LPCTSTR"), num); ShowTime st(msg); for (int i=0; i<num; i++) { String* s = new String(_T("This is a TCHAR string")); }

StrTime then does the same thing using a managed S literal. Here are the results:

C:\>StrTime 50000000 Test 1: Allocate 50000000 strings using LPCTSTR: 32797 msec Test 2: Allocate 50000000 strings using .NET S literal: 100 msec

Wow! As you can see, S literals are much faster. But hey—wait a minute, what's really going on here? Is ldstr really 327 times faster? This test is suspicious for a couple of reasons. First, a function call shouldn't be so expensive; second, did I really allocate 50 million strings when my machine has only 785MB? Of course not! Since the variable s lives inside the for loop, it goes out of scope with every iteration—which makes it eligible for garbage collection. In fact, you could write the following program and it would loop forever without running out of memory:

while (1) { String* s = new String(L"foo"); }

The main reason the S literal appears so much faster in the test is that the C++ literal calls newobj to create a new object whereas the S literal generates ldstr, which simply pushes the same object reference on the stack over and over again, 50 million times (remember "string interning"?) To prove this, I modified StrTime (StrTime2) to display the actual address of the managed String. To get it, you need to use GCHandle:

String* s = new String(S"foo"); GCHandle h = GCHandle::Alloc(s, GCHandleType::Pinned); IntPtr ptr = h.AddrOfPinnedObject(); _tprintf(_T(" string at %p\n"), ptr.ToInt32()); h.Free();

This creates a pinned handle, whereupon you can call GCHandle::AddrOfPinnedObject to get the address of the pinned object. Here's the result of running this new test:

C:\>StrTime2 5 Test 1: Allocate 5 strings using LPCTSTR string at 04AC21E0 string at 04AC22D8 string at 04AC2318 string at 04AC2358 string at 04AC2398 Test 2: Allocate 5 strings using .NET S literal string at 04AC218C string at 04AC218C string at 04AC218C string at 04AC218C string at 04AC218C

No need to allocate 50 million strings; 5 is enough to get the idea. As you can see, calling new String(_T("foo")) five or a zillion times generates a new object each time, whereas calling new String(S"foo") loads the same object each time. So the real reason the 50-million-string test takes so much longer with C++ literals is that the runtime has to allocate a lot of memory and perform garbage collection.

All of this may already be more than you bargained for, but just for fun I wrote yet another modification, StrTime3, that forces a garbage collection before allocating the strings, then displays the amount of memory "thought to be allocated" (as the documentation for GC::GetTotalMemory describes it):

GC::Collect(); GC::WaitForPendingFinalizers(); // allocate strings printf("TotalMemory = %d\n", GC::GetTotalMemory(false));

And here are the results:

C:\>StrTime3 50000000 Test 1: Allocate 50000000 strings using LPCTSTR TotalMemory = 190952, time: 33147 msec Test 2: Allocate 50000000 strings using .NET S literal TotalMemory = 43484, time: 110 msec

As you can see, 50 million managed Strings take up a lot less space—especially when there's really only one! Even with TCHARs, the garbage collector uses only 190,952 bytes. Advanced mathematical reasoning suggests that 50 million strings can't fit in 191KB, so it's safe to conclude that the Framework has made a few trips to the dumpster.

In the end I'm afraid all my test programs have failed to answer your question—how much faster are managed String literals? Without a timer with microsecond or nanosecond granularity, it's impossible to tell. But at least my investigations will help you understand what's going on under the hood, and you can see why ldstr is significantly more efficient than newobj.

So which type of literal should you use? The truth is, a user probably will never notice the difference. But it's best practice to use S literals wherever you call a function that takes a String. Note that whereas in MFC you can use LPCTSTR and CString interchangeably, you can't pass a String to C/C++ functions that require TCHARs. That's because managed strings are always Unicode (wide-character) and there's no automatic conversion from String to wchar_t. For that, you need the PtrToStringChars function provided in vcclr.h. The next question shows an example of this.

One final minor note on managed versus native string literals. If you examine the ILDASM code in Figure 3 and Figure 4, you'll see that while the text "This is a String literal" is plainly visible in the disassembled code, the text for the C++ literal "This is a TCHAR string" is nowhere to be found. It's hidden in the mysterious variable ?A0x1f1e2151.unnamed-global-0. Does this mean C++ literals provide more obfuscation—that is, let you hide your strings from prying eyes? Not really. The MSIL code in Figure 3 reports that the mysterious "unnamed-global-0" is "at D_0000B14C." Out of curiosity, I examined StrLit.exe in a hex viewer. Sure enough, there's the string, precisely at location 0xB14C. Figure 6 shows what the hex dump looks like.

Figure 6 The Hex Dump

Figure 6** The Hex Dump **

Q I'm presented with the problem of protecting our intellectual property in an upcoming C# application. I understand that all code can be disassembled with ILDASM, which makes it easy for someone to recover our mathematical formulas. To resolve this problem, I can write a C++ DLL and use DllImport to import my functions in C#, but I prefer to use managed C++ to write a managed __gc class so I can expose both the properties and methods. When I do this and compile my DLL, I can still use ILDASM to disassemble it, which defeats what I'm trying to do. I thought the C++ compiler would generate native machine code, not MSIL. What's going on?

Matt Hess

A When you compile a C++ program with /clr, the compiler treats all functions as managed and compiles them to MSIL by default (except in a few situations, which I'll discuss shortly). Structs and classes can be managed or unmanaged, depending on your use of __gc. If you want to compile a particular function to native code (perhaps in order to hide its implementation, or for performance reasons), you can use #pragma unmanaged:

#pragma unmanaged // this fn and all subsequent ones are compiled // to native code void func(/* args */) { ••• } #pragma managed

#pragma unmanaged // this fn and all subsequent ones are compiled // to native code void func(/* args */) { ••• } #pragma managed

The #pragma managed directive switches back to managed mode. Figure 7 shows a simple C++ program I wrote with two functions, PrintFive1 and PrintFive2, that each print a string five times. The first function is native and the second managed. If you compile with /FAs and look at the .asm file produced, you'll see that the for loop in PrintFive1 compiles to something like this:

; for (int i=0; i<5; i++) { mov DWORD PTR _i$1662[ebp], 0 jmp SHORT $L1663 ••• ; etc... (more native assembly)

Whereas the same for loop in PrintFive2 compiles like so:

; for (int i=0; i<5; i++) { ldc.i.0 0 ; i32 0x0 stloc.0 ; _i$1669 ••• ; etc... (more MSIL)

Figure 7 Managed and Unmanaged Code

//////////////////////////////////////////////////////////////// // MSDN Magazine — July 2004 // If this code works, it was written by Paul DiLascia. // If not, I don't know who wrote it. // Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3. // To see what #pragma unmanaged does, compile this file with // // cl /FAs /clr mixed.cpp // // and look at the .asm generated. You can see that unmanaged functions // generate native machine code whereas managed ones generate MSIL. // By default, when you use /clr, all functions are managed. // #include <stdio.h> #include <vcclr.h> typedef wchar_t WCHAR; #using <mscorlib.dll> using namespace System; #pragma unmanaged // This function will be compiled to native machine language void PrintFive1(const WCHAR* msg) { for (int i=0; i<5; i++) { wprintf(msg); } } #pragma managed // This function and main will be compiled to MSIL void PrintFive2(const WCHAR* msg) { for (int i=0; i<5; i++) { wprintf(msg); } } int main() { String* s2 = new String(S"Hello, world\n"); const WCHAR __pin* pstr = PtrToStringChars(s2); PrintFive1(pstr); PrintFive2(pstr); return 0; }

I said all functions are managed by default when you use /clr, but that's not quite true. While MSIL is general enough to express most C/C++ constructs, it can't express them all. There are a few situations that force a function to be compiled natively, even without #pragma unmanaged. They tend to be obvious: for example, if your function has __asm blocks, setjmp/longjmp instructions, or a variable parameter list (varargs). Since MSIL can't handle these constructs, the compiler automatically compiles such functions to native code.

You can only use #pragma unmanaged to compile pure C++ functions—that is, functions that don't use managed types. And you can't use #pragma unmanaged to compile the methods of a managed __gc class (that wouldn't make sense.) So if you want to use C++ to hide your algorithms, you have to implement them as pure C++ functions with native C/C++ types, which your __gc class methods can call. You must convert any managed parameters to unmanaged types before calling your implementation:

__gc class Widget { public: MyMethod(String *s) { // convert managed types to C++ types const WCHAR __pin* pstr = PtrToStringChars(s); // call unmanaged C++ function (in #pragma unmanaged block) DoSecretAlgorithm(pstr); } };

Managed C++ is the only Microsoft language that lets you combine native and MSIL code in the same file, which is one reason it's so powerful! Another way managed C++ mixes code is through linking. For example, whenever you call C library functions like printf or memset, they get statically linked from the appropriate libraries. When you use /clr, it automatically implies /MT to ensure your program links with the multithreaded versions of the C runtime libraries. The CLR requires them because the garbage collector runs finalizers in a separate thread.

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at https://www.dilascia.com.

Additional resources