Multiple Threads in
Visual Basic 5.0, Part II:
Writing a Win32 Debugger
This article assumes you're familiar with Win32, Visual Basic
Code for this article: MultiVB2.exe (28KB)
John Robbins is a software engineer at NuMega Technologies, Inc., who specializes in debuggers. He can be reached at firstname.lastname@example.org.
I realize that most developers don't write debuggers for a living, so I'll begin by presenting an overview of debuggers in Win32®. My overview will concentrate on the basics in VBDebug, but
if you're really curious, the DEB sample and the article, "The Win32 Debugging Application Programming Interface," on the MSDN CD go into more detail on Win32 debuggers. If you want to go hog wild and write a real debugger, I suggest starting with MSDN and the CPU architecture books. Nothing out there fully describes all the gyrations needed to write a debugger, but there are enough hints in these materials to take you a long way.
There are three concepts to keep in mind about Win32 debuggers. First, a debugger consists of two components: the debugger and the debuggee. Simply put, a debugger is a process that can control another process in a debugging relationship, and a debuggee is a process that is started under a debugger. Some operating systems refer to the debugger as the parent process and the debuggee as the child process. Second, the debugger and the debuggee are completely separate processes. This makes the Win32 operating systems much more robust when debugging. If the debuggee has wild memory writes, it will not crash the debugger. The 16-bit Windows® and Macintosh operating systems have the debugger and the debuggee running in the same process context. Third, like debuggers in all operating systems, Win32 debuggers generally sit in
some sort of loop waiting for the operating system to report that the debuggee did something. This is commonly referred to as the debug loop.
From a distance, a Win32 debugger is a pretty simple thing, with only a couple of code requirements. The first is that the debugger must pass a special flag in dwCreationFlags to CreateProcess: DEBUG_ONLY_THIS_PROCESS. This tells the operating system that the calling process is to be treated as a debugger. After the debuggee is started, the debugger must sit in a loop calling the WaitForDebugEvent API to receive debugging notifications. When it's finished processing them, it calls ContinueDebugEvent. The pseudo code below shows just how little is required to create a Win32 debugger.
Do While (1 = WaitForDebugEvent(...))
If EXIT_PROCESS Then
Notice that a Win32 debugger does not require multithreading, a user interface, or much of anything else. But
as with most things in Windows, the difference between minimal and reasonable is pretty big. In reality, the Win32 Debug API almost dictates that the actual debug loop needs to sit in a separate thread. As the name implies, WaitForDebugEvent blocks on an internal operating system event until the debuggee does something to make the operating system stop the debuggee so it can tell the debugger about the event. If your debugger had a single thread, then your user interface would be totally hung until a debug event was triggered.
While a debugger sits in the debug loop, it receives various notifications that certain events took place in the debuggee. Unfortunately, the DEBUG_EVENT used by WaitForDebugEvent is a C structure that uses a union and cannot be expressed in straight Visual Basic® terms. In C, a union is a collection of other data types that are lumped together, with the total size of the union being the size of the largest structure. It is a convenient way to pack a great deal of data into a small space. To get around the union issue, I defined the DEBUG_EVENT structure in VBDebug using the largest of the unioned structures, EXCEPTION_DEBUG_INFO, to pad DEBUG_EVENT to the correct size:
dwDebugEventCode As Long
dwProcessID As Long
dwThreadID As Long
dwUnionData As EXCEPTION_DEBUG_INFO
To get the appropriate data out of DEBUG_EVENT, use the Visual Basic LSet statement to copy the data out of the dwUnionData into the appropriate user-defined type.
Dim stEvtDbg As DEBUG_EVENT
Dim stCreateProcess AS CREATE_PROCESS_DEBUG_INFO
WaitForDebugEvent ( stEvtDbg , INFINITE )
If ( CREATE_PROCESS_DEBUG_EVENT = stEvtDbg.dwDebugEventCode ) then
' Here's the magic LSet.
LSet stCreateProcess = stEvtDbg.dwUnionData
The dwDebugEventCode field indicates which type of event has been returned. Figure 1 describes all the different events returned by the Win32 Debug API.
When the debugger is processing the debug events returned by WaitForDebugEvent, it can do anything that it wants to the debuggee because all the threads in the debuggee are completely stopped. If the debugger needs to read or write to the debuggee's address space, it can use ReadProcessMemory and WriteProcessMemory. Of course, if the debugger patches the debuggee's code with a call to WriteProcessMemory, it should call FlushInstructionCache to clear out the instruction cache. If the debugger needs to get or set the debuggee's current context or CPU registers, it can call GetThreadContext or SetThreadContext.
The only Win32 debug event that needs special handling is the loader breakpoint. Right after the CREATE_PROCESS_DEBUG_EVENT is received, the debugger will receive an EXCEPTION_DEBUG_EVENT. This is the loader breakpoint, the result of the first assembler instruction executed by the debuggee. The debuggee executes this breakpoint because the CREATE_PROCESS_DEBUG_EVENT indicates only that the process was loaded, not executed. The loader breakpoint, which the operating system forces each debuggee to execute, is the first time the debugger actually knows when the debuggee is truly running. In real-world debuggers, all of the main data structures, such as symbol tables, are initialized in process creation, and the debugger starts showing code disassembly or doing necessary debuggee patching in the loader breakpoint.
When the loader breakpoint is hit, the debugger should record that it saw the breakpoint so that subsequent breakpoints can be handled accordingly. The only other processing needed for the first breakpoint (and for all breakpoints in general) depends on the CPU. On the Intel platform, the debugger has to continue processing by calling ContinueDebugEvent and passing it the DBG_CONTINUE flag so that the debuggee resumes execution. On an Alpha CPU, the debugger must get the CONTEXT structure that represents the current CPU state with GetThreadContext, then increment the Fir (Fault Instruction) field by the size of an instruction, which is four bytes. After incrementing the Fir, the debugger must call SetThreadContext to change the debuggee's registers. This operation manually skips the breakpoint. Unlike the Intel CPU, which automatically increments the EIP (Instruction Pointer) register after executing a breakpoint, the Alpha CPU does not.
When handling the various debug events, there are a number of handles that are passed around, including process handles, thread handles, and file handles. While it's usually good practice to close any open handles an application is given, the Win32 Debug API automatically closes all process and thread handles that are passed to the debugger. If you close them yourself, you might end up closing a random handle in your application because Windows NT® will recycle handle values. This will lead to bugs that are almost impossible to find. I learned this the hard way always check the return values of CloseHandle.
VBDebug: The Big Picture
When I first started thinking about VBDebug (see Figure 2), my goal was to design it to use as a prototyping vehicle for some of my debugger ideas. To achieve this, I needed a way to get the interesting partsthe actual Win32 debug event handlersisolated so that I could experiment with different things without having to rewrite the application. There are three main objects at the highest level in VBDebug: the User Interface (UI), the Executive, and the Core Debugger. The UI is the main way the user controls the application, and it runs completely in its own thread. The Executive is the portion that handles the actual debug events (the real meat of a debugger), and it runs in the same thread as the Core Debugger. The Core Debugger is where the actual debug loop resides.
These three objects are pretty much equally important and they do have to interact with one another. The UI object is responsible for creating the Executive and Core Debugger objects, getting them running in the background thread, and stopping the background thread to release the objects. Since the Executive object is responsible for handling the debugging events, it needs to get the information it controls up to the UI object so the user knows what's going on. This means that the UI and Executive objects need to coordinate some sort of interface that the Executive can use to display the information the user requested.
The Core Debugger object is simple and just encapsulates the debugger loop. It has a reference to the Executive object, so it calls into the Executive object for processing when it receives a debug event. If you set everything correctly, the Core Debugger object should only have to be written once.
From a multithreading standpoint, the only object that will be accessed from both the UI thread and the debug thread is the Executive object. This means the Executive must use one of the Win32 data protection synchronization types, like a critical section, on each of its interfaces.
Once I decided on my design goals for VBDebug, I had to wrestle with some implementation issues. Since I needed a way to plug in various Executives and wanted as much flexibility as possible, I implemented the Executive and Core Debugger portions as Visual Basic classes. By using classes and taking advantage of the late binding offered by COM, the Executive can be easily replaced.
The source files frmVBDebug.frm, SimpleExecutive.cls, and DebuggerClass.cls implement the UI, Executive, and Core Debugger objects, respectively. As you look through the code for SimpleExecutive.cls, you might notice that the application implements the interface from a class called BaseExecutive. This is the abstract class that DebuggerClass uses to access the Executive. This means that DebuggerClass only needs to be implemented once, and it can have any number of different Executive objects passed to it.
As you peruse the source code, pay careful attention to the interactions between the three main objects. Probably the best way to see everything in action is to start with the user interface, frmVBDebug.frm, and carefully track how the other classes are created and accessed. There is a good deal of code in VBDebug, so I will only cover some of the more interesting highlights in this article. I will start with the debugger portions and move into the multithreaded sections. As with all new code, you might want to load VBDebug in a debugger, set breakpoints everywhere, and start it. When a breakpoint is hit, note what thread it came from.
Another interesting feature of the VBDebug code is just how much of it can be reused by another debugger UI and Executive implementation. The source code tree is set up so that all the specific VBDebug code resides in the VBDebug directory, and all the generic code that can be used for other debuggers using my architecture is in the VBDebug\Reuse directory. I hope this will help you when you sit down and play with your own debugger implementations.
More On Win32 Debuggers
VBDebug does not pretend to be a full-fledged debugger by any means, but it does enough work to be useful. And it provides a good starting point for developing more advanced support. All of the good debugging support in VBDebug can be found in the SimpleExecutive class. As you look through the class, you will notice that the functions that are derived from BaseExecutive are the usual debug event handlers. For the most part, these are pretty straightforward.
The only thing the event handlers do is keep two lists: one for dealing with threads and one for dealing with DLLs. The methods that deal with thread creation, BaseExecutive_
DebugCreateProcess and BaseExecutive_DebugCreateThread, put the thread handle into a collection class, g_colThreads. The methods that deal with thread destruction, BaseExecutive_DebugExitProcess and BaseExecutive_
DebugExitThread, remove the thread handle from g_colThreads. While VBDebug only uses these thread handles to show which ones started and stopped, a full-fledged debugger needs to have these handles so that it can set breakpoints in a thread, query the thread's registers, walk the thread's stack, and let the user see the priority of threads.
The methods that deal with DLL loading and unloading, BaseExecutive_DebugDllLoad and BaseExecutive_DebugDllUnload, place the load address into a g_colDlls collection, much like the thread handler methods. Likewise, the DLL handler methods don't do a whole lot. They simply report that a DLL loaded or unloaded at a particular address. If you looked at the information returned in a LOAD_DLL_DEBUG_INFO structure, you would notice a field called lpImageName, which the documentation says is the name of the file. In reality, the name of the DLL is never filled in. This means that you must wind through the actual PE image in memory to get the export section and then find the name. I'll leave this as an exercise for the reader.
A slightly more interesting method is BaseExecutive_
DebugODS, where the debuggee calls to OutputDebugString are handled. When a debuggee calls OutputDebugString, the operating system notifies the debugger and it actually reads the string from the debuggee's address space. While this may sound difficult to initiate, there is a simple API call that handles it for you pretty easilyReadProcessMemory.
Another interesting method in the SimpleExecutive class is BaseExecutive_DebugException, which offers plenty of room for more advanced development. The special handling in this method is needed only if the first breakpoint has not been seen and the exception is a breakpoint, as mentioned. If those conditions are true and VBDebug is compiled for an Alpha CPU, the SkipBreakPoint private method is called. Looking at SkipBreakPoint, you will see the GetThreadContext and SetThreadContext APIs are used to skip over the fault instruction. After skipping the breakpoint on the Alpha, which is done automatically by Intel CPUs, BaseExecutive_DebugException tells the DebuggerClass to pass DBG_CONTINUE onto ContinueDebugEvent. If the exception passed to BaseExecutive_DebugException is not the first breakpoint or is something other than a breakpoint, it passes the exception to the debuggee for it to handle the exception. If the exception is a second-chance exception, then I output the formal name of the exception and where it occurred. Here is where some of the advanced debugger operations need to be added, such as stack walking and variable dumping.
Since there is more to debuggers than just responding to the basic events from the Debug API, there are a couple of methods that do things outside the event architecture. In VBDebug, these are BaseExecutive_PauseProcess and BaseExecutive_ResumeProcess. As the names suggest, they are used to pause and restart the debuggee. What makes pausing and restarting the debuggee interesting is that the debugger must actually call the SuspendThread and ResumeThread APIs on all the debuggee's threads individually. There isn't a single API call to instantly suspend or resume a debuggee. While this might seem odd, the idea makes sense. When I first looked at the Win32 Debug API, I thought that the entire debuggee was stopped when I did not call WaitForDebugEvent. But when you think about the Debug API, it makes sense that it is not.
The debuggee is running full tilt until it does something that the operating system notices will cause an event to trigger, at which point the debuggee is stopped dead in its tracks. This means that, even though I could tell the debug thread to block and not call WaitForDebugEvent (because I wanted to pause the debuggee), it is still running until the debuggee does something that causes a debug event. The way to properly pause the debuggee is to call SuspendThread on each of the active threads in the debuggee, then block on the debug thread. Keep in mind that suspending a thread is not the same as halting a thread. To halt a thread, you would need to suspend it, write a breakpoint instruction to the current program counter, and then resume the thread to cause the breakpoint to be hit.
Senior Project Time!
Since the title of this article concerns multithreading, it's high time that I talk about the hardcore multithreading in VBDebug. While some interesting pieces of the actual debugger were left as exercises for the reader, the multithreaded portions are complete and good enough for you to consider for your applications. When you read over this section, you should look at the code very carefully, because it sometimes gets difficult to imagine exactly what happens in a multithreaded system. There are three major areas that need to be covered here: getting the debug thread started, protecting items that can be accessed from multiple threads, and handling the coordination between the UI thread and the debugger thread.
Of the items that I will discuss in this section, getting the debugger thread cranked up is the easiest to explain. After opening an executable and selecting Start from the Debug menu, the mnuDebugStart_Click method in frmVBDebug.frm does all the work (see Figure 2). As described earlier, the UI is responsible for creating the Core Debugger and the Executive class, so this is where the DebuggerClass from DebuggerClass.cls (see Figure 3) and the SimpleExecutive class are instantiated. After instantiating the classes, mnuDebugStart_Click tells the DebuggerClass which Executive to use, then calls the StartDebugThread function from DebugThread.bas. There are some other things that go on before the call to StartDebugThread, but I'll discuss them later as part of the thread synchronization. StartDebugThread calls CreateThread with the DebugThread function (also in DebugThread.bas) as the thread function, then passes the DebuggerClass as the thread parameter. Since I only want to write the actual DebugThread function once, I have it take the DebugClass passed in and start calling methods on it. The first is StartDebuggee. If the debuggee starts correctly, it lets the DebuggerClass spin in the ProcessDebugEvents method until it returns.
Once the debug thread is up and running, you need to protect the data accessed by both threads. Fortunately, the SimpleExecutive class is the only class accessed by multiple threads. Since VBDebug is designed so that it will never have multiple processes accessing the debugger data, the critical sectionthe simplest of synchronization objectscan protect everything. I wanted to make the critical section as easy to use as possible, so I created a class, CriticalSection, that encapsulates the critical section operations. In CriticalSection.cls, you can see that I use the Initialize and Terminate methods to call InitializeCriticalSection and DeleteCriticalSection, respectively. This is good object-oriented design because it hides the grunge work from the user. When you need to enter a critical section to block access to some data, you simply call the Enter method. When you are finished accessing the data, you simply call the Leave method. Whatever you do, make sure that you match up the calls to Enter and Leave. If you don't, you'll have an instant deadlock because the critical section will be owned by one thread permanently. Using the CriticalSection class requires that you have an error handler in each function that calls the Enter method so that you guarantee a matching Leave method is called, even if something bad happens. This will take a lot of planning to get it right.
There is only one CriticalSection instantiation in VBDebug, the g_clsCritSec private variable in the SimpleExecutive class. The best places to see it used are in the DumpActiveThreads and DumpLoadedDLLs methods. As their names suggest, they dump the current active threads and the loaded DLLs to the UI. When either of these methods is called, the first thing it does is grab the class critical section g_clsCritSec by calling the Enter method. Every class event in SimpleExecutive that can be accessed from multiple threads must grab the class critical section before it does anything. If DumpActiveThreads is in the middle of listing the current threads to the screen and a new thread event notification is sent to the debug loop, then the BaseExecutive_DebugCreateThread method will block until DumpActiveThreads releases the class critical section. All of these gyrations make sure the data is safe and accessed by only one thread at a time. If the new thread was added by BaseExecutive_DebugCreateThread before DumpActiveThreads was finished, there is no telling what output you would get from DumpActiveThreads. It could be wrong, or it might even crash.
After the debug thread is running and the data in SimpleExecutive is protected, you have to do the really hard part: synchronize the UI thread and the debugger thread. To me, getting multiple threads properly synchronized is one of the most difficult aspects of multithreading. One of the main reasons that I did VBDebug as the big sample program for this article is that the thread synchronization for a debugger is pretty difficult; I wanted to show you how it could be done in a real-world instance.
In VBDebug, the debugger thread really does not tell the UI thread much, but the UI thread needs to tell the debug thread a lot. Specifically, the debugger thread needs to tell the UI thread when the debuggee startedor failed to startand when the debug thread ended. The UI thread needs to tell the debugger thread when to start debugging, when to stop debugging, when to pause debugging, and when to restart debugging.
As I demonstrated in Part I of this article (MSJ, August 1997), Win32 event objects are an excellent way for one thread to signal a state change to another thread. Here, I put all of the events that are needed, and the times that you would want to wait on an event, in a single Visual Basic class, DebugSynchClass.cls, which is designed so the debug and UI threads can each create their own instances and use them appropriately. Looking at DebugSynchClass, you will see that the concept of getting threads to communicate is simple, but the implementation is a little more involved.
If you take a look in the VBDebug\ReUse subdirectory, you'll see another interesting item: DebugSynchClass.cls is the largest of the reusable files by a good bit. A lot of work goes on in there to hide the nasty implementation details so that each thread just has to call a single method when appropriate. Although I placed all the synchronization code for the UI and debug threads into DebugSynchClass, you might consider splitting them into separate classes. Since the key to using Win32 event objects is to keep the names straight and unique, I thought it was better to have everything lumped together and just document which thread is allowed to call what method.
Before getting the debug thread started, the UI thread is responsible for calling the DebugSynchClass PrepareWaitForStartup method to prepare for startup. After creating the debug thread, the UI thread must call the DebugSynchClass WaitForStartup method to block and wait for the debug thread to start. The debug thread, after attempting to start the debuggee, must call the DebugSynchClass SignalGoodStartup or SignalBadStartup methods to indicate how the CreateProcess call worked. The code for the UI-required methods is rather simple: PrepareWaitForStartup creates two manual reset events, WaitForStartup tells WaitForMultipleObjects to wait until one of the events is signaled, and WaitForStartup returns True if the startup was successful. The debug thread-required methods, SignalGoodStartup or SignalBadStartup, create and signal the appropriate event.
The startup synchronization code itself is also very simple and looks like just about every other piece of sample code that demonstrates event object synchronization. There are two things that make the code interesting. If you look at the parameters passed to CreateEvent in the code, there is a value appended to the end of the constant string which is initialized to the current process ID in the Class_Initialize method. The reason for appending this value is logical when you think about it. First, remember that in Win32 events themselves can be accessed across processes, so it's possible that two instances of VBDebug could be running at the same time. If there is some twist of the scheduler, both UIs could be waiting to see if the debuggee started. If the event doesn't have a unique name, the instant that one of the debug threads signals how the CreateProcess worked, both process's UI threads would continue when only one should. In VBDebug, you might be able to play around with autoreset events, but I want to prepare you for worst-case scenarios. By adding the unique process ID to the event name, you avoid any problems with multiple processes.
The other interesting part of the startup synchronization is that the UI thread has to prepare the startup events before it can create the debug thread. I originally developed the code so that the DebugSynchClass WaitForStartup method created the events to wait on and then did the WaitForMultipleObjects. This worked fine when I tested VBDebug on Intel machines. However, when I tested it on a 400MHz DEC Alpha, I ran into a small problem: if the CreateProcess failed on the debuggee, then the UI thread would block and hang in the DebugSynchClass WaitForStartup method. Obviously, I had a synchronization problem, but I didn't see how it could have happened.
After setting breakpoints with WinDBG everywhere and running VBDebug many times in the debugger, I found that it sometimes runs correctly under the debugger. After placing calls to OutputDebugString in strategic places, I was surprised to find that after the CreateProcess failed in the debug thread, the call to the DebugSynchClass SignalBadStartup method created the event, signaled it, and destroyed it before the UI thread could get the two events created in the DebugSynchClass WaitForStartup method! I suspect that either the DEC Alpha's excellent speed got in the way, or there are some subtle differences in the Windows NT thread handling between Intel and Alpha CPUs. When I created the DebugSynchClass PrepareWaitForStartup and called it before creating the debug thread, everything worked out fine.
After the UI and debug threads are running, the synchronization gets a little easier. As both threads are purring along, the only time something happens is when the UI needs to tell the debug thread how to behave. Essentially, the UI thread needs to create the appropriate event and signal it so the debug thread can respond. Obviously, the debug thread needs to be waiting on some events, and these correspond to the pause, restart, and quit events discussed earlier. Again, like the startup, this arrangement is nothing unusual.
The debug thread in VBDebug needs to continue running so that it can process debug events but still be responsive to the events signaled by the UI thread. In the debug loop, I wait to see what UI-debug synchronization events are signaled. If the UI thread does not signal anything, I take a quick peek to see if there are any debug events to process and loop back to the UI-debug synchronization. All of the UI-debug synchronization is wrapped in the DebugSynchClass WaitForSynchObject method and it is called in the DebuggerClass ProcessDebugEvents method.
The DebugSynchClass WaitForSynchObject method returns one of four values, zero through 3, which correspond to what the UI thread tells the debug thread to do. The only time something does not match up to the synchronization conditions described earlier is when the DebugSynchClass WaitForSynchObject method returns 3, which means that the UI thread did not signal anything so it is safe to call WaitForDebugEvent. Each of the return values for the DebugSynchClass WaitForSynchObject method indicates which event was signaled.
Inside the DebugSynchClass is an array of events created by calling the CreateSynchObjects method. The first three handles in the array are quit, pause, and resume events. These three events are manual-reset events that are created in the nonsignaled state. The final event is the debug active event, which is a manual-reset event created in the signaled state. Since the debug and UI threads need unique events for a particular instance of VBDebug, I append the debuggee process ID to the event names to keep them unique. While I could have appended the VBDebug process ID, this would have limited me to a single debug loop per process, which might be fine for now but would limit future growth.
The DebugSynchClass WaitForSynchObject method calls WaitForMultipleObjects on the array of event handles. Since the debug active event is always signaled, WaitForMultipleObjects returns immediately every time. If the UI does signal a synchronization event, WaitForSynchObject takes care of getting all the event object states set in the debug thread since all the events are manual-reset events. For example, if the UI thread signals that it wants to pause, the debug active event is set to nonsignaled and the pause event is set to nonsignaled. The reset event is similar except that it sets the debug active event back to signaled and clears the reset event. I probably could have made pause and reset autoreset events instead of manual-reset events, but I wanted to make sure I kept everything consistent. Remember the motto of multithreading: Trust NothingVerify Everything!
Handling Thread Demise
So far, I have discussed handling the synchronization for startup and when the UI thread needs to tell the debug thread to do something, but I have not discussed what happens when the debug thread terminates normally. Specifically, I am now going to cover how the UI thread knows that the debug thread ended without polling some global variable. This turned out to be a much nastier problem than I thought because I had to work around the Visual Basic UI portionswhich do not all appear to be thread-safeto solve it.
If I were writing a C++ debugger, I would have simply passed an HWND (the main UI window) into the DebuggerClass, specifying that it uses PostMessage to send a private message to that window. Once that message was received in the UI, I could reset the state of the UI so the debugger was ready to debug again. Instead of passing the HWND all the way down into the Core Debugger portions, the idea is to create a secondary background thread whose sole purpose is to wait on the debug thread handle with WaitForSingleObject. Waiting on the actual debug thread handle is the only way to guarantee that the debug thread truly ended. After the debug thread handle is signaled, the secondary thread calls PostMessage to notify the main window. This is a fairly tried-and-true method of handling this situation.
Since Visual Basic does not, without subclassing, allow you to handle private messages, I set a protocol where the HWND for the form that would receive the message also handles the MouseDown event, indicating that the debug thread was finished. I chose the MouseDown event because I only had to do a PostMessage with the WM_
MBUTTONDOWN message. To ensure that I was not handling a legitimate message, I required that the X and Y parameters to the MouseDown event both be -15 (remember, these are client coordinates, so a multiple monitor system as David Campbell described in the June 1997 MSJ won't break my code). When I posted the message as the last thing in the debug thread before it ended, it worked perfectly and made it quite easy to know when the debug thread ended. Then I ran VBDebug on a multiprocessor machine and, unfortunately, the UI never received the WM_MBUTTONDOWN message so the UI would never get notified that the debug thread ended. Little problems like these are why it is absolutely vital that you test your multithreaded applications on as many different machines as possible.
I should have expected this because the Visual Basic Books Online specifically say that the apartment threading model support is only for non-UI components. On the single-processor machines, I was getting lucky that it worked. When I tried tracking the lost message down, all I could tell was that the PostMessage went off and the message was lost deep down in the bowels of MSVBVM50.DLL. I must point out here that this is not a bug in Visual Basic. Multithreading Visual Basic-based programs is undocumented hacking, and Microsoft never said that they support full multithreading.
Unfortunately, VBDebug was one of those applications that needed to do something with the UI in response to another thread ending. This meant that even if I called a form method directly from the debug thread, I could still cause some problems with the Visual Basic runtime.
Since I had no other choice, I decided that I would force the UI update myself by calling one of the methods directly on the form. Since I did not want to lose all the benefits of the reusable code I had developed, I did not want to be passing a specific form type all the way through the Executive and DebuggerClass. I decided to create a secondary background thread that waits for the debug thread to end, making it easier to isolate the pieces that have to change in future versions. This secondary thread, the WaitForEndOfDebugThread function in EndDbgThread.bas, will call the DebugThreadEnded method from the main VBDebug form to help the UI change its state.
The code in the frmVBDebug DebugThreadEnded method simply calls the private SetUIState where the work of setting the menus and the new caption takes place. While this code might look simple, it brushes up against one of the areas where the runtime is not thread-safe. Even though the menu and the caption are all set correctly, the fact that the form Caption property is set from another thread ends up not quite getting the titlebar text set. After the call to SetUIState in the DebugThreadEnded method, I have to call the SetWindowText API with the same settings to get the title text updated.
Like GDI objects, it looks like Visual Basic UI objects should not be passed for use by different threads. There might be a few other ways that you could get an indication that a thread has ended. In a commercial multithreaded application, the best way to overcome the small problem on the nonthread-safe UI would be to create a couple of small ActiveX controls that handle the synchronization. The control would signal an event that would indicate to the UI that a thread has ended. Whatever mechanism that you finally decide on, I cannot stress enough: test, test, test, and test some more!
Now that you have the groundwork for a debugger, here are some additional ideas to try on your own:
The whole point is to experiment a little and see what you can come up with. By the time you get done writing a moderately featured debugger, you will have an excellent understanding of the entire operating system.
- Add some PE-reading code so that you can retrieve the names of the DLLs that are being loaded and unloaded. Looking at the executable format structures and trying to do them in straight Visual Basic might be a pretty formidable challenge. Since Visual Basic is not designed for the structure manipulation that is needed to deal with PE files, it might not be possible without writing an ActiveX control. However, at one time I thought multithreading in Visual Basic would be impossible, too.
- Incorporate the generic stack-walking code found in the Windows NT 4.0 IMAGEHLP.DLL into VBDebug so that the call stack is displayed when an exception occurs. The stack-walking code is generic, so just by setting it up, you would handle both Intel and Alpha CPUs.
- Add the IMAGEHLP.DLL symbol engine support to VBDebug. Unfortunately, the symbol engine in IMAGEHLP.DLL is rather rudimentary and does nothing more than return the addresses of public items. If you wanted source and line information, for example, you would have to write your own symbol engine. If you do end up writing your own full-featured symbol engine, I will be sure to grant you a Ph.D in debuggers; symbol engines are extremely difficult to develop.
- Put a better user interface on VBDebug. Even though Visual Basic makes it a complete snap to do user interface work, I think you can tell from the existing VBDebug UI that I am not a UI kind of guy. One nice enhancement would be to set up the UI so that different output could go to different windows. For example, you could have a CPU registers window, a stack window, an OutputDebugString window, and a command window. On a crash, you could fill out the CPU window and the stack window with the state of the debuggee. Setting up additional UI would also mean that you would have to extend the Executive so it knows where to put things.
- Do some normal debugger things like adding single step and breakpoints.
The new Visual Basic 5.0 AddressOf operator is a pretty magical thing. Now you can do things in Visual Basic that I never expected. By utilizing some of the Win32 APIs, you are able to support multithreading in Visual Basic. Be sure to check your code carefully, since some portions of Visual Basic do not support multithreading completely. However, if you use multithreading carefully and you test well, you will be able to make your applications more responsive and useful to the user. The extra functionality you gain might get you more cash in your pocket, or at least happier users.
See Part I of Multiple Threads in Visual Basic 5.0