Chapter 28 - OS/2 Compatibility

This chapter describes the OS/2 subsystem in Windows NT. It describes the types of applications that the subsystem currently supports, as well as those that it does not support. It describes the supported, unsupported, and partially supported OS/2 application programming interfaces (APIs). This chapter also describes how the OS/2 subsystem is implemented and the Win32 thunking mechanism.

Running Applications

The OS/2 subsystem allows OS/2 16-bit character-based applications to run directly with Windows NT with essentially no modification. You can launch a character-based or video I/O (VIO) application from the Windows NT command prompt, from My Computer, from the Windows NT Explorer, or indirectly from within a Win32 or OS/2 application. You can create a single batch file that can launch any combination of MS-DOS, Windows, or OS/2 programs. Windows NT recognizes an OS/2 application from information stored in the header of the executable file; it then calls the OS/2 subsystem to load the application.

If you never run an OS/2 application, the subsystem does not use any Windows NT resources. When you run an application, the OS2SRV process is loaded and continues to exist even after you've quit the application. To free up the minimal resources that the OS2SRV process uses, run the PViewer utility (which is provided with the Windows NT Resource Kit) and quit the OS2SRV process. If you later run another OS/2 application, the OS2SRV process is reloaded.

Supported Applications

You can run the following types of applications with the OS/2 subsystem:

OS/2 1.x 16-bit applications on x86 computers only

Character-based applications

Unsupported Applications

You cannot run the following types of applications with the OS/2 subsystem:

OS/2 2.x applications.

Presentation Manager (PM) applications (unless you install the Windows NT Add-On Subsystem for Presentation Manager, which can be ordered separately from Microsoft)

Advanced video I/O (AVIO) applications (unless you install the Windows NT Add-On Subsystem for Presentation Manager)

OS/2 applications on RISC-based computers.

Applications that directly access hardware memory or I/O ports at Ring 2 or below.

For example, applications that directly access video memory to manipulate text or graphics are not supported. Some OS/2 applications, which rely on the statement IOPL=YES in the Config.sys file to run Ring 2 code segment, will run nevertheless under the OS/2 subsystem as long as the privileged instructions they issue in those segments are CLI/STI instructions and not IN/OUT instructions. For more details, see "I/O Privilege Mechanism" later in this chapter.

You cannot run custom device drivers (those not included with OS/2 itself). These must be rewritten to the Windows NT device driver interface.

If you want to run an OS/2 application that is not supported, you have the following choices:

If this is a bound application (one that can run under both OS/2 and MS-DOS), you can try to run it with the MS-DOS subsystem. To do so, run the forcedos command from the command line: 

FORCEDOS [/D directory] filename [parameters]

where /directory is the current directory for the application to use, filename is the application to start, and parameters is the parameters to pass to the application.

If this is not a bound application and you have the source code, you can recompile the source without the unsupported APIs, which are specified in the error message that is displayed when you try to run the application. If you don't have the source, contact the application's developer.

Partially Supported Applications

Video input/output (VIO) applications are partially supported. Some will work and some will not, depending on the API functions that the applications use. The robustness and security of Windows NT restrict access to physical hardware, which restricts the use of VIO physical buffer APIs, certain DosDevIOCtl functions, and I/O privilege level (IOPL). For more information, see the following section on APIs.

Note Presentation Manager and AVIO applications are supported by the Windows NT Add-on Subsystem for Presentation Manager, available from Microsoft.


A complete list of the APIs that are supported, unsupported, or partially supported is provided in the Os2api.txt file on the Resource Kit disk.

Supported APIs

APIs with the following prefixes are supported:

Dos (except DosDevIOCtl and DosDevIOCtl2, which are partially supported)

Kbd (except those that conflict with the security and robustness of Windows NT)

Mou (except those that require Presentation Manager or AVIO)

Vio (except those that conflict with the security of Windows NT by accessing the physical video hardware and those that require PM or AVIO)

WinQueryProfile and WinWriteProfile 

Net (selected APIs based on their commercial use)

Unsupported APIs

APIs with the following prefixes are not supported:



Kbd (those that conflict with the security and robustness of Windows NT)

Mou (those that require PM or AVIO)

Vio (those that conflict with the security of Windows NT by accessing
the physical video hardware and those that require PM or AVIO)

Win (except WinQueryProfile and WinWriteProfile APIs)

Partially Supported APIs

The following APIs are partially supported:

DosDevIOCtl and DosDevIOCtl2 


VioGetMode and VioSetMode 

VioGetState and VioSetState 

Note APIs with the Mou or Vio prefixes that require Presentation Manager (PM) or advanced video I/O (AVIO) are supported by the Windows NT Add-on Subsystem for Presentation Manager, available from Microsoft.

Implementation of the OS/2 Subsystem

This section describes how the OS/2 subsystem is implemented.

Memory Map of an OS/2 Application

The following is a map of memory usage while the OS/2 subsystem is running an application.


Figure 28.1 OS/2 Subsystem Memory Map 

The tiled area is 512 MB of virtual address space that is reserved up-front and then committed or decommitted when 16-bit applications need segments. The OS/2 subsystem maintains a local descriptor table (LDT) for each process, with shared memory segments at the same LDT slot for all OS/2 processes.

Architecture Diagram

The OS/2 subsystem is implemented as a protected server; OS/2 applications communicate with the subsystem by using the local procedure call (LPC) message-passing facility. The subsystem and each application run in their own protected address spaces, which protects them from other processes running with Windows NT.


Figure 28.2 OS/2 Subsystem in Windows NT 

In native OS/2, applications run in user mode (Ring 3) and communicate with the OS/2 kernel by using calls to the DLLs. Some application programs and DLLs contain I/O privilege segments and are allowed to perform I/O operations in Ring 2. The OS/2 subsystem will attempt to run such programs but those using the I/O privilege to perform IN/OUT instructions (to access some hardware device) violate the robustness features of Windows NT and thus will be terminated with a general protection fault. For more information, see "I/O Privilege Mechanism" later in this chapter.


Figure 28.3 Native OS/2 



The OS/2 subsystem uses OS/2 semantics to maintain the various OS/2 objects. Examples of this include process IDs, the process tree, handles, local and global infosegs, thread-1 semantics, exit-list processing, signals, and semaphores. Windows NT objects are used only when they are relevant; they are then embedded inside OS/2 objects (for example, file handles).

The process tree records the descendent processes of a given process. The subsystem uses the process tree in all related operations, such as ending a program by pressing CTRL+C.


Every thread created by an OS/2 application is implemented with a Windows NT thread in the same process. The thread receives the priority and ID that are relevant in OS/2. The exact OS/2 semantics (such as contents of the register and the stack) are retained when the thread function starts.


The Windows NT scheduler handles the scheduling of OS/2 threads, with the OS/2 priorities 0–63 mapping to Windows NT variable priorities 0–15. (OS/2 priorities are changed only by the application; they are not changed by the scheduler). OS/2 threads never receive Windows NT real-time priorities 16–31.

VIO User Interface

The VIO user interface is partially supported. Applications cannot get direct control of the video hardware. The use of a logical video buffer, as opposed to a physical video buffer, is allowed. For specific information, see the lists of APIs earlier in this chapter.

Dynamic Linking

The OS/2 subsystem implements a full OS/2 loader, which loads DLLs, executables, and resources in exactly the same way as in OS/2. Static linking, load-time dynamic linking, and run-time dynamic linking all function as they do in OS/2.

Memory Management

Protection Model

The OS/2 subsystem implements the protection between OS/2 applications. It constructs their address spaces (both the flat address space and LDTs) and implements the same protection as exists in OS/2.

Some of the memory management limitations of OS/2 1.x are removed. The most important of these is the limit of 16 MB of physical RAM; the OS/2 subsystem uses the large memory capability of Windows NT. This translates into increased performance for applications that can use the additional memory, such as Microsoft's SQL Server. SQL Server asks for the physical memory available inthe system at setup time. It then uses this number to determine the level of caching it will use. In OS/2, you can't use more than 16 MB; however, in the OS/2 subsystem in Windows NT, you can use 32 MB (for example) and double your caching capability.

Segment Swapping

The OS/2 subsystem uses the Windows NT paging mechanism; no segment swapping is performed. Segment swapping is inferior to paging and exists in OS/2 only to support the 80286 processor, which is not supported for Windows NT.

Interprocess Communication

The OS/2 subsystem implements all OS/2 IPC mechanisms (semaphores, pipes, shared memory, queues, and signals).

Named Pipes

The OS/2 subsystem implements named pipes on top of the Windows NT named-pipe file system. These are supported transparently between Win32, MS-DOS, Win16, and OS/2 applications, both locally and remotely. Microsoft LAN Manager 2.x named pipe functionality is supported in its entirety.

Anonymous Pipes

Anonymous pipes, including inheritance, are fully supported. They are integrated into the OS/2 file handle space.

Shared Memory

The full functionality of OS/2 1.x shared memory, including Get and Give semantics, is implemented using Windows NT shared memory features. The discardable segments property is ignored. (It is invisible to the OS/2 application.)


The OS/2 subsystem supports the full range of OS/2 1.x semaphore APIs, including RAM semaphores in private and shared memory, system semaphores, and fast-safe RAM semaphores. Association of semaphores with timers and named pipes is fully supported. The OS/2 subsystem uses a combination of the Windows NT semaphore object and the Windows NT event object to implement an OS/2 semaphore.


OS/2 1.x queues are fully supported, using shared memory between OS/2 processes and OS/2 semaphores as required.


OS/2 signals are fully supported, using Windows NT APIs to manipulate thread context. The OS/2 subsystem controls the address space of OS/2 processes and uses it to manipulate the register content and the stack of thread 1 of the process to be signaled.

I/O Architecture

Device Drivers

Existing private OS/2 device drivers will not be supported in the OS/2 subsystem directly, but must be rewritten for the Windows NT device driver model. In this context, private device driver means a driver that a particular application requires but that is not included in the OS/2 operating system itself.

Examples of such drivers include those that provide custom support for security, fax, MIDI, or 3270 communication cards. Once an OS/2 device driver has been rewritten for the Windows NT model, however, an OS/2 application can communicate with that device driver using the same OS/2 API, DosDevIoctl; no changes will be required within the application itself. Additionally, support exists for the native device drivers included with Windows NT, such as the display, printer, disk, communications, keyboard, and mouse devices.

For example, suppose that a corporation has written a custom device driver to control a security card. The OS/2 device driver for this card uses an internal name, SECDEV, and an entry for this device driver appears in the Config.sys file. In OS/2, the operating system reads the Config.sys file and adds SECDEV to the device driver list. When an application calls the OS/2 API, DosOpen, this list is searched first. The OS/2 subsystem will read this file during initialization and add symbolic links that will allow the OS/2 application to call the Windows NT device driver from the subsystem. For information about how to set the Config.sys file for the OS/2 subsystem to load a Windows NT device driver, see "OS/2 Configuration" later in this chapter.

The OS/2 application code, as opposed to the device driver code, can still load and run in a binary-compatible manner because the device-specific parameters passed by DosDevIoctl(2) APIs are just PVOID buffers. Of course, the new Windows NT version of the ported device driver would have to be made compatible with the original by accepting the same set of parameters within the buffers. Other related OS/2 APIs, such as DosOpen, are supported compatibly, just as they are for supporting native Windows NT system device drivers such as the communications device, the keyboard, and the screen.

File System

The OS/2 subsystem supports long names and extended attributes but no longer supports HPFS. The subsystem does not utilize or expose recoverability and C2 security functions.

Network Connectivity

The OS/2 subsystem implements many LAN Manager APIs. It also implements NetBIOS (both version 2.x and version 3.0 functionality), named pipes, and mail slots.

The OS/2 subsystem maintains remote drives compatible with OS/2. With these, any OS/2 application can use redirected drives transparently with the file I/O APIs. Uniform naming convention (UNC) naming is supported as well. Redirected drives of various network operating systems can be used, provided that the related Win32 Windows NT device drivers (redirectors) have been installed.

I/O Privilege Mechanism

Under native OS/2, if the statement IOPL=YES is present in the Config.sys file, applications may include Ring 2 segments in which it is possible to execute CLI/STI instruction (disable-enable hardware interrupts), but not IN/OUT.

The OS/2 subsystem of Windows NT allows OS/2 applications to run Ring 2 code segments (no special statement required in the OS/2 C:\Config.sys) but with the following important restrictions:

CLI/STI instructions will work.

The OS/2 subsystem will suspend all the other OS/2 applications in the system and all the other threads in the OS/2 process issuing the CLI instructions, until an STI instruction follows. This emulation of CLI/STI instruction is much more costly in run-time overhead than on native OS/2 (where the CPU simply disables external interrupts, which would violate the Windows NT robustness design rules) and also much more costly than semaphore calls. Therefore, when it is possible to modify the OS/2 application, semaphore calls are the preferred way to implement critical sections.

IN/OUT instructions are not supported.

Such instructions will cause a general-protection fault and the application will be terminated.


Filters are supported and are integrated with Win32 and MS-DOS; that is, you can redirect input and output between OS/2, MS-DOS, and Win32 applications transparently.

Device Monitors

Device monitors are a feature that OS/2 provides in the device driver level, which violates Windows NT security if given across the system. Therefore, the OS/2 subsystem implements device monitors within an OS/2 session (an OS/2 application and all of its descendants). Within the session the implementation of device monitors is complete and compatible with OS/2. The vast majority of OS/2 applications use monitors within a session already.


Printing from the OS/2 subsystem is identical to base-level printing on OS/2. For example, you can connect to a remote printer by typing the following at the command prompt:

NET USE LPT1: \\myprinter\pscript 

You can then use the dialog boxes within an application to set up a printer and print.


The OS/2 subsystem is subject to the security measures imposed by Windows NT. OS/2 processes, among themselves, have only the security restrictions of OS/2 (no ACLs attached, and so on). OS/2 processes run under the logged-on user token, just as Win32 processes do.

Communication with Other Subsystems

Subsystems communicate by passing messages to one another. When an OS/2 application calls an API routine, for example, the OS/2 subsystem receives a message and implements it by calling Windows NT system services or by passing messages to other subsystems. When it's finished, the OS/2 subsystem sends a message containing the return values back to the application. The message passing and other activities of the subsystem are invisible to the user.

Communication between OS/2 and Windows NT processes can be accomplished by means of named pipes, mail slots, NetBIOS, files, and COM devices. The Win32 subsystem directs user input to an OS/2 application; it handles all screen I/O for OS/2 applications.

Calling 32-bit DLLs

The OS/2 subsystem provides a general mechanism to allow 16-bit OS/2 and PM applications to load and call any Win32 DLL. This feature could be extremely useful in the following cases:

When you need to call from your OS/2 application some functionality available under Windows NT only as Win32 code.

Without the ability to call Win32 DLLs, the alternative would be to split the application into an OS/2 application and a Win32 application, then communicate between them using, for example, named pipes. This would be much more complicated to implement and may not yield a good performance.

When you want to port your OS/2 application to Win32 but would like to do so in stages, by porting only part of the application at first.

A small set of new APIs is provided. See "Win32 Thunking Mechanism" later in this chapter.

OS/2 Configuration

The OS/2 subsystem handles Os2.ini compatibly with OS/2. The WINxxx APIs supported in this release of Windows NT are provided for this purpose. Startup.cmd is just a batch file.

When the OS/2 subsystem starts for the first time, it checks the Registry for OS/2 subsystem configuration information. If it doesn't find any, it looks for information in the original Config.sys file and adds the information to the Registry. If the original Config.sys file does not exist or is not an OS/2 configuration file, the subsystem adds the following default information to the Registry:

PROTSHELL=c:\os2\pmshell.exe c:\os2\os2.ini c:\os2\os2sys.ini 
SET COMSPEC=%SystemRoot%\system32\cmd.exe

The subsystem updates the environment variable, Os2LibPath, with LIBPATH information found in the original Config.sys file. The updated Os2LibPath is <systemroot>\SYSTEM32\OS2\DLL concatenated with the list of directories specified in the LIBPATH line of the original Config.sys file.

The PATH information found in the original Config.sys file is not entered automatically into the default Windows NT path. To add the location of OS/2 applications, use the System applet in the Control Panel to add a PATH variable to the user environment variables. This information is appended automatically by Windows NT each time a user logs on to the system.

Windows NT supports the OS/2 configuration commands shown in the following table. If you use commands that are not supported, Windows NT ignores them.




Specifies the command interpreter. Only the Windows NT command interpreter is supported.


Specifies a user-defined Windows NT device driver used by OS/2 applications.


Specifies the location of OS/2 16-bit dynamic-link libraries.


Sets environment variables.


Sets a country code that defines country-dependent information such as time, date, and currency conventions.


Specifies the code pages your system is prepared to use.


Specifies the information the keyboard needs in order to use a particular code page.

The libpath, set, and devicename commands are processed as follows:

The libpath command appends path information to the OS/2 library path in the Windows NT environment. At the command prompt, you can change the library path for OS/2 applications by using the os2libpath command.

The following set commands are ignored:

set path 

set comspec 

set video_devices 

set vio_ibmvga 

set vio_vga 

set prompt 

The devicename command specifies a device driver compatible with Windows NT for use with an OS/2 application. The syntax for the devicename command is as follows: 

DEVICENAME=OS/2devicename [[path][NTdevicename]] 

OS/2devicename is the logical name that OS/2 applications use to address the device. Path and NTdevicename specify the Windows NT device driver to which the OS/2 device name is mapped. If these are not specified, the device is mapped to \DEVICE\os/2devicename.

Changing OS/2 Configuration Information

Although the OS/2 configuration information is stored in the Registry, you can edit that information just as you would edit an OS/2 Config.sys file. To edit the information, you must use an OS/2 text editor.

Note To change configuration information, you must be logged on as a member of the Administrators group.

To change configuration information


While running Windows NT, start an OS/2 text editor in a window.


Open a file called C:\CONFIG.SYS.

Windows NT retrieves the configuration information from the Registry and stores it in a temporary file that you can edit.


Edit the configuration information.


Save and close the file.


Quit the editor.

Windows NT stores the new information in the registry.


Log off from Windows NT, and restart your computer.

File List

The main files that make up the OS/2 subsystem are listed in the following table. Many additional files, not listed here, are needed when running the Windows NT Add-On Subsystem for Presentation Manager, such as all the 16-bit PM DLLs (Pmwin.dll, Pmgre.dll etc.) and 16-bit EXEs (Pmshell.exe, Pmspool.exe, etc.).



This file is the subsystem server. It is invoked when you run the first OS/2 application, and it remains to serve new applications as they are run.


This file is the client side of every OS/2 application. There is an instance of Os2.exe for each OS/2 application that is running.


This file contains the DOSxxx APIs. The other DLLs that are used in OS/2, such as KBDCALLS and VIOCALLS, are provided in memory by the OS/2 subsystem.


This file contains the LM APIs.

1 This file is located in the SYSTEM32\OS2\DLL or C:\OS2\DLL directories when running the Windows NT Add-On Subsystem for Presentation Manager. 

Win32 Thunking Mechanism

As mentioned earlier in this document, the OS/2 subsystem provides a general mechanism to allow 16-bit OS/2 and PM applications to load and call any Win32 DLL. To take advantage of this feature, you typically need to complete the following tasks:

Write a small Win32 DLL thunking layer that will be called by the 16-bit OS/2 application.

This Win32 thunking layer will in turn call the real Win32 API, using the parameters passed by the 16-bit code. The need for such a thunking layer (rather than calling the real Win32 DLL directly from 16-bit) stems from the fact that the OS/2 subsystem thunking mechanism allows only for one generic pointer parameter. Most Win32 APIs require more parameters or of different type so that a small Win32 thunking layer is required to retrieve parameters via the parameter pointer. This parameter pointer points to application-defined data, which will typically be a structure with the parameters for the actual call to the real Win32 API.

Change your 16-bit application to include calls to the Win32 thunking APIs described below.

The following 16-bit APIs are to be used by the OS/2 application code. (These APIs are defined in the same manner as OS/2 APIs. See the OS/2 1.2 Programmer's Reference Manual.)

USHORT pascal far Dos32LoadModule (
PULONG pDllHandle);

Purpose: Load a Win32 thunk DLL that will intermediate between an OS/2 application and Win32 APIs.

Returns: If NO_ERROR is returned, the value pointed to by pDllHandle is used for other Win32 thunk APIs as described below. It is invalid for usage with regular OS/2 APIs. If ERROR_MOD_NOT_FOUND is returned, the value pointed to by pDLLHandle is undefined.

USHORT pascal far Dos32GetProcAddr (
ULONG DllHandle,
PSZ pszProcName,
PULONG pWin32Thunk);

Purpose: Get a cookie (flat pointer) to a routine in a Win32 thunk DLL, previously opened by Dos32LoadModule. For example, if the OS/2 application wants to call the WinSocketFoo API, it builds a Win32 intermediate DLL, named Mysock.dll, that exports MyWinSocketFoo. The application calls Dos32LoadModule with Mysock.dll and then Dos32GetProcAddr with pszProcName of value MyWinSocketFoo. If no error is returned, it can use the value pointed to by pWin32Thunk in a later call to Dos32Dispatch, for calling the MyWinSocketFoo routine, which in turn will call a real Win32 API (for example, WinSocketFoo).

Returns: NO_ERROR if the pszProcName is exported by the Win32 intermediate DLL which relates to DllHandle. If ERROR_PROC_NOT_FOUND or ERROR_INVALID_HANDLE is returned, the value pointed to by pWin32Thunk is undefined.

USHORT pascal far Dos32Dispatch (
ULONG Win32Thunk,
PVOID pArguments,
PULONG pRetCode);

Purpose: Dos32Dispatch calls the 32-bit thunk routine Win32Thunk, previously obtained by Dos32GetProcAddr. It returns the error code returned by Win32Thunk in pRetCode. It translates the pArguments 16:16 pointer to a flat pointer and passes it to the Win32Thunk call. The structure pointed to by pArguments, and the values of pRetCode are application specific and are not interpreted or modified by the OS/2 subsystem.

On the Win32 side, i.e. in the Win32 DLL, the Win32 thunk must be defined as follows:

ULONG MyWinSocketFoo (
PVOID pFlatArg);

The return code from MyWinSocketFoo is application-defined and is copied by the OS/2 subsystem to pRetCode.

Returns: NO_ERROR if the pFlatArg argument is a valid pointer and no exception occurred in the call to it.

USHORT pascal far Dos32FreeModule (
ULONG DllHandle);

Purpose: Unload a Win32 thunk DLL that intermediates between an OS/2 application and Win32 APIs.

Returns: NO_ERROR if DllHandle indeed corresponds to a Win32 DLL previously loaded by Dos32LoadModule (after the call, DllHandle is no longer valid). Otherwise, ERROR_INVALID_HANDLE is returned.

USHORT pascal far FarPtr2FlatPtr(
PULONG pFlatPtr);

Purpose: Translates the segmented pointer FarPtr to a flat pointer pointed to by pFlatPtr.

Returns: NO_ERROR if FarPtr is a valid 16:16 pointer: in this case, upon completion of the call pFlatPtr contains a valid 32-bit flat pointer to be used by Win32 code. ERROR_INVALID_PARAMETER is returned if the 16:16 pointer is not valid: in this case the value pointed to by pFlatPtr is undefined.

USHORT pascal far FlatPtr2FarPtr(
ULONG FlatPtr,
PULONG pFarPtr);

Purpose: Translates the flat pointer FlatPtr to a far pointer which it stores into pFarPtr.

Returns: NO_ERROR if the 32-bit FlatPtr maps to a valid 16:16 pointer in the 16-bit application's context: in this case, upon completion of the call pFarPtr contains a valid 16:16 segmented pointer to be used by the 16-bit OS/2 code. Otherwise, i.e. if the 16:16 pointer is not a valid address in the 16-bit application's context, ERROR_INVALID_PARAMETER is returned and pFarPtr is undefined.

The following are the .H file and .DEF file that should be compiled and linked with the 16-bit OS/2 application:

The .H File

// Definition of WIN32 thunk APIs.

extern USHORT pascal far
Dos32LoadModule(PSZ DllName, PULONG pDllHandle);

extern USHORT pascal far
Dos32GetProcAddr(ULONG Handle, PSZ pszProcName, PULONG pWin32Thunk);

extern USHORT pascal far
Dos32Dispatch(ULONG Win32Thunk, PVOID pArguments, PULONG pRetCode);

extern USHORT pascal far
Dos32FreeModule(ULONG DllHandle);

extern USHORT pascal far
FarPtr2FlatPtr(ULONG FarPtr, PULONG pFlarPtr);

extern USHORT pascal far
FlatPtr2FarPtr(ULONG FlatPtr, PULONG pFarPtr);

The .DEF File


© 2017 Microsoft Corporation. All rights reserved. Contact Us |Terms of Use |Trademarks |Privacy & Cookies