Commerce Server 2000 User Profile Management Performance Characteristics: Transaction Cost Analysis

Published: April 1, 2001

On This Page

Executive Summary

Chapter 1 User Profile Manager Description

Chapter 2 Capacity Planning

Chapter 3 Performance and Scalability of the User Profile System

Web Farm: Scaling the UPM System

Appendix A Hardware Details

Appendix B - Test Scripts Samples

Appendix C - Test Tools

Executive Summary

This document evaluates the performance and scalability characteristics of Microsoft® Commerce Server 2000 User Profile Management Service.

This document also includes procedures for identifying these characteristics. Using these procedures, administrators can determine how user load impacts hardware resources, and which resources are likely to be bottlenecks in performance. Administrators can then calculate maximum capacity for a particular hardware configuration, and can assess the value of adding resources, and identify those resources that are needed to satisfy greater capacity needs.

Included in this document is a brief explanation of the method the Commerce Server 2000 development team used to improve the performance of the site code and software/hardware configuration.

Also described is the usage profile, the key performance counters for performance and scalability, and event sequence charts for a subset of page interactions.

The methodology used to perform this work is called Transaction Cost Analysis (TCA). For an in depth discussion of this methodology, review the document "Capacity Model for Internet Transactions" located at http://research.microsoft.com/scripts/pubs/view.asp?TR_ID=MSR-TR-99-18 .

Based on the data collected in this document, the following concurrent capacities should be obtainable by the reference hardware. The Microsoft® SQL Server™ and Microsoft® Active Directory™ (AD) capacities are an extrapolation of the CPU cost. The systems will also require a disk subsystem that can support the throughput and the size of the user's information. The projected systems will also require enough physical memory to maintain the same cache hit rate as the system under test.

Per Server Concurrent User Capacity

From this data, we show that SQL server can support the most concurrent users on the Web server, followed by Active Directory, and third is a mix of Active Directory and SQL server.

The user profile is 9 UPM operations per 10 minute session. The online concurrence rate of 1 percent of the total user population.

The User Profile System scales linearly in a Web farm, by adding additional Web server to the Web Farm. This solution requires a load balancing technique implementation. The graph below shows the number of ASP pages per second obtained up to four Web servers.

Horizontal Scaling vs. Single SQL

The back end also scales out using partitioning built in to the User Profile System. This algorithm spreads the users across multiple Data Bases. (There will be a paper that will describe how to scale Active Directory Stores beyond one million users. This information is not contained in this document.) The system uses a hashing algorithm on the hashing key to determine which data store the user's object is stored. The User Profile System deployment must be considered at the time of initial deployment. It cannot be implemented or changed after the system is put into use.

Definition of Terms

Top of page

Chapter 1 User Profile Manager Description

The User Profile Manager (UPM) system provides an easy to use User Management and Personalization API. The API abstracts the Data Store organization and type for Web developers to use to develop a rich and personable Web experience.

The diagram above depicts the core elements of the UPM. Each of these objects and data stores were designed to handle different loads based upon the expected usage characteristics. The Bizdata store is not expected to handle a large load so its API and storage architecture are not covered in this document. The middle group shows the objects and data stores that must be carefully understood when considering the online user experience. These elements will be the primary focus of this report. The analysis portion of the UPM is designed for high scalability in terms of number of users it supports. The User Profile Management Data Transformation Service (UPM DTS) task has been optimized for fast processing of the user information into the Commerce Server 2000 data warehouse.

Performance Counters

All of the Commerce server 2000 performance counters are interesting to observe, but for the purposes of this analysis, it is necessary to focus on the following key counters.

Processor -> %Processor Time

The "limit of comfort" under which the processor should be kept during performance testing for capacity. This limit is usually the target of monitoring tools, or a limit that has been set by data center personnel. For the purposes of our testing, the limit was set at 85 percent.

Memory -> Pages/sec

Indicates a lack of memory resources or a poorly laid out solution. This counter measures the actual memory requests that are made to the hard disk.

Memory -> Available Mbytes

If the number of available megabytes is too low, then the system will start paging. The absolute minimum number is four, but it is recommended that you keep headroom on a server to account for usage peaks.

Active Server Pages -> Request Execution Time

The time spent processing the actual script. If this counter rises dramatically or the time goes over one second, then the system is beyond its limit. We designed all of our pages to process well under one second.

Active Server Pages -> Request Wait Time

The amount of time a new request waits in the queue before it begins processing.

Active Server Pages -> Requests/sec

The actual throughput of the dynamic pages.

Web Service -> Get Requests/sec

Represents the sum of the dynamic pages and static pages. This is the key counter for throughput.

Network Interface-> Bytes Total (per card)

Represents the sum of network throughput for the card. This is the key counter for network throughput.

Physical Disk-> %Disk Time
Physical Disk-> Disk Reads/sec
Physical Disk-> Disk Writes/sec

Three counters that track the activity in the disk subsystem. The disk subsystem can very easily become a bottleneck on any system. On the front-end Web server, the disk utilization should be quite low because the content and images for a page should fit well within the file cache. The primary disk activity is the log file that is now well tuned for performance in Windows 2000.

However, SQL Server makes extensive use of the physical disk subsystem. Planning and calibrating this subsystem is the key component of a fast database management system.

System -> Context Switches/sec

Represents the number of times the system swaps from one thread to another. If this counter goes above 5000 per processor, it indicates poor symmetric multiprocessing (SMP) scalability for the server and/or application. The components of Windows 2000 and Commerce Server 2000 were designed to scale well.

SQL Server:Databases ->Transactions/sec

The key indicator for activity in the back-end SQL server.

LDAP Searches/sec

Tracks the number of lookups that the Active Directory does from its database. This occurs during the initial login of a user that is not in the cache, or it may occur when a page requests user information that was not cached in the initial lookup.

NTLM Authentications/sec

The actual number of times that the authentication check is performed.

CS2000: User Profile Management -> UPM: Cache Hit Rate

The number of cache hits on the server per second for any SQL Server-based properties. This counter is for the server, not an instance.

CS2000: User Profile Management -> UPM: Cache Miss Rate

The number of cache misses on the server per second for any SQL Server-based properties. This counter is for the server, not an instance.

CS2000: User Profile Management -> UPM: Cache Objects

Number of users in a local cache. This should always be equal to or greater than the number of current users on the system. This counter is for the server, not an instance.

CS2000: User Profile Management -> UPM: Object Creation per Second

The actual number of user objects were created. This counter is for the server, not an instance. As this number increases, you need to monitor the disk queue on the back-end data store.

CS2000: User Profile Management -> UPM: object Modifies per Second

The actual number of user objects modified. This counter is for the server, not an instance.

CS2000: User Profile Management -> UPM: object Deletes per Second

The actual number of user objects deleted. This counter is for the server, not an instance.

CS2000: User Profile Management -> UPM: object Reads per Second

The actual number of user objects retrieved. This counter is for the server, not an instance.

Top of page

Chapter 2 Capacity Planning

This chapter describes a detailed capacity analysis using Transaction Cost Analysis (TCA).

Transactions in the UPM System

The following sections detail the operations of a user visiting the site. Each of the costs represents the system running at optimal throughput, which for these tests was determined to be 85 percent CPU utilization on the front-end Web servers.

Profile Retrieval

Profile Retrieval uses dynamic page requests to return an attribute for a user. Sample pages are shown in Appendix B, "Test Script Samples." The SQL Server and Active Directory server had the user's information in the server's local memory cache so disk utilization was lower than at a typical site. The cache hit rate was between 85 and 90 percent.

Costing Table

Profile Retrieval Event Sequence Chart

Modifying an Attribute

This operation modifies a single attribute for an existing user in the UPM system. A modify request does a Get request of the object before performing the modify.

Costing Table

Event Sequence Chart for Object Creation and Attribute Modification

Create User

This operation creates a new user with 12 attributes in the UPM system.

Costing Table

Usage Profile

A usage profile must be developed before you perform a TCA analysis. The process of developing an online usage profile is covered in detail in the document, "Creating a Usage Profile for Site Capacity Planning" available at the same location as this document.

Online Usage Profile

The online usage profile is described in the following table:.

This following table enumerates the statistically determined users' page request distribution, given that 90.53 percent of the operations are profile retrievals, 0.5 percent of the operations are Attribute Modifications, and 8.97 percent of the operations are user creations.

Net UPM operations per session for Profile Retrieval = .9053 * 9 = 8.15

Site Profile

The User Profile Manager (UPM) database we tested contained 100,000 users for Active Directory, and Active Directory and SQL Server data stores. The SQL Server-only system contained 500,000 users.

The Web server was on the same 100BaseT Ethernet switch as the storage servers in order to minimize the latency of network traffic.

The following software configuration changes were made:

• Increased the performance slider bar to "best performance"

• Increased the UPM cache size in the AD/SQL configuration to 100,000

• Increased the TCP window size on the AD server to 0xFFFF.

To increase the performance slider bar, select Properties on the virtual server for your Web server inside the Microsoft® Internet Information Services plug-in found in the Microsoft Management Console (MMC). Then select the Performance tab in the Properties window. A three-position slider bar for Performance tuning appears at the top of the screen. Using the cursor, move the indicator to the far right position marked More than 100,000.

The TCP window size is set in the registry. The registry key is HKLM/SYSTEM/CurrentControlSet/Services/Tcpip/Parameters/TCPWindowSize (REG_DWORD). You must create this value. We then populated it with 0xFFFF.

Transaction Cost Analysis

The following sections detail the mathematical calculations for capacity planning.

This methodology allows each operation to be isolated for tuning or complexity. It also allows users at sites with a different usage profile, but similar page groups, to calculate their capacity. Alternatively, if a site will change only a single page group, the new capacity can be calculated by measuring the new costs for that single page group. This is a powerful concept that can be used in many different ways.

Transaction cost analysis involves the following steps.

1. Creating costing tables for all transactions

2. Calculating the per user frequency per transaction

3. Multiplying the frequency by cost

4. Calculating users based upon CPU capacity

Cost Table for All Transactions

The following tables shows the core cost for each transaction represented. For mathematical purposes, a matrix format is used for displaying data.

The table below shows the CPU cost of the Web server and the SQL Server CPU and Disk copy for each UPM transaction. This deployment is for a SQL server-only Data Store.

SQL Server Data Store

The table below shows the CPU cost of the Web server and the SQL Server CPU and Disk Transfer for each UPM transaction. This deployment is for a SQL server and Active Directory Data Store.

SQL Server/Active Directory Data Store

The table below shows the CPU cost of the Web server and the SQL Server CPU and Disk copy for each UPM transaction. This deployment is for a Active Directory-only Data Store.

Active Directory Server Data Store

Per User Frequency per Transaction

The following table represents the per transaction frequency, or the request rate of the operation per concurrent user, for a user. It is a statistical determination based on the defined usage profile. The first column calculates the frequency..

Frequency in Gets/sec = normalized request rate / time of visit

Thus, for the Retrieve Profile operation:

8.15 Read UPM operations per session / (10 minutes * 60 seconds) = 0.013579 Reads per sec

Matrix Multiply Frequency by Cost Tables

The next step is to multiply the frequency by the hardware resource costs for Web CPU, Active Directory disk, CPU, and so forth. For example, the CPU cost of the operation is:

page cost in P3EM = frequency * P3MC cost

Thus, for default operation:

0.013579 Reads per sec * 6.47 P3MC(for SQL Server only) = 0.087856 P3EM

Yields the following cost matrix for per concurrent user.

Calculating Users Based Upon CPU Capacity

The CPU capacity for a system is the number of processors multiplied by the MHz rating of the CPU. Thus, for a dual processor 800 MHz:

CPU capacity = 2 x 800 MHz = 1600 P3EM

The Information Technologies (IT) department usually determines the target CPU capacity for the system under load. If no standard exists then we recommend determining this goal based on an analysis of peak versus average sustained load to ensure that the CPU is under 100 percent utilization.

Target CPU capacity = CPU capacity x 85% = 1360 P3EM

The following shows how to calculate the capacity for the Web server based on the CPU target and the user cost. From the preceding table, the Web CPU cost per user is 0.103646 P3EM. Divide this cost into the target CPU capacity.

User capacity = Target CPU Capacity / Web CPU cost per user

= 1360 / 0.103646= 13,121 concurrent users

This capacity planning equation is an easy-to-use capacity planning solution for any Web site.

Top of page

Chapter 3 Performance and Scalability of the User Profile System

This chapter briefly discusses the steps that the Commerce Server team took to achieve the throughput and scalability requirements for the site code and the actual deployment. This chapter does not address ASP coding practices or Microsoft Internet Information Services (IIS)/SQL Server tuning parameters.

To optimize performance, the development team investigated the following:

• Tuning the servers

• Tuning the software

Hardware Tuning

Choosing the correct hardware for the Web servers, SQL Servers, and Active Directory plays an important part in a performance analysis. Additionally, the goal of this work was to be able to recommend the appropriate hardware for other users.

Web Servers

The Web servers were given more than enough RAM to perform their tasks. We calculated the maximum working set for the server under load and thus determined how much we could lower the physical RAM in the server.

The disk subsystem of the front-end Web server is used as a read-only device for storing the boot partition and the site content. It needs a read/write device for the pagefile, but these operations are minimal given sufficient physical memory to support the system. The Web server does use the disk subsystem to write event logs and Web logs. This activity is well tuned by the operating system and rarely requires more than a single spindle for performance.

The network system on the Web server should consist of at least a single 100BaseT card. For improved security, manageability, and availability, the server should have two or even three network cards. The network throughput of the Web servers was not sufficient to saturate even a 100-megabit card.

Finally, the CPU and processing subsystem for the server should be the best currently available. This particular hardware subsystem remains the bottleneck on this server for the near future. This is due to the dynamic and process intensive nature of the ASP Web pages.

SQL Servers

Typically, the most important tuning option for the SQL Servers is setting up the physical disk subsystem. For optimal performance, the databases should be separated from their transaction logs on different physical drives. All of the databases, transaction logs, and the TempDB should be set up so that the individual disk subsystem is not a bottleneck. In our test scenario, the physical disk subsystem was not an issue, but a working site should carefully correlate disk costs with transactions in order to plan for increased disk requirements.

SQL Server takes advantage of large amounts of physical memory, so the amount of RAM available should be weighed against the working set of the database. During a run-time test the network input/output and the processing load on the SQL Server is a direct function of the number of front-end servers accessing the SQL Server database as well as the profile of the load. In our testing, we set up the SQL Server so that it was not a bottleneck. Commerce Server 2000 is designed for horizontal scalability and partitioning of the back-end database systems. The databases for marketing, user profile management, catalog, data warehouse, transactions, and administration can be separated into physical SQL Server databases. Thus, the deployed system can easily be distributed onto a separate server or cluster per database. For more information about this topic, see Commerce Server 2000 Help.

Active Directory Server

Like SQL Server, the Active Directory server heavily uses the disk subsystem. Furthermore, the authentication of users into the system is a CPU-intensive process that is best improved with a processor running at a higher clock rate.

Software Tuning

For the purposes of this analysis, we limited the tuning of the front-end Web server to the minimum: in Internet Service Manager, the performance slider bar was set to maximum. All other settings were left as is. If you need to change any parameters in testing or in a live site, change them one at a time, and then compare the new results carefully with the old.

Caution: Unwanted changes to any of the various parameters can make site administration and management difficult.

On the SQL Server servers, all of the defaults were left in place.

On the Active Directory Server, we increased the TCP window size. For more information on this, consult the Microsoft® Windows® 2000 Resource Kit.

Minimize the object creation in the site. One common mistake is to have the following lines in each ASP page:

set objUPM = CreateObject("Commerce.ProfileService")
objUPM.Initialize "Provider=CSOLEDB;Data Source=cs2kperfsql02;Initial
Catalog=BlankSite_commerce;User ID=sa;Password=;", "Profile Definitions"

Having this in each ASP page is very expensive. Instead you should include the following lines in the Global.asa file.

<SCRIPT LANGUAGE=VBScript RUNAT=Server>
Const PRODUCTION  = 1
Const DEVELOPMENT = 0
Const ENGLISH_UNITED_STATES = &H0409
Sub Application_OnStart()  
    Call Main()
End Sub    
Function InitProfileService()
    Dim mscsProfileService
    Set mscsProfileService = Server.CreateObject("Commerce.ProfileService")
    MSCSProfileService.Initialize "Provider=CSOLEDB;Data Source=cs2kperfsql02;Initial 
Catalog=BlankSite_commerce;User ID=sa;Password=;", "Profile Definitions"
    Set InitProfileService = mscsProfileService
End Function
Sub Main()
      Dim MSCSProfileService
.
    Set MSCSProfileService = InitProfileService()
    Set Application("MSCSProfileService") = MSCSProfileService 
End Sub
</SCRIPT>

And then at the top of your ASP pages you only will need set objUPM = Application("MSCSProfileService")

Performance Settings

Cache Size

PsObjectCacheSize is an option in the connection string for initializing the profile service. The default is 32768 objects. The maximum setting is 131072 objects. The setting should be equal to or greater than the maximum number of concurrent users on a Web server. The back ground thread is started every 30 second. The primary performance monitor counter to watch for this is the number of objects in the cache. If it is equal to or within one or two percent of the current setting, you should increase the cache size if you have available memory.

Changes to this option require an IISRESET.

Caution: Use caution when performing IISRESET. This action shuts down dependant services and they may not be restarted. Also, IISRESET only restarts services that start automatically. If a service is set to manual start, it will not start automatically.

Cache Time to Live

PsObjectAgeoutPeriod is the option that describes the length of time a cached object can be stored with out being accessed. The default is 5 minutes. The maximum is 1hour.

Changes to this option require an IISRESET.

Caution: Use caution when performing IISRESET. This action shuts down dependant services and they may not be restarted. Also, IISRESET only restarts services that start automatically. If a service is set to manual start, it will not start automatically.

Top of page

Web Farm: Scaling the UPM System

You can scale out the Supplier Web farm. The decisions surrounding this should be a function of the analysis provided above. Simply put, if the required CPU P3EM is greater than the processing capacity you can purchase in a single server, then the farm will require multiple Web servers.

Likewise, the back-end SQL Servers or DTS task processing computers should be added to the farm if they experience a hardware resource bottleneck.

Scaled Deployment

The above figure shows how a Commerce Server 2000 deployment might be scaled horizontally.

Top of page

Appendix A Hardware Details

Authentication Machine test bed setup

Detailed System Configuration

One Microsoft Web Application Stress (WAS) Test tool client is used in the following configuration:

Two servers were used in these tests in the following configurations:

Top of page

Appendix B - Test Scripts Samples

Read User Profile ASP Script

The following is a sample of the ASP script that retrieves a user profile from an Active Directory server test

<%@ Language=VBScript %>
<%
Option Explicit
dim objUPM
dim objProfile
dim strName
dim lNum
dim localvalue
'Get an instance of the Profile service that was created and initialized in the global.asa
set objUPM = Application("MSCSProfileService")
'intialize the random number generator
randomize 
'For Counter = 1 to 1000
 'randomly get a value between 0 and 1000000
 lNum = Int(100000 * rnd)
 'Generate the name we are going to retrieve
 strName = "blkusr" & lNum
 'Get the User
 set objProfile = objUPM.GetProfile(strName,"UserObject")
 localvalue = objProfile.GeneralInfo.logon_name
 Response.Write(localvalue & "<BR>")
 set objProfile = nothing
%>
Modify User Profile ASP Script
This is a sample of the ASP script that modifies an existing user profile in an Active directory server test
<%@ Language=VBScript %>
<%
Option Explicit
dim objUPM
dim objProfile
dim strName
dim lNum
dim lnum2
set objUPM = Application("MSCSProfileService")
randomize  
lnum2 = "test3"
lNum = Int(100000 * rnd)
strName = "blkusr" & lNum  
'Get the User
on error resume next
set objProfile = objUPM.GetProfile(strName,"UserObject")
if err.number <> 0 then 
response.write "1err.number=" & hex(err.number) & " err.description=" & err.description
response.end
end if
Response.Write "<html><body>"
 objprofile.GeneralInfo.First_name=lnum2
if err.number <> 0 then 
response.write "2err.number=" & hex(err.number) & " err.description=" & err.description
response.end
end if
 objProfile.Update
if err.number <> 0 then 
response.write "3err.number=" & hex(err.number) & " err.description=" & err.description
response.end
end if
 response.Write "User " & strName & " modified"
Response.Write "</body></html>"
set objProfile = nothing
%>

Create User Profile ASP Script

The following is a sample of the ASP script that creates a new user profile in an Active directory server test

<%@ Language=VBScript %>
<%
Option Explicit
dim objUPM
dim objProfile
dim strEmailName
dim lNum
dim strstring
dim strname
set objUPM = Application("MSCSProfileService")
randomize  
lNum = (Int(1000000 * rnd)+1000000)
strname = "blkusr" & lNum
'Create the User
on error resume next
set objProfile = objUPM.CreateProfile(strName ,"userobject")
response.write "<html><body>"
response.write "created.."
if err.number = 0 then
'Generate a string of the generated length and character
strString = "test12"
 objProfile.GeneralInfo.user_id = strName
 objProfile.GeneralInfo.logon_name = strName
 objProfile.GeneralInfo.user_security_password = "password"
 objProfile.ProfileSystem.user_account_control = 512
'response.write "set IDInfo...."
 objProfile.GeneralInfo.first_name= strString
 objProfile.GeneralInfo.last_name = strString
 objProfile.AccountInfo.account_status = 1
 objProfile.GeneralInfo.user_title = strstring
 objProfile.GeneralInfo.tel_number = strString
 objProfile.GeneralInfo.tel_extension = strString
 objProfile.GeneralInfo.fax_number = strString
 objProfile.GeneralInfo.company = strString
 objProfile.GeneralInfo.state = strString
 objProfile.GeneralInfo.department = strString
 objProfile.GeneralInfo.zip = strString
 objProfile.update
 set oGenID = Nothing
 set objProfile = Nothing
 Response.Write strname & " created"
else
 err.clear
 response.write "User " & strname & " already existed"
end if
Response.Write objProfile.Errors
Response.Write "</body></html>"
%>

Top of page

Appendix C - Test Tools

The Microsoft Web Application Stress (WAS) tool is available at http://www.microsoft.com/technet/archive/itsolutions/intranet/downloads/webstres.mspx . There is no charge for this download.

[Note: Here's what was written for the Auth paper. You might want to reference the same location.]

The Microsoft Web Application Stress (WAS) Testing tool can be found at:

http://www.microsoft.com/technet/archive/itsolutions/intranet/downloads/webstres.mspx

The TCA tool kit can be found in the Microsoft Commerce Server 2000 Performance Resource Kit.

Top of page