Top Ten Tips for Secure Testing
On This Page
Introduction
This paper provides information about testing applications for the Microsoft Windows family of operating systems in a secure fashion. The topics are divided into ten tips that any developer or tester can readily apply when testing an application or hardware device driver.
For access to the tools and resources discussed in this paper, see "Resources" at the end of this paper.
1. Don't Run Tests as an Administrator
Far too often developers and testers run as the local administrator, because it's the path of least resistance. This practice can lead to a false sense of stability, because administrators can do virtually anything on the system, so most things will work just fine. As soon as the current user isn't an administrator, the operating system security features really come into play, and things might not work as smoothly. Fortunately, there's an easy fix: don't run any tests as an administrator unless absolutely necessary.
Use the RUNAS Command
Microsoft added a great feature to recent versions of Windows that lets the currently logged on user run another program as a different account. Using the RUNAS command from either the Windows Explorer interface or from the command line is a quick and easy way to specify a non-administrative account.
Use a Secondary Local Machine or Domain Account
Create a user on the local machine that has only local user level permissions. Thus, someone running a test can quickly see whether features don't work as the result of certain permission levels being assumed for file system or network access. After installing the program, log off and log back on as the local user before running tests.
If network permissions are required, use a second domain account that has only user privileges on the test system. That way, network permissions don’t get in the way, and administrator rights don't provide a false sense of security.
A Few Exceptions…
Here are a few exceptions where requiring administrator access is acceptable:
| • | Setups and installations need to be able to change system state in a substantial way. |
| • | Maintenance and configuration tools might require administrator access as necessary. |
In these cases, the best programming practice is for an application to check the permissions available when it is launched and then warn the user if administrative access is required for full functionality.
2. Test on a Secure File System--NTFS
It does little good to run as an administrator while testing if the file system isn't secure. FAT12, FAT16, and FAT32 have only rudimentary file permissions, but NTFS provides a full, robust base for testing all types of permissions scenarios.
Use NTFS
Set up all test machines with an NTFS system drive from the start. That way the operating system is configured with base default permissions. On Windows XP and later versions, running the CONVERT command automatically applies the setup default permissions to the system volume.
Also, consider turning on disk quotas for users.
Test on NTFS First, Others as Necessary
Because NTFS is a secure file system, if the software being tested works on NTFS, chances are it will work just fine when using other non-secure file systems. Software that works well in a restrictive environment usually works just fine in an unrestrictive one.
Beware of Differences
Independent of the file system permissions, there are some other differences between file systems to note, such as maximum file, cluster, and partition sizes. These are prime areas for testing border cases.
Start Restrictive; Add Permissions as Necessary
Does the software work if the current user isn't allowed to access a particular folder? What are the minimum permissions required for a particular file operation? Can the program save files into a "drop box" folder?
Start with restrictive permissions on a folder or file share, and then add privileges as testing progresses.
3. Don't Change Anything in HKLM or %PROGRAMFILES%
Far too many programs assume unrestricted access to the HKEY_LOCAL_MACHINE section of the registry and the %PROGRAMFILES% folder on %SYSTEMDRIVE%. Both are read-only for normal users. Don't change anything there except at installation time.
Required for Logo Compliance
An application that is designed to comply with the "Designed for Windows XP" for Applications logo must not try to write to either HKLM or %PROGRAMFILES%. For information, see section 3.0 of the Designed for Windows XP Applications Specification.
Store Data on a Per-User Basis
It's reasonably safe to assume a user has access to the %USERPROFILE% folder. Use the %TEMP% folder for temporary data files. Use %APPDATA% for storing application-specific data per user. Consider storing data under the user's My Documents folder. For registry settings, use HKEY_CURRENT_USER.
4. Don't Store or Send Secrets in Clear Text
Far too often applications attempt to hide something in plain sight using simple obfuscation. This might keep casually prying eyes away, but it doesn't provide any significant protection.
Use the System.Security.Cryptography
Microsoft .NET Framework has an entire class library devoted to cryptography functions for hashing, encrypting, decrypting, signing, and so on.
Don't Store the Secret; Store the Hash
If something sensitive must be stored but not unencrypted, don't store the secret; instead, store a one-way hash of the secret. Then compare the computed hash with the stored hash of the secret.
Use Some Salt
When computing or storing a hash, add some unique randomization (that is, some "salt"), so that the hash is unique per user or per machine for the same data.
Know the Trust Boundary
Where can the program no longer trust data to be transmitted discreetly? A well-defined trust boundary is essential
Legal Concerns
Several legal requirements should be considered in testing, such as the Health Insurance Portability and Accountability Act of 1996 (HIPAA), Computer Fraud and Abuse Act (CFAA), and California (CA) SB1386.
5. Get Adequate Feature Documentation
The realities of a development cycle sometimes save documentation for the last minute, but documentation is necessary and prudent. Notice the title of this tip isn't about perfection, just that the team should have adequate documentation for each feature. Not having such information will make diagnosing potential security problems much harder in the future.
Know the 5 Ws
The bare minimum is knowing the who, what, where, when, and why for each feature.
List the Files
What source files does the feature use? What temporary files does the feature write out? What data file types does the feature let the user open or save? Never assume a data file will be unchanged between execution sessions. Any data file could be changed or manipulated by a malicious user while not opened by the application. Test the file opening code for robust error handling. And never assume a specific file name and so on in code or while testing unless you can be absolutely certain the file name cannot be changed by a malicious user.
List the Registry Entries
Which registry entries does the feature read? What data formats are stored in the registry entries? Does the feature need to write data to the registry?
List the Data Formats and Types
Does the feature store or transmit data in XML or other format? What schema is used? What ranges are valid for fields inside the schema (or related format)? Does the program store data in a proprietary binary format? What happens if the data is altered?
List the Error Messages
Does the feature fail gracefully in certain scenarios? Does the error message provide too much information? For example, rather than telling the user the password was wrong, it's better to indicate some combination of username and password was wrong, without specifying which was inaccurate.
6. Press the Limits of Bad Data
Using the documentation from tip 5, analyze the boundary conditions for each data field, and then design tests that check the validation of each field.
Check All Numeric Ranges
Is the field numeric? Is the field an integer? Are negative numbers allowed? What's the maximum value? What's the minimum value?
Check All Buffer Sizes
If the field size is 255 characters, what happens when the input string is 256 characters? What happens when the input string is 256 Unicode characters?
User Data is Untrustworthy until Proven Otherwise
Does the software accept input from a user? Does the program download files from the internet? Is that data fully and completely range checked? What happens if a malicious data stream is received? All input must be properly and fully validated before use. Software must never assume that a user data file hasn't changed between executions.
Change the Date Formatting
If the feature expects a date in US format (for example, 10/13/2003), what happens when the date is in European format (13/10/2003)? Don't forget to test impossible dates such as 02/29/2013.
Change the Number Formatting
If the feature expects numbers in US format (for example, 3.14159), what happens when the input is in a European format (3,1415926)?
Inject Scripts
If the feature accepts input from a web page form, can a script function be inserted into a text field such that it can be executed? View the web page source to see whether any sensitive details were unintentionally left behind, for example, SQL database server passwords, comments from development, and so on.
7. Test on Two or More Processors
Symmetric multiprocessing (SMP) is becoming much more common today with the advent of technologies such as Hyper-Threading technology in Intel Pentium 4 processors. More and more customers are likely to have SMP-capable systems. Every driver should be tested on an SMP-enabled system. Every multi-threaded application should be tested on a multi-CPU system to assure proper behavior with one or more simultaneous threads of execution. Similarly, every hardware driver should be tested on an SMP system. Pay particular attention to multi-user software, as threads executing in one user's process space shouldn't allow for illicit access by another. Drivers should also be tested on 64-bit platforms, with special attention to issues such as PAE for 64-bit systems.
Every Tester and Developer Should Have One
Make sure every tester and developer on the software project has at least one SMP-capable system to use.
Turn on Hyper-Threading
If a test system is running Windows XP or Windows Server 2003 on a Pentium 4 Hyper-Threading CPU, testing on SMP is as simple as making sure Hyper-Threading is enabled in the BIOS settings.
Add a Second Processor
Many workstation-class computers can be made to support SMP by simply adding a processor. A fast, cheap (compared to a new computer), and easy way to enable SMP is to purchase and install a second processor.
Caveats for Older Windows Releases
Windows 2000 and older Windows operating systems don't support dynamically switching between single and multiple CPUs. Adding a second processor to an older system might require reinstalling the operating system. Test the complete package on all the acceptable systems, and document which service packs might be required to install.
8. Know the 20 Laws of Security
Everyone developing software should acquaint themselves with the articles "The 10 Immutable Laws of Security" and the "The 10 Immutable Laws of Security Administration"--such as testing to make sure that a failed installation or usage doesn't leave the system in an exploitable state. These rules, in addition to the extensive information in Writing Secure Code, 2nd Edition, provide an excellent starting point for understanding what makes well-designed, secure software.
Practice Defense in Depth
Just because there is a security hole that is possible doesn't mean the game is over. Define ways to mitigate the direct risks and the secondary risks if a particular security technology fails. Once the mitigation methods are defined, designed, and built, then test them. Remember, well designed software fails gracefully and to a secure mode. When in doubt, deny access. Installation programs, while permitted to require administrative access, shouldn't leave a system in an insecure state. Be sure setup doesn't configure anything more than the least privileges required for the software to run.
Design Security into the Product
Consider security from the beginning of the development process; don't try to add it later. Create a threat model as an integral part of the design process. Analyzing potential threats before writing code can reduce the need for mitigation later. Careful design can decrease the "surface area" exposed to potentially malicious users.
Secure All Test Systems
It's of little value to test security on an insecure system! Protect all test systems with patches, anti-virus software, and firewalls as appropriate. This is especially important for programs that communicate over the Internet.
9. Use the tools available
Microsoft and other independent software vendors have a wealth of affordable tools for securing, updating, scanning, testing, and evaluating software security.
Windows Update
Windows Update and the Automatic Updates feature built in to Windows 2000 Service Pack (SP) 3, Windows XP, and Windows Server 2003 provide the best ways to get a test system up to date on all the latest patches. Keep all test systems regularly updated.
Microsoft Baseline Security Analyzer, v. 1.2
Microsoft Baseline Security Analyzer (MBSA) is a free tool for scanning and evaluating any system (client or server) for compliance with security best practices. MBSA looks for patching status, and it also checks various other system settings and then provides a concise report with details about how to correct any problems found.
Microsoft Windows Application Compatibility Toolkit, v. 3.0
The Windows Application Compatibility Toolkit, version 3.0, contains several helpful tools that detect many of the problems mentioned in earlier tips. Application Verifier is particularly helpful for detecting common software problems. The kit is available at no charge.
Network Scanning Tools
Network Monitor (NetMon) and other network scanning and analysis tools are useful for watching and analyzing traffic among computers while a particular software test is underway. Careful dissection of the data can reveal where certain risks are inadequately mitigated.
10. Analyze the source code
Careful reviews of source code with an eye for security can reveal potential security holes and risks. Writing Secure Code, Second Edition, has some excellent recommendations on what to watch for.
Hold Code Reviews
Gather all the parties working on a particular feature and have them review the source code line by line. Any issues found should be verified as changed in the manner agreed upon by multiple people.
Search for Risky Functions
Writing Secure Code, Second Edition, has an excellent set of appendices that list standard functions that are commonly used incorrectly and in such a way as to leave a potential security hole. Consider replacing all references to unsafe functions with the recommended alternatives.
Use Automated Scanning
Using automated source code analysis tools, such as PREfast for drivers, can speed up the code review process and provide consistent feedback. Although some tools may occasionally give false positives, be sure to review any feedback carefully.
Resources
Designed for Microsoft Windows XP Application Specification
http://www.microsoft.com/downloads/details.aspx?FamilyID=209e3d65-f0be-4eef-8602-73bb9bc29d54&DisplayLang=en
Matt’s Top 10 Tips for Securely Testing on MSDN TV
http://www.microsoft.com/downloads/details.aspx?displaylang=en&familyid=6dd3da05-399a-40ec-a14e-ba3fbdff41ed
Microsoft Baseline Security Analyzer v1.2
http://www.microsoft.com/technet/security/tools/mbsahome.mspx
Microsoft .NET Framework Developer Center
http://msdn.microsoft.com/netframework/default.aspx
Microsoft Windows Application Compatibility Toolkit 5.0
http://www.microsoft.com/downloads/details.aspx?FamilyID=24DA89E9-B581-47B0-B45E-492DD6DA2971&displaylang=en
Microsoft Windows Driver Development Kit (DDK)
http://www.microsoft.com/whdc/devtools/ddk/default.mspx
PREfast for Drivers
Article at http://www.microsoft.com/whdc/DevTools/tools/PREfast.mspx
Tool in the Windows DDK
The 10 Immutable Laws of Security
http://www.microsoft.com/technet/archive/community/columns/security/essays/10imlaws.mspx
The 10 Immutable Laws of Security Administration
http://www.microsoft.com/technet/archive/community/columns/security/essays/10salaws.mspx
Threat Modeling for Drivers
http://www.microsoft.com/whdc/driver/security/threatmodel.mspx
WHDC
http://www.microsoft.com/whdc/default.mspx
Microsoft Update
http://update.microsoft.com/microsoftupdate/
Writing Secure Code, 2nd Edition
http://www.microsoft.com/mspress/southpacific/books/book19911.htm