Examining World Wide Web Designs --Lessons from Pilot Studies

By Donald E. Zimmerman, Michel Muraski, Michael Palmquist,
Emily Estes, Catherine McClintoch, &Linda Bilsing
Center for Research on Writing & Communication Technologies
C-225 Clark Bldg.
Dept. of Journalism & Technical Communication
Colorado State University
Fort Collins, CO 80523-1785
Telephone 1-970-491-5674
FAX 1-970-491-2908
dzimmerman@vines. colostate. edu

Introduction

Since 1994, our faculty and graduate students have studied a variety of design issues critical to enhancing the effectiveness of World Wide Web (WWW) sites. Guided by earlier literature from such wide-ranging disciplines as information design, text legibility, hypertext, multimedia, interface design, human computer interaction, and communication science, we have completed five studies. Further, we developed a research protocol designed to enhance the effectiveness of WWW sites for education and information delivery. Our research protocol was guided by Berger and Chaffee's (1987) communication science orientation where communication science seeks to understand the production, processing, and effects of symbol and signal systems by developing testable theories, containing lawful generalizations, that explain phenomena associated with production, processing and effects.

Accordingly, we employed a variety empirical research methodologies, such as in-depth interviews, focus groups, nominal group techniques, surveys, usability testing/protocol analyses, and content analysis of video tapes to our examination of effective WWW design for education and information delivery.

Below, we summarize three studies and highlight selected findings.

The Online Catalog Project

In Spring 1994, we compared the usability of Gopher and WWW prototypes for the university catalog. Our methodology consisted of having three students complete navigational tasks using Gopher and three students complete tasks using the WWW. We videotaped the usability tests, conducted a content analysis of the video tapes, and asked students to complete a short survey about their reactions to the version that they used. We found that participants (1) wanted the information chunked into smaller nodes to preclude their scrolling through the text; (2) wanted the information to be organized hierarchically by frequency of information use; (3) felt that disorientation could be minimized by keeping the information brief, matching headings to menu items, and providing a recognizable home page; (4) felt that the hypertext hierarchies helped them remain oriented and find information. Additionally, experienced participants found that the WWW version provided them with shortcuts that helped them navigate efficiently.

The Graduate School Application Forms Project

Our Graduate School provided us with the opportunity to pilot test students' ability to complete both the US graduate school application form and the international graduate school application form in WWW format. For the initial design, our project programmer converted the hardcopy versions of each Graduate School application form into a continuous, scrollable, online form with "save" buttons about every two screens. To test the usability of this format, we developed a scenario, observed and videotaped students (US students, n=5; international students, n=4) as they completed the appropriate WWW version of the Graduate School application form.

Participants, who were US citizens, filled out the US form in an average of 25 minutes and found three sections of the US form easy to use. However, participant unfamiliarity with University terminology used in the form confused users. Further, participants reported that some blank spaces were not long enough to enter the information requested. Invoking the "save" button every two screens made participants anxious and participants expressed fear that their application would be lost. Even more disconcerting to these participants was the programming flaw that jumped participants back to the first screen of the form when they invoked the "save" button. Thus, the program required participants to repeatedly scroll through the form and multiple screens as they filled out sections and clicked on the "save" button.

Participants, who were international students, filled out the form in an average 36 minutes and were extremely concerned about being "correct" when filling out the form. Generally, they took longer to read the form's introduction, about four minutes, when compared to US students, who took about three minutes. As with the US students, the international students expressed confusion about university terminology used on the form, and they too found some form spaces did not provide sufficient space to enter the requested information. Again, clicking on the "save" button jumped these participants back to the first screen, which they found to be a problem.

WWW and Multimedia Instructional Modules Comparison Project

In 1993, we launched a five-year project designed to provide online communication instruction for the university's emerging communication across the curriculum program (Palmquist et al., 1995; Vest et al., 1995; Zimmerman et al., 1994; Zimmerman et al. 1994; Zimmerman et al. 1995; Zimmerman et al., 1996).

One segment of the project consisted of developing tutorial and reference modules that provided guidance for completing writing and speaking assignments in engineering, science, technical communication, composition, and speech courses. The project was heavily research-driven, and we reported initial survey results of computer uses by electrical engineering majors (Zimmerman et al., 1994) and usability testing of the multimedia interface (Zimmerman et al., 1995).

Concurrently, we began developing the reference and tutorial modules with Asymetrix Multimedia ToolBook. Programming the basic interface, using Aysmetrix Multimedia ToolBook, required about six months of a programmer's time and six months of a graduate student assistant's time (about 20 hours per week) over the six-month period.

To evaluate the interface, we developed a reference module based on an initial assignment for a junior-level technical communication course. To begin, we worked with a printed version of the assignment and then segmented the information into nodes, prepared a hypertext map, and rewrote the text for online reading. We worked in WordPerfect, saved the text files in ASCII format, then imported them into Multimedia ToolBook. Our interface included drop-down windows, overlays, navigational aids, buttons, and color illustrations. Creating the first reference model required an additional 160 hours of writing and programming--preparing the hypertext maps and rewriting the text for reading online.

In early 1995, WWW browsers were added to personal computers in our campus computer laboratories. We realized that the WWW might provide an alternative delivery of our online reference and tutorial modules. We realized that HTML may provide an alternative to programming in Multimedia Toolbook and that it might require less programming time.

We were influenced, in part, by IBM researchers (Peper, Maclntyre and Keenan, 1989; Peper, Williams, Maclntyre and Vandall, 1990) who had compared the requirements for programming in an artificial intelligence program versus programming a simple DOS-based hypertext program (IBM's HyperWIN). The IBM researchers found that a hypertext document (written using IBM's HyperWIN) was as effective as a rule-based, expert system, even when the problem and knowledge domain were quite complex.

Therefore, we used the same hypertext map and ASCII files that we used to create the Multimedia ToolBook module to create a WWW version of the reference module. Writing the HTML code took less than 80 hours, including navigating the Net, learning HTML code, and tagging the ASCII files using an HTML editor.

Not only were we interested in comparing the difference in programming times, we were interested in learning students' reactions to the modules and what they learned from the modules. In addition, we were interested in learning what computer expertise students brought to our junior-level technical communication class, their levels of computer anxiety, and their demographic characteristics.

Accordingly, we conducted two studies designed to evaluated the multimedia and WWW designs. In the initial study, 24 students used the WWW version of the module and 26 used the multimedia version of the module. Students, mostly university juniors and seniors, brought substantial computer experience to the project, and they reported a wide range of computer skills. They used the WWW an average of 3 hours a week. Most students exhibited low computer anxiety. Students rated the HTML and multimedia modules comparable in legibility of text, quality of writing, ease of navigation, and ease of recovering their orientation if they felt lost (Zimmerman, Muraski, and Palmquist, 1996). Overall, students using the HTML module scored it easier to use, more enjoyable, and more satisfying than students using the multimedia module.

In the second study, 30 students evaluated the WWW version of the module and 34 students evaluated the multimedia version. As with the initial study, students brought substantial computer experience to the study, and they reported a wide range of computer skills. In the second study, students reported using the WWW about three and half hours a week. Again, most students reported low computer anxiety. In the second study, students rated the HTML and multimedia modules comparable in their enjoyability, legibility of text, quality of writing, satisfaction with using the module, ease of recovery if they became lost, number of errors made using the modules, levels of frustration, and ease of use of links or buttons. In this second study, students reported the multimedia module easier to use and rated the graphics higher in the multimedia module.

In both studies, students gave the modules better than average to high ratings. Since the ratings for both studies were similar and programming in HTML is faster than in the multimedia software, we are now developing our communication modules using HTML and posting the materials on the Colorado State's Writing Center Web site (http:/www.ColoState.EDU/Depts/Writing Center).

Future Plans

Like many universities, Colorado State University has a master WWW Site and many colleges, departments, and administrative offices have developed home pages while others are completing their efforts. To provide some continuity and consistency, the University has established a WWW Committee that is developing university policy and guidelines for WWW content and design.

As part of the WWW Committee, we have been asked to conduct usability tests of the home pages on the Colorado State University WWW site and to establish a set of design guidelines. In addition, the university is actively encouraging faculty to create individual home pages for their courses and to post their syllabuses, assignments, and supporting materials online. To enhance the overall quality, Dr. Jud Harper, our vice-president for Research and Information Technologies, has provided funding--i.e., a graduate research assistantship for a student in the M.S. in Technical Communication program, and operating expenses--so that we can conduct usability tests and develop guidelines for designing home pages for all university offices, departments, and colleges.

At this point, a graphic illustrator in our Office of Instructional Services has developed different prototypes for a new university home page that we will be evaluating. In addition, we will conduct usability tests on more than 50 home pages from university departments, colleges and administrative offices.

Based on our reviews of WWW sites, guidelines existing on the WWW, and the print and online information design literature, we have identified the following areas to investigate, using empirical research designs:

1. Students and off-campus visitors about our university's WWW sites. Using surveys, we are seeking answers to the following questions:

   • How are they accessing our WWW site--i.e., are they coming in via modems or via optical fiber connections?

   • For what purposes they are accessing our WWW site?

   • How long they are willing to wait for home pages to download?

   • What are their levels of computer skills?

   • How frequently do they use the WWW and multimedia usage skills?

   • How familiar are they with the different functions and options of the WWW browser they use?

   • What are the levels of computer anxiety among them?

2. Issues associated with WWW site design, including:

   • The impact of colored and textured backgrounds on the legibility of text presented online.

   • The impact of legibility of paragraph depth--i.e., the number of lines per paragraph and line length.

   • The legibility of serif versus sans serif text and its impact on the speed of reading.

   • The legibility of boldface, italics and all capitalization of text on screen.

   • The impact of different line lengths on the ease of reading text on screen.

   • The effectiveness of bullet lists in helping users locate key information.

3. Alternative designs that will:

   • Minimize the problem of users becoming "lost in hyperspace."

   • Minimize the problem of excessive scrolling required to read the text.

   • Help users find key information more quickly--i.e., brief elaborations on bullet lists to clue users into where to find specific information.

4. We plan to explore, from users' perspective, the issue of using one, continuous file for a WWW site or home page--i.e., simply putting a printed document online in its original form--versus the approach of breaking the text into a series of nodes and presenting them in a hypertext structure.

5. We plan to develop alternative designs for supporting pages--i.e., once the user views the home page and begins navigating through the subsequent nodes in a specific WWW site, what design elements need to be included in subsequent nodes (screens). Tentatively, we will explore design elements that will

   • Keep the user oriented.

   • Break up the screen visually.

   • Allow users to easily return to specific nodes or the home page.

References

Berger, Charles R., & Steve H. Chaffee.. The Study of Communication as a Science. In Charles R. Berger and Steven H. Chaffee (eds.). Handbook of Communication Science. Sage. Newbury Park, CA.1987.

Palmquist, Michael, Dawn Rodrigues, Kate Kiefer, & Donald E. Zimmerman. "Enhancing the Audience for Writing Across the Curriculum." Computers and Composition 12 (1995): 335-53.

Peper, G., C. Maclntyre, & J. Keenan. "Hypertext: A New Approach for Implementing an Expert System." Expert Systems Conference. Boulder, Colorado: IBM. 1989.

-------, D. Williams, C. MacIntyre, & M. Vandall. Comparing a Hypertext Document to an Expert System. Boulder, Colorado: IBM, 1990.

Vest, David, Michael Palmquist, & Donald E. Zimmerman. "Enhancing Engineering Students' Communication Skills Through Multimedia Instruction." Journal of Engineering Education 84, no. 4 (1995): 383-87.

Vest, David, Marilee Long, Laura Thomas, & Michael Palmquist. "Relating Communication Training to Workplace Requirements: The Perspective of New Engineers." IEEE Transactions on Professional Communication 38, No. 1 (1995): 1-5.

Zimmerman, Donald E., Michel Muraski, & Michael Palmquist. "A Comparative Analysis of HTML and Windows-based Multimedia Software--An Exploratory Study. pp. 173-184. IEEE International Professional Communication Conference '96. Saratoga Springs, NY; IEEE, New York, 1996.

Zimmerman, Donald E., Michael Palmquist, David Vest, Greg Boiarsky, Marilee Long, Ron Tjachman, Thadd Anderson, Karin Criswell, & Cathy Crim. "Developing Online Writing Aids for Electrical Engineering Majors--A Progress Report," pp. 230- 33. IEEE International Professional Communication Conference '95. Savannah, Georgia: IEEE, New York, 1995.

-------, & Michael Palmquist. "Enhancing Electrical Engineering Students' Communication Skills," pp. 428-431. Proceedings of the IEEE International Professional Communication Conference '93.New York, New York: IEEE, 1993.

------, Michael Palmquist, Kate Kiefer, Marilee Long, David Vest, Martha Tipton, & Laura Thomas. "Enhancing Electrical Engineering Students' Communication Skills: The Baseline Findings.," pp. 412-417. Proceedings of the IEEE International Professional Communication Conference '93. Banff, Canada: IEEE, New York, New York, 1994.