Examining World Wide Web Designs --Lessons from Pilot Studies
By Donald E. Zimmerman, Michel Muraski, Michael
Emily Estes, Catherine McClintoch, &Linda Bilsing
Center for Research on Writing & Communication Technologies
C-225 Clark Bldg.
Dept. of Journalism & Technical Communication
Colorado State University
Fort Collins, CO 80523-1785
dzimmerman@vines. colostate. edu
Since 1994, our faculty and graduate students have studied a variety of design issues critical to enhancing the effectiveness of World Wide Web (WWW) sites. Guided by earlier literature from such wide-ranging disciplines as information design, text legibility, hypertext, multimedia, interface design, human computer interaction, and communication science, we have completed five studies. Further, we developed a research protocol designed to enhance the effectiveness of WWW sites for education and information delivery. Our research protocol was guided by Berger and Chaffee's (1987) communication science orientation where communication science
seeks to understand the production, processing, and effects of symbol and signal systems by
developing testable theories, containing lawful generalizations, that explain phenomena
associated with production, processing and effects.
Accordingly, we employed a variety empirical research methodologies, such as in-depth
interviews, focus groups, nominal group techniques, surveys, usability testing/protocol analyses,
and content analysis of video tapes to our examination of effective WWW design for education
and information delivery.
Below, we summarize three studies and highlight selected findings.
The Online Catalog Project
In Spring 1994, we compared the usability of Gopher and WWW prototypes for the university
catalog. Our methodology consisted of having three students complete navigational tasks using
Gopher and three students complete tasks using the WWW. We videotaped the usability tests,
conducted a content analysis of the video tapes, and asked students to complete a short survey
about their reactions to the version that they used. We found that participants (1) wanted the
information chunked into smaller nodes to preclude their scrolling through the text; (2) wanted
the information to be organized hierarchically by frequency of information use; (3) felt that
disorientation could be minimized by keeping the information brief, matching headings to menu
items, and providing a recognizable home page; (4) felt that the hypertext hierarchies helped
them remain oriented and find information. Additionally, experienced participants found that the
WWW version provided them with shortcuts that helped them navigate efficiently.
The Graduate School Application Forms Project
Our Graduate School provided us with the opportunity to pilot test students' ability to complete
both the US graduate school application form and the international graduate school application
form in WWW format. For the initial design, our project programmer converted the hardcopy
versions of each Graduate School application form into a continuous, scrollable, online form
with "save" buttons about every two screens. To test the usability of this format, we
developed a scenario, observed and videotaped students (US students, n=5; international
students, n=4) as they completed the appropriate WWW version of the Graduate School
Participants, who were US citizens, filled out the US form in an average of 25 minutes and found
three sections of the US form easy to use. However, participant unfamiliarity with University
terminology used in the form confused users. Further, participants reported that some blank spaces were not long enough
to enter the information requested. Invoking the "save"
button every two screens made participants anxious and participants expressed fear that their
application would be lost. Even more disconcerting to these participants was the programming
flaw that jumped participants back to the first screen of the form when they invoked the "save"
button. Thus, the program required participants to repeatedly scroll through the form and
multiple screens as they filled out sections and clicked on the "save" button.
Participants, who were international students, filled out the form in an average 36 minutes and
were extremely concerned about being "correct" when filling out the form. Generally, they took
longer to read the form's introduction, about four minutes, when compared to US students, who
took about three minutes. As with the US students, the international students expressed
confusion about university terminology used on the form, and they too found some form spaces
did not provide sufficient space to enter the requested information. Again, clicking on the "save"
button jumped these participants back to the first screen, which they found to be a problem.
WWW and Multimedia Instructional Modules Comparison Project
In 1993, we launched a five-year project designed to provide online communication instruction
for the university's emerging communication across the curriculum program (Palmquist et al.,
1995; Vest et al., 1995; Zimmerman et al., 1994; Zimmerman et al. 1994; Zimmerman et al.
1995; Zimmerman et al., 1996).
One segment of the project consisted of developing tutorial and reference modules that provided
guidance for completing writing and speaking assignments in engineering, science, technical
communication, composition, and speech courses. The project was heavily research-driven, and
we reported initial survey results of computer uses by electrical engineering majors (Zimmerman
et al., 1994) and usability testing of the multimedia interface (Zimmerman et al., 1995).
Concurrently, we began developing the reference and tutorial modules with Asymetrix
Multimedia ToolBook. Programming the basic interface, using Aysmetrix Multimedia
ToolBook, required about six months of a programmer's time and six months of a graduate
student assistant's time (about 20 hours per week) over the six-month period.
To evaluate the interface, we developed a reference module based on an initial assignment for a
junior-level technical communication course. To begin, we worked with a printed version of the
assignment and then segmented the information into nodes, prepared a hypertext map, and
rewrote the text for online reading. We worked in WordPerfect, saved the text files in ASCII
format, then imported them into Multimedia ToolBook. Our interface included drop-down windows,
overlays, navigational aids, buttons, and color illustrations. Creating the first reference model
required an additional 160 hours of writing and programming--preparing the hypertext maps and rewriting the text for reading online.
In early 1995, WWW browsers were added to personal computers in our campus computer laboratories. We realized that the WWW might provide an
alternative delivery of our online reference and tutorial modules. We realized that HTML may
provide an alternative to programming in Multimedia Toolbook and that it might require less
We were influenced, in part, by IBM researchers
(Peper, Maclntyre and Keenan, 1989; Peper,
Williams, Maclntyre and Vandall, 1990) who had compared the requirements for programming
in an artificial intelligence program versus programming a simple DOS-based hypertext program
(IBM's HyperWIN). The IBM researchers found that a hypertext document (written using IBM's
HyperWIN) was as effective as a rule-based, expert system, even when the problem and
knowledge domain were quite complex.
Therefore, we used the same hypertext map and ASCII files that we used to create the
Multimedia ToolBook module to create a WWW version of the reference
module. Writing the HTML code took less than 80 hours, including navigating the Net,
learning HTML code, and tagging the ASCII files using an HTML editor.
Not only were we interested in comparing the difference in programming times, we were
interested in learning students' reactions to the modules and what they learned from the modules.
In addition, we were interested in learning what computer expertise students brought to our
junior-level technical communication class, their levels of computer anxiety, and their
Accordingly, we conducted two studies designed to evaluated the multimedia and WWW
designs. In the initial study, 24 students used the WWW version of the module and 26 used the
multimedia version of the module. Students, mostly university juniors and seniors, brought
substantial computer experience to the project, and they reported a wide range of computer skills.
They used the WWW an average of 3 hours a week. Most students exhibited low computer
anxiety. Students rated the HTML and multimedia modules comparable in legibility of text,
quality of writing, ease of navigation, and ease of recovering their orientation if they felt lost
(Zimmerman, Muraski, and Palmquist, 1996). Overall, students using the HTML module scored it
easier to use, more enjoyable, and more satisfying than students using the multimedia module.
In the second study, 30 students evaluated the WWW version of the module and 34 students
evaluated the multimedia version. As with the initial study, students brought substantial
computer experience to the study, and they reported a wide range of computer skills. In the
second study, students reported using the WWW about three and half hours a week. Again, most students reported low computer anxiety. In the second study, students rated the HTML and
multimedia modules comparable in their enjoyability, legibility of text, quality of writing,
satisfaction with using the module, ease of recovery if they became lost, number of errors made
using the modules, levels of frustration, and ease of use of links or buttons. In this second study,
students reported the multimedia module easier to use and rated the graphics higher in the
In both studies, students gave the modules better than average to high ratings. Since the ratings
for both studies were similar and programming in HTML is faster than in the multimedia
software, we are now developing our communication modules using HTML and posting the
materials on the Colorado State's Writing Center Web site (http:/www.ColoState.EDU/Depts/Writing Center).
Like many universities, Colorado State University has a master WWW Site and many colleges,
departments, and administrative offices have developed home pages while others are completing
their efforts. To provide some continuity and consistency, the University has established a
WWW Committee that is developing university policy and guidelines for WWW content and
As part of the WWW Committee, we have been asked to conduct usability tests of the home
pages on the Colorado State University WWW site and to establish a set of design guidelines. In
addition, the university is actively encouraging faculty to create individual home pages for their
courses and to post their syllabuses, assignments, and supporting materials online. To enhance
the overall quality, Dr. Jud Harper, our vice-president for Research and Information Technologies, has provided
funding--i.e., a graduate research assistantship for a student in the M.S. in Technical
Communication program, and operating expenses--so that we can conduct usability tests and
develop guidelines for designing home pages for all university offices, departments, and
At this point, a graphic illustrator in our Office of Instructional Services has developed
different prototypes for a new university home page that we will be evaluating. In
addition, we will conduct usability tests on more than 50 home pages from university
departments, colleges and administrative offices.
Based on our reviews of WWW sites, guidelines existing on the WWW, and the print and online
information design literature, we have identified the following areas to investigate, using
empirical research designs:
Students and off-campus visitors about our university's WWW sites. Using surveys, we are seeking answers to the following questions:
How are they accessing our WWW site--i.e., are they coming in via modems or via
optical fiber connections?
For what purposes they are accessing our WWW site?
How long they are willing to wait for home pages to download?
What are their levels of computer skills?
How frequently do they use the WWW and multimedia usage skills?
How familiar are they with the different functions and options of the WWW browser
What are the levels of computer anxiety among them?
Issues associated with WWW site design, including:
The impact of colored and textured backgrounds on the legibility of text
The impact of legibility of paragraph
depth--i.e., the number of lines per
paragraph and line length.
The legibility of serif versus sans serif text and its impact on the speed of reading.
The legibility of boldface, italics and all capitalization of text on screen.
The impact of different line lengths on the ease of reading text on screen.
The effectiveness of bullet lists in helping users locate key information.
Alternative designs that will:
Minimize the problem of users becoming "lost in hyperspace."
Minimize the problem of excessive scrolling required to read the text.
Help users find key information more
quickly--i.e., brief elaborations on
bullet lists to clue users into where to find specific information.
We plan to explore, from users' perspective, the issue of using one, continuous file for a
WWW site or home page--i.e., simply putting a printed document online in its original
form--versus the approach of breaking the text into a series of nodes and presenting them in a
We plan to develop alternative designs for supporting
pages--i.e., once the user views
the home page and begins navigating through the subsequent nodes in a specific WWW site,
what design elements need to be included in subsequent nodes (screens). Tentatively, we will
explore design elements that will
Berger, Charles R., & Steve H. Chaffee.. The Study of Communication as a Science. In
Charles R. Berger and Steven H. Chaffee (eds.). Handbook of Communication Science. Sage. Newbury Park, CA.1987.
Palmquist, Michael, Dawn Rodrigues, Kate Kiefer, & Donald E. Zimmerman. "Enhancing the
Audience for Writing Across the Curriculum." Computers and Composition 12 (1995): 335-53.
Peper, G., C. Maclntyre, & J. Keenan. "Hypertext: A New Approach for Implementing
an Expert System." Expert Systems Conference. Boulder, Colorado: IBM. 1989.
-------, D. Williams, C. MacIntyre, & M.
Vandall. Comparing a Hypertext Document to
an Expert System. Boulder, Colorado: IBM, 1990.
Vest, David, Michael Palmquist, & Donald E. Zimmerman. "Enhancing Engineering Students'
Communication Skills Through Multimedia Instruction." Journal of Engineering Education 84,
no. 4 (1995): 383-87.
Vest, David, Marilee Long, Laura Thomas, & Michael
Palmquist. "Relating Communication
Training to Workplace Requirements: The Perspective of New Engineers." IEEE Transactions
on Professional Communication 38, No. 1 (1995): 1-5.
Zimmerman, Donald E., Michel Muraski, & Michael
Palmquist. "A Comparative Analysis of
HTML and Windows-based Multimedia Software--An Exploratory Study. pp. 173-184. IEEE International
Professional Communication Conference '96. Saratoga Springs, NY; IEEE, New York, 1996.
Zimmerman, Donald E., Michael Palmquist, David Vest, Greg
Boiarsky, Marilee Long, Ron Tjachman, Thadd Anderson, Karin Criswell, & Cathy
Crim. "Developing Online Writing Aids
for Electrical Engineering Majors--A Progress Report," pp. 230- 33. IEEE International
Professional Communication Conference '95. Savannah, Georgia: IEEE, New York, 1995.
-------, & Michael Palmquist. "Enhancing Electrical Engineering Students' Communication
Skills," pp. 428-431. Proceedings of the IEEE International Professional Communication Conference '93.New York, New York: IEEE, 1993.
------, Michael Palmquist, Kate Kiefer, Marilee Long, David Vest, Martha Tipton, & Laura
Thomas. "Enhancing Electrical Engineering Students' Communication Skills: The Baseline Findings.," pp. 412-417. Proceedings of the IEEE International Professional Communication Conference '93. Banff, Canada: IEEE, New York, New York, 1994.