Usability Testing - Netscape Navigator Vs. Microsoft Internet Explorer

 TEST GOALS:
Two similar pieces of software were evaluated in this usability test - Netscape Navigator (Netscape) and Microsoft Internet Explorer (IE). The purpose of this test was to: Testing of the more advanced features in the browsers, or sub-features such as e-mail or news group reading was not done, since the evaluation of such items would need a more extensive test. In addition, due to the numerous features contained in both Internet Explorer and Netscape Navigator, only a limited number of browser functions were evaluated.

TYPE OF USABILITY TEST - COMPARISON TEST
The comparison test is not associated with any particular point in the product development life cycle. Toward the end of the life cycle, a comparison test can be used to see how the released product stacks up against a competitor's product. One of it's functions, among others, is to compare one product with another, usually competitors. The comparison test is typically used to establish which design is easier to use or learn, or to better understand the advantage and disadvantages of different designs (Rubin, 1994).

TEST SUBJECTS
Three users were tested, and all had basic Windows 95 experience and who understood Windows 95 concepts, which was important since that was the operating system being used. All users were also familiar with the basic functions of web browsers such as:

TASK SCENARIO
A total of four sets of tasks were developed which were not specifically tailored for either browser to ensure neutrality, and were "jargon-free". I read out-loud the tasks to the participants so as to make sure they understood exactly what they were to do (Rubin, 1994). Each browser was capable of performing each task in primarily the same fashion. During the test sessions, tasks were rotated - I switched the order for each task set. Meaning, that the user would be told to do something in Netscape, and then told to do the same task in IE. For the next task, I would tell them to perform the task in IE first, then Netscape. I started out with a different browser for different test subjects - two started the task sets with Netscape, and one test subject started out with IE. This way, there wouldn't be a bias towards whichever browser they performed the task in first, since users tend to be more productive with the second of the two browsers tested (Microsoft Corporation January 13, 1997).

The tasks were designed to simulate skills fundamental to typical browser use. Also, the tasks were developed to allow each user to use the same functions of each web browser including:

Tasks for IE

Part I

1. Someone has told you that the address for a really cool music site is located at the URL: "http://www.imusic.com". Use the web browser to visit this site.

2. You notice a link for "music news" on the top of the page. Click on it to see what is there.

3. Once you get to the Music news page, click on the first hyperlink in the list under "Top Stories". Wait for the site to completely finish downloading.

4. You think the first picture on the page you are currently viewing is really cool and you want to save it to your hard drive. Right -click over the image and save it to the desktop.

5. Return to the first imusic page you were on (http://www.imusic.com)

6. You would like to be able to return to the imusic page easily each time you use the browser without having to type in the URL again. Save the page in your favorites folder it so you can quickly return to this page each time without typing the entire address again.

Part II

1. Go to the home page of this browser by hitting the "home button" on the task bar at the top of the browser.

2. You like this page and want to print it. Print out the page you are now viewing.

3. You have an HTML file on your desktop that you want to see in your browser. Open up that file in your browser. (hint: You start at the "file" option on your browser toolbar)

4. The Whitehouse has a web site! It's been put in your favorites list, so go to the site using the favorites list.

5. Click on the image of Bill Clinton and Al Gore. Add this page to your favorites list.

6. Return to the White House Home Page.

7. Go to "The White House for Kids" page and add this page to your Favorites list.

Part III

1. Click on the Favorites menu item and look at all of the items in the list. You decide that this list is too confusing. IE provides a way to organize your Favorites list. Organize it by grouping some things together. Create two groupings for your list, "Government" and "Music".

2. Put the 2 White House pages that you placed in your Favorites in the "Government" group.

3. Put the page of the imusic site in the "Music" group.

4. You want to find information about Marvin the Marvin on the Internet. Find information about Marvin on the Internet using your browser.

5. Go to your Favorites file and remove the imusic entry.

Part IV

1. You don't think you are seeing the most recent updates on the web pages due to old versions stored in your browser. Clear out these old files so that there is no old web page information in your browser (Hint: to find this section, you need to start with "view" on your toolbar)

2. Find a place in the browser that has links to interesting new pages.

3. You want to make it so that the imusic site is the page that appears when you first start your browser. Change the default page location in your browser.

Tasks for Netscape

Part I

1. Someone has told you that the address for a really cool music site is located at the URL: "http://www.imusic.com". Use the web browser to visit this site.

2. You notice a link for "music news" on the top of the page. Click on it to see what is there.

3. Once you get to the Music news page, click on the first hyperlink in the list under "Top Stories". Wait for the site to completely finish downloading.

4. You think the first picture on the page you are currently viewing is really cool and you want to save it to your hard drive. Right -click over the image and save it to the desktop.

5. Return to the first imusic page you were on (http://www.imusic.com)

6. You would like to be able to return to the imusic page easily each time you use the browser without having to type in the URL again. Save the page in your bookmark folder it so you can quickly return to this page each time without typing the entire address again.

Part II

1. Go to the home page of this browser by hitting the "home button" on the task bar at the top of the browser.

2. You like this page and want to print it. Print out the page you are now viewing.

3. You have an HTML file on your desktop that you want to see in your browser. Open up that file in your browser. (hint: You start at the "file" option on your browser toolbar)

4. The Whitehouse has a web site! It's in your bookmark list, so go to the site using the bookmarks list.

5. Click on the image of Bill Clinton and Al Gore. Add this page to your bookmarks.

6. Return to the White House Home Page.

7. Go to "The White House for Kids" page and add this page to your Bookmarks list.

Part III

1. Click on the Bookmarks menu item and look at all of the items in the list. You decide that this list is too confusing. Netscape provides a way to organize your Bookmarks list. Organize it by grouping some things together. Create two groupings for your list, "Government" and "Music".

2. Put the 2 White House pages that you placed in your bookmarks in the "Government" group.

3. Put the page of the imusic site in the "Music" group.

4. You want to find information about Marvin the Marvin on the Internet. Find information about Marvin on the Internet using your browser.

5. Go to your bookmark file and remove the imusic entry.

Part IV

1. You don't think you are seeing the most recent updates on the web pages due to old versions stored in your browser. Clear out these old files so that there is no old web page information in your browser (Hint: to find this section, you need to start with Options on the toolbar)

3.Find a place in the browser that has links to interesting new pages.

4. You want to make it so that the imusic site is the page that appears when you first start your browser. Change the default page location in your browser.

Prior to the test, both browsers were set up for maximum similarity. Both browsers were set so that all links were marked as "never visited" and all the temporary Internet files and cache files were cleared out. Both browsers were set so that the "Home" button went to that particular browser's home page.

Upon completing the tasks for each product, the users filled out a satisfaction questionnaire to rate their satisfaction with the different areas of each web browser. The questionnaire was "tried out" on some colleagues who fit the users' profiles prior to the test to avoid ambiguity (Rubin, 1994).

TESTING AND DATA COLLECTION
All testing was done at a desk using a Pentium Pro 200 on a 28.8 modem connection, since that is the modem speed the majority of users would use to connect to the Internet. Since only three people were participating in the study, it was feasible to test each user individually rather than in a group setting, and individual testing often provides more accurate results (Nielson, Mack, 1994).

To measure task timing, start time was recorded when the tester (me) finished reading the task aloud and said then said "go." The stop time was recorded when the user declared that he/she had completed the task. Users were allotted three minutes for each task. If they exceeded the three minute time limit they were told to stop, and the I moved on to the next task. The one exception dealt with download time. In one of the tasks, I asked the user to wait until the site was completely downloaded before I would continue. If the page was not completely downloaded within three minutes, another minute would be allotted. If the page was not finished within that time, the user would be asked to hit the browsers "stop" button. In other instances, I would stop the timer after the actual task was completed, and did not wait for an entire web page to download.

However, sometimes the user could not complete a task until the browser connected to a web site. In some cases, a browser would have trouble connecting, and the time it took to complete the task would be increased. This is a prominent usability problem, since this could cause a user not to finish a task because of the software, or can only do so with more difficulty than is expected or is acceptable (Nielson, Mack, 1994).

Users were asked to complete a 10 question product satisfaction questionnaire after testing each web browser. In this questionnaire, they ranked the product on a scale of 1-7, where 1 meant they strongly agreed with the statement and 7 meant they strongly disagreed with the statement.

For each browser, users rated the following questions:

  1. The functions I expected to complete the tasks were available
  2. The browser was intuitive
  3. I was satisfied with how the browser works
  4. The browser was simple to use
  5. I could effectively complete the tasks
  6. I could complete the tasks quickly
  7. I could complete the tasks efficiently
  8. I thought the "look" of browser was pleasant
  9. I could easily recover from errors
  10. I really liked using the browser. These questions were tailored after the post test given to participants in a Microsoft sponsored usability test for Netscape and Internet Explorer. (Microsoft Corporation January 13, 1997)

    The users were also asked after completing the questionnaire:

    • Which browser they would prefer to use to access the Internet.
    • The specific reasons for their preferences or rating in the questionnaire.
    • If they felt the tasks were biased toward either browser.
    TASK TIMINGS
    The users were timed on each task and an overall time was taken for each browser. The times (in seconds) are the raw timings and do not show whether the users completed a task correctly or incorrectly. The times were added together for each task at the completion of the study.

    The times were broken down as follows:
    NETSCAPE IE
    USER 1 588 seconds 753 seconds
    USER 2 473 seconds 589 seconds
    USER 3 492 seconds 644 seconds


    All users completed the tasks faster using Netscape than IE. These times do not reflect the actual time for the entire test, just the actual tasks themselves.

    TASK SUCCESS RATE
    The Task Success Rate is the percent of tasks that users successfully completed within the three minute time limit. Each user had to complete four sets of tasks for each web browser. If a user incorrectly completed a task, gave up on the task, or exceeded the three minute time limit, the task was not counted toward the total number of tasks completed.

    All but one task was completed properly in this study. In one instance, IE refused to connect to a site and brought back an error message. The site was checked in Netscape (after clearing out the cache) and the site was indeed functional, so the error was not due to the remote server, but was due to the browser itself.

    SATISFACTION QUESTIONNAIRE
    After combining the results of all the satisfaction questionnaires, IE received an average rating of 2.10 on the 7-point scale and Netscape Navigator received an average score of 1.17 on the 7-point scale, meaning that users were, overall, more satisfied with Netscape as opposed to IE.

    COMMENTS
    There were several areas that the users commented on while taking the usability test and filling out the questionnaire. I used the "Thinking out loud" technique to capture what the participants were thinking while they were working (Rubin, 1994). The following is a list of all major comments, both positive and negative for both browsers:

    CONNECTING TO SITES:
    "Why is Internet Explorer taking so long to find a site?"

    "The main problem I had with Internet Explorer was the fact that it took forever to connect to sites."

    "IE is so %$!# slow getting to a site!" "Have we lost the Internet connection? IE can't find this si...oh there it is."

    BROWSER SCROLL BARS:
    "I like how smoothly the IE scroll bar scrolls!"

    "I don't like the way IE scrolls the page - it's weird"

    ON APPEARANCE:
    "Netscape looks "blah" compared to IE"

    "I like the spinning world on the IE logo - it's easier to see than the shooting stars on Netscape."

    "I can't stand how those buttons pop up on the IE toolbar!"

    "It's annoying that those little pop up descriptions are not shown for all tool bar options in IE - it's inconsistent"

    SEARCHING THE INTERNET:
    "Why does IE only provide one search option? Netscape gives you a quite a few."

    "AOL Netfind?? (in IE) That's it? Where are the other search engines?"

    NAVIGATION
    "When IE opens a new browser window, you can still use the "back" button to go back to a previous site from another window. Netscape doesn't allow you to do that, which is annoying."

    BOOKMARKS
    "Netscape's bookmarks aren't as nice looking as IE's Favorites and IE's are more intuitive."

    "When I delete a bookmark in IE it's annoying to wait just so I can see a little folder with paper flying out of it."

    "You can't see the different Favorites in the folders! I hate that! You can see the whole list in Netscape within each category! You have to scroll, but at least you know what is there."

    CONCLUSIONS
    All users were able to complete their tasks faster with Netscape, and all had a higher satisfaction rating for Netscape, though the overall satisfaction was quite high for both browsers. Users also concurred that the test did not seem biased towards either browser. All three users preferred Netscape Navigator over Internet Explorer to access the Internet, though two agreed that both browsers were very good, and that in the absence of Netscape, they wouldn't mind using IE. One user was not "very happy" with IE, yet this user's satisfaction rating for the browser was similar to the other participants.

    Out of a total of 21 tasks, all tasks were completed successfully with Netscape, and 20 of the 21 were completed successfully with IE. The major problem seemed to arise in IE's connection, or lack thereof, to Internet sites. (Not downloading - actual connecting to the site.) The only major problem that arose while testing is that IE refused to connect to a site at one point, and had trouble connecting to sites at other points. This connection time was the major factor in the increased task time for IE over Netscape. In the other areas tested, there did not seem to be a significant difference with regards to the speed at which the tasks were completed though after factoring out connection times, the tasks were still completed faster in Netscape but the difference was not as significant.

    Another possible reason for the increased task completion time for IE could be due to an increased number of keystrokes needed to complete a task. For instance, I observed that users has to click through more options to clear out temporary Internet files, and add items to the favorites section (known as bookmarks in Netscape) .

    One task where users had a specific preference was the search task. Users were asked to search the web for a specific topic. Each browser contains a button that launches a browser specific Web page containing access to various Internet search engines. Users preferred Netscape's search tool because it offered many search engines while IE provided only one. Some users seemed to prefer the look of IE to Netscape, as well as some navigational features.

    Overall, however, users felt that Netscape was an overall better web browser than IE in all the areas surveyed. Users were also able to finish the tasks with faster and with fewer errors using Netscape.

    BIBLIOGRAPHY
    Microsoft Corporation (1997, January 13) [On-Line] "Usability Study - Microsoft Internet Explorer 3.0 vs. Netscape Navigator 3.0 " Microsoft Corporation
    Available: http://www.microsoft.com/ie/ie3/usability.htm

    Nielson, J. Robert, M. (1994). Usability Inspection Methods. New York, NY. John Wiley and Sons, Inc

    Rubin, J. (1994). Handbook of Usability Testing. New York, NY. John Wiley and Sons, Inc.