AAO and City of Vancouver Usability testing
Main Page > Development > Development/Projects > Development/Projects/AAO CVA Usability testing
Note
This is historical development documentation, that refers to an earlier version of ICA-AtoM. This page was first created February 9th, 2011, and its content was last updated on April 21, 2011. The content below was moved to the new AtoM wiki on July 3, 2015. Some changes have since been made to fix the formatting for greater readability, or to remove dead links, etc. No changes to the content itself have been made since 2011.
Background
While the developers of the ICA-AtoM software were able to do functionality testing of the software, they were only able to speculate as to possible end-user issues as they relate to the effectiveness and usability of the software. The Developers were interested in determining how effective, efficient and satisfying it is for an end-user to accomplish certain tasks in the ICA-AtoM software. The Developers were primarily concerned about user feedback on the core functionality of the software; namely searching and browsing for archival materials and creating and editing archival descriptions. A simulated ‘real world’ Usability Study was identified as desirable to identify any potential end-user issues as ICA-AtoM expands its user base.
One set of usability studies were conducted in collaboration with the University of Toronto iSchool and the Archives Association of Ontario as part of its ARCHEION migration project.
Additional usability testing was conducted with staff archivists at the City of Vancouver Archives to establish the effectiveness and usefulness of ICA-AtoM and to detect any major usability deficiencies. This work is part of the CVA's Archivematica and ICA-AtoM integration project.
Definition
Usability is defined as the extent to which a product can be used by specific users to achieve specified goals with effectiveness, efficiency and satisfaction.
- Effectiveness: The accuracy and completeness with which users achieve specified goals.
- Efficiency: The resources expended in relation to the accuracy and completeness with which users achieve goals.
- Satisfaction: The comfort and acceptability of use.
See, ISO 9241-11:1998 Guidance on Usability
Usability can address both the detailed software design activity and the goal that the software meets user needs.
See: Nigel Bevan, Quality in Use: "Meeting User Needs for Quality" Journal of System and Software(1999)
Methodology
Our goal is to conduct “simple user testing” as a method to debug our design. Our approach is simple, small-sample, do-it-yourself usability testing that fits within the short timelines presented by working within an Agile development methodology.
A literature review of usability testing undertaken in the disciplines of archival, library and information sciences led us to decide upon a qualitative framework utilizing a small, but selected sample group of archivists, researchers and inexperienced users. The most relevant usability study reviewed was Joyce Celeste Chapman's Empirical Analysis of User Interaction with Online Finding Aids, conducted at the Usability Lab at the Davis Library at UNV-Chapel Hill from April to May 2009.
Our Usability Study for the Archives Association of Ontario (AAO) at the University of Toronto School of Information Studies and at the City of Vancouver Archives (CVA) will consist of two parts:
- series of tasks to be completed by each participant through interaction with ICA-AtoM software;
- and a post-task questionnaire.
Our AAO study will test three types of online archival management software users. The first group will consist of 7 Archivists that have experience with data entry into ARCHEION, 5 Researchers with experience searching ARCHEION and 8 inexperienced internet Users. By keeping the groups small, it will enable us to test in the morning and review results in the afternoon. The degree of familiarity with ICA-AtoM software will be part of our documented questions, and it would be great to have participants with a spectrum of prior experience with the software; however, this may not be possible depending upon scheduling availability. Ideally we will schedule regular Usability testing in tandem with development cycles.
Our CVA study will test 12 archives staff members (project, part-time and full) over a period of two days. Three out of twelve tested ICA-AtoM using the Researchers set tasks and nine out of twelve tested ICA-AtoM using the Archivists set tasks.
A list of tasks will be drafted for testing of each user group.
- Archivists: The tasks will be based on workflow that supports archivist's routine data input, search actions, and export operations.
- Researchers: The tasks will be based on resource discovery procedures, such as browse and search actions for repositories, fonds-level descriptions and a variety of textual and non-textual items. General areas to be targeted by scenarios: navigation, structure, browse and search functions, help features, and archival terminology and labelling.
- Inexperienced User: The tasks will be based on general resource discovery procedures in an online environment.
Schedule
Archival Association Ontario (AAO) / ICA-AtoM Usability Testing Dates
- March 09.2011 - March 11.2011
- March 14.2011
- March 16.2011 & March 17.2011
City of Vancouver Archives / ICA-AtoM Usability Testing Dates
- April 05.2011 & April 06.2011
Findings
AAO Usability Testing of ICA-AtoM Findings
The overall response to ICA-AtoM by the twenty participants that performed the set tasks and answered the post-task questionnaire was positive:
- 95% of all participants found the layout of ICA-AtoM to be clear,
- 85% of all participants felt that ICA-AtoM was easy to use and quick to learn,
- 90% of participants were satisfied with ICA-AtoM, and
- 70% of the Archivists that tested the software said they would recommend ICA-AtoM to a colleague.
The study provided valuable feedback regarding shortfalls in navigation and functionality, such as:
- 70% of Archivists testing the software felt that ICA-AtoM did not work the way they wanted it to work,
- 43% of Archivists felt they could not use ICA-AtoM successfully every time, and
- 25% of all participants felt lost at some point during the set tasks.
Changes suggested by participants to improve Usability of ICA-AtoM:
- Implement an Advanced Search,
- Clarify Browse Tab navigation and functionality,
- Provide contextual information for labels,
- Revise Design Interface to support larger text, minimize scrolling, and highlight edit operations.
All post-task questions and their responses are available:
- File:AAO Charts.pdf (PDF, 91 KB)
CVA Usability Testing of ICA-AtoM Findings
The general response to ICA-AtoM by the twelve participants that performed the set tasks and answered the post-task questionnaire was positive:
- 83% of all participants found the layout of ICA-AtoM to be clear,
- 67% of all participants felt that ICA-AtoM was both fun and satisfying to use, and
- 75% of all participants said they would recommend ICA-AtoM to a colleague.
The study provided valuable feedback regarding shortfalls in navigation and functionality, such as:
- 67% of all participants testing the software felt that ICA-AtoM did not work the way they wanted it to work,
- 58% of all participants felt they could not use ICA-AtoM successfully every time, and
- 58% of all participants felt lost at some point during the set tasks.
Changes suggested by participants to improve Usability of ICA-AtoM:
- Implement an Advanced Search,
- Clarify functionality and labels for digital object linking and import,
- Provide contextual information for labels,
- Implement spell checking in data entry fields, and
- Revise Design Interface to support larger text, minimize scrolling, and highlight edit operations.
All post-task questions and their responses are available:
- File:CVA Charts.pdf (PDF, 98 KB)
Further Activities
AAO User Interface Mock-ups
In response to the findings from the AAO ICA-AtoM Usability Study, changes to the current ICA-AtoM user interface have been suggested.
These have been translated into a set of UI feature mock-ups presented here:
- File:AAO UI features mockups v03.pdf (PDF, 460 KB)
CVA Feature enhancements
An additional question was included in the CVA post-task questionnaire asking participants to comment on how the software could be improved to support their workflow. The majority of comments made by participants in response to this question highlighted the need for Advanced Search capabilities, the option to create record skeletons for data entry for specific record series or media types and bulk updating and global corrections. With the exception of the record skeleton (this functionality is addressed in ICA-AtoM through the use of “Duplicate” record.)
List of feature upgrade suggestions:
- Advanced Search: Need ways to narrow result sets. Narrow by hierarchical level. Drop-downs lists of field to search such as date or geographic area. Search using subject or keyword. Faceted searching.
- Search results hit list at top and bottom of page.
- Incorporate hover text over labels to explain performance.
- Sort default set to Alphabetical.
- Global Corrections and Bulk Updating needed.
- Digital objects import and link terms need clarification and possible redesign.
- Confirmation of changed/saved record and confirmation of imported digital object needed.
- Spellcheck function in data entry needs to be added.
- Graphic designer edit of Text (size, font, colour, spacing), Search Box Alignment, Tabs (size, colour, spacing), and Labels (size, colour, spacing.)
Selected Websites
This is a List of Websites with Noteworthy Usability
- Online Archive of California Finding Aid Interface
- Features:
- Browse access to an Overview of the Collection and the Finding Aid
- Collapsible areas for easy navigation through the Finding Aid
- PDF and HTML versions of Collection Guide Available
- Easy to find links to online images and specific repository
- Northwest Digital Archives Finding Aid Interface
- Features:
- Collapsible areas for easy navigation though the Finding Aid and Container List
- Easy to find links to PRINT and EMAIL Finding Aid
- Yale University Search Repository and Finding Aids
- Features:
- Search individual institutional websites
- Federated searching of holdings held across all institutions
- Advanced search with drop-down lists relating to specific areas of a finding aid
- West Vancouver Archives Advanced Search
- Features:
- Combination of features offers the User a high degree of control and wide range of support/tips to enable the creation of precise searching throughout the Fonds/Collection and down to the specific Item.
- Lesley University Subject Browse
- Features:
- Combination of features offers the User a unique browse interface that is built upon controlled vocabularies and authority files.
- Bentley Historical Library/UMichigan Finding Aid Support
- Features:
- Interesting approach to educating the User about Finding Aids and EAD, in an effort to support more successful searching and discovery of the holdings.
