Development/Projects/AAO CVA Usability testing - Revision history

Dan: Fix images, add page category

2015-07-03T20:36:52Z

Fix images, add page category

Dan: /* Selected Websites */ fix header

2015-07-03T19:17:35Z

‎Selected Websites: fix header

Dan: Fix attachment formatting in some places

2015-07-03T19:13:31Z

Fix attachment formatting in some places

Dan: /* Findings */ Fix pdf links

2015-07-03T19:07:35Z

‎Findings: Fix pdf links

Dan: /* Findings */ Fix CVA PDF link

2015-07-03T19:05:42Z

‎Findings: Fix CVA PDF link

Dan at 18:59, 3 July 2015

2015-07-03T18:59:23Z

Dan: /* Background */ Move image, adjust size

2015-07-03T18:54:24Z

‎Background: Move image, adjust size

Dan: /* Background */ Remove dead link, adjust image size

2015-07-03T18:51:54Z

‎Background: Remove dead link, adjust image size

Dan: migrate basic content to page from defunct Artefactual wiki

2015-07-03T18:50:43Z

migrate basic content to page from defunct Artefactual wiki

New page

{{#pagetitle:AAO and City of Vancouver Usability testing}}
[[Main Page]] > [[Development]] > [[Development/Projects]] > Development/Projects/AAO CVA Usability testing

<admonition type="note">
This is historical development documentation, that refers to an earlier version of ICA-AtoM. This page was first created February 9th, 2011, and its content was last updated on April 21, 2011. The content below was moved to the new AtoM wiki on July 3, 2015.
</admonition>

=Background=
While the developers of the ICA-AtoM software were able to do functionality testing of the software, they were only able to speculate as to possible end-user issues as they relate to the effectiveness and usability of the software. The Developers were interested in determining how effective, efficient and satisfying it is for an end-user to accomplish certain tasks in the ICA-AtoM software. The Developers were primarily concerned about user feedback on the core functionality of the software; namely searching and browsing for archival materials and creating and editing archival descriptions. A simulated ‘real world’ Usability Study was identified as desirable to identify any potential end-user issues as ICA-AtoM expands its user base.

One set of usability studies were conducted in collaboration with the University of Toronto iSchool and the Archives Association of Ontario as part of its ARCHEION migration project.

* [http://www.aao-archivists.ca/resources/document-downloads/doc_download/65-invite-archeion-volunteers-wanted AAO call] for usability testing volunteers
Additional usability testing was conducted with staff archivists at the City of Vancouver Archives to establish the effectiveness and usefulness of ICA-AtoM and to detect any major usability deficiencies. This work is part of the CVA's Archivematica and ICA-AtoM integration project.

[[File:AAO-Usability.png|680px|thumb|right|]]

=Definition=

'''Usability is defined as the extent to which a product can be used by specific users to achieve specified goals with effectiveness, efficiency and satisfaction.'''
* Effectiveness: The accuracy and completeness with which users achieve specified goals.
* Efficiency: The resources expended in relation to the accuracy and completeness with which users achieve goals.
* Satisfaction: The comfort and acceptability of use.

See, [http://www.iso.org/iso/iso_catalogue/cacatalogue_tc/catalogue_detail.htm?csnumber=16883 ISO 9241-11:1998 Guidance on Usability]

'''Usability can address both the detailed software design activity and the goal that the software meets user needs.'''

See, [http://www.usabilitynet.org/papers/qiuse.pdf| Nigel Bevan, Quality in Use: "Meeting User Needs for Quality" ''Journal of System and Software''(1999)]

=Methodology=

Our goal is to conduct “simple user testing” as a method to debug our design.
Our approach is simple, small-sample, do-it-yourself usability testing that fits within the short timelines presented by working within an Agile development methodology.

A literature review of usability testing undertaken in the disciplines of archival, library and information sciences led us to decide upon a qualitative framework utilizing a small, but selected sample group of archivists, researchers and inexperienced users. The most relevant usability study reviewed was Joyce Celeste Chapman's Empirical Analysis of User Interaction with Online Finding Aids, conducted at the Usability Lab at the Davis Library at UNV-Chapel Hill from April to May 2009.

Our Usability Study for the Archives Association of Ontario (AAO) at the University of Toronto School of Information Studies and at the City of Vancouver Archives (CVA) will consist of two parts:

*series of tasks to be completed by each participant through interaction with ICA-AtoM software;
*and a post-task questionnaire.

Our AAO study will test three types of online archival management software users. The first group will consist of 7 Archivists that have experience with data entry into ARCHEION, 5 Researchers with experience searching ARCHEION and 8 inexperienced internet Users. By keeping the groups small, it will enable us to test in the morning and review results in the afternoon. The degree of familiarity with ICA-AtoM software will be part of our documented questions, and it would be great to have participants with a spectrum of prior experience with the software; however, this may not be possible depending upon scheduling availability. Ideally we will schedule regular Usability testing in tandem with development cycles.

Our CVA study will test 12 archives staff members (project, part-time and full) over a period of two days. Three out of twelve tested ICA-AtoM using the Researchers set tasks and nine out of twelve tested ICA-AtoM using the Archivists set tasks.

A list of tasks will be drafted for testing of each user group.

* Archivists: The tasks will be based on workflow that supports archivist's routine data input, search actions, and export operations.
* Researchers: The tasks will be based on resource discovery procedures, such as browse and search actions for repositories, fonds-level descriptions and a variety of textual and non-textual items. General areas to be targeted by scenarios: navigation, structure, browse and search functions, help features, and archival terminology and labelling.
* Inexperienced User: The tasks will be based on general resource discovery procedures in an online environment.

=Schedule=

=== Archival Association Ontario (AAO) / ICA-AtoM Usability Testing Dates ===

* March 09.2011 - March 11.2011
* March 14.2011
* March 16.2011 & March 17.2011

=== City of Vancouver Archives / ICA-AtoM Usability Testing Dates ===

* April 05.2011 & April 06.2011

=Findings=

===AAO Usability Testing of ICA-AtoM Findings===

'''The overall response to ICA-AtoM by the twenty participants that performed the set tasks and answered the post-task questionnaire was positive:'''
* 95% of all participants found the layout of ICA-AtoM to be clear,
* 85% of all participants felt that ICA-AtoM was easy to use and quick to learn,
* 90% of participants were satisfied with ICA-AtoM, and
* 70% of the Archivists that tested the software said they would recommend ICA-AtoM to a colleague.
 
'''The study provided valuable feedback regarding shortfalls in navigation and functionality, such as:'''
* 70% of Archivists testing the software felt that ICA-AtoM did not work the way they wanted it to work,
* 43% of Archivists felt they could not use ICA-AtoM successfully every time, and
* 25% of all participants felt lost at some point during the set tasks.
 
'''Changes suggested by participants to improve Usability of ICA-AtoM:'''
* Implement an Advanced Search,
* Clarify Browse Tab navigation and functionality,
* Provide contextual information for labels,
* Revise Design Interface to support larger text, minimize scrolling, and highlight edit operations.
 
All post-task questions and their responses are available [[File:AAO_Charts.pdf|200px|right|thumb|AAO_Charts.pdf]]

===CVA Usability Testing of ICA-AtoM Findings===

'''The general response to ICA-AtoM by the twelve participants that performed the set tasks and answered the post-task questionnaire was positive:'''
* 83% of all participants found the layout of ICA-AtoM to be clear,
* 67% of all participants felt that ICA-AtoM was both fun and satisfying to use, and
* 75% of all participants said they would recommend ICA-AtoM to a colleague.
 

'''The study provided valuable feedback regarding shortfalls in navigation and functionality, such as:'''
* 67% of all participants testing the software felt that ICA-AtoM did not work the way they wanted it to work,
* 58% of all participants felt they could not use ICA-AtoM successfully every time, and
* 58% of all participants felt lost at some point during the set tasks.
 

'''Changes suggested by participants to improve Usability of ICA-AtoM:'''
* Implement an Advanced Search,
* Clarify functionality and labels for digital object linking and import,
* Provide contextual information for labels,
* Implement spell checking in data entry fields, and
* Revise Design Interface to support larger text, minimize scrolling, and highlight edit operations.
 

All post-task questions and their responses are available [[File:CVA_Charts.pdf|CVA_Charts.pdf]]

=Further Activities: User Interface Mock-ups=

===AAO UI Mock-ups===

In response to the findings from the AAO ICA-AtoM Usability Study, changes to the current ICA-AtoM user interface have been suggested.

These have been translated into a set of UI feature mock-ups presented here [[File:AAO_UI_features_mockups_v03.pdf|AAO_UI_Features.pdf]]

=Further Activities: Feature enhancements=

An additional question was included in the CVA post-task questionnaire asking participants to comment on how the software could be improved to support their workflow. The majority of comments made by participants in response to this question highlighted the need for Advanced Search capabilities, the option to create record skeletons for data entry for specific record series or media types and bulk updating and global corrections. With the exception of the record skeleton (this functionality is addressed in ICA-AtoM through the use of “Duplicate” record.)

'''List of feature upgrade suggestions''':
# Advanced Search: Need ways to narrow result sets. Narrow by hierarchical level. Drop-downs lists of field to search such as date or geographic area. Search using subject or keyword. Facetted searching.
# Search results hit list at top and bottom of page.
# Incorporate hover text over labels to explain performance.
# Sort default set to Alphabetical.
# Global Corrections and Bulk Updating needed.
# Digital objects import and link terms need clarification and possible redesign.
# Confirmation of changed/saved record and confirmation of imported digital object needed.
# Spellcheck function in data entry needs to be added.
# Graphic designer edit of Text (size, font, colour, spacing), Search Box Alignment, Tabs (size, colour, spacing), and Labels (size, colour, spacing.)

=Selected Websites=

===== This is a List of Websites with Noteworthy Usability =====
* [http://www.oac.cdlib.org/findaid/ark:/13030/tf196nb4nt/ Online Archive of California] Finding Aid Interface
[[Image:OAC_FindingAidView.jpg|200px|right|thumb|OAC Finding Aid Interface]]
:Features:
:* Browse access to an Overview of the Collection and the Finding Aid
:* Collapsible areas for easy navigation through the Finding Aid
:* PDF and HTML versions of Collection Guide Available
:* Easy to find links to online images and specific repository
 

* [http://nwda-db.wsulibs.wsu.edu/nwda-search/fstyle.aspx?doc=ALKPCA277.xml&t=k&q=dog Northwest Digital Archives] Finding Aid Interface
[[Image:NWDA_FindingAidView.jpg|200px|right|thumb|NWDA Finding Aid Interface]]
:Features:
:* Collapsible areas for easy navigation though the Finding Aid and Container List
:* Easy to find links to PRINT and EMAIL Finding Aid
 

* [http://drs.library.yale.edu:8083/fedoragsearch/rest Yale University] Search Repository and Finding Aids
[[Image:YALE_MultiInsPortalSearch.jpg|200px|right|thumb|YALE Search Interface]]
:Features:
:* Search individual institutional websites
:* Federated searching of holdings held across all institutions
:* Advanced search with drop-down lists relating to specific areas of a finding aid
 

* [http://www.wvma.net/AdvSearch.aspx West Vancouver Archives] Advanced Search
[[Image:WVan_AdvSearch.jpg|200px|right|thumb|WEST VANCOUVER Advanced Search Interface]]
:Features:
:* Combination of features offers the User a high degree of control and wide range of support/tips to enable the creation of precise searching throughout the Fonds/Collection and down to the specific Item.
 

* [http://lesleyuniversityarchives.org/Archon/index.php?p=subjects/subjects Lesley University] Subject Browse
[[Image:LesUniv_SubjectBrowseOption.jpg|200px|right|thumb|LESLEY UNIVERSITY Subject Browse Interface]]
:Features:
:* Combination of features offers the User a unique browse interface that is built upon controlled vocabularies and authority files.
 

* [http://bentley.umich.edu/EAD/index.php Bentley Historical Library/UMichigan] Finding Aid Support
[[Image:UMich_HelpEAD.jpg|200px|right|thumb|BENTLEY HISTORICAL LIBRARY Finding Aid Support Interface]]
:Features:
:* Interesting approach to educating the User about Finding Aids and EAD, in an effort to support more successful searching and discovery of the holdings.

__NOTOC__

@@ Line 97: / Line 97: @@
 * Revise Design Interface to support larger text, minimize scrolling, and highlight edit operations.
 <br />
 All post-task questions and their responses are available: [[File:AAO_Charts.pdf|AAO_Charts.pdf]]
@@ Line 122: / Line 123: @@
 <br />
-All post-task questions and their responses are available: [[File:CVA_Charts.pdf|CVA_Charts.pdf]] (PDF, 97.4KB)
+All post-task questions and their responses are available: [[File:CVA_Charts.pdf|CVA_Charts.pdf]] (PDF, 98 KB)
 =Further Activities: User Interface Mock-ups=

@@ Line 162: / Line 162: @@
 * [http://www.oac.cdlib.org/findaid/ark:/13030/tf196nb4nt/ Online Archive of California] Finding Aid Interface
 :Features:
 :* Browse access to an Overview of the Collection and the Finding Aid
@@ Line 168: / Line 167: @@
 :* PDF and HTML versions of Collection Guide Available
 :* Easy to find links to online images and specific repository
 <br />
 * [http://nwda-db.wsulibs.wsu.edu/nwda-search/fstyle.aspx?doc=ALKPCA277.xml&t=k&q=dog Northwest Digital Archives] Finding Aid Interface
 :Features:
 :* Collapsible areas for easy navigation though the Finding Aid and Container List
 :* Easy to find links to PRINT and EMAIL Finding Aid
 <br />
 * [http://drs.library.yale.edu:8083/fedoragsearch/rest Yale University] Search Repository and Finding Aids
 :Features:
 :* Search individual institutional websites
 :* Federated searching of holdings held across all institutions
 :* Advanced search with drop-down lists relating to specific areas of a finding aid
 <br />
 * [http://www.wvma.net/AdvSearch.aspx West Vancouver Archives] Advanced Search
-[[Image:WVan_AdvSearch.jpg|200px|right|thumb|WEST VANCOUVER Advanced Search Interface]]
 :Features:
 :* Combination of features offers the User a high degree of control and wide range of support/tips to enable the creation of precise searching throughout the Fonds/Collection and down to the specific Item.
 <br />
 * [http://lesleyuniversityarchives.org/Archon/index.php?p=subjects/subjects Lesley University] Subject Browse
 :Features:
 :* Combination of features offers the User a unique browse interface that is built upon controlled vocabularies and authority files.
 <br />
 * [http://bentley.umich.edu/EAD/index.php Bentley Historical Library/UMichigan] Finding Aid Support
 :Features:
 :* Interesting approach to educating the User about Finding Aids and EAD, in an effort to support more successful searching and discovery of the holdings.
 __NOTOC__

@@ Line 159: / Line 159: @@
 =Selected Websites=
-===== This is a List of Websites with Noteworthy Usability =====
+'''This is a List of Websites with Noteworthy Usability'''
 * [http://www.oac.cdlib.org/findaid/ark:/13030/tf196nb4nt/ Online Archive of California] Finding Aid Interface
 [[Image:OAC_FindingAidView.jpg|200px|right|thumb|OAC Finding Aid Interface]]

@@ Line 3: / Line 3: @@
 <admonition type="note">
-This is historical development documentation, that refers to an earlier version of ICA-AtoM. This page was first created February 9th, 2011, and its content was last updated on April 21, 2011. The content below was moved to the new AtoM wiki on July 3, 2015.
+This is historical development documentation, that refers to an earlier version of ICA-AtoM. This page was first created February 9th, 2011, and its content was last updated on April 21, 2011. The content below was moved to the new AtoM wiki on July 3, 2015. Some changes have since been made to fix the formatting for greater readability, or to remove dead links, etc. No changes to the content itself have been made since 2011.
 </admonition>
@@ Line 98: / Line 98: @@
 <br />
-All post-task questions and their responses are available: [[File:AAO_Charts.pdf|AAO_Charts.pdf]] (PDF, 91 KB)
+All post-task questions and their responses are available:
@@ Line 123: / Line 125: @@
 <br />
-All post-task questions and their responses are available: [[File:CVA_Charts.pdf|CVA_Charts.pdf]] (PDF, 98 KB)
+All post-task questions and their responses are available:
-=Further Activities: User Interface Mock-ups=
+=Further Activities=
-===AAO UI Mock-ups===
+===AAO User Interface Mock-ups===
 In response to the findings from the AAO ICA-AtoM Usability Study, changes to the current ICA-AtoM user interface have been suggested.
-These have been translated into a set of UI feature mock-ups presented here [[File:AAO_UI_features_mockups_v03.pdf|AAO_UI_Features.pdf]]
+These have been translated into a set of UI feature mock-ups presented here:
-=Further Activities: Feature enhancements=
+===CVA Feature enhancements===
 An additional question was included in the CVA post-task questionnaire asking participants to comment on how the software could be improved to support their workflow. The majority of comments made by participants in response to this question highlighted the need for Advanced Search capabilities, the option to create record skeletons for data entry for specific record series or media types and bulk updating and global corrections. With the exception of the record skeleton (this functionality is addressed in ICA-AtoM through the use of “Duplicate” record.)
 '''List of feature upgrade suggestions''':
-# Advanced Search: Need ways to narrow result sets. Narrow by hierarchical level. Drop-downs lists of field to search such as date or geographic area. Search using subject or keyword. Facetted searching.
 # Search results hit list at top and bottom of page.
 # Incorporate hover text over labels to explain performance.

@@ Line 25: / Line 25: @@
 See, [http://www.iso.org/iso/iso_catalogue/cacatalogue_tc/catalogue_detail.htm?csnumber=16883 ISO 9241-11:1998 Guidance on Usability]
 '''Usability can address both the detailed software design activity and the goal that the software meets user needs.'''
-See, [http://www.usabilitynet.org/papers/qiuse.pdf| Nigel Bevan, Quality in Use: "Meeting User Needs for Quality" ''Journal of System and Software''(1999)]
+See: Nigel Bevan, Quality in Use: "Meeting User Needs for Quality" ''Journal of System and Software''(1999)

@@ Line 11: / Line 11: @@
 One set of usability studies were conducted in collaboration with the University of Toronto iSchool and the Archives Association of Ontario as part of its ARCHEION migration project.
 Additional usability testing was conducted with staff archivists at the City of Vancouver Archives to establish the effectiveness and usefulness of ICA-AtoM and to detect any major usability deficiencies. This work is part of the CVA's Archivematica and ICA-AtoM integration project.
 =Definition=