AtoM wiki - User contributions [en]

User:Samason

2025-02-27T17:05:39Z

Sromkey: Creating user page for new user.

I am the Contributor Success Specialist at Artefactual Systems Inc.

User:Mkosta

2024-03-22T16:24:17Z

Sromkey: Creating user page for new user.

I am an archivist who lives in London, Ontario, Canada. I'm a strong supporter of open-source software and of AtoM.

Development/Functional testing/Rights module tests

2016-06-17T21:42:54Z

Sromkey: Added Administer section

{{#pagetitle: Right module tests }}
[[Main Page]] > [[Development]] > [[Development/Functional testing]] > Development/Functional testing/Rights module tests

== Create ==
A logged-in User can create one or more rights for an accession, archival description(s) and digital object(s).

*In the Edit accession record screen, User can Click on Rights area.(Additionally, test the Edit archival descriptions screen and the Edit digital objects screen. Issue 2363 )
*Can Click Add new hyperlink and rights data entry dialog box opens
**Can select Acts from drop-down pick list provides users with 7 acts to select: delete, discover, display, disseminate, modify, migrate and replicate
**Can select restriction drop-down list, two options: Allow and Disallow
**Can modify date field "start" and enter data into date field "end"
**Can select existing names from dynamic list in Rights holder field, or can input new name into field.
**Can enter free text in Rights notes field
**Can select Basis from drop-down list: copyright, license, statute, policy and donor - further relevant contextual fields should appear depending on basis
**Can Click on Submit. After submitting rights, User can view Rights records information: Act, Restriction, Start and End.
***Option to click save [description; accession; digital object; etc] record to make sure rights holder name is presented in the view record screen as a hyperlink. Click on hyperlink rights holder name and User is redirected to view rights holder screen.
*After completing all data input, Click on Save button and view accession record screen.
*Click create information object and view archival description screen. All rights should remain linked.
*Click import digital objects and view archival description screen. All rights should remain linked.

== Edit ==
A logged-in User can edit one or more rights records for an accession record and archival description(s).

*User can view inherited/existing Rights records in the Rights area of the view accession record screen and view archival description screen.
**Can view and edit rights area information: Act, Restriction, Start and End.
**Can edit existing rights by clicking on pencil icon, delete existing rights by clicking on "x" or click on Add new hyperlink to create new rights
**Can edit existing rights and submit
**Can Add new rights and submit
**Can edit existing rights and submit, then delete rights and Click on Save button Issue 2335
**Can delete existing rights and Click on Save button
*Click on Save button

A logged-in User can add one or more rights records to the archival description of digital object(s)

*User can view inherited/existing Rights records in the view archival description screen for a digital object
**Cannot view inherited/existing Rights record in the edit archival description screen.
**Can Add new rights record to digital object, click Submit.
*Click Save button and view archival description screen for digital object. New rights record should be added at the digital object level, while inherited/existing Rights record should not be overwritten by new rights record.

A logged-in User can cascade rights from a parent record to its children
*User can click manage rights inheritance at the parent level.
**Can choose to apply rights to all children or only children with digital objects
**Can choose to delete current rights and replace with parent rights or keep current rights and add parent rights.
*Click apply and check child records to ensure the rights were inherited correctly.

==Administer==
A logged-in administrator can set PREMIS access permissions.

Navigate to Administer - Settings - Permissions

*Can choose a PREMIS act to base permissions on
**Can set permissions for Allow, Conditional, and Disallow per-Basis
*Can set Disallow and Conditional Statements per Basis
* Can enable and edit Copyright statement for Conditional copyrights

== Manage Rights holders ==
A logged-in User can manage information about rights holders

*Can select Rights holders from Manage menu
**Can view Browse rights holder screen
**Can sort order of rights holder list by alphabetic or recent changes
**Can add name into secondary search box and Click on Search rights holder button
**Search rights holder screen, can Click hyperlink rights holder name

[[Category:Development documentation]]

__NOTOC__

Development/Functional testing/Rights module tests

2016-06-17T20:23:17Z

Sromkey: /* Edit */

{{#pagetitle: Right module tests }}
[[Main Page]] > [[Development]] > [[Development/Functional testing]] > Development/Functional testing/Rights module tests

== Create ==
A logged-in User can create one or more rights for an accession, archival description(s) and digital object(s).

*In the Edit accession record screen, User can Click on Rights area.(Additionally, test the Edit archival descriptions screen and the Edit digital objects screen. Issue 2363 )
*Can Click Add new hyperlink and rights data entry dialog box opens
**Can select Acts from drop-down pick list provides users with 7 acts to select: delete, discover, display, disseminate, modify, migrate and replicate
**Can select restriction drop-down list, two options: Allow and Disallow
**Can modify date field "start" and enter data into date field "end"
**Can select existing names from dynamic list in Rights holder field, or can input new name into field.
**Can enter free text in Rights notes field
**Can select Basis from drop-down list: copyright, license, statute, policy and donor - further relevant contextual fields should appear depending on basis
**Can Click on Submit. After submitting rights, User can view Rights records information: Act, Restriction, Start and End.
***Option to click save [description; accession; digital object; etc] record to make sure rights holder name is presented in the view record screen as a hyperlink. Click on hyperlink rights holder name and User is redirected to view rights holder screen.
*After completing all data input, Click on Save button and view accession record screen.
*Click create information object and view archival description screen. All rights should remain linked.
*Click import digital objects and view archival description screen. All rights should remain linked.

== Edit ==
A logged-in User can edit one or more rights records for an accession record and archival description(s).

*User can view inherited/existing Rights records in the Rights area of the view accession record screen and view archival description screen.
**Can view and edit rights area information: Act, Restriction, Start and End.
**Can edit existing rights by clicking on pencil icon, delete existing rights by clicking on "x" or click on Add new hyperlink to create new rights
**Can edit existing rights and submit
**Can Add new rights and submit
**Can edit existing rights and submit, then delete rights and Click on Save button Issue 2335
**Can delete existing rights and Click on Save button
*Click on Save button

A logged-in User can add one or more rights records to the archival description of digital object(s)

*User can view inherited/existing Rights records in the view archival description screen for a digital object
**Cannot view inherited/existing Rights record in the edit archival description screen.
**Can Add new rights record to digital object, click Submit.
*Click Save button and view archival description screen for digital object. New rights record should be added at the digital object level, while inherited/existing Rights record should not be overwritten by new rights record.

A logged-in User can cascade rights from a parent record to its children
*User can click manage rights inheritance at the parent level.
**Can choose to apply rights to all children or only children with digital objects
**Can choose to delete current rights and replace with parent rights or keep current rights and add parent rights.
*Click apply and check child records to ensure the rights were inherited correctly.

== Manage Rights holders ==
A logged-in User can manage information about rights holders

*Can select Rights holders from Manage menu
**Can view Browse rights holder screen
**Can sort order of rights holder list by alphabetic or recent changes
**Can add name into secondary search box and Click on Search rights holder button
**Search rights holder screen, can Click hyperlink rights holder name

[[Category:Development documentation]]

__NOTOC__

Development/Functional testing/Rights module tests

2016-06-17T19:50:47Z

Sromkey: Removed: "Rights are also present on the Digital object edit page. "

{{#pagetitle: Right module tests }}
[[Main Page]] > [[Development]] > [[Development/Functional testing]] > Development/Functional testing/Rights module tests

== Create ==
A logged-in User can create one or more rights for an accession, archival description(s) and digital object(s).

*In the Edit accession record screen, User can Click on Rights area.(Additionally, test the Edit archival descriptions screen and the Edit digital objects screen. Issue 2363 )
*Can Click Add new hyperlink and rights data entry dialog box opens
**Can select Acts from drop-down pick list provides users with 7 acts to select: delete, discover, display, disseminate, modify, migrate and replicate
**Can select restriction drop-down list, two options: Allow and Disallow
**Can modify date field "start" and enter data into date field "end"
**Can select existing names from dynamic list in Rights holder field, or can input new name into field.
**Can enter free text in Rights notes field
**Can select Basis from drop-down list: copyright, license, statute, policy and donor - further relevant contextual fields should appear depending on basis
**Can Click on Submit. After submitting rights, User can view Rights records information: Act, Restriction, Start and End.
***Option to click save [description; accession; digital object; etc] record to make sure rights holder name is presented in the view record screen as a hyperlink. Click on hyperlink rights holder name and User is redirected to view rights holder screen.
*After completing all data input, Click on Save button and view accession record screen.
*Click create information object and view archival description screen. All rights should remain linked.
*Click import digital objects and view archival description screen. All rights should remain linked.

== Edit ==
A logged-in User can edit one or more rights records for an accession record and archival description(s).

*User can view inherited/existing Rights records in the Rights area of the view accession record screen and view archival description screen.
**Can view and edit rights area information: Act, Restriction, Start and End.
**Can edit existing rights by clicking on pencil icon, delete existing rights by clicking on "x" or click on Add new hyperlink to create new rights
**Can edit existing rights and submit
**Can Add new rights and submit
**Can edit existing rights and submit, then delete rights and Click on Save button Issue 2335
**Can delete existing rights and Click on Save button
*Click on Save button

A logged-in User can add one or more rights records to the archival description of digital object(s)

*User can view inherited/existing Rights records in the view archival description screen for a digital object
**Cannot view inherited/existing Rights record in the edit archival description screen.
**Can Add new rights record to digital object, click Submit.
*Click Save button and view archival description screen for digital object. New rights record should be added at the digital object level, while inherited/existing Rights record should not be overwritten by new rights record.

== Manage Rights holders ==
A logged-in User can manage information about rights holders

*Can select Rights holders from Manage menu
**Can view Browse rights holder screen
**Can sort order of rights holder list by alphabetic or recent changes
**Can add name into secondary search box and Click on Search rights holder button
**Search rights holder screen, can Click hyperlink rights holder name

[[Category:Development documentation]]

__NOTOC__

Community/Users

2015-10-16T15:37:54Z

Sromkey: /* AtoM (2.x) sites */

{{#pagetitle: Users }}
[[Main Page]] > [[Community]] > Community/Users

On this page you'll find some examples of institutions and organizations using AtoM and/or ICA-AtoM. Don't see your institution on this list? Please feel free to [[Special:UserLogin|add a link]] to your AtoM installation!

==AtoM (2.x) sites==

* [http://www.albertaonrecord.ca/ Alberta on Record: Archives Society of Alberta Portal]
* [http://archeion.ca Archeion: Ontario's Archival Information Network]
* [https://archie.library.carleton.ca/ Archives & Research Collections, Carleton University Library]
* [http://www.archivescommunales-vd.ch/ Archives comunales du canton de Vaud] (Swiss)
* [http://anglicanarchives.ica-atom.org/ Archives of the Anglican Diocese of New Westminster and Provincial Synod of BC and Yukon]
* [http://archivesdemontreal.ica-atom.org/ Archives de Montréal]
* [http://arc.anla.nf.ca/ ARC: the Archival Resource Catalogue of the Association of Newfoundland and Labrador Archives]
* [http://acervo.cidarq.ufg.br/index.php/iuryy;isad Acervos Arquivísticos] (Brazilian Portuguese)
* [http://archives.canbarchives.ca/ ArchivesCANB: the Council of Archives New Brunswick's archival database]
* [http://fotos.psd.pt/atom/index.php/ Arquivo da Social-Democracia] (Portuguese)
* [http://arquivo.cm-albufeira.pt/ Arquivo Municipal Albufeira] (Portuguese)
* [http://arquivo.cm-valedecambra.pt/atom/ Arquivo Municipal de Vale de Cambra] (Portuguese)
* [http://arquivo.cm-viladobispo.pt/ Arquivo Municipal de Vila do Bispo] (Portuguese)
* [http://archivescollection.anu.edu.au/ Australian National University Archives]
* [http://icaatom.coc.fiocruz.br/ Base Arch - Fundação Oswaldo Cruz] (Brazilian Portuguese)
* [http://beatoninstitute.com/ The Beaton Institute (Cape Breton University)]
* [http://collections.westbeyondthewest.ca British Columbia Digitization Coalition's West Beyond the West Portal]
* [http://archivos.cedinci.org/ CeDInCI] Centro de Documentación e Investigación de la Cultura de Izquierdas en Argentina (Spanish)
* [https://archives.cendari.dariah.eu/ CENDARI Archives Directory]
* [http://searcharchives.coquitlam.ca/atom-2.0.1/ City of Coquitlam Archives]
* [http://searcharchives.vancouver.ca/ City of Vancouver Archives]
* [http://concordia.accesstomemory.org/ Concordia University]
* [http://archives.gcu.ac.uk/ Glasgow Caledonian University Archives]
* [http://lslirc-tarrarchives.accesstomemory.org/ Lesser Slave Lake IRC Treaty Aboriginal Rights Research Archives]
* [http://nanna.lib.umanitoba.ca/atom/ MAIN: Manitoba Archival Information Network]
* [http://catalogue.millsarchive.org/ Mills Archive]
* [http://atom.hmarchives.com/ Human and Memory Archives] (Korean)
* [http://archives.jewishmuseum.ca/ The Jewish Museum & Archives of BC]
* [http://www.memorybc.ca/ MemoryBC - British Columbia's Archival Information Network]
* [https://memoryns.ca/ MemoryNS, the Council of Nova Scotia Archives' archival database]
* [http://archives.museeheritage.ca Musée Héritage Museum Archives]
* [http://archives.eclibrary.ca/ Nipissing University and Canadore College Archives and Special Collections]
* [http://archives.ppcli.com/ Princess Patricia’s Canadian Light Infantry Museum & Archives]
* [http://search-bcarchives.royalbcmuseum.bc.ca/ Royal British Columbia Museum - BC Archives]
* [http://atom.archives.sfu.ca/ SFU AtoM - Simon Fraser University]
* [http://squamishlibrary.digitalcollections.ca/ Squamish Public Library - Digital Collections]
* [https://archive.sro.wa.gov.au/ State Records Office of Western Australia]
* [http://archivessearch.twu.ca/ Trinity Western University Archives and Special Collections]
* [http://search.nbca.unbc.ca/index.php/ UNBC Northern BC Archives]
* [http://atom.archives.unesco.org/ UNESCO Archives]
* [http://search.archives.un.org/ United Nations Archives and Records Management Section]
* [http://rbscarchives.library.ubc.ca/ University of British Columbia Rare Books and Special Collections]
* [https://searcharchives.ucalgary.ca/ University of Calgary Archives and Special Collections]
* [http://sca.glos.ac.uk/index.php/ University of Gloucestershire Special Collections and Archives]
* [http://search.nbca.unbc.ca/ University of Northern British Columbia Archives]
* [http://strathclyde.ica-atom.org/ University of Strathclyde Archives]
* [http://whistler.ica-atom.org/ Whistler Museum and Archives]
* [https://archivesholdings.worldbank.org/ World Bank Group Archives Holdings]

==ICA-AtoM (1.x) sites==

* [https://archives.aber.ac.uk/index.php/ Aberystwyth University Archives]
* [http://icaatom.almg.gov.br/ Acervo Arquivístico da Assembleia Legislativa do Estado de Minas Gerais] (Portuguese)
* [http://web2.santamaria.rs.gov.br/arquivohistorico/sistema_descricao_documental/index.php/ Acervo digital do Arquivo Histórico Municipal de Santa Maria] (Portuguese)
* [http://archivo.ateneodemadrid.es Archivo del Ateneo de Madrid] (Spanish)
* [http://www.madrid.org/icaatom_pub/index.php/ Archivos de la Comunidad de Madrid] (Spanish)
* [http://archivo.plasencia.es/ Archivo Municipal de Plasencia] (Spanish)
* [http://archief.vioe.be/ Archief van het Vlaams Instituut voor het Onroerend Erfgoed] (Dutch)
* [http://archivocti.uma.es/icaatom/ Base de datos fotográfica del Centro de Tecnología de la Imagen de la Universidad de Málaga] (Spanish)
* [http://cad.cgtp.pt/ica/ Centro de Arquivo e Documentação CGTP-IN] (Portuguese)
* [http://www.dart.ipsr.mahidol.ac.th/index.php/n56S2;isad Digital Archive of Research on Thailand]
* [http://archive.womenslibrary.org.uk/index.php/ Glasgow Women's Library Archive Catalogue]
* [http://www.highlandparkhistory.org/ Highland Park Historical Society]
* [http://www.ltiarchives.ca Leeds and the Thousand Islands Archives]
* [http://www.migan.org MIGAN - the directory of Caribbean archival institutions] (English/French)
* [http://www.archive.coop/icaatom-1.3.0/index.php/ National Co-operative Archive]
* [http://gta.nise.eu/ NISE (National movements and Intermediary Structures in Europe) Guide to the Archives]
* [http://archives.nato.int/ North Atlantic Treaty Organization (NATO) Archives]
* [http://asiapacific.anu.edu.au/pambu/catalogue/ Pacific Manuscripts Bureau]
* [http://archivesdatabase.royalroads.ca/icaatom/index.php/ Royal Roads University]
* [http://sain.scaa.sk.ca/collections/ Saskatchewan Archival Information Network Collections Database]
* [http://ica-atom.tasglann.org.uk/ Tasglann nan Eilean Siar (Hebridean Archives)]

Please feel free to [[Special:UserLogin|add a link]] to your AtoM installation.

[[Category:Community]]

Community/Users

2015-10-05T16:14:11Z

Sromkey: /* AtoM (2.x) sites */

{{#pagetitle: Users }}
[[Main Page]] > [[Community]] > Community/Users

On this page you'll find some examples of institutions and organizations using AtoM and/or ICA-AtoM. Don't see your institution on this list? Please feel free to [[Special:UserLogin|add a link]] to your AtoM installation!

==AtoM (2.x) sites==

* [http://www.albertaonrecord.ca/ Alberta on Record: Archives Society of Alberta Portal]
* [http://archeion.ca Archeion: Ontario's Archival Information Network]
* [https://archie.library.carleton.ca/ Archives & Research Collections, Carleton University Library]
* [http://www.archivescommunales-vd.ch/ Archives comunales du canton de Vaud] (Swiss)
* [http://anglicanarchives.ica-atom.org/ Archives of the Anglican Diocese of New Westminster and Provincial Synod of BC and Yukon]
* [http://archivesdemontreal.ica-atom.org/ Archives de Montréal]
* [http://arc.anla.nf.ca/ ARC: the Archival Resource Catalogue of the Association of Newfoundland and Labrador Archives]
* [http://acervo.cidarq.ufg.br/index.php/iuryy;isad Acervos Arquivísticos] (Brazilian Portuguese)
* [http://archives.canbarchives.ca/ ArchivesCANB: the Council of Archives New Brunswick's archival database]
* [http://fotos.psd.pt/atom/index.php/ Arquivo da Social-Democracia] (Portuguese)
* [http://arquivo.cm-albufeira.pt/ Arquivo Municipal Albufeira] (Portuguese)
* [http://arquivo.cm-valedecambra.pt/atom/ Arquivo Municipal de Vale de Cambra] (Portuguese)
* [http://arquivo.cm-viladobispo.pt/ Arquivo Municipal de Vila do Bispo] (Portuguese)
* [http://archivescollection.anu.edu.au/ Australian National University Archives]
* [http://icaatom.coc.fiocruz.br/ Base Arch - Fundação Oswaldo Cruz] (Brazilian Portuguese)
* [http://beatoninstitute.com/ The Beaton Institute (Cape Breton University)]
* [http://collections.westbeyondthewest.ca British Columbia Digitization Coalition's West Beyond the West Portal]
* [http://archivos.cedinci.org/ CeDInCI] Centro de Documentación e Investigación de la Cultura de Izquierdas en Argentina (Spanish)
* [https://archives.cendari.dariah.eu/ CENDARI Archives Directory]
* [http://searcharchives.coquitlam.ca/atom-2.0.1/ City of Coquitlam Archives]
* [http://searcharchives.vancouver.ca/ City of Vancouver Archives]
* [http://concordia.accesstomemory.org/ Concordia University]
* [http://archives.gcu.ac.uk/ Glasgow Caledonian University Archives]
* [http://lslirc-tarrarchives.accesstomemory.org/ Lesser Slave Lake IRC Treaty Aboriginal Rights Research Archives]
* [http://nanna.lib.umanitoba.ca/atom/ MAIN: Manitoba Archival Information Network]
* [http://catalogue.millsarchive.org/ Mills Archive]
* [http://atom.hmarchives.com/ Human and Memory Archives] (Korean)
* [http://archives.jewishmuseum.ca/ The Jewish Museum & Archives of BC]
* [http://www.memorybc.ca/ MemoryBC - British Columbia's Archival Information Network]
* [https://memoryns.ca/ MemoryNS, the Council of Nova Scotia Archives' archival database]
* [http://archives.museeheritage.ca Musée Héritage Museum Archives]
* [http://archives.eclibrary.ca/ Nipissing University and Canadore College Archives and Special Collections]
* [http://archives.ppcli.com/ Princess Patricia’s Canadian Light Infantry Museum & Archives]
* [http://search-bcarchives.royalbcmuseum.bc.ca/ Royal British Columbia Museum - BC Archives]
* [http://atom.archives.sfu.ca/ SFU AtoM - Simon Fraser University]
* [http://squamishlibrary.digitalcollections.ca/ Squamish Public Library - Digital Collections]
* [https://archive.sro.wa.gov.au/ State Records Office of Western Australia]
* [http://archivessearch.twu.ca/ Trinity Western University Archives and Special Collections]
* [http://search.nbca.unbc.ca/index.php/ UNBC Northern BC Archives]
* [http://atom.archives.unesco.org/ UNESCO Archives]
* [http://search.archives.un.org/ United Nations Archives and Records Management Section]
* [http://rbscarchives.library.ubc.ca/ University of British Columbia Rare Books and Special Collections]
* [https://searcharchives.ucalgary.ca/ University of Calgary Archives and Special Collections]
* [http://search.nbca.unbc.ca/ University of Northern British Columbia Archives]
* [http://strathclyde.ica-atom.org/ University of Strathclyde Archives]
* [http://whistler.ica-atom.org/ Whistler Museum and Archives]
* [https://archivesholdings.worldbank.org/ World Bank Group Archives Holdings]

==ICA-AtoM (1.x) sites==

* [https://archives.aber.ac.uk/index.php/ Aberystwyth University Archives]
* [http://icaatom.almg.gov.br/ Acervo Arquivístico da Assembleia Legislativa do Estado de Minas Gerais] (Portuguese)
* [http://web2.santamaria.rs.gov.br/arquivohistorico/sistema_descricao_documental/index.php/ Acervo digital do Arquivo Histórico Municipal de Santa Maria] (Portuguese)
* [http://archivo.ateneodemadrid.es Archivo del Ateneo de Madrid] (Spanish)
* [http://www.madrid.org/icaatom_pub/index.php/ Archivos de la Comunidad de Madrid] (Spanish)
* [http://archivo.plasencia.es/ Archivo Municipal de Plasencia] (Spanish)
* [http://archief.vioe.be/ Archief van het Vlaams Instituut voor het Onroerend Erfgoed] (Dutch)
* [http://archivocti.uma.es/icaatom/ Base de datos fotográfica del Centro de Tecnología de la Imagen de la Universidad de Málaga] (Spanish)
* [http://cad.cgtp.pt/ica/ Centro de Arquivo e Documentação CGTP-IN] (Portuguese)
* [http://www.dart.ipsr.mahidol.ac.th/index.php/n56S2;isad Digital Archive of Research on Thailand]
* [http://archive.womenslibrary.org.uk/index.php/ Glasgow Women's Library Archive Catalogue]
* [http://www.highlandparkhistory.org/ Highland Park Historical Society]
* [http://www.ltiarchives.ca Leeds and the Thousand Islands Archives]
* [http://www.migan.org MIGAN - the directory of Caribbean archival institutions] (English/French)
* [http://www.archive.coop/icaatom-1.3.0/index.php/ National Co-operative Archive]
* [http://gta.nise.eu/ NISE (National movements and Intermediary Structures in Europe) Guide to the Archives]
* [http://archives.nato.int/ North Atlantic Treaty Organization (NATO) Archives]
* [http://asiapacific.anu.edu.au/pambu/catalogue/ Pacific Manuscripts Bureau]
* [http://archivesdatabase.royalroads.ca/icaatom/index.php/ Royal Roads University]
* [http://sain.scaa.sk.ca/collections/ Saskatchewan Archival Information Network Collections Database]
* [http://ica-atom.tasglann.org.uk/ Tasglann nan Eilean Siar (Hebridean Archives)]

Please feel free to [[Special:UserLogin|add a link]] to your AtoM installation.

[[Category:Community]]

File:Accesstomemory-contributor-agreement-2014.pdf

2015-07-14T23:36:11Z

Sromkey:

Development/Projects/DCB/Templates/DC

2015-07-13T22:44:10Z

Sromkey: Created page with "Main Page > > Development > Development/Projects > Development/Projects/DCB > Data-entry Templates > Dublin Core '''Purpose''' * Provide Qubit interface f..."

[[Main Page]] > > [[Development]] > [[Development/Projects]] > [[Development/Projects/DCB]] > [[Data-entry Templates]] > Dublin Core

'''Purpose'''
* Provide Qubit interface for describing resources using the [http://dublincore.org/ Dublin Core Metadata Element Set (DC).]
* Create separate templates for Simple DC (15 elements) and Qualified DC (7 additional elements plus a number of qualifiers that extend or refine the original elements).
* Note that DC does not define a prescribed order of elements.

'''Mapping:'''
* Not always a direct one-to-one mapping between DC elements and Qubit database fields.
* For mapping, see [[Qubit Input / Export - Dublin Core|Metadata Mapping: Qubit Input / Export - Dublin Core]].

'''Design issues'''
* Good to break up the data entry screen in collapsible divs, but DC standard provides no basis for grouping elements.
* Mock-ups include options for:
** Simple DC "unstructured" (all DC elements in one div).
** Simple DC "structured" (DC elements grouped in several divs).
** Qualified DC "structured".

'''Field labels'''
* Follow prescriptions of the [http://dublincore.org/documents/usageguide/ Using Dublin Core User Guide].
* User Guide practice not always consistent with the ''element labels'' given in the [http://dublincore.org/documents/dces/ Dublin Core Metadata Element Set Version 1.1] (ISO 15836).
* Where the ISO standard generally give element ''name'' and ''label'' the same value but different cases (lower vs sentence), the User Guide labels sometimes add words (e.g. "Rights Management" vs "Rights").
* Which authority to use? The mock-ups follow the User Guide; discrepancies are indicated in the notes.

Mock-up for Simple DC "unstructured":
* [[Template:Dublin Core - Simple DC Unstructured|Simple DC unstructured]]

Mock-ups for Simple and Qualified DC "structured" approach:
* [[Template:Dublin Core - Identity Area|Identity area]]
* [[Template:Dublin Core - Creator / Publisher / Contributor / Dates Area|Creator / publisher / contributor / dates area]]
* [[Template:Dublin Core - Description Area|Description area]]
* [[Template:Dublin Core - Access and Use Area|Access and use area]] (Qualified only)
* [[Template:Dublin Core - Indexing Area|Indexing area]]
* [[Template:Dublin Core - Related Resources Area|Related resources area]]

'''Control area''', '''Digital object''', and '''Storage location''' are the same standard Qubit sections.

Development/Projects/DCB/Templates

2015-07-13T22:38:04Z

Sromkey: Created page with "Main Page > Development > Development/Projects > Development/Projects/DCB > Data-entry Templates This section contains links to data-entry template mock-ups..."

[[Main Page]] > [[Development]] > [[Development/Projects]] > [[Development/Projects/DCB]] > Data-entry Templates

This section contains links to data-entry template mock-ups and notes.

*[[Template:Dublin Core]]

*[[Template:MODS]]

Development/Projects/DCB

2015-07-13T22:34:05Z

Sromkey: Created page with "{{#pagetitle:Alouette Toolkit (Digital Collection Builder)}} Main Page > Development > Development/Projects > Alouette Toolkit (Digital Collection Builder) <admon..."

{{#pagetitle:Alouette Toolkit (Digital Collection Builder)}}
[[Main Page]] > [[Development]] > [[Development/Projects]] > Alouette Toolkit (Digital Collection Builder)

<admonition type="note">
This is historical documentation that refers to the development of the Digital Collecton Builder. This page was first created in September of 2008, and its content was last updated in February 2010. Its contents were migrated to the new AtoM wiki on July 13, 2015: no changes have been made to the original content, though some formatting may have been changed for greater readability, or to account for differences in styles between the two wikis. Dead external links may also have been removed. Otherwise, the content itself has not been altered.
</admonition>

== Project Status ==
'''Note: the development work has been completed (May 2009) and the 1.0.7 release of the software was launched in June 2009 and re-branded as the Canadiana.org Digital Collection Builder.''' See the [http://dcb-gcn.canadiana.org/ project website] for the most current information.

== Background ==

As the lead developers of the [http://ica-atom.org ICA-AtoM] open source archival description application, [http://artefactual.com Artefactual Systems] has been collaborating with [http://www.canadiana.org/eco.php?doc=cihm Canadiana.org]'s [http://www.alouettecanada.ca/ Alouette Toolkit] Project since June 2007.

Both projects decided to create the [http://qubit-toolkit.org Qubit] open information management toolkit as a common, underlying platform for development of their respective applications. Having common project objectives and almost identical functional requirements, both ICA-AtoM and the Alouette Toolkit are now simply themed versions of the Qubit Toolkit.

Since the summer of 2007 Artefactual Systems has continued to grow Qubit's project infrastructure, development team, and software feature set. The current version of Qubit (1.0.3) contains approximately 70-80% of the features envisioned for the first release of the Alouette Toolkit. Artefactual will complete development on Alouette specific requirements to provide a first public release of the Toolkit by winter 2009. SFU Library is acting as the administrative and technical liaison for this phase of development.

=== Partners ===
* [http://www.canadiana.org/eco.php?doc=cihm Canadiana.org]
* [http://www.lib.sfu.ca/ SFU Library]
* [http://www.eln.bc.ca/ BC Electronic Library Network]
* [http://www.ikebarberlearningcentre.ubc.ca/ UBC Irving K. Barber Learning Centre]
* [http://artefactual.com Artefactual Systems]

== Project Tasks ==

# MODS & Dublin core templates
## Qubit core crosswalk
## [[Data-entry Templates|data-entry templates]]
## output templates
## XML import/export profiles
# [http://qubit-toolkit.org/wiki/index.php?title=Theming Theming/Menuing module]
## create a Wordpress-style theming directory structure
## provide a Drupal-style theming user interface for administrators
## build navigation menus from database values
## allow adminstrators to configure menu options
# [http://qubit-toolkit.org/wiki/index.php?title=Multi-page_digital_content_handling Multi-page digital content handling]
## upload multi-page images
## provide a browsing user interface
# Alouette Metadata Toolkit distro
## Alouette theming (logo, terminology, default templates, menu structure, etc.)
## Create web installer for Alouette distro

File:Pdi nbca transform final.png

2015-07-08T17:51:09Z

Sromkey:

Development/Projects/AABC Data Migration Toolkit/UNBC

2015-07-08T17:48:01Z

Sromkey: Created page with "{{#pagetitle:UNBC data migration}} Main Page > Development > Development/Projects > Development/Projects/AABC Data Migration Toolkit > UNBC data migration =..."

{{#pagetitle:UNBC data migration}}
[[Main Page]] > [[Development]] > [[Development/Projects]] > [[Development/Projects/AABC Data Migration Toolkit]] > UNBC data migration

= Overview =

[[File:Pdi nbca transform final.png|thumb|none|800px|Screenshot of final NBCA -> ICA-AtoM '''PDI Spoon''' migration job]]

[http://library.unbc.ca/UNBCArchives/archDisplay/Default.aspx UNBC Archives] is the first participant to be migrated as part of the [[AABC Data Migration Toolkit]] project. The original project specification included migrating data from MS SQL Server, MS Access and ICA-AtoM (Release 1.0.9) MySQL databases as well as linking over 5,000 digital asset files (image and video).

Data migration from the MS SQL Server data was performed via the [http://wiki.pentaho.com/display/EAI/Spoon+User+Guide '''Pentaho Data Integration (PDI) Spoon'''] (pictured) extract, transform, load tool - see [[AABC_Data_Migration_Toolkit#Technical_Tool_Options|Technical Tool Options]] for other options considered, and reasons for selecting '''PDI'''.

Due to the pilot nature of the project, there were a number of unforeseen technical challenges and the project ran over schedule.

== Original sources for migration ==

One (1) MS SQL Server database backup file
# NBCA_backup_201004080030.bak

Four (4) MS Access databases:
# Fawcett.mdb
# Ray Williston Collection.mdb
# Cassiar_Box1_on.mdb
# Cassiar-Box751onwards.mdb

One (1) archive (ZIP) file containing digital objects (image and video files):
# Uploads.zip

Data from one (1) ICA-AtoM Release 1.0.9 instance (set up in late 2010?), including description and digital assets
# http://www.ica-atom.org/unbc

== Data migrated ==

The final migrated data includes
* The ''NBCA'' MS SQL Server database data as detailed in the [[#NBCA database mappings]] section below
* The digital files from the ''Uploads.zip'' archive
* The ICA-AtoM Release 1.0.9 UNBC website (hosted by Artefactual) including digital assets

== Data not migrated ==

=== MS Access databases ===

None of the four MS Access databases provided were migrated, for the following reasons
* the Cassiar fonds files (''Cassiar_Box1_on.mdb'', ''Cassiar-Box751onwards.mdb'') were never scheduled for migration
* in the course of the project it was determined that the "Ray Williston Collection" data was present in the NBCA (MS SQL Server) database, so migration of the ''Ray Williston Collection.mdb'' database was not necessary
* The ''Fawcett.mdb'' database will be migrated after development of a [[AABC_Data_Migration_Toolkit#CSV Import]] methodology

=== Accession data ===

[http://www.ica-atom.org/doc/Release_1.1 ICA-AtoM Release 1.1] has no provision for storing accession data. The future [http://qubit-toolkit.org/wiki/index.php?title=Release_1.2 ICA-AtoM Release 1.2] '''will''' include an accession module, and this data will be migrated at a future date.

Accession data is stored in several tables in the NBCA database, and these were not migrated:
# ''Accession'' - accession and appraisal data
# ''Individual'' - most of the data in this table is specific to appraisers and donors
# ''AppraiserSpecialty'' - links appraisers (Individual) to their area of specialty
# ''Receipt'' - accession receipts?

=== Other MS SQL Server tables not migrated ===

# ''SpecifiedLocation'' - locations are specified at the Fonds level, and are very general (e.g. "Photograph room", "Mobile shelving", "On display in library"). UNBC agreed that migrating this data was not necessary.
# ''Lookup'' - contained controlled vocabulary maintained in the ''Lookups'' table was not applicable in the ICA-AtoM application.

== Project deliverables ==

# Migration of all source data to a single [http://www.ica-atom.org ICA-AtoM] [[ica-atom:Release_1.1|Release 1.1]] website (hosted by Artefactual)
# Integration of linked digital assets
# This process documentation

= Methodology =

# Migrate ICA-AtoM Release 1.0.9 data (with attached digital assets) to ICA-AtoM Release 1.1 data schema. See: [[qubit:Upgrading|Upgrading]]
# Install MS SQL Server 2005 Express and load the ''NBCA'' database from backup file provided
# Use the [http://wiki.pentaho.com/display/EAI/Spoon+User+Guide '''Pentaho Data Integration (PDI) Spoon'''] tool to create a data migration script to migrate the ''NBCA'' database to the ICA-AtoM data model
# Attach digital assets from ''Uploads.zip'' archive to ''NBCA'' descriptions
# Refine the data migration script through several iterations
# Set up an install of ICA-AtoM for UNBC to audit the migrated data - including Release 1.0.9 data, ''NBCA'' database data, and all digital assets from both
# UNBC audits migrated data on the auditing site
# Correct any problems with the migration script found in the data audit
# Do the final migration of the data to the live UNBC site (hosted by Artefactual)

= Extract =

== Restore UNBC MSSQL Backup ==

[[File:MSSQL restore database.png|thumb|Restore database dialog]]

* Download and install [http://www.microsoft.com/Sqlserver/2005/en/us/express-down.aspx SQL Server 2005 Express Edition with Advanced Services SP3 ]
* Installion options
** Components: Management Studio Express, Full-text search, Connectivity Components, Replication?
** Mixed Authentication Mode (Windows and SQL Server)
** Add user to the SQL admin role
* Copy NBCA_backup_201004080030.bak to C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Backup\NBCA_backup_201004080030.bak
* Run SQL Server Management Studio express
* Right-click "Databases" > Restore
* To database "NBCA" (or whatever).
* From device (browse to NBCA_backup_201004080030.bak)
* Click checkbox next to backup filename
* OK

== Setup MS SQL Server for remote connections ==

=== Create new SQL Server login ===

[[File:MSSQL new login.png|thumb|Create SQL Server login]]

* In left pane of "Microsoft SQL Server Management Studio Express" click to expand "Security > Logins"
* Right-click either "Login" folder or in Login pane and select "New Login..."
* Enter username
* Select "SQL Server authentication"
* Enter password, and confirm
* Uncheck "Enforce password expiration"
* Default database "NBCA"
* OK

=== Give read access to new login ===

[[File:MSSQL user mapping.png|thumb|Give read access to new login]]

* Right-click newly created login, and select "Properties"
* Click "User-Mappings" in left pane
* Click checkbox next to "NBCA" database in top-right pane
* Click "db_datareader" permission in lower-right pane
* Click OK

=== Turn on TCP/IP access ===

[[File:MSSQL configure TCPIP.png|thumb|Turn on TCP/IP access]]

* "Start > Microsoft SQL Server 2005 > Configuration Tools > SQL Server Configuration Manager"
* In left pane, click "SQL Server 2005 Network Configuration > Protocols for SQLEXPRESS"
* Right-click "TCP/IP" in right pane, and click "Properties"
* click "IP Addresses" tab
* For desired TCP/IP connection select "Enabled: Yes", "TCP Dynamic Ports: <blank>", "TCP Port: 1433". OK (You will get a warning that server must be restarted for changes to effect) (See screenshot)
* Right-click "TCP/IP" and click "Enable" (Warning again)
* Click "SQL Server 2005 Services" in left pane
* In right pane, right-click "SQL Server (SQLEXPRESS)" and click "Restart"

=== Troubleshooting TCP/IP config ===

In several instances the static port settings above did not work properly with SQL Server 2005. Fixing this issue required modification the following Windows registry keys to set the ''Port'' value to ''1433''.
* HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\<InstanceName>\MSSQLServer\SuperSocketNetLib\Tcp
* HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\<MSSQL.x>\MSSQLServer\SuperSocketNetLib\Tcp\IPAll

== Connecting to MSSQL with PDI Spoon via JDBC ==

[[File:Spoon jdbc connection MSSQL.png|thumb|PDI Spoon JDBC connection dialog]]

'''PDI Spoon''' is a Java GUI for creating '''PDI''' Transformation scripts. We will use it to connect to MSSQL with the JDBC connector. Please read the [http://wiki.pentaho.com/display/EAI/Spoon+User+Guide Spoon documentation] for general instructions on using the tool.

* Add new "Input > Table input" step
* Double-click the Table input step
* Click the "New" connection button,
* Select the MS SQL Server connector, and fill in the form like the screenshot below
* Click "Test"
* Click OK

<div class="note">''NOTE: The "MS SQL Server (Native)" connector did not work in our tests, so the vanilla "MS SQL Server" connector was used''</div>

 

= Transform =

== NBCA database mappings ==

=== Table: Fond ===

{|
!Source column
!Dest. column
!Notes
|-class="odd"
|Id||keymap.source_id||keymap table maps source keys to target keys
|-
|Accession||note_i18n.content||Store as RAD "Alpha-numeric designations" note
|-class="odd"
|Title||information_object_i18n.title||
|-
|DerivedNameOne||''none''||calculated value based on Title. Not migrated
|-class="odd"
|DerivedNameTwo||''none''||calculated value based on Title. Not migrated
|-
|BeginCirca||event.start_date||Parse regular dates (e.g. "22 February 1960", "Jul. 1983", "1922") to event.start_date (date type). Copy all (regular and irregular) dates to event_i18n.date (string type)
|-class="odd"
|EndCirca||event.end_date||Parse regular dates (e.g. "22 February 1960", "Jul. 1983", "1922") to event.end_date (date type). Copy all (regular and irregular) dates to event_i18n.date (string type)
|-
|Restrictions||information_object_i18n.access_conditions||RAD label "Restrictions on access"
|-class="odd"
|Copyright||''none''||All rows are blank in source table
|-
|SourceOfTitle||note_i18n.content||RAD note type: "Source of title proper"
|-class="odd"
|TitleAndAccessNotes||''none''||All rows are blank in source table
|-
|PhysicalDescription||information_object_i18n.extent_and_medium||
|-class="odd"
|Conservation||''none''||All rows are blank in source table
|-
|ScopeAndContent||information_object_i18n.scope_and_content||
|-class="odd"
|Series||''none''||Series list will be derived from parent/child relationship
|-
|PhysicalDescriptionNotes||''none''||All rows are blank in source table
|-class="odd"
|CustodialHistoryPrivate||''none''||ICA-AtoM doesn't currently have a column for storing "private" data. Migrate with accession data after Release 1.2
|-
|CustodialHistoryPublic||information_object_i18n.archival_history||The "archival_history" column is labeled "Custodial history" in the RAD template.
|-class="odd"
|CreatorHistory||''none''||Source column contains a mix of: creator name, dates of existence and immediate source of acquisition. Normalized creator name is imported from UNBC edited csv file "creator.csv"
|-
|ProvenanceNotes||''none''||All rows are blank in source table
|-class="odd"
|KeyWords||term_i18n.name, actor_i18n.authorized_form_of_name||Exported to csv file. UNBC manually normalized CSV data to name, place or subject "access points" using Google Refine. The modified csv file was then imported using '''PDI Spoon'''
|-
|BioAdminHistory||actor_i18n.history||
|-class="odd"
|BioAdminHistNotes||''none''||All rows are blank in source table
|-
|LangaugeNotes||''none''||Only three rows in Fonds table have non-English languages, migrate manually
|-class="odd"
|Record||''none''||One row contains "FOR-17800", migrate manually
|-
|Arrangement||information_object_i18n.arrangement||
|-class="odd"
|FindAids||information_object_i18n.finding_aids||
|-
|GeneralNotes||notes_i18n.content||General note type
|-class="odd"
|ArchivistId||''none''||Ran out of time to migrate the archivist name to "revision history"
|-
|ShowOnWeb||status.status_id||false = "Draft" (status_id: 159), true = "published" (status_id: 160)
|-class="odd"
|CreatedBy||''none''||Machine generated data that is not relevant to ICA-AtoM
|-
|CreatedOn||object.created_at||
|-class="odd"
|ModifiedBy||''none''||Machine generated data that is not relevant to ICA-AtoM
|-
|ModifiedOn||object.updated_at||
|-class="odd"
|Deleted||''none''||Manually remove 'deleted' rows from Qubit. Can't do this during migration because of data integrity issues, some deleted rows have non-deleted children
|}

=== Table: FondSeries ===

{|
!Source column
!Dest. column
!Notes
|-class="odd"
|Id||keymap.source_id||The keymap table maps source keys to target keys
|-
|Accession||note_i18n.content||Store as RAD "Alpha-numeric designations" note
|-class="odd"
|Title||information_object_i18n.title||''Trim trailing periods from title in transformation''
|-
|DerivedNameOne||''none''||Calculated value based on Title. Not migrated
|-class="odd"
|DerivedNameTwo||''none''||Calculated value based on Title. Not migrated
|-
|FondId||information_object.parent_id||Foreign key self-relation. Use keymap table to lookup parent (Fond) target key
|-class="odd"
|BeginCirca||event.start_date||Parse regular dates (e.g. "22 February 1960", "Jul. 1983", "1922") to event.start_date (date type). Copy all (regular and irregular) dates to event_i18n.date (string type)
|-
|EndCirca||event.end_date||Parse regular dates (e.g. "22 February 1960", "Jul. 1983", "1922") to event.start_date (date type). Copy all (regular and irregular) dates to event_i18n.date (string type)
|-class="odd"
|SourceOfTitle||note_i18n.content||RAD note type: "Source of title proper"
|-
|TitleAndAccessNotes||''none''||All rows are blank in source table
|-class="odd"
|PhysicalDescription||information_object_i18n.extent_and_medium||
|-
|ScopeAndContent||information_object_i18n.scope_and_content||
|-class="odd"
|PhysicalDescriptionNotes||''none''||All rows are blank in source table
|-
|BioAdminHistory||''none''||Field data is often inconsistent with intended use. Can't determine authoritative name for person or organization
|-class="odd"
|KeyWords||''none''||All rows are blank in source table
|-
|CustodialHistoryPrivate||''none''||All rows are blank in source table
|-class="odd"
|CustodialHistoryPublic||information_object_i18n.archival_history||The "archival_history" column is labeled "Custodial history" in the RAD template.
|-
|CreatorHistory||''none''||All rows are blank in source table
|-class="odd"
|ProvenanceNotes||''none''||All rows are blank in source table
|-
|Record||''none''||16 rows in database, all part of "Aleza Lake Research Forest Archival Records fonds". Migrate manually
|-class="odd"
|Arrangement||information_object_i18n.arrangement||
|-
|GeneralNotes||notes_i18n.content||General note type
|-class="odd"
|ArchivistId||''none''||Ran out of time to migrate the archivist name to "revision history"
|-
|ShowOnWeb||status.status_id||false = "Draft" (status_id: 159), true = "published" (status_id: 160)
|-class="odd"
|CreatedBy||''none''||Machine generated data that is not relevant to ICA-AtoM
|-
|CreatedOn||object.created_at||
|-class="odd"
|ModifiedBy||''none''||Machine generated data that is not relevant to ICA-AtoM
|-
|ModifiedOn||object.updated_at||
|-class="odd"
|Deleted||''none''||Manually remove 'deleted' rows from Qubit. Can't do this during migration because of data integrity issues, some deleted rows have non-deleted children
|}

=== Table: FondSeriesFile ===

{|
!Source column
!Dest. column
!Notes
|-class="odd"
|Id||keymap.source_id||The keymap table maps source keys to target keys
|-
|Accession||note_i18n.content||Store as RAD "Alpha-numeric designations" note
|-class="odd"
|Title||information_object_i18n.title||''Trim trailing periods from title in transformation''
|-
|DerivedNameOne||''none''||Calculated value based on Title. Not migrated
|-class="odd"
|DerivedNameTwo||''none''||Calculated value based on Title. Not migrated
|-
|FondSeriesId||information_object.parent_id||Foreign key self-relation. Use keymap table to lookup parent (FondSeries) target key
|-class="odd"
|CreationCirca||event.date||Start date may be copied to event.start_date, if the format is recognized
|-
|SourceOfTitle||note_i18n.content||RAD note type: "Source of title proper"
|-class="odd"
|Copyright||''none''||All rows are blank in source table
|-
|AccessRestrictions||information_object_i18n.access_conditions||RAD label "Restrictions on access"
|-class="odd"
|TitleAndAccessNotes||note_i18n.content||Contains a mix of "Physical condition", "Cartographic scale", Physical location, previous Accession #'s and general notes. Migrated as "General Note"
|-
|GeneralMaterialDesignation ||''none''||All rows are blank in source table
|-class="odd"
|PhysicalDescription||information_object_i18n.extent_and_medium||
|-
|LocationId ||''none''||All rows are blank in source table
|-class="odd"
|ScopeAndContent||information_object_i18n.scope_and_content||
|-
|Conservation||note_i18n.content||"Conservation" note type
|-class="odd"
|PhysicalDescriptionNotes||''none''||All rows are blank in source table
|-
|BioAdminHistory||''none''||Field data is mostly inconsistent with intended use. Can't algorithmically determine authoritative name for person or organization
|-class="odd"
|KeyWords||''none''||All rows are blank in source table
|-
|CustodialHistoryPrivate||''none''||ICA-AtoM doesn't currently have a column for storing private data. Migrate with accession data after Release 1.2?
|-class="odd"
|CustodialHistoryPublic||''none''||All rows are blank in source table
|-
|ProvenanceNotes||''none''||All rows are blank in source table
|-class="odd"
|PhotographerOrArtist||note_i18n.content||Migrated as "General note"
|-
|Record||''none''||Appropriate RAD designation is unclear. Ran out of time to migrate (146 rows contain data)
|-class="odd"
|Arrangement||''none''||All rows are blank in source table
|-
|GeneralNotes||notes_i18n.content||General note type
|-class="odd"
|ArchivistId||''none''||Ran out of time to migrate the archivist name to "revision history"
|-
|ShowOnWeb||status.status_id||false = "Draft" (status_id: 159), true = "published" (status_id: 160)
|-class="odd"
|CreatedBy||''none''||Machine generated data that is not relevant to ICA-AtoM
|-
|CreatedOn||object.created_at||
|-class="odd"
|ModifiedBy||''none''||Machine generated data that is not relevant to ICA-AtoM
|-
|ModifiedOn||object.updated_at||
|-class="odd"
|Deleted||''none''||Manually remove 'deleted' rows from Qubit. Can't do this during migration because of data integrity issues, some deleted rows have non-deleted children
|}

=== Table: FondSeriesItem ===

{|
!Source column
!Dest. column
!Notes
|-class="odd"
|Id||keymap.source_id||The keymap table maps source keys to target keys
|-
|Accession||note_i18n.content||Store as RAD "Alpha-numeric designations" note
|-class="odd"
|Title||information_object_i18n.title||''Trim trailing periods from title in transformation''
|-
|DerivedNameOne||''none''||Calculated value based on Title. Not migrated
|-class="odd"
|DerivedNameTwo||''none''||Calculated value based on Title. Not migrated
|-
|FondSeriesFileId||''none''||All rows blank in source table
|-class="odd"
|FondSeriesId||information_object.parent_id||Foreign key self-relation. Use keymap table to lookup parent (FondSeries) target key
|-
|CreationCirca||event.date||In addition, start date may be copied to event.start_date, if the format is recognized
|-class="odd"
|SourceOfTitle||note_i18n.content||RAD note type: "Source of title proper"
|-
|Copyright||''none''||One value for ''"Bob Bossin’s Home Remedy for Nucelar War" Show''. Migrate manually
|-class="odd"
|AccessRestrictions||information_object_i18n.access_conditions||RAD label "Restrictions on access"
|-
|TitleAndAccessNotes||note_i18n.content||Contains a mix of "Source of title proper", "Terms governing use, reproduction and publication", "Attributions and conjectures" (?) and general notes. Migrated to "General notes"
|-class="odd"
|GeneralMaterialDesignation ||object_term_relation.term_id||Match closest available term in "General material designation" taxonomy
|-
|PhysicalDescription||information_object_i18n.extent_and_medium||
|-class="odd"
|ScopeAndContent||information_object_i18n.scope_and_content||
|-
|Conservation||note_i18n.content||"Conservation" note type
|-class="odd"
|PhysicalDescriptionNotes||''none''||All rows are blank in source table
|-
|BioAdminHistory||''none''||Field data is mostly inconsistent with intended use (mostly "Part of" designation). Can't algorithmically determine authoritative name for person or organization
|-class="odd"
|KeyWords||term_i18n.name, actor_i18n.authorized_form_of_name||Exported to csv file. UNBC manually normalized CSV data to name, place or subject "access points" using Google Refine. The modified csv file was then imported using '''PDI Spoon'''
|-
|CustodialHistoryPrivate||''none''||ICA-AtoM doesn't currently have a column for storing private data. Migrate with accession data after Release 1.2?
|-class="odd"
|CustodialHistoryPublic||''none''||All rows are blank in source table
|-
|ProvenanceNotes||''none''||All rows are blank in source table
|-class="odd"
|PhotographerOrArtist||note_i18n.content||Migrated as "General note"
|-
|Record||''none''||Appropriate RAD designation is unclear. Ran out of time to migrate (3561 rows contain data)
|-class="odd"
|Arrangement||''none''||All rows are blank in source table
|-
|GeneralNotes||notes_i18n.content||General note type
|-class="odd"
|ArchivistId||''none''||Ran out of time to migrate the archivist name to "revision history"
|-
|ShowOnWeb||status.status_id||false = "Draft" (status_id: 159), true = "published" (status_id: 160)
|-class="odd"
|CreatedBy||''none''||Machine generated data that is not relevant to ICA-AtoM
|-
|CreatedOn||object.created_at||
|-class="odd"
|ModifiedBy||''none''||Machine generated data that is not relevant to ICA-AtoM
|-
|ModifiedOn||object.updated_at||
|-class="odd"
|Deleted||''none''||Manually remove 'deleted' rows from Qubit. Can't do this during migration because of data integrity issues, some deleted rows have non-deleted children
|}

=== Table: Attachment ===

{|
!Source column
!Dest. column
!Notes
|-class="odd"
|SourcePath||digitalobject.path||See [[#Digital asset import]] below for details
|-
|colspan="3" | ''Remaining columns in this table were not migrated''
|}

=== Table: FondSeriesItemAttachement ===

{|
!Source column
!Dest. column
!Notes
|-class="odd"
|AttachmentId||digitalobject.information_object_id||See [[#Digital asset import]] below for details
|-
|FileSeriesItemId||information_object.id||See [[#Digital asset import]] below for details
|- class="odd"
|colspan="3" | ''Remaining columns in this table were not migrated''
|}

== ICA-AtoM Release 1.0.9 data ==

Data from the existing http://ica-atom.org/unbc database was migrated to ICA-AtoM Release 1.1 data schema via the standard ICA-AtoM [[qubit:Upgrading|upgrade]] process.

== Other transform sources ==

[[File:PDI-AccessPointEntities.png|thumb|'''AccessPointEntity.ktr''' '''PDI Spoon''' transform script creating actor (name) and term (subject, place) rows from access points]]

[[File:Unbc-access-point-relations-ktr.png|thumb|'''AccessPointRelation.ktr''' '''PDI Spoon''' transform script creating description->actor (name) and description->term (subject, place) relations from access points]]

=== Creator name ===

The NBCA.Fond table had no standard field for recording the creator of a Fonds, so the ''CreatorHistory'' and ''Keywords'' columns were used in an ad-hoc manner. Because neither the name format nor the column used was consistent, there was no automated way to extract the name of the Fonds' creator. The solution was to create a CSV export file from the NBCA ''Fond'' table and for the UNBC archivists to manually enter a creator name, where the creator was known.

The ''creator.csv'' file was created by export of the NBCA ''Fond'' table (columns: Id, Title, CreatorHistory and BioAdminHistory) and was then manually edited by the UNBC Archives staff. The edited ''creator.csv'' file is loaded into ICA-AtoM by the '''PDI Spoon''' Transformation "Creators.ktr".

=== Keywords ===

In the NBCA ''Fond'' and ''FondSeries'' source tables, the ''Keywords'' column contains a semi-colon separated list of strings. This data is defined via "Access Points" in the ICA-AtoM model. Because ICA-AtoM distinguishes between name, subject and place access points, and there is no automated way to categorize all of the original strings, this required a multi-part process:
# Export a comma-separated file containing all ''Keywords'' column data from the NBCA database (only present in ''Fond'' and ''FondSeries'' tables)
# Run automated normalization and data-cleaning using [http://code.google.com/p/google-refine/ Google Refine] (Artefactual)
# Manual data cleaning via '''Google Refine''' (UNBC)
# Import of cleaned and normalized CSV file via '''PDI Spoon'''

The complex data structure of ICA-AtoM further complicated the import process (step #4) as can been seen in the '''AccessPointEntity.ktr''' and '''AccessPointRelation.ktr''' screenshots.

=== Digital asset import ===

Digital assets locations were exported from the NBCA MS SQL Server database to a CSV file via '''PDI Spoon''' transformation script then the actual digital assets were imported and linked to a description via the [[AABC_Data_Migration_Toolkit#Digital_object_load_task|digitalobject:load]] task. The transformation script also mapped the source id (''FondSeriesItemAttachement.FondSeriesItemId'') to the target key (''information_object.id'') via the ''keymap'' table, and did a string search & replace on the ''Attachment.SourcePath'' column to substitute the local path for each digital asset.

 

= Load =

== Load existing ICA-AtoM install data ==

Data from the http://ica-atom.org/unbc site can be loaded via [http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html mysqldump] backup file. This step must be performed before any other migrations because it destroys any existing data in the target database.

== Load NBCA data ==

The '''PDI Spoon''' ''NBCA_Transform.kjb'' job writes data directly to the target ICA-AtoM MySQL database.

== Load digital objects ==

Digital objects are loaded using the [[AABC_Data_Migration_Toolkit#Digital_object_load_task|digitalobject:load]] task using the [[#Digital assets|nbca_digitalobjects.csv]] file to link assets to archival descriptions.

= Post-processing =

Several final tasks must be performed on the final ICA-AtoM instance data which require direct access to the ICA-AtoM/symfony framework

# Build nested set - See: [[AABC Data Migration Toolkit#Build nested set task|Build nested set task]]
# Generate permalink slugs - See: [[AABC Data Migration Toolkit#Generate slugs task|Generate slugs task]]
# Build search index - See: [[qubit:Rebuild search index|Rebuild search index]]

Development/Projects/AABC Data Migration Toolkit/CVA

2015-07-08T17:31:42Z

Sromkey: Created page with "{{#pagetitle:City of Vancouver data migration}} Main Page > Development > Development/Projects > Development/Projects/AABC Data Migration Toolkit > City of Van..."

{{#pagetitle:City of Vancouver data migration}}
[[Main Page]] > [[Development]] > [[Development/Projects]] > [[Development/Projects/AABC Data Migration Toolkit]] > City of Vancouver data migration

== Data sources ==

=== Content server data ===

* approx. 125,000 Private Records (from private donors)
* approx. 87,000 Public Records (City created documents)
* approx. 9,700 Maps and Pamphlets
* approx. 3,350 Public documents (published materials issued by the City)
* 812 Accession records
* 224 Authority records (for Public Records only)
* 195 Authority records relationships
* 88 Ephemera records (private sources)

=== Other sources ===

* ??? ICA-AtoM 1.1 records
* approx. 70,000 digital objects (TIFF)

== Process ==

For Content Server data:
* Exported data to CSV from Content Server,
* Transformed data to match ICA-AtoM RAD CSV template via "Pentaho Data Integration" ('''PDI''') tool,
* imported into ICA-AtoM via CSV import command line ('''CLI''') script.

For ICA-AtoM 1.1 data:
* Migrate and import via standard ICA-AtoM [[qubit:Upgrading|upgrade]] procedure.

For digital objects:
* Processed RAW TIFFs via Archivematica to produce DIPs (expand?),
* Then imported using CLI DIP import script

== Importing ==

Import of the private records initially took 1 second per record on a conventional workstation, but an import speed of 0.12 seconds per record, initially, was achieved using a ~15GB RAM Rackspace Cloud instance (~$0.90/hour?) with an Ubuntu 11.04 LAMP stack. Import would slow down as it went, however, so the CSV dump was split into 10000 row chunks. The import slows down as it goes, but we were able to import 125K rows in ~12 hours after using a patch to disable nested set (hierarchy) updating. Post-import search indexing is very slow still.

Following are the steps to do the import (using Gnu Screen to keep the session alive).

# SSH to import server
# Download CSV data to <tt>/root</tt>
# Enter <tt>apt-get update</tt>
# Enter <tt>apt-get install -y tasksel subversion screen</tt>
# Enter <tt>tasksel install lamp-server</tt>
# Enter <tt>/usr/sbin/apache2ctl restart</tt>
# Enter <tt>cd /var/www</tt>
# Check out qubit Subversion trunk to <tt>qubit</tt>
# Enter <tt>chown -R www-data:www-data qubit</tt>
# Create <tt>qubit</tt> MySQL database
# Visit Qubit with browser to complete installation
# Enter <tt>cd qubit</tt>
# Enter <tt>screen</tt>
# Enter <tt>./symfony csv:import path/to/first/import/file --rows-until-update=5 --error-log=/root/errors.log --source-name=source_name</tt>
# In the import command, "source_name" is used when recording mapping of legacy IDs to new Qubit IDs

Steup after initial import.

# Update nested set data: ./symfony propel:build-nested-set (took 3.33 minutes for 125K imported objects)

== Challenges ==

* Private Records CSV export included 100+ columns, and over 1 million data points.
* Google Refine could not handle this much data, and could not be used for data transformation or normalization

== Data mapping ==

{| style="border:1px black solid; border-collapse:collapse"
!style="border:1px black solid"|LEGEND
|-
|"B" - [mostly] blank (haven't checked all records)
|-
|"T" - human readable text
|-
|"R" - machine readable (i.e. populated from a value list)
|-
|Problem mapping
|-
|Requires pre-processing
|-
|Do not migrate
|}

=== Private record data mapping ===

{|
!Original CVA column
!RAD CSV template column
!Qubit column
!Notes
|-
|ACCESSION NUMBER (B)
|accessionNumber
|accession.identifier (via `relation`)
|Add an accession record related via `relation` table (many-to-many)
|-
|ADMIN HIST OR BIO (T)
|creatorsHistory
|actor_i18n.history, event.type_id = QubitTerm::CREATION_ID
|Creators are an `actor` linked to information_object via the many-to-many `event` table (event.information_object_id <-> event.actor_id). The event.type_id column is required and is defined in ''lib/model/QubitTerm.php'' as a class constant (i.e. CREATION_ID = 111). The related `actor` row needs to be the same one used for '''Creators''' (actor_i18n.authorized_form_of_name) column. The `event` row needs to be the same one used for '''Creation date''' (event_i18n.dates), '''START YEAR''' (event.start_date) and '''END YEAR''' (event.end_date).
|- style="color:orange"
|ARCH ADDRESS (B)
|scopeAndContent
|information_object_i18n.scope_and_content
|This column will be merged into the "Scope and content" column before import (See "Migration Instructions - CS Records to ICA AtoM - Public Records Export Set.XLS")
|- style="color:orange"
|ARCHIVIST (RT)
|revision
|information_object_i18n.revision
|Prefix with "Created by "
|-
|ARTIST (B)
|nameAccessPoints
|actor_i18n.authorized_form_of_name (relation.type_id = QubitTerm::NAME_ACCESS_POINT_ID)
|Name access point is related to information object via many-to-many "relation" table (relation.subject_id = information_object.id, relation.object_id = actor.id). Need to test if an actor already exists with the specified name, if actor name DOES exist, then just create a new relation between actor and information object, if actor DOES NOT exist, then create a new actor row with ''actor_i18n.authorized_form_of_name'' equal to column value, then link via relation.
|-
|ASSOCIATED MATERIAL (B)
|relatedUnitsOfDescription
|information_object_i18n.related_units_of_description
|
|-
|CONSERVATION NOTE (B)
|radNoteConservation
|note_i18n.content, note.type_id = ???
|Each "Rad note" type is a separate row in the "note" table, related to information object by (one-to-many) fk ''note.information_object_id''. Getting the ''note.type_id'' (required) is tricky, you'll have to search like <code lang="SQL">SELECT term.id FROM term, term_i18n i18n WHERE term.id = i18n.id and term.taxonomy_id = 51 and i18n.culture = 'en' and i18n.name = 'Conservation';</code> (N.B. QubitTaxonomy::RAD_NOTE_ID = 51)
|-
|COPYRIGHT EXPIRES (B)
|copyrightExpires
|rights.end_date
|Not part of RAD. Would be preferable to include copyright information in a "Rights" import module, but out-of-scope for current project.
|- style="color:orange"
|COPYRIGHT STATUS (B)
|copyrightHolder
|rights_holder.authorized_form_of_name (via rights.rights_holder_id)
|Not part of RAD. Would be preferable to include copyright information in a "Rights" import module, but out-of-scope for current project. Split rights holder name into separate "copyrightHolder" field. Requires creation of a new `rights_holder` row, or matching an existing rights holder name.
|- style="color:orange"
|COPYRIGHT STATUS (B)
|copyrightStatus
|rights.copyright_status_id
|Not part of RAD. Would be preferable to include copyright information in a "Rights" import module, but out-of-scope for current project. Split rights holder name into separate "copyrightHolder" field. Will need to match `term_i18n.name` to get foreign key for `rights.copyright_status_id`.
|-
|DATE NOTE (B)
|datesOfCreationNote
|event_i18n.note, event.type_id = QubitTerm::CREATION_ID?
|'''I think this only applies to "Creation" date, but need to check with Glenn'''
|-
|DATE OF CREATION (T)
|datesOfCreation
|event_i18n.dates, event.type_id = QubitTerm::CREATION_ID
|This event row needs to link to the actor defined by '''Creators''' (actor_i18n.authorized_form_of_name) and '''Creator history''' (actor_i18n.history) columns, and be the same event row used for '''START YEAR''' (event.start_date) and '''END YEAR''' (event.end_date) columns.
|-
|EDITION (B)
|edition
|information_object_i18n.edition
|
|-
|END YEAR (T)
|datesOfCreationEnd
|event.end_date, event.type_id = QubitTerm::CREATION_ID
|This event row needs be the same event row used for '''DATE OF CREATION''' (event_i18n.date) and '''NAME OF CREATOR''' (event.actor_id) columns.
|
|-
|GENERAL NOTE (B)
|radNoteGeneral
|note_i18n.content, note.type_id = ???
|Each "Rad note" type is a separate row in the "note" table. See note for '''Rad note – conservation'''.
|- style="color:orange"
|ITEM NUMBER (B)
|identifier
|information_object.identifier
|CVA wants to prefix with a delimiter (e.g. ": ") to differentiate from other identifiers. Needs to be merged to single "identifier" column
|-
|LANG NOTE (B)
|language
|property.name = 'language', property_i18n.value
|Language is written to ''property_i18n.value'' as a '''serialized''' PHP array of the form Array('en', 'fr', 'de') where the values are a two letter [http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes ISO639-1 language code]. The ''property.source_culture'' = 'en' and ''property_i18n.culture'' = 'en' in all cases (e.g. the culture is always forced to English) and translation of language name is handled by symfony's [http://www.symfony-project.org/book/1_2/13-I18n-and-L10n#chapter_13_sub_outputting_data_in_the_user_s_culture format_language()] helper function.
|- style="color:orange"
|LOCATION (R)
|physicalStorageName
|physical_object_i18n.name
|Copy value to Physical storage name and location fields
|- style="color:orange"
|LOCATION (R)
|physicalStorageLocation
|physical_object_i18n.location
|Copy value to Physical storage name and location fields
|- style="color:grey"
|TYPE (R)
|''n/a''
|''None''
|Do not migrate
|-
|NAME OF CREATOR (R)
|creators
|actor_i18n.authorized_form_of_name, event.type_id = QubitTerm::CREATION_ID
|Creators are an `actor` linked to information_object via the many-to-many `event` table (event.information_object_id <-> event.actor_id). If multiple creators are indicated (we ask for a pipe "<nowiki>|</nowiki>" delimiter for multi-value fields) then link, in order, to creator history (e.g. if 2 creators and 1 history, then link the *first* creator to the creator history). Use the same principle (e.g. match in order) for Creation dates. The related `actor` row needs to be the same one used for '''Creator history''' (actor_i18n.history) column. The `event` row needs to be the same one used for '''Creation date''' (event_i18n.dates), '''START YEAR''' (event.start_date) and '''END YEAR''' (event.end_date).
|-style="color:orange"
|OLD PHOTO NO (B)
|radNoteAlphaNumericDesignation
|note_i18n.content, note.type_id = ???
|Prefix with "Old Photo Number: " and map as "Rad note - alpha-numeric designation". Create a separate note entry or concatenate with other alpha-numeric notes?
|- style="color:orange"
|OTHER CREATORS (R)
|creators
|actor_i18n.authorized_form_of_name
|Creator names without history or creation dates. Concatenate to "Creators" column using pipe delimiter.
|- style="color:orange"
|PART OF FILE (B)
|''n/a''
|''None''
|Use to calculate "parent_id" value
|- style="color:orange"
|PART OF FONDS (R)
|''n/a''
|''None''
|Use to calculate "parent_id" value
|- style="color:orange"
|PART OF SERIES (R)
|''n/a''
|''None''
|Use to calculate "parent_id" value
|- style="color:orange"
|PART OF SOUSFONDS (R)
|''n/a''
|''None''
|Use to calculate "parent_id" value
|- style="color:orange"
|PART OF SUBSERIES (R)
|''n/a''
|''None''
|Use to calculate "parent_id" value
|- style="color:orange"
|PHOTO CLIENT (B)
|actorCommissionedBy
|actor_i18n.authorized_form_of_name (via event)
|For any "actor..." field use the string after "actor" (i.e. "Commissioned by") as the event label. Note in this case this will require creating a new term in the "Event type" taxonomy with term_i18n.name = 'Commissioned' a related scope note of 'Commissioned by' (See existing "event type" terms).
|-
|PHOTO GEOG (B)
|placeAccessPoints
|term_i18n.name (via object_term_relation table)
|Place and subject access points live in the `term` (+ `term_i18n`) table, and have a many-to-many relationship to `information_object` via the `object_term_relation` table [object_term_relation.object_id (information_object.id), object_term_relation.term_id (term.id)]. The term.taxonomy_id field is required and must equal to QubitTaxonomy::PLACE_ID (42) for Places and QubitTaxonomy::SUBJECT_ID (35) for Subjects.
|-
|PHOTO SUBJECTS (B)
|subjectAccessPoints
|term_i18n.name (via object_term_relation table)
|Place and subject access points live in the `term` (+ `term_i18n`) table, and have a many-to-many relationship to `information_object` via the `object_term_relation` table [object_term_relation.object_id (information_object.id), object_term_relation.term_id (term.id)]. The term.taxonomy_id field is required and must equal to QubitTaxonomy::PLACE_ID (42) for Places and QubitTaxonomy::SUBJECT_ID (35) for Subjects.
|-
|PHOTOGRAPHER (B)
|actorPhotographedBy
|note_i18n.content, note.type_id = ???
|For any "actor..." field use the the text after "actor" (i.e. PhotographedBy) for the term name (Photographed) and scope note (Photographed by)
|-
|PHY COND NOTE (B)
|physicalCharacteristics
|information_object_i18n.physical_characteristics
|
|- style="color: orange"
|PHYS CHAR NOTE (B)
|physicalCharacteristics
|information_object_i18n.physical_characteristics
|Must be concatenated to physicalCharacteristics column
|- style="color:orange"
|PRI REC NO (R)
|identifier
|information_object_i18n.identifier
|Need to merge this with other identifier fields. Should only be applied to top-level descriptions (ignore at lower levels). If possible then automate creation of an auto-incremented integer number (within the current branch) as an identifier for lower-level descriptions.
|-
|PUB SERIES (B)
|titleProperOfPublishersSeries
|property_i18n.value
|Create a new `property` row related to the information object (one-to-many) by property.object_id = information_object.id. Set property.name = 'titleProperOfPublishersSeries'.
|- style="color:orange"
|RECORD_ID
|sources AND UNIQUE_ID
|information_object_i18n.sources
|Prefix with "CS-Records ID: " in sources field. Use as "UNIQUE_ID" for mapping source to target rows.
|-
|RELATED RECORDS (R)
|Related units of description
|information_object_i18n.related_units_of_description
|
|- style="color:orange"
|RRDB_* (B)
|accessConditions
|information_object_i18n.access_conditions
|All of the RRDB_* (Rights) columns will be concatenated into a single 'access conditions' column according to rules laid out by CVA
|- style="color:orange"
|SCALE (B)
|statementOfScaleArchitectural
|property_i18n.value
|Write to "statement of scale (architectural)" or "statement of scale (cartographic)" column based on mediatype - property.name = 'statementOfScaleArchitectural' or property.name = 'statementOfScaleCartographic' respectively.
|-
|SCALE (B)
|statementOfScaleCartographic
|property_i18n.value
|Write to "statement of scale (architectural)" or "statement of scale (cartographic)" column based on mediatype - property.name = 'statementOfScaleArchitectural' or property.name = 'statementOfScaleCartographic' respectively.
|-
|START YEAR (T)
|datesOfCreationStart
|event.start_date, event.type_id = QubitTerm::CREATION_ID
|Use the same event row that is used for '''DATE OF CREATION''' (event_i18n.date) and '''NAME OF CREATOR''' (event.actor_id) columns.
|
|- style="color:grey"
|THUMBNAIL (B)
|''n/a''
|''None''
|Do not migrate
|- style="color:grey"
|TIFF and JPEG Number (B)
|''n/a''
|''None''
|Do not migrate
|- style="color:grey"
|TIFFIMAGE (B)
|''n/a''
|''None''
|Do not migrate
|- style="orange"
|TITLE NOTE (B)
|radTitleNoteContinuationOfTitle
|note_i18n.content, note.type_id = ???
|Split into "RAD note - continuation of title" or "RAD note - variations in title" based on prefix within field. Note type (note.type_id) is one of <code language="SQL">SELECT id FROM term WHERE taxonomy_id = QubitTaxonomy::RAD_TITLE_NOTE_ID;</code>
|- style="orange"
|TITLE NOTE (B)
|radTitleNoteVariationsInTitle
|note_i18n.content, note.type_id = ???
|Split into "RAD note - continuation of title" or "RAD note - variations in title" based on prefix within field. Note type (note.type_id) is one of <code language="SQL">SELECT id FROM term WHERE taxonomy_id = QubitTaxonomy::RAD_TITLE_NOTE_ID;</code>
|-
|TITLE SOURCE (R)
|radTitleNoteSourceOfTitleProper
|note_i18n.content, note.type_id = ???
|New related note object with note.type_id = <code language="sql">SELECT id FROM term, term_i18n i18n WHERE term.id = i18n.id AND term.taxonomy_id = QubitTaxonomy::RAD_TITLE_NOTE_ID AND i18n.culture='en' AND i18n.name = 'Source of title proper';</code>
|
|}

File:Crosswalkcapture.png

2015-07-08T17:28:00Z

Sromkey:

Development/Projects

2015-07-08T17:24:54Z

Sromkey:

{{#pagetitle: Projects }}
[[Main Page]] > [[Development]] > Development/Projects

This area of the wiki is used to hold descriptions of AtoM development or research projects. Most projects listed here have concluded and are listed here for historical purposes.

==Current projects==

We'll add new or upcoming development or research projects here!

==Historical projects==

*[[Development/Projects/ArchivesCanada Upgrade|ArchivesCanada Upgrade]]
*[[Development/Projects/|AABC Migration Toolkit|AABC Data Migration Toolkit]]
*[[Development/Projects/LAC Scalability testing|LAC ICA-AtoM Scalability Testing]]
*[[Development/Projects/AAO CVA Usability testing|AAO and City of Vancouver ICA-AtoM Usability Testing (2011)]]
*[[Development/Projects/BCAUL Pilot|BCAUL Pilot Project (MemoryBC) (2008-2009)]]
*[[Development/Projects/DCB|Alouette Toolkit (Digital Collections Builder)]]

[[Category:Development documentation]]

Development/Projects

2015-07-08T17:24:13Z

Sromkey:

{{#pagetitle: Projects }}
[[Main Page]] > [[Development]] > Development/Projects

This area of the wiki is used to hold descriptions of AtoM development or research projects. Most projects listed here have concluded and are listed here for historical purposes.

==Current projects==

We'll add new or upcoming development or research projects here!

==Historical projects==

*[[Development/Projects/ArchivesCanada Upgrade|ArchivesCanada Upgrade]]
*[[Development/Projects/AABC Migration Toolkit|AABC Data Migration Toolkit]]
*[[Development/Projects/LAC Scalability testing|LAC ICA-AtoM Scalability Testing]]
*[[Development/Projects/AAO CVA Usability testing|AAO and City of Vancouver ICA-AtoM Usability Testing (2011)]]
*[[Development/Projects/BCAUL Pilot|BCAUL Pilot Project (MemoryBC) (2008-2009)]]
*[[Development/Projects/DCB|Alouette Toolkit (Digital Collections Builder)]]

[[Category:Development documentation]]

Development/Projects/

2015-07-08T17:23:38Z

Sromkey: Sromkey moved page Development/Projects/ to Development/Projects/AABC Data Migration Toolkit: Correct title

#REDIRECT [[Development/Projects/AABC Data Migration Toolkit]]

Development/Projects/AABC Data Migration Toolkit

2015-07-08T17:23:37Z

Sromkey: Sromkey moved page Development/Projects/ to Development/Projects/AABC Data Migration Toolkit: Correct title

{{#pagetitle:AABC Data Migration Tookkit}}
[[Main Page]] > [[Development]] > [[Development/Projects]] > Development/Projects/AABC Data Migration Toolkit

<admonition type="note">
This is historical documentation that refers to the migration of data from legacy systems to ICA-AtoM. This page was first created in April 2011, and its content was last updated in April 2012. Its contents were migrated to the new AtoM wiki on July 8, 2015: no changes have been made to the original content, though some formatting may have been changed for greater readability, or to account for differences in styles between the two wikis. Dead external links may also have been removed. Otherwise, the content itself has not been altered.
</admonition>

=Background and Objectives=
The AABC Data Migration Toolkit project is a collaboration between the Archives Association of British Columbia (AABC), Artefactual Systems, the Anglican Diocese of New Westminster Archives, Simon Fraser University Archives, the University of Victoria, and the University of Northern British Columbia Archives to migrate data from legacy database systems used in the BC archival community to ICA-AtoM. The project will begin November 2010 and be completed by April 2011.

The project will analyze and select appropriate technical tool(s) to use in the migration process as well as develop a generic methodology to assist those doing data migration projects. This 'toolkit' should also be useful to other data migration projects, not just those migrating data into ICA-AtoM.

=Project Participants=
* National Archival Development Program (funding)
* Archives Association of British Columbia (coordination)
* Artefactual Systems (technical lead & coordination)
* Anglican Church Archives (InMagic migration & funding)
* Simon Fraser University Archives (FileMaker migration & funding)
* University of Victoria Archives (MS-Word migration & funding)
* University of Northern British Columbia Archives (MS-SQLserver migration & funding)

=Technical Tool Options=

==Drupal-based tool(s)==

There are a number of [http://groups.drupal.org/node/21338 Drupal import/export modules] available.

The [http://drupal.org/project/migrate Migrate module] developed by [http://cyrve.com Cyrve] looks interesting. Cyrve has also published their migration methodology [http://drupal.org/node/915102] [http://drupal.org/node/661762].

Presumably we can use Drupal CCK and Views to model almost any data structure.
Then we can use export functionality or other internal data migration tools to
map it and export it to Qubit data structure?

Also, once we get external data into Drupal we could, theoretically, expose it
as RDF data to get it out into other systems:

* http://drupal.org/project/rdf
* http://openspring.net/blog/2009/10/22/produce-and-consume-linked-data-with-drupal

See also: [http://drupal.org/project/xc The eXtensible Catalog (XC) Drupal Toolkit]

===Reasons for elimination of Drupal-based tools===

Ultimately any Drupal based tool is going to involve a two step migration process
# Migrate data Drupal schema (via CCK?)
# Migrate data to Qubit schema (via RDF?)

The extra step of migrating data to Drupal is a significant amount of extra work, without any concrete benefits. The difficulty of migrating the data to Drupal is exacerbated by the complex nature of the source and Qubit schemas. Implementing an RDF schema for Drupal and Qubit would have benefits beyond the scope of this project, but requires a significant amount of extra work that is not within the scope of the project. In addition any Drupal modules are, by nature, a extension on the Drupal framework, which will involve compromises that are not required for task specific ETL tools.

Using Drupal as a migration tool also requires installing an configuring Drupal, plus the necessary Drupal modules (CCK, View, Migrate, RDF, etc.); This involves significantly more work then installing a stand-alone ETL desktop application.

==Google Refine (formerly Freebase Gridworks)==

* http://code.google.com/p/google-refine/wiki/GettingStarted?tm=6

===Reasons for elimination of Google Refine===
Google refine only works with flat-files [http://code.google.com/p/google-refine/wiki/Importers] (version 2.0 2010-11-25) so it's not an option for Filemaker, MSSQL, MySQL or other SQL formats, which are part of our requirements.

==Open-source ETL tool(s)==

[http://en.wikipedia.org/wiki/Extract,_transform,_load#Open-source_ETL_frameworks Wikipedia list] of [http://en.wikipedia.org/wiki/Extract,_transform,_load Extract, transform, load] (ETL) tools

It's tough finding any decent (unbiased) information on any ETLs and the open source ones especially. A couple of names cropped up multiple times in my searches:

[http://kettle.pentaho.com/ Pentaho Data Integration (Kettle)] - community edition (FOSS)
* screencast: http://www.pentaho.com/products/demos/PDI_overview/PDI_overview.html (skip to ~5m00s for live demo)

[http://www.talend.com/products-data-integration/talend-open-studio.php Talend Open Studio]
* screencast: http://www.talend.com/demo/index.php

It sounds like either tool should be fine for small-scale projects like the ones we are tackling. Both Kettle and Talend OS use a Java GUI and thus run on Linux, Windows and Mac. [http://www.jonathanlevin.co.uk/2008/03/open-source-etl-tools-vs-commerical-etl.html Blog comparison of Pentaho Kettle and Talend]

Talend immediately annoyed me by offering the dreaded "feature comparison matrix" for it's own products and being pretty obtuse about which one is their open-source offering, as well as asking me to register to view their demos. >:-( So, Kettle beats Talend for first try by virtue of having the least annoying website.

I also checked out the screencast for [http://www.apatar.com/ Apatar] (which has an [http://aws.amazon.com/amis/819?_encoding=UTF8&jiveRedirect=1 EC2 AMI] - cool) but I didn't find much information about it other than Wikipedia and their own website. Apatar has a similar GUI to Kettle and Talend. Apatar's puts a lot of emphasis on it's web-service [http://www.apatarforge.org/wiki/display/GUI/Apatar+Connector+Guides connectors] - which include Salesforce, SugarCRM, Amazon S3, but this functionality is not required or beneficial for this project.

Also checked out the [http://scriptella.javaforge.com/ Scriptella] site. Scriptella is very appealing at first glance because of it's non-corporate web presence and it's claims of simplicity. I was thrown off a bit when I found a post questioning whether Scriptella was dead [http://www.javaforge.com/proj/forum/viewMessage.do?thread_id=42103&msg_id=42103&thread_members=true], but they released the 1.0 version in May, 2010 which is somewhat reassuring. However Scriptella does look like a one-man show which is a bit worrying. Scriptella looks like it's command line + config files, so no GUI , which significantly raises the barrier to entry for non-programmers. Random [http://www.javaworld.com/community/node/5234 article about Scriptella]

===Reasons for selection of Pentaho Data Integration (Kettle)===

Pentaho Data Integration (PDI) Community Edition (also known as Kettle) is a purpose built ETL tool with a feature set that matches and exceeds our requirements.

The Community Edition of PDI is open-source (LGPL version 2), which meshes well with the open-source philosophy of Qubit. PDI uses a graphical user interface (GUI) running on a Java Virtual Machine (several open-source JVMs are available) which allows the application to run, without functional differences, on Mac, Linux or Windows platforms.

Installing PDI is straight-forward, though it does require installing a suitable JVM as well. This compares favorably with having to install a Drupal-based migration toolset.

There is a steep learning curve for using PDI, especially with regard to the concurrent (multi-thread) and non-sequential property of the transformations. However, [http://wiki.pentaho.com/display/EAI/Latest+Pentaho+Data+Integration+%28aka+Kettle%29+Documentation PDI's documentation] is good, and includes several video and text tutorials, and the PDI community forum is quite active. The migrations that we develop for our pilot organizations and our own documentation should also help ease the learning process.

==Comma Seperated Values (CSV) import==
===Reasons for selection of CSV import as preferred migration path===

'''needs update (note: reserve Pentaho as option for complex migrations but not likely a tool that will be easily implemented by most archivists)'''

=Methodology=

"Real life ETL Cycle" (from Wikipedia [http://en.wikipedia.org/wiki/Extract,_transform,_load], 2010-12-02)

# Cycle initiation
# Build reference data
# Extract (from sources)
# Validate
# Transform (clean, apply business rules, check for data integrity, create aggregates or disaggregates)
# Stage (load into staging tables, if used)
# Audit reports (for example, on compliance with business rules. Also, in case of failure, helps to diagnose/repair)
# Publish (to target tables)
# Archive
# Clean up

==1. Migration using Pentaho==
===Extract===
Source Data Formats:
* MS SQL Server: [[UNBC data migration]]
* Filemaker Pro 10: [[SFU data migration]]

===Transform===

Pentaho Data Integration (PDI) includes a GUI interface for designing and running transformations named "spoon". The screenshots below show the spoon interface.

PDI organizes the individual "Steps" for transforming and mapping data into "Transformations". Data flows between Steps through "Hops" (the arrows) in an non-sequential and simultaneous fashion. This simultaneous execution of steps can be non-intuitive so the user must be careful not to assume what data will already be migrated (e.g. for foreign key relations).

[[File:FondtoInfoObject_ktr.png|600px]]

Several "Transformations" can be further grouped in to "Jobs". Jobs run as series of Transformations sequentially (with error handling, state checking, etc.)

[[File:UNBC Transform Job.png|600px]]

===Load===

There are several possible options for loading the transformed data into Qubit, each with pros and cons.

====Write directly to MySQL====

My initial instinct was to use the ETL tool to write directly to MySQL. This approach avoids the pain points we've had in the past with data load/import via the ORM: bad performance, large memory requirements and vexing segfaults.

However, This approach has proved difficult and time-consuming due to the highly relational nature of Qubit. To create one archival description record we must write to the following tables:

* information_object
* information_object_i18n
* note
* note_i18n
* object
* object_term_relation
* property
* property_i18n
* relation
* slug
* status
* term
* term_i18n
* ... more?

Many of these tables (e.g. property, note) require writing many rows per archival description.

In addition we've embedded the equivalent of SQL "triggers" in the Qubit ORM in an attempt to be database agnostic. Examples of this are the nested set logic, and slug name sanitization. Having a separate implementation of this logic in our ETL tool will make the transform fragile and very difficult to maintain.
* Thought: Can we run a php script after import to leverage the ORM and fire these "triggers"?

====Import via EAD====

We have experience with EAD import into ICA-AtoM so we are clear about it's limitations - high resource requirements (low performance and scalability), mapping ambiguities and an enforcement of one fonds/collection per EAD-XML file. However, the very fact that the problem space is well defined and that solutions will benefit the application as a whole (such as round-tripping) make EAD appealing.

;Pro
:We already have EAD import done (with a few outstanding issues, see below)
:Don't have to build table hierarchy (e.g. object, actor, repository, slug) manually
:Data-model integrity and logic enforced by embedded ORM logic (don't have to replicate in transform step)
:portable to any system that will import EAD

;Con
:EAD format is a very detailed and finicky, and therefore time-consuming to implement
:Data in the Qubit schema that we can not represent in EAD? (maybe we can "bend" EAD to capture this data?)
:EAD import has several outstanding issues [http://code.google.com/p/qubit-toolkit/issues/list?can=2&q=Component%3DEAD]
:Slow load, large memory requirements, segfaults?
:EAD enforces one top-level description per file, which makes bulk load very work-intensive (need to import many files)

Some initial research with the [http://kettle.pentaho.com/ Pentaho Kettle] ETL makes it clear that the Kettle XML output step will ''not'' support a highly structured XML format like EAD. This leaves us with several possible options:
* Write a simple XML file from Kettle, then use [http://en.wikipedia.org/wiki/XSLT XLST] to convert this intermediary file to EAD-XML
* Use [http://en.wikipedia.org/wiki/ECMAScript_for_XML e4x] and Kettle's javascript capabilities to write EAD directly ([http://forums.pentaho.com/showthread.php?68265-How-to-XML e4x example])
* Write a simple XML file from Kettle then use another tool (Google refine?) to transform to EAD-XML

====Automated http input====

Another possibility is loading data using an HTTP POST to the web front-end, via CURL or another tool. Like the EAD import, this takes advantage of current data model logic embedded in Qubit ORM. This method also has the advantage over EAD import that we don't have to worry about mapping data from the source to EAD and then to Qubit, so we don't have to worry about what data EAD may not capture gracefully.

Automated form input may be especially useful for the digital object load, as we are currently doing in Archivematica. The big issues we've seen with the Archivematica digital object scripting is fragility of script due to UI changes in Qubit, and finding the URI for the related information object (how do we get the right slug?).

Matching URIs is actually a huge problem with automated form input in general - for example how do we easily find the URI for a related name access point to select the correct value in the "name access point" drop-down? Performance is also a serious concern here, because the page load times for creating/editing a single description can be greater than 5 seconds.

=== Post-load scripts===

Several scripts must be run after data load to add secondary data that is essential for the Qubit application, but is not easily generated during [[#Transform]].

==== Build nested set task ====

;Syntax
:<pre>php symfony propel:build-nested-set</pre>
;Description
:Qubit uses the [http://en.wikipedia.org/wiki/Nested_set_model Nested set model] for arranging hierarchical data. The 'build-nested-set' task generates the required column values (lft, rgt) for the loaded data, based on the parent_id self-relationship. Creating the nested set script was quite complex, as it requires recursively descending down the hierarchy tree to derive the proper lft and rgt values. This script could be useful outside of the context of migrations for re-building the nested set after corruption (e.g. due to direct editing of the database) or data loss.

==== Generate slugs task ====

;Syntax
:<pre>php symfony propel:generate-slugs</pre>
;Description
:Qubit generates [http://en.wikipedia.org/wiki/Slug_%28web_publishing%29 Slugs] for creating [[Qubit:Permalinks|Permalinks]] to application resources. The 'generate-slugs' task generates these slugs for any resources that do not already have a slug. Because this script is non-destructive it can be run any time.

==== Digital object load task====

;Syntax
:<pre>php symfony digitalobject:load filename.csv</pre>
;Description
:Takes a CSV file as input which contains two 'columns': "filename" and "information_object_id"; the script will fail if these column headers are not present in the first row of the CSV file. The filename column contains the full (current) path to the digital asset (file). The information_object_id identifies the linked information object. Qubit does not allow more than one digital object per information object (with the exception of derivatives), and each digital object must have a corresponding information object to describe it, so this one-to-one relationship must be respected in the CSV import file.

==2. Migration using CSV import==

===Extract===

Most database applications have an export function to save data to a flat-file format. CSV and tab-delimited exports are especially ubiquitous, which is one of the reasons for using the CSV format.

====Filemaker Pro 10====

Pilot participants: [[SFU data migration]]

Filemaker has an advanced export function that allows export to many flat-file formats, including CSV, tab-delimited, XML and Excel (and others). In addition Filemaker allows the user to select exactly which fields are exported.

We used the "Excel" format for export as it includes column headers, which is not true of the CSV or tab-delimited formats.

====Inmagic====

Pilot participants: [[Anglican Archives data migration]]

Inmagic includes nice export functionality that allows specifying custom delimiters for field, line and multiple values within a field as well as string delimiters (e.g. quotes). Inmagic also allows exporting a found set of records, or the entire database and can include or exclude column headers.

We exported data using the standard CSV format with column headers.

====MS Word====

Pilot participants: [[University of Victoria data migration]]

''TODO''

====MS Excel====

MS Excel will save directly to a CSV or tab-delimited format.

===Validate===

Two comma-separated value (CSV) file templates are provided, one for ISAD(G) and one for RAD. These files can be edited with any common spreadsheet application (Microsoft Excel, LibreOffice Calc, etc.) but must be saved in CSV format to be imported by ICA-AtoM.
# [[Media:Ica atom import template isad.csv|ICA-AtoM ISAD(G) CSV template]]
# [[Media:Ica_atom_import_template_rad.csv|ICA-AtoM RAD CSV template]]

It became clear that a Crosswalk between ISAD(G), RAD, ICA-AtoM, and our .csv template was needed to support Archivists beginning the process of cleaning-up & transferring their data into ICA-AtoM.

Here is a screen capture of the spreadsheet: 
[[Image:Crosswalkcapture.png|Crosswalk Screen Capture]]

Download the whole spreadsheet for your own use:[[File:Crosswalk.ods|Crosswalk.ods]]

===Transform===

*Export data from legacy database as a comma separated values (CSV) text file format. CSV is used to store tabular data and can be accessed using a spreadsheet or database application
*Using the appropriate CSV template, compare legacy database CSV file with either .csv RAD template or .csv ISAD(G) template
*The templates are based on descriptive standards RAD and ISAD(G)
*Many legacy databases are populated with non-standardized data fields. When comparing column headers, the .csv template provides the standardized data fields and the legacy database CSV file may need to be rectified to match
*Rectify column headers in legacy database CSV file
*If necessary, merge data in more than one column to match individual column headers in specific .csv template

*Review CSV document for any anomalies in the data cells (e.g., misspelled names, inconsistent date formats, typos, incomplete records etc...)
*If data is inaccurate or inconsistent, [http://code.google.com/p/google-refine/ Google Refine] is an excellent tool for data clean-up

= Implementation =

==[[City_of_Vancouver_data_migration|City of Vancouver Archives]] data migration==

==[[UNBC data migration]]==

==[[SFU data migration]]==

==[[Anglican Archives data migration]]==

==[[University of Victoria data migration]]==

File:UNBC Transform Job.png

2015-07-08T17:22:40Z

Sromkey:

File:FondtoInfoObject ktr.png

2015-07-08T17:22:07Z

Sromkey:

Development/Projects/AABC Data Migration Toolkit

2015-07-08T17:15:45Z

Sromkey:

Development/Projects/AABC Data Migration Toolkit

2015-07-08T17:06:24Z

Sromkey:

{{#pagetitle:AABC Data Migration Tookkit}}
[[Main Page]] > [[Development]] > [[Development/Projects]] > Development/Projects/AABC Data Migration Toolkit

<admonition type="note">
This is historical documentation that refers to the migration of the British Columbia Archival Union Catalogue (BCAUL) maintained by the AABC to the beta 1.0 version of ICA-AtoM, in 2008 and 2009. This page was first created in April 2011, and its content was last updated in April 2012. Its contents were migrated to the new AtoM wiki on July 8, 2015: no changes have been made to the original content, though some formatting may have been changed for greater readability, or to account for differences in styles between the two wikis. Dead external links may also have been removed. Otherwise, the content itself has not been altered.
</admonition>

=Background and Objectives=
The AABC Data Migration Toolkit project is a collaboration between the Archives Association of British Columbia (AABC), Artefactual Systems, the Anglican Diocese of New Westminster Archives, Simon Fraser University Archives, the University of Victoria, and the University of Northern British Columbia Archives to migrate data from legacy database systems used in the BC archival community to ICA-AtoM. The project will begin November 2010 and be completed by April 2011.

The project will analyze and select appropriate technical tool(s) to use in the migration process as well as develop a generic methodology to assist those doing data migration projects. This 'toolkit' should also be useful to other data migration projects, not just those migrating data into ICA-AtoM.

=Project Participants=
* National Archival Development Program (funding)
* Archives Association of British Columbia (coordination)
* Artefactual Systems (technical lead & coordination)
* Anglican Church Archives (InMagic migration & funding)
* Simon Fraser University Archives (FileMaker migration & funding)
* University of Victoria Archives (MS-Word migration & funding)
* University of Northern British Columbia Archives (MS-SQLserver migration & funding)

=Technical Tool Options=

==Drupal-based tool(s)==

There are a number of [http://groups.drupal.org/node/21338 Drupal import/export modules] available.

The [http://drupal.org/project/migrate Migrate module] developed by [http://cyrve.com Cyrve] looks interesting. Cyrve has also published their migration methodology [http://drupal.org/node/915102] [http://drupal.org/node/661762].

Presumably we can use Drupal CCK and Views to model almost any data structure.
Then we can use export functionality or other internal data migration tools to
map it and export it to Qubit data structure?

Also, once we get external data into Drupal we could, theoretically, expose it
as RDF data to get it out into other systems:

* http://drupal.org/project/rdf
* http://openspring.net/blog/2009/10/22/produce-and-consume-linked-data-with-drupal

See also: [http://drupal.org/project/xc The eXtensible Catalog (XC) Drupal Toolkit]

===Reasons for elimination of Drupal-based tools===

Ultimately any Drupal based tool is going to involve a two step migration process
# Migrate data Drupal schema (via CCK?)
# Migrate data to Qubit schema (via RDF?)

The extra step of migrating data to Drupal is a significant amount of extra work, without any concrete benefits. The difficulty of migrating the data to Drupal is exacerbated by the complex nature of the source and Qubit schemas. Implementing an RDF schema for Drupal and Qubit would have benefits beyond the scope of this project, but requires a significant amount of extra work that is not within the scope of the project. In addition any Drupal modules are, by nature, a extension on the Drupal framework, which will involve compromises that are not required for task specific ETL tools.

Using Drupal as a migration tool also requires installing an configuring Drupal, plus the necessary Drupal modules (CCK, View, Migrate, RDF, etc.); This involves significantly more work then installing a stand-alone ETL desktop application.

==Google Refine (formerly Freebase Gridworks)==

* http://code.google.com/p/google-refine/wiki/GettingStarted?tm=6

===Reasons for elimination of Google Refine===
Google refine only works with flat-files [http://code.google.com/p/google-refine/wiki/Importers] (version 2.0 2010-11-25) so it's not an option for Filemaker, MSSQL, MySQL or other SQL formats, which are part of our requirements.

==Open-source ETL tool(s)==

[http://en.wikipedia.org/wiki/Extract,_transform,_load#Open-source_ETL_frameworks Wikipedia list] of [http://en.wikipedia.org/wiki/Extract,_transform,_load Extract, transform, load] (ETL) tools

It's tough finding any decent (unbiased) information on any ETLs and the open source ones especially. A couple of names cropped up multiple times in my searches:

[http://kettle.pentaho.com/ Pentaho Data Integration (Kettle)] - community edition (FOSS)
* screencast: http://www.pentaho.com/products/demos/PDI_overview/PDI_overview.html (skip to ~5m00s for live demo)

[http://www.talend.com/products-data-integration/talend-open-studio.php Talend Open Studio]
* screencast: http://www.talend.com/demo/index.php

It sounds like either tool should be fine for small-scale projects like the ones we are tackling. Both Kettle and Talend OS use a Java GUI and thus run on Linux, Windows and Mac. [http://www.jonathanlevin.co.uk/2008/03/open-source-etl-tools-vs-commerical-etl.html Blog comparison of Pentaho Kettle and Talend]

Talend immediately annoyed me by offering the dreaded "feature comparison matrix" for it's own products and being pretty obtuse about which one is their open-source offering, as well as asking me to register to view their demos. >:-( So, Kettle beats Talend for first try by virtue of having the least annoying website.

I also checked out the screencast for [http://www.apatar.com/ Apatar] (which has an [http://aws.amazon.com/amis/819?_encoding=UTF8&jiveRedirect=1 EC2 AMI] - cool) but I didn't find much information about it other than Wikipedia and their own website. Apatar has a similar GUI to Kettle and Talend. Apatar's puts a lot of emphasis on it's web-service [http://www.apatarforge.org/wiki/display/GUI/Apatar+Connector+Guides connectors] - which include Salesforce, SugarCRM, Amazon S3, but this functionality is not required or beneficial for this project.

Also checked out the [http://scriptella.javaforge.com/ Scriptella] site. Scriptella is very appealing at first glance because of it's non-corporate web presence and it's claims of simplicity. I was thrown off a bit when I found a post questioning whether Scriptella was dead [http://www.javaforge.com/proj/forum/viewMessage.do?thread_id=42103&msg_id=42103&thread_members=true], but they released the 1.0 version in May, 2010 which is somewhat reassuring. However Scriptella does look like a one-man show which is a bit worrying. Scriptella looks like it's command line + config files, so no GUI , which significantly raises the barrier to entry for non-programmers. Random [http://www.javaworld.com/community/node/5234 article about Scriptella]

===Reasons for selection of Pentaho Data Integration (Kettle)===

Pentaho Data Integration (PDI) Community Edition (also known as Kettle) is a purpose built ETL tool with a feature set that matches and exceeds our requirements.

The Community Edition of PDI is open-source (LGPL version 2), which meshes well with the open-source philosophy of Qubit. PDI uses a graphical user interface (GUI) running on a Java Virtual Machine (several open-source JVMs are available) which allows the application to run, without functional differences, on Mac, Linux or Windows platforms.

Installing PDI is straight-forward, though it does require installing a suitable JVM as well. This compares favorably with having to install a Drupal-based migration toolset.

There is a steep learning curve for using PDI, especially with regard to the concurrent (multi-thread) and non-sequential property of the transformations. However, [http://wiki.pentaho.com/display/EAI/Latest+Pentaho+Data+Integration+%28aka+Kettle%29+Documentation PDI's documentation] is good, and includes several video and text tutorials, and the PDI community forum is quite active. The migrations that we develop for our pilot organizations and our own documentation should also help ease the learning process.

==Comma Seperated Values (CSV) import==
===Reasons for selection of CSV import as preferred migration path===

'''needs update (note: reserve Pentaho as option for complex migrations but not likely a tool that will be easily implemented by most archivists)'''

=Methodology=

"Real life ETL Cycle" (from Wikipedia [http://en.wikipedia.org/wiki/Extract,_transform,_load], 2010-12-02)

# Cycle initiation
# Build reference data
# Extract (from sources)
# Validate
# Transform (clean, apply business rules, check for data integrity, create aggregates or disaggregates)
# Stage (load into staging tables, if used)
# Audit reports (for example, on compliance with business rules. Also, in case of failure, helps to diagnose/repair)
# Publish (to target tables)
# Archive
# Clean up

==1. Migration using Pentaho==
===Extract===
Source Data Formats:
* MS SQL Server: [[UNBC data migration]]
* Filemaker Pro 10: [[SFU data migration]]

===Transform===

Pentaho Data Integration (PDI) includes a GUI interface for designing and running transformations named "spoon". The screenshots below show the spoon interface.

PDI organizes the individual "Steps" for transforming and mapping data into "Transformations". Data flows between Steps through "Hops" (the arrows) in an non-sequential and simultaneous fashion. This simultaneous execution of steps can be non-intuitive so the user must be careful not to assume what data will already be migrated (e.g. for foreign key relations).

[[File:FondtoInfoObject_ktr.png|600px]]

Several "Transformations" can be further grouped in to "Jobs". Jobs run as series of Transformations sequentially (with error handling, state checking, etc.)

[[File:UNBC Transform Job.png|600px]]

===Load===

There are several possible options for loading the transformed data into Qubit, each with pros and cons.

====Write directly to MySQL====

My initial instinct was to use the ETL tool to write directly to MySQL. This approach avoids the pain points we've had in the past with data load/import via the ORM: bad performance, large memory requirements and vexing segfaults.

However, This approach has proved difficult and time-consuming due to the highly relational nature of Qubit. To create one archival description record we must write to the following tables:

* information_object
* information_object_i18n
* note
* note_i18n
* object
* object_term_relation
* property
* property_i18n
* relation
* slug
* status
* term
* term_i18n
* ... more?

Many of these tables (e.g. property, note) require writing many rows per archival description.

In addition we've embedded the equivalent of SQL "triggers" in the Qubit ORM in an attempt to be database agnostic. Examples of this are the nested set logic, and slug name sanitization. Having a separate implementation of this logic in our ETL tool will make the transform fragile and very difficult to maintain.
* Thought: Can we run a php script after import to leverage the ORM and fire these "triggers"?

====Import via EAD====

We have experience with EAD import into ICA-AtoM so we are clear about it's limitations - high resource requirements (low performance and scalability), mapping ambiguities and an enforcement of one fonds/collection per EAD-XML file. However, the very fact that the problem space is well defined and that solutions will benefit the application as a whole (such as round-tripping) make EAD appealing.

;Pro
:We already have EAD import done (with a few outstanding issues, see below)
:Don't have to build table hierarchy (e.g. object, actor, repository, slug) manually
:Data-model integrity and logic enforced by embedded ORM logic (don't have to replicate in transform step)
:portable to any system that will import EAD

;Con
:EAD format is a very detailed and finicky, and therefore time-consuming to implement
:Data in the Qubit schema that we can not represent in EAD? (maybe we can "bend" EAD to capture this data?)
:EAD import has several outstanding issues [http://code.google.com/p/qubit-toolkit/issues/list?can=2&q=Component%3DEAD]
:Slow load, large memory requirements, segfaults?
:EAD enforces one top-level description per file, which makes bulk load very work-intensive (need to import many files)

Some initial research with the [http://kettle.pentaho.com/ Pentaho Kettle] ETL makes it clear that the Kettle XML output step will ''not'' support a highly structured XML format like EAD. This leaves us with several possible options:
* Write a simple XML file from Kettle, then use [http://en.wikipedia.org/wiki/XSLT XLST] to convert this intermediary file to EAD-XML
* Use [http://en.wikipedia.org/wiki/ECMAScript_for_XML e4x] and Kettle's javascript capabilities to write EAD directly ([http://forums.pentaho.com/showthread.php?68265-How-to-XML e4x example])
* Write a simple XML file from Kettle then use another tool (Google refine?) to transform to EAD-XML

====Automated http input====

Another possibility is loading data using an HTTP POST to the web front-end, via CURL or another tool. Like the EAD import, this takes advantage of current data model logic embedded in Qubit ORM. This method also has the advantage over EAD import that we don't have to worry about mapping data from the source to EAD and then to Qubit, so we don't have to worry about what data EAD may not capture gracefully.

Automated form input may be especially useful for the digital object load, as we are currently doing in Archivematica. The big issues we've seen with the Archivematica digital object scripting is fragility of script due to UI changes in Qubit, and finding the URI for the related information object (how do we get the right slug?).

Matching URIs is actually a huge problem with automated form input in general - for example how do we easily find the URI for a related name access point to select the correct value in the "name access point" drop-down? Performance is also a serious concern here, because the page load times for creating/editing a single description can be greater than 5 seconds.

=== Post-load scripts===

Several scripts must be run after data load to add secondary data that is essential for the Qubit application, but is not easily generated during [[#Transform]].

==== Build nested set task ====

;Syntax
:<pre>php symfony propel:build-nested-set</pre>
;Description
:Qubit uses the [http://en.wikipedia.org/wiki/Nested_set_model Nested set model] for arranging hierarchical data. The 'build-nested-set' task generates the required column values (lft, rgt) for the loaded data, based on the parent_id self-relationship. Creating the nested set script was quite complex, as it requires recursively descending down the hierarchy tree to derive the proper lft and rgt values. This script could be useful outside of the context of migrations for re-building the nested set after corruption (e.g. due to direct editing of the database) or data loss.

==== Generate slugs task ====

;Syntax
:<pre>php symfony propel:generate-slugs</pre>
;Description
:Qubit generates [http://en.wikipedia.org/wiki/Slug_%28web_publishing%29 Slugs] for creating [[Qubit:Permalinks|Permalinks]] to application resources. The 'generate-slugs' task generates these slugs for any resources that do not already have a slug. Because this script is non-destructive it can be run any time.

==== Digital object load task====

;Syntax
:<pre>php symfony digitalobject:load filename.csv</pre>
;Description
:Takes a CSV file as input which contains two 'columns': "filename" and "information_object_id"; the script will fail if these column headers are not present in the first row of the CSV file. The filename column contains the full (current) path to the digital asset (file). The information_object_id identifies the linked information object. Qubit does not allow more than one digital object per information object (with the exception of derivatives), and each digital object must have a corresponding information object to describe it, so this one-to-one relationship must be respected in the CSV import file.

==2. Migration using CSV import==

===Extract===

Most database applications have an export function to save data to a flat-file format. CSV and tab-delimited exports are especially ubiquitous, which is one of the reasons for using the CSV format.

====Filemaker Pro 10====

Pilot participants: [[SFU data migration]]

Filemaker has an advanced export function that allows export to many flat-file formats, including CSV, tab-delimited, XML and Excel (and others). In addition Filemaker allows the user to select exactly which fields are exported.

We used the "Excel" format for export as it includes column headers, which is not true of the CSV or tab-delimited formats.

====Inmagic====

Pilot participants: [[Anglican Archives data migration]]

Inmagic includes nice export functionality that allows specifying custom delimiters for field, line and multiple values within a field as well as string delimiters (e.g. quotes). Inmagic also allows exporting a found set of records, or the entire database and can include or exclude column headers.

We exported data using the standard CSV format with column headers.

====MS Word====

Pilot participants: [[University of Victoria data migration]]

''TODO''

====MS Excel====

MS Excel will save directly to a CSV or tab-delimited format.

===Validate===

Two comma-separated value (CSV) file templates are provided, one for ISAD(G) and one for RAD. These files can be edited with any common spreadsheet application (Microsoft Excel, LibreOffice Calc, etc.) but must be saved in CSV format to be imported by ICA-AtoM.
# [[Media:Ica atom import template isad.csv|ICA-AtoM ISAD(G) CSV template]]
# [[Media:Ica_atom_import_template_rad.csv|ICA-AtoM RAD CSV template]]

It became clear that a Crosswalk between ISAD(G), RAD, ICA-AtoM, and our .csv template was needed to support Archivists beginning the process of cleaning-up & transferring their data into ICA-AtoM.

Here is a screen capture of the spreadsheet: 
[[Image:Crosswalkcapture.png|Crosswalk Screen Capture]]

Download the whole spreadsheet for your own use:[[File:Crosswalk.ods|Crosswalk.ods]]

===Transform===

*Export data from legacy database as a comma separated values (CSV) text file format. CSV is used to store tabular data and can be accessed using a spreadsheet or database application
*Using the appropriate CSV template, compare legacy database CSV file with either .csv RAD template or .csv ISAD(G) template
*The templates are based on descriptive standards RAD and ISAD(G)
*Many legacy databases are populated with non-standardized data fields. When comparing column headers, the .csv template provides the standardized data fields and the legacy database CSV file may need to be rectified to match
*Rectify column headers in legacy database CSV file
*If necessary, merge data in more than one column to match individual column headers in specific .csv template

*Review CSV document for any anomalies in the data cells (e.g., misspelled names, inconsistent date formats, typos, incomplete records etc...)
*If data is inaccurate or inconsistent, [http://code.google.com/p/google-refine/ Google Refine] is an excellent tool for data clean-up

= Implementation =

==[[City_of_Vancouver_data_migration|City of Vancouver Archives]] data migration==

==[[UNBC data migration]]==

==[[SFU data migration]]==

==[[Anglican Archives data migration]]==

==[[University of Victoria data migration]]==

Development/Projects/AABC Data Migration Toolkit

2015-07-08T17:02:10Z

Sromkey: Created page with "Main Page > Development > Development/Projects AABC Data Migration Toolkit =Background and Objectives= The AABC Data Migration Toolkit project is a collaboration..."

[[Main Page]] > [[Development]] > [[Development/Projects]] AABC Data Migration Toolkit

=Background and Objectives=
The AABC Data Migration Toolkit project is a collaboration between the Archives Association of British Columbia (AABC), Artefactual Systems, the Anglican Diocese of New Westminster Archives, Simon Fraser University Archives, the University of Victoria, and the University of Northern British Columbia Archives to migrate data from legacy database systems used in the BC archival community to ICA-AtoM. The project will begin November 2010 and be completed by April 2011.

The project will analyze and select appropriate technical tool(s) to use in the migration process as well as develop a generic methodology to assist those doing data migration projects. This 'toolkit' should also be useful to other data migration projects, not just those migrating data into ICA-AtoM.

=Project Participants=
* National Archival Development Program (funding)
* Archives Association of British Columbia (coordination)
* Artefactual Systems (technical lead & coordination)
* Anglican Church Archives (InMagic migration & funding)
* Simon Fraser University Archives (FileMaker migration & funding)
* University of Victoria Archives (MS-Word migration & funding)
* University of Northern British Columbia Archives (MS-SQLserver migration & funding)

=Technical Tool Options=

==Drupal-based tool(s)==

There are a number of [http://groups.drupal.org/node/21338 Drupal import/export modules] available.

The [http://drupal.org/project/migrate Migrate module] developed by [http://cyrve.com Cyrve] looks interesting. Cyrve has also published their migration methodology [http://drupal.org/node/915102] [http://drupal.org/node/661762].

Presumably we can use Drupal CCK and Views to model almost any data structure.
Then we can use export functionality or other internal data migration tools to
map it and export it to Qubit data structure?

Also, once we get external data into Drupal we could, theoretically, expose it
as RDF data to get it out into other systems:

* http://drupal.org/project/rdf
* http://openspring.net/blog/2009/10/22/produce-and-consume-linked-data-with-drupal

See also: [http://drupal.org/project/xc The eXtensible Catalog (XC) Drupal Toolkit]

===Reasons for elimination of Drupal-based tools===

Ultimately any Drupal based tool is going to involve a two step migration process
# Migrate data Drupal schema (via CCK?)
# Migrate data to Qubit schema (via RDF?)

The extra step of migrating data to Drupal is a significant amount of extra work, without any concrete benefits. The difficulty of migrating the data to Drupal is exacerbated by the complex nature of the source and Qubit schemas. Implementing an RDF schema for Drupal and Qubit would have benefits beyond the scope of this project, but requires a significant amount of extra work that is not within the scope of the project. In addition any Drupal modules are, by nature, a extension on the Drupal framework, which will involve compromises that are not required for task specific ETL tools.

Using Drupal as a migration tool also requires installing an configuring Drupal, plus the necessary Drupal modules (CCK, View, Migrate, RDF, etc.); This involves significantly more work then installing a stand-alone ETL desktop application.

==Google Refine (formerly Freebase Gridworks)==

* http://code.google.com/p/google-refine/wiki/GettingStarted?tm=6

===Reasons for elimination of Google Refine===
Google refine only works with flat-files [http://code.google.com/p/google-refine/wiki/Importers] (version 2.0 2010-11-25) so it's not an option for Filemaker, MSSQL, MySQL or other SQL formats, which are part of our requirements.

==Open-source ETL tool(s)==

[http://en.wikipedia.org/wiki/Extract,_transform,_load#Open-source_ETL_frameworks Wikipedia list] of [http://en.wikipedia.org/wiki/Extract,_transform,_load Extract, transform, load] (ETL) tools

It's tough finding any decent (unbiased) information on any ETLs and the open source ones especially. A couple of names cropped up multiple times in my searches:

[http://kettle.pentaho.com/ Pentaho Data Integration (Kettle)] - community edition (FOSS)
* screencast: http://www.pentaho.com/products/demos/PDI_overview/PDI_overview.html (skip to ~5m00s for live demo)

[http://www.talend.com/products-data-integration/talend-open-studio.php Talend Open Studio]
* screencast: http://www.talend.com/demo/index.php

It sounds like either tool should be fine for small-scale projects like the ones we are tackling. Both Kettle and Talend OS use a Java GUI and thus run on Linux, Windows and Mac. [http://www.jonathanlevin.co.uk/2008/03/open-source-etl-tools-vs-commerical-etl.html Blog comparison of Pentaho Kettle and Talend]

Talend immediately annoyed me by offering the dreaded "feature comparison matrix" for it's own products and being pretty obtuse about which one is their open-source offering, as well as asking me to register to view their demos. >:-( So, Kettle beats Talend for first try by virtue of having the least annoying website.

I also checked out the screencast for [http://www.apatar.com/ Apatar] (which has an [http://aws.amazon.com/amis/819?_encoding=UTF8&jiveRedirect=1 EC2 AMI] - cool) but I didn't find much information about it other than Wikipedia and their own website. Apatar has a similar GUI to Kettle and Talend. Apatar's puts a lot of emphasis on it's web-service [http://www.apatarforge.org/wiki/display/GUI/Apatar+Connector+Guides connectors] - which include Salesforce, SugarCRM, Amazon S3, but this functionality is not required or beneficial for this project.

Also checked out the [http://scriptella.javaforge.com/ Scriptella] site. Scriptella is very appealing at first glance because of it's non-corporate web presence and it's claims of simplicity. I was thrown off a bit when I found a post questioning whether Scriptella was dead [http://www.javaforge.com/proj/forum/viewMessage.do?thread_id=42103&msg_id=42103&thread_members=true], but they released the 1.0 version in May, 2010 which is somewhat reassuring. However Scriptella does look like a one-man show which is a bit worrying. Scriptella looks like it's command line + config files, so no GUI , which significantly raises the barrier to entry for non-programmers. Random [http://www.javaworld.com/community/node/5234 article about Scriptella]

===Reasons for selection of Pentaho Data Integration (Kettle)===

Pentaho Data Integration (PDI) Community Edition (also known as Kettle) is a purpose built ETL tool with a feature set that matches and exceeds our requirements.

The Community Edition of PDI is open-source (LGPL version 2), which meshes well with the open-source philosophy of Qubit. PDI uses a graphical user interface (GUI) running on a Java Virtual Machine (several open-source JVMs are available) which allows the application to run, without functional differences, on Mac, Linux or Windows platforms.

Installing PDI is straight-forward, though it does require installing a suitable JVM as well. This compares favorably with having to install a Drupal-based migration toolset.

There is a steep learning curve for using PDI, especially with regard to the concurrent (multi-thread) and non-sequential property of the transformations. However, [http://wiki.pentaho.com/display/EAI/Latest+Pentaho+Data+Integration+%28aka+Kettle%29+Documentation PDI's documentation] is good, and includes several video and text tutorials, and the PDI community forum is quite active. The migrations that we develop for our pilot organizations and our own documentation should also help ease the learning process.

==Comma Seperated Values (CSV) import==
===Reasons for selection of CSV import as preferred migration path===

'''needs update (note: reserve Pentaho as option for complex migrations but not likely a tool that will be easily implemented by most archivists)'''

=Methodology=

"Real life ETL Cycle" (from Wikipedia [http://en.wikipedia.org/wiki/Extract,_transform,_load], 2010-12-02)

# Cycle initiation
# Build reference data
# Extract (from sources)
# Validate
# Transform (clean, apply business rules, check for data integrity, create aggregates or disaggregates)
# Stage (load into staging tables, if used)
# Audit reports (for example, on compliance with business rules. Also, in case of failure, helps to diagnose/repair)
# Publish (to target tables)
# Archive
# Clean up

==1. Migration using Pentaho==
===Extract===
Source Data Formats:
* MS SQL Server: [[UNBC data migration]]
* Filemaker Pro 10: [[SFU data migration]]

===Transform===

Pentaho Data Integration (PDI) includes a GUI interface for designing and running transformations named "spoon". The screenshots below show the spoon interface.

PDI organizes the individual "Steps" for transforming and mapping data into "Transformations". Data flows between Steps through "Hops" (the arrows) in an non-sequential and simultaneous fashion. This simultaneous execution of steps can be non-intuitive so the user must be careful not to assume what data will already be migrated (e.g. for foreign key relations).

[[File:FondtoInfoObject_ktr.png|600px]]

Several "Transformations" can be further grouped in to "Jobs". Jobs run as series of Transformations sequentially (with error handling, state checking, etc.)

[[File:UNBC Transform Job.png|600px]]

===Load===

There are several possible options for loading the transformed data into Qubit, each with pros and cons.

====Write directly to MySQL====

My initial instinct was to use the ETL tool to write directly to MySQL. This approach avoids the pain points we've had in the past with data load/import via the ORM: bad performance, large memory requirements and vexing segfaults.

However, This approach has proved difficult and time-consuming due to the highly relational nature of Qubit. To create one archival description record we must write to the following tables:

* information_object
* information_object_i18n
* note
* note_i18n
* object
* object_term_relation
* property
* property_i18n
* relation
* slug
* status
* term
* term_i18n
* ... more?

Many of these tables (e.g. property, note) require writing many rows per archival description.

In addition we've embedded the equivalent of SQL "triggers" in the Qubit ORM in an attempt to be database agnostic. Examples of this are the nested set logic, and slug name sanitization. Having a separate implementation of this logic in our ETL tool will make the transform fragile and very difficult to maintain.
* Thought: Can we run a php script after import to leverage the ORM and fire these "triggers"?

====Import via EAD====

We have experience with EAD import into ICA-AtoM so we are clear about it's limitations - high resource requirements (low performance and scalability), mapping ambiguities and an enforcement of one fonds/collection per EAD-XML file. However, the very fact that the problem space is well defined and that solutions will benefit the application as a whole (such as round-tripping) make EAD appealing.

;Pro
:We already have EAD import done (with a few outstanding issues, see below)
:Don't have to build table hierarchy (e.g. object, actor, repository, slug) manually
:Data-model integrity and logic enforced by embedded ORM logic (don't have to replicate in transform step)
:portable to any system that will import EAD

;Con
:EAD format is a very detailed and finicky, and therefore time-consuming to implement
:Data in the Qubit schema that we can not represent in EAD? (maybe we can "bend" EAD to capture this data?)
:EAD import has several outstanding issues [http://code.google.com/p/qubit-toolkit/issues/list?can=2&q=Component%3DEAD]
:Slow load, large memory requirements, segfaults?
:EAD enforces one top-level description per file, which makes bulk load very work-intensive (need to import many files)

Some initial research with the [http://kettle.pentaho.com/ Pentaho Kettle] ETL makes it clear that the Kettle XML output step will ''not'' support a highly structured XML format like EAD. This leaves us with several possible options:
* Write a simple XML file from Kettle, then use [http://en.wikipedia.org/wiki/XSLT XLST] to convert this intermediary file to EAD-XML
* Use [http://en.wikipedia.org/wiki/ECMAScript_for_XML e4x] and Kettle's javascript capabilities to write EAD directly ([http://forums.pentaho.com/showthread.php?68265-How-to-XML e4x example])
* Write a simple XML file from Kettle then use another tool (Google refine?) to transform to EAD-XML

====Automated http input====

Another possibility is loading data using an HTTP POST to the web front-end, via CURL or another tool. Like the EAD import, this takes advantage of current data model logic embedded in Qubit ORM. This method also has the advantage over EAD import that we don't have to worry about mapping data from the source to EAD and then to Qubit, so we don't have to worry about what data EAD may not capture gracefully.

Automated form input may be especially useful for the digital object load, as we are currently doing in Archivematica. The big issues we've seen with the Archivematica digital object scripting is fragility of script due to UI changes in Qubit, and finding the URI for the related information object (how do we get the right slug?).

Matching URIs is actually a huge problem with automated form input in general - for example how do we easily find the URI for a related name access point to select the correct value in the "name access point" drop-down? Performance is also a serious concern here, because the page load times for creating/editing a single description can be greater than 5 seconds.

=== Post-load scripts===

Several scripts must be run after data load to add secondary data that is essential for the Qubit application, but is not easily generated during [[#Transform]].

==== Build nested set task ====

;Syntax
:<pre>php symfony propel:build-nested-set</pre>
;Description
:Qubit uses the [http://en.wikipedia.org/wiki/Nested_set_model Nested set model] for arranging hierarchical data. The 'build-nested-set' task generates the required column values (lft, rgt) for the loaded data, based on the parent_id self-relationship. Creating the nested set script was quite complex, as it requires recursively descending down the hierarchy tree to derive the proper lft and rgt values. This script could be useful outside of the context of migrations for re-building the nested set after corruption (e.g. due to direct editing of the database) or data loss.

==== Generate slugs task ====

;Syntax
:<pre>php symfony propel:generate-slugs</pre>
;Description
:Qubit generates [http://en.wikipedia.org/wiki/Slug_%28web_publishing%29 Slugs] for creating [[Qubit:Permalinks|Permalinks]] to application resources. The 'generate-slugs' task generates these slugs for any resources that do not already have a slug. Because this script is non-destructive it can be run any time.

==== Digital object load task====

;Syntax
:<pre>php symfony digitalobject:load filename.csv</pre>
;Description
:Takes a CSV file as input which contains two 'columns': "filename" and "information_object_id"; the script will fail if these column headers are not present in the first row of the CSV file. The filename column contains the full (current) path to the digital asset (file). The information_object_id identifies the linked information object. Qubit does not allow more than one digital object per information object (with the exception of derivatives), and each digital object must have a corresponding information object to describe it, so this one-to-one relationship must be respected in the CSV import file.

==2. Migration using CSV import==

===Extract===

Most database applications have an export function to save data to a flat-file format. CSV and tab-delimited exports are especially ubiquitous, which is one of the reasons for using the CSV format.

====Filemaker Pro 10====

Pilot participants: [[SFU data migration]]

Filemaker has an advanced export function that allows export to many flat-file formats, including CSV, tab-delimited, XML and Excel (and others). In addition Filemaker allows the user to select exactly which fields are exported.

We used the "Excel" format for export as it includes column headers, which is not true of the CSV or tab-delimited formats.

====Inmagic====

Pilot participants: [[Anglican Archives data migration]]

Inmagic includes nice export functionality that allows specifying custom delimiters for field, line and multiple values within a field as well as string delimiters (e.g. quotes). Inmagic also allows exporting a found set of records, or the entire database and can include or exclude column headers.

We exported data using the standard CSV format with column headers.

====MS Word====

Pilot participants: [[University of Victoria data migration]]

''TODO''

====MS Excel====

MS Excel will save directly to a CSV or tab-delimited format.

===Validate===

Two comma-separated value (CSV) file templates are provided, one for ISAD(G) and one for RAD. These files can be edited with any common spreadsheet application (Microsoft Excel, LibreOffice Calc, etc.) but must be saved in CSV format to be imported by ICA-AtoM.
# [[Media:Ica atom import template isad.csv|ICA-AtoM ISAD(G) CSV template]]
# [[Media:Ica_atom_import_template_rad.csv|ICA-AtoM RAD CSV template]]

It became clear that a Crosswalk between ISAD(G), RAD, ICA-AtoM, and our .csv template was needed to support Archivists beginning the process of cleaning-up & transferring their data into ICA-AtoM.

Here is a screen capture of the spreadsheet: 
[[Image:Crosswalkcapture.png|Crosswalk Screen Capture]]

Download the whole spreadsheet for your own use:[[File:Crosswalk.ods|Crosswalk.ods]]

===Transform===

*Export data from legacy database as a comma separated values (CSV) text file format. CSV is used to store tabular data and can be accessed using a spreadsheet or database application
*Using the appropriate CSV template, compare legacy database CSV file with either .csv RAD template or .csv ISAD(G) template
*The templates are based on descriptive standards RAD and ISAD(G)
*Many legacy databases are populated with non-standardized data fields. When comparing column headers, the .csv template provides the standardized data fields and the legacy database CSV file may need to be rectified to match
*Rectify column headers in legacy database CSV file
*If necessary, merge data in more than one column to match individual column headers in specific .csv template

*Review CSV document for any anomalies in the data cells (e.g., misspelled names, inconsistent date formats, typos, incomplete records etc...)
*If data is inaccurate or inconsistent, [http://code.google.com/p/google-refine/ Google Refine] is an excellent tool for data clean-up

= Implementation =

==[[City_of_Vancouver_data_migration|City of Vancouver Archives]] data migration==

==[[UNBC data migration]]==

==[[SFU data migration]]==

==[[Anglican Archives data migration]]==

==[[University of Victoria data migration]]==

Development/Projects

2015-07-02T22:59:36Z

Sromkey: Created page with "{{#pagetitle: Projects }} Main Page > Development > Development/Projects This area of the wiki is used to hold descriptions of AtoM development or research projects...."

{{#pagetitle: Projects }}
[[Main Page]] > [[Development]] > Development/Projects

This area of the wiki is used to hold descriptions of AtoM development or research projects. Most projects listed here have concluded and are listed here for historical purposes.

==Current projects==

We'll add new or upcoming development or research projects here!

==Historical projects==

*[[Development/Projects/ArchivesCanada Upgrade]]
*[[Development/Projects/AABC Data Migration Toolkit]]
*[[Development/Projects/LAC ICA-AtoM Scalability Testing]]
*[[Development/Projects/AAO & Vancouver ICA-AtoM Usability Testing]]
*[[Development/Projects/BCAUL Pilot Project (MemoryBC)]]
*[[Development/Projects/Alouette Toolkit (Digital Collections Builder)]]

[[Category:Development documentation]]

Releases/Release announcements/Release 1.3.1

2015-07-01T20:17:53Z

Sromkey: Created page with "{{#pagetitle: Release 1.3.1 }} Main Page > Releases > Releases/Release announcements > Releases/Release announcements/Release 1.3.1 '''Released May 7, 2013''' |..."

{{#pagetitle: Release 1.3.1 }}
[[Main Page]] > [[Releases]] > [[Releases/Release announcements]] > Releases/Release announcements/Release 1.3.1

'''Released May 7, 2013''' | Download [http://pear.qubit-toolkit.org/get/icaatom-1.3.1.tgz]

This is a bug fix release intended to fix several major bugs that were found in Release 1.3.

In our tracker:
https://projects.artefactual.com/versions/53

== Bug fixes ==

* Digital object upload path is wrong when user culture is not English [https://projects.artefactual.com/issues/4372 4372]
* Fix the translation bar not showing up for some themes [https://projects.artefactual.com/issues/552 552]
* Fix a glitch in the search bar only affecting IE9 users
* Multiple fixes in the search index [https://projects.artefactual.com/issues/4713 4713]
* Tooltips can now be disabled (regression introduced in 1.3, [https://projects.artefactual.com/issues/4286 4286]
* Compatibility fix for PHP 5.4
* Optimization of slug generation
* Fix a problem with the user notes [https://projects.artefactual.com/issues/4539 4539]
* Update IP security filter to avoid IP spoofing
* Fix a bug in the dialogs where XHR requests were not being processed properly [https://projects.artefactual.com/issues/4869 4869]
** Thanks to Kehan Harman for his contribution!
* Validate the RAD field Statement of scale (architectural) field properly [https://projects.artefactual.com/issues/4165 4165]
* Allow to reset the accession number to zero [https://projects.artefactual.com/issues/4604 4604]
* Hide draft descriptions in the imageflow [https://projects.artefactual.com/issues/3887 3887]
* Fix in the viewDraft permission [https://projects.artefactual.com/issues/3306 3306]
* '''Treeview fixes'''
** Show child-level descriptions instead of siblings [https://projects.artefactual.com/issues/4357 4357]
** Removed "Show all" button [https://projects.artefactual.com/issues/573 573]
** Fix culture fallback for title in treeview [https://projects.artefactual.com/issues/574 574]
** Fix pop-ups issues in IE8 within the treeview and the digital object browser [https://projects.artefactual.com/issues/4452 4452] [https://projects.artefactual.com/issues/561 561]
* '''Better integration with Archivematica'''
** Updates to use net_gearman instead of pecl/gearman [https://projects.artefactual.com/issues/4767 4767]
** Improved the import of METS documents
** Stability tweaks in the SWORD worker [https://projects.artefactual.com/issues/4182 4182]
** Update the upstart service found in /init

[[Category:Releases]]

__NOTOC__

Releases/Release announcements/Release 1.3

2015-07-01T20:01:30Z

Sromkey: Created page with "{{#pagetitle: Release 1.3 }} Main Page > Releases > Releases/Release announcements > Releases/Release announcements/Release 1.3 '''Released Aug 27, 2012''' | Down..."

{{#pagetitle: Release 1.3 }}
[[Main Page]] > [[Releases]] > [[Releases/Release announcements]] > Releases/Release announcements/Release 1.3

'''Released Aug 27, 2012''' | Download:

'''Release 1.3''' includes PDF Fulltext Search, security enhancements, improvements to Treeview functionality, and many bug fixes to make this the most stable and fully-functional ICA-AtoM release to date.

Release 1.3 comes nine months after the 1.2 release in November 2011. Release 1.3 includes updated translations for Dutch, French, Polish, Portuguese, Slovenian and Spanish, along with the addition of Galician, Georgian, Indonesian, Japanese and Thai!

== New Features ==

* PDF fulltext search Issue #1252
* Batch search index updates have been optimized to run 10 - 100x faster!
* Command line and web interface CSV import for archival descriptions (RAD and ISAD), authority records, accessions and events
* IE8 compatibility with Trillium theme
* Addition of ellipses to expand or contract lengthy text fields Issue #537
* Administrator control of the public visibility of some archival description elements (Visibility Elements) Issue #1931

== Enhancements ==

==== Security Enhancements ====

* Enhance login process to force all authenticated user requests to https (requires SSL certificate on server) Issue #2226
* Enhance login validation to force use of strong passwords Issue #2227: at least 8 characters long, contains characters from 3 of the following classes:
* # Upper case letters
* # Lower case letters
* # Numbers
* # Special characters
* Limit incoming requests for all Administrator functionality to a static IP address range Issue #2228

==== Treeview enhancements ====

* Brand new treeview!
* Drag & drop sorting Issue #1289
* View large number of child records in context Issue #1943
* Pop-up to show long description names when hovering over name

==== Import ====

* Integratration of command line and web interface for CSV imports Issue #140
* Allow XML or CSV import to an existing Fonds, Collection or Series Issue #1993

==== Accessions ====

* Add fulltext search for accessions Issue #2323
* Add ability to link pre-existing information objects to accession records Issue #2178

==== Browse ====

* Default sort order setting (Alphabetic or Last Updated) can be configured by the administrator Issue #2235

==== Searching ====

* Allow searching on textual dates Issue #2294
* Allow search by date range Issue #2261
* Add copyright status to advanced search Issue #2314
* Add extra search fields for authority records and repositories issue #1965

==== Usability ====

* Addition of ellipses to expand or contract text boxes Issue #537

==== Misc ====

* Display Archivematica AIP and File UUID for matching ICA-AtoM digital object Issue #2379
* SQL-only data schema upgrades for Release 1.1 thru Release 1.3 Issue #2270

== Internationalization ==

*New! Galician, Georgian, Indonesian, Japanese and Thai translations of the user interface.
*Merged new and improved user interface translations for Dutch, French, Polish, Portuguese, Slovenian and Spanish (thank you to all our volunteer translators).
*Complete French translation of Rules for Archival Description (RAD) elements. (Translation funded by Manitoba Heritage Grant courtesy of the Department of Culture, Heritage and Tourism.)

== Bug fixes ==

* Issue# 2420: CSV Import throws warning
* Issue #2417: Treeview: first click on "arrow" doesn't show child levels
* Issue #2415: filters.yml problem under qtSwordPlugin
* Issue #2413: Deleted records showing up in search results
* Issue #2411: Publication status in treeview is adding double parentheses in all themes except Trillium
* Issue #2408: Pop-up display in treeview not visible wen browser window is resized
* Issue #2407: Publication status in treeview should come at end of information object title
* Issue #2402: Admin user gets deactivated
* Issue #2395: Taxonomy/term treeview is broken
* Issue #2394: Installer theme is broken
* Issue #2390: CSV: check -import task fails with error. Invalid column "culture"
* Issue #2384: EAD - dates not importing
* Issue #2382: Unauthorized users should not be able to view PREMIS rights
* Issue #2377: Search for donor name produces no results for authenticated user/admin
* Issue #2376: Repository names mixed with Authority record names in drop-down, selection of Repository name results in 404 error
* Issue #2368: User orange circle "a" favicon for release 1.3
* Issue #2365: In Edit digital object screen, substitute new representation results in white screen
* Issue #2364: In Edit digital object screen, changing mediatype causes white screen
* Issue #2360: Rights holder name appears in drop-down/suggested list for Name of Creator data entry field
* Issue #2358: Move button not working below fonds level in 1.3
* Issue #2348: User Groups unavailable in drop-down when Admin tried to add new user in Trillium theme and Chrome browser
* Issue #2344: New information objects created from the accession page are not linked
* Issue #2337: User with translation rights is unable to edit the information in ISAAR fields
* Issue #2334: Draft status records not appearing in search results for authenticated users
* Issue #2330: Admin cannot view user permissions by repository
* Issue #2323: Add fulltext search for accessions
* Issue #2314: Add copyright status to advanced search
* Issue #2313 Hide accession numbers in archival description from public
* Issue #2312: Trillium css compression of page counter in browse digital objects view
* Issue #2311: Delete reference copy of digital object and replace or regenerate copy causes error
* Issue #2310: "Language and script notes" field should not repeat
* Issue #2298: Can't delete final UF field from taxonomy term
* Issue #2291: Translation interface broken
* Issue #2286: Using "+" in Treeview at Series-level, reveals child-levels without Publication Status "Draft"
* Issue #2284: Administrator unable to edit new user account's active status or edit new user account
* Issue #2276: Search results for PDF, TIFF and JPEG produce multiples.
* Issue #2273: Deleted archival description included in search results.
* Issue #2272: Change field label "Archival history" to "Archival /Custodial history" in Accessions template
* Issue #2270: SQL-only data schema upgrades / Switch migration task to direct db updates instead of YAML upgrades
* Issue #2263: Solve taxonomy and term constant
* Issue #2246: Moving an information object can break the nested set
* Issue #2238: Full reference code for archival descriptions is truncated in search results screen
* Issue #2235: Defaut sort order setting (Alphabetic or Last Updated) should be Admin Configurable
* Issue #2234: Qubit-sword daemon stops working after a long time running (MySQL server has gone away)
* Issue #2220: Migrate task in 1.2 release is not bumping database version during upgrades
* Issue #2212: Lucene Range Search broken in ICA-AtoM 1.2
* Issue #2209: Server 500 error on multiple record deletion
* Issue #2207: Apache (with mod_rewrite?) returns Forbidden error if permalink slug is more than 255 characters
* Issue #2206: Reports - Date field at file and item level populated with date from fonds-level
* Issue #2203: Browse menu translated in Caribou but NOT other themes. Browse results page title not translated.
* Issue #2198: Fatal error viewing file/item report in language other than English
* Issue #2197: In ICA-AtoM Selecting Dublin Core as the default template in Admin>settings does not change resource edit template. Defaults to ISAD(G).
* Issue #2194: Search and Replace results in Fatal error.
* Issue #2176: Can't print file or item lists in DC or MODS
* Issue #2168: Translated UI labels appear in English
* Issue #2147: Trillium theme css not sticking in narrow Safari browser window
* Issue #2143: Trillium: pager alignment is inconsistent
* Issue #2141: Unable to activate a theme when browsing in a language distinct to english
* Issue #2048: Link to ALOUETTE.ca on ABOUT page is wrong, replace with DCB link.
* Issue #2035: RAD Note field for Language needs to be text, not a list of languages
* Issue #1995: Segmentation faults when calling _toString() magic method implicitly
* Issue #1993: Specify parent for EAD import
* Issue #1983: Actor records (users, authority records and repositories) cause incorrect hit count for authenticated users
* Issue #1975: ICA-AtoM virtual appliance does not work under VirtualBox 4.0.4
* Issue #1943: Context menu "see all" links are not useful for a large number (100+) of children
* Issue #1915: User profile should include institutional (repository) affiliation
* Issue #1628: String fields in the db are limited to 255 characters but Users demand bigger sizes and there is no warning
* Issue #1519: When logged in user doesn't have update permission to update archival descriptions, can partially drag and drop in hierarchy treeview
* Issue #1409: Users without view draft permissions can navigate to drafts in information object treeview hierarchy
* Issue #1344: Information div titles in edit screens disappear when scrolled over or clicked on
* Issue #1327: Sorting treeview error when mixing identifiers with none identifiers
* Issue #1280: Prevent same user from being created more than once
* Issue #1113: Can't delete a user if they have created a note
* Issue #767: Include physical objects (i.e. storage containers) in the search index

[[Category:Releases]]

__NOTOC__

Releases/Release announcements/Release 1.2.1

2015-07-01T19:23:07Z

Sromkey:

{{#pagetitle: Release 1.2.1 }}
[[Main Page]] > [[Releases]] > [[Releases/Release announcements]] > Releases/Release announcements/Release 1.2.1

'''Released June 28, 2012''' | Download: [http://pear.qubit-toolkit.org/get/icaatom-1.2.1.tgz icaatom-1.2.1.tgz]

'''Release 1.2.1''' is a patch release intended to fix several major bugs that were found in Release 1.2, including compatibility of the Trillium theme with IE8.

== Bug fixes ==

* Level of description not linked to Imported digital objects, Issue #2172
* Error editing and/or replacing contact details in Repository record, Issue #2219
* Migrate task in 1.2 Release is not bumping database version during upgrades, Issue #2220
* Fatal error in advanced search when selecting mediatype filter, Issue #2223
* Advanced search using mediatype doesn't return results, Issue #2255
* User cannot see admin menu after admin privileges are added to their account, Issue #2266
* Admin menu caching problems, Issue #2231
* Broken links to Authority records in search results, Issue #2249
* Can't delete final UseFor (UF) Field from taxonomy term, Issue #2298
* Apache returns forbidden error if permalink slug is more than 255 characters, Issue #2207
* Draft status records are not appearing in search results for authenticated users, Issue #2334
* User with translation rights is unable to edit the information in Authority record fields, Issue #2337
* User groups are unavailable in drop-down when Admin tries to add new user in Trillium theme and Chrome browser, Issue #2348
* Trillium theme compatibility with IE8, Issue #2150

__NOTOC__

[[Category:Releases]]

Releases/Release announcements/Release 1.2.1

2015-07-01T19:18:13Z

Sromkey: Created page with "{{#pagetitle: Release 1.2.1 }} Main Page > Releases > Releases/Release announcements > Releases/Release announcements/Release 1.2.1 '''Released June 28, 2012'''..."

Releases/Release announcements

2015-07-01T18:48:54Z

Sromkey:

{{#pagetitle: Release announcements }}

[[Main Page]] > [[Releases]] > Releases/Release announcements

Here are detailed lists of the new features, enhancements, and bug fixes in each ICA-AtoM and AtoM release.

<admonition type="tip">
Looking for information on upcoming releases? Check the [[Releases/Roadmap|Roadmap]] page!
</admonition>

== AtoM (2.x) releases ==

AtoM 2 releases are available on the AtoM website, at: https://www.accesstomemory.org/download/

* [[Releases/Release announcements/Release 2.2|Release 2.2]] (June 2015)
* [[Releases/Release announcements/Release 2.1.2|Release 2.1.2 translation pack]] (February 10, 2015 )
* [[Releases/Release announcements/Release 2.1.1|Release 2.1.1 security upgrade]] (January 28, 2015)
* [[Releases/Release announcements/Release 2.1|Release 2.1.0]] (September 22, 2014)
* [[Releases/Release announcements/Release 2.1.1|Release 2.0.2 security upgrade]] (January 28, 2015)
* [[Releases/Release announcements/Release 2.0.1|Release 2.0.1]] (December 16, 2013)
* [[Releases/Release announcements/Release 2.0.0|Release 2.0.0]] (October 7, 2013)

== ICA-AtoM (1.x) releases ==

For more information about ICA-AtoM, our earlier version of the AtoM software, please visit the ICA-AtoM wiki: https://www.ica-atom.org/doc/Main_Page

* [[Releases/Release announcements/Release 1.4|Release 1.4]] (unscheduled)
* [[Releases/Release announcements/Release 2.1.1|Release 1.3.2 security upgrade]] (January 28, 2015)
* [[Releases/Release announcements/Release 1.3.1|Release 1.3.1]] (May 7, 2013)
* [[Releases/Release announcements/Release 1.3|Release 1.3]] (August 27, 2012)
* [[Releases/Release announcements/Release 1.2.1|Release 1.2.1]] (June 28, 2012)
* [[Releases/Release announcements/Release 1.2|Release 1.2]] (November 29, 2011)
* [[Releases/Release announcements/Release 1.1|Release 1.1]] (November 22, 2010)
* [[Releases/Release announcements/Release 1.0.9-beta|Release 1.0.9-beta]] (May 17, 2010)
* [[Releases/Release announcements/Release 1.0.8-beta|Release 1.0.8-beta]] (November 2, 2009)
* [[Releases/Release announcements/Release 1.0.7-beta|Release 1.0.7-beta]] (June 1, 2009)
* [[Releases/Release announcements/Release 1.0.6-beta|Release 1.0.6-beta]] (April 27, 2009)
* [[Releases/Release announcements/Release 1.0.5-beta|Release 1.0.5-beta]] (March 11, 2009)
* [[Releases/Release announcements/Release 1.0.4-beta|Release 1.0.4-beta]] (November 17, 2008)
* [[Releases/Release announcements/Release 1.0.3-beta|Release 1.0.3-beta]] (October 9, 2008)
* [[Releases/Release announcements/Release 1.0.2-beta|Release 1.0.2-beta]] (August 8, 2008)
* [[Releases/Release announcements/Release 1.0.1-beta|Release 1.0.1-beta]] (July 22, 2008)
* [[Releases/Release announcements/Release 1.0-beta|Release 1.0-beta]] (July 1, 2008)

[[Category:Releases]]

Releases/Release announcements/Release 2.1.1, 2.0.2, 1.3.2

2015-07-01T18:47:52Z

Sromkey:

{{#pagetitle: Release 2.1.1, 2.0.2, 1.3.2 }}
[[Main Page]] > [[Releases]] > [[Releases/Release announcements]] > Releases/Release announcements/Release 2.1.1, 2.0.2, 1.3.2

'''Released January 28, 2015'''

<admonition type="note"> This release announcement was originally posted in the AtoM User Forum, and has been copied here. The original post can be seen at: https://groups.google.com/d/msg/ica-atom-users/ScS-JMW-nh8/p-DuSq3_X_QJ
</admonition>

Greetings AtoM users!

A new security upgrade is now available for all our major branches. '''We strongly encourage all users to upgrade to the most recent security release'''. Security upgrade releases include:

* AtoM 2.1.1 - the most current release
* AtoM 2.0.2 - security upgrade for 2.0.1 users who have chosen not to upgrade to 2.1
* ICA-AtoM 1.3.2 - security upgrade for ICA-AtoM users who have chosen not to upgrade to AtoM 2

All security releases are available on our Downloads page:

https://accesstomemory.org/download/

Upgrade instructions for AtoM 2 users can be found here:

https://accesstomemory.org/docs/2.1/admin-manual/installation/upgrading/#installation-upgrading

For ICA-AtoM users looking to upgrade from 1.3.1 or earlier to the 1.3.2 security release, upgrade instructions can be found here:

https://www.ica-atom.org/doc/Upgrading

'''PLEASE NOTE:''' We are not actively maintaining the legacy ICA-AtoM 1.x branch. At this time we do not expect to be making any further ICA-AtoM 1.x releases, as all active development is going to our 2.x branch. We encourage our global ICA-AtoM users to take this opportunity to explore AtoM 2, and consider upgrading to 2.1.2 to enjoy all the newest features. AtoM 2 shares the same code base, is managed by the same lead developers, and is released under the same open-source license. Key differences include a new look and feel, a vastly improved search index, and a completely rewritten set of documentation, as well as hundreds of bug fixes, dozens of new features, and a bright future as we continue to develop the application. Some links for more information:

* AtoM 2 homepage: https://accesstomemory.org/
* AtoM 2.0.0 release announcement (explains some of the differences further): https://groups.google.com/d/msg/ica-atom-users/_zgOnNxM1mE/ODGTv_Bxox4J
* AtoM 2.1 demo site: http://demo.accesstomemory.org/
* AtoM 2.1 documentation: https://accesstomemory.org/docs/2.1/

These security releases were first announced 2 weeks ago, to give system administrators time to schedule an upgrade window. See the original advance announcement here:
https://groups.google.com/d/msg/ica-atom-users/JH8c8eBl2P0/c6MSBObn0y0J

Note that if you are a hosting client with Artefactual systems, your site has already been upgraded to the security release.

AtoM 2.1.1 also includes several minor bug fixes including:

* #7352 - Search text box in AtoM clears text when clicking in / out of the text box
* #7649 - Add --index option to Accessions CSV import CLI task
* #7417 - Hide Archival history in public Advanced search if field is hidden via Visible elements
* #7418 - Fix multiple deletion problem for Alternative identifiers in edit form
* #7419 - Fix public search of Alternative identifier fields
* #7282 - Fix URL export links in EAD when roundtripping
* #7328 - Minor EAD mapping improvements

For a full list of issue tickets related to the 2.1.1 release, please see:
https://projects.artefactual.com/versions/86

TRANSLATORS - Please note, there are NO new translation files included in the 2.1.1 release. Instead, we will be including them in an upcoming translation release, 2.1.2, which we anticipate releasing on February 9, 2015. For more information, please see this User forum post:

https://groups.google.com/d/msg/ica-atom-users/vlnl21EQvOo/ApTpgWg1XSsJ

Meanwhile, we are hard at work on our 2.2 release, which will include a number of exciting new features. We expect the 2.2 release to be available later in Q1, or early Q2, of 2015.

Regards,

Dan Gillean

__NOTOC__

[[Category:Releases]]

Releases/Release announcements/Release 2.1.1, 2.0.2, 1.3.2

2015-07-01T18:32:24Z

Sromkey: Sromkey moved page Releases/Release announcements/Release 2.1.1 to Releases/Release announcements/Release 2.1.1, 2.0.2, 1.3.2: To prevent making 3 security release pages

{{#pagetitle: Release 2.1.1, 2.0.1, 1.3.2 }}
[[Main Page]] > [[Releases]] > [[Releases/Release announcements]] > Releases/Release announcements/Release 2.1.1, 2.0.1, 1.3.2

'''Released January 28, 2015'''

<admonition type="note"> This release announcement was originally posted in the AtoM User Forum, and has been copied here. The original post can be seen at: https://groups.google.com/d/msg/ica-atom-users/ScS-JMW-nh8/p-DuSq3_X_QJ
</admonition>

Greetings AtoM users!

A new security upgrade is now available for all our major branches. '''We strongly encourage all users to upgrade to the most recent security release'''. Security upgrade releases include:

* AtoM 2.1.1 - the most current release
* AtoM 2.0.2 - security upgrade for 2.0.1 users who have chosen not to upgrade to 2.1
* ICA-AtoM 1.3.2 - security upgrade for ICA-AtoM users who have chosen not to upgrade to AtoM 2

All security releases are available on our Downloads page:

https://accesstomemory.org/download/

Upgrade instructions for AtoM 2 users can be found here:

https://accesstomemory.org/docs/2.1/admin-manual/installation/upgrading/#installation-upgrading

For ICA-AtoM users looking to upgrade from 1.3.1 or earlier to the 1.3.2 security release, upgrade instructions can be found here:

https://www.ica-atom.org/doc/Upgrading

'''PLEASE NOTE:''' We are not actively maintaining the legacy ICA-AtoM 1.x branch. At this time we do not expect to be making any further ICA-AtoM 1.x releases, as all active development is going to our 2.x branch. We encourage our global ICA-AtoM users to take this opportunity to explore AtoM 2, and consider upgrading to 2.1.2 to enjoy all the newest features. AtoM 2 shares the same code base, is managed by the same lead developers, and is released under the same open-source license. Key differences include a new look and feel, a vastly improved search index, and a completely rewritten set of documentation, as well as hundreds of bug fixes, dozens of new features, and a bright future as we continue to develop the application. Some links for more information:

* AtoM 2 homepage: https://accesstomemory.org/
* AtoM 2.0.0 release announcement (explains some of the differences further): https://groups.google.com/d/msg/ica-atom-users/_zgOnNxM1mE/ODGTv_Bxox4J
* AtoM 2.1 demo site: http://demo.accesstomemory.org/
* AtoM 2.1 documentation: https://accesstomemory.org/docs/2.1/

These security releases were first announced 2 weeks ago, to give system administrators time to schedule an upgrade window. See the original advance announcement here:
https://groups.google.com/d/msg/ica-atom-users/JH8c8eBl2P0/c6MSBObn0y0J

Note that if you are a hosting client with Artefactual systems, your site has already been upgraded to the security release.

AtoM 2.1.1 also includes several minor bug fixes including:

* #7352 - Search text box in AtoM clears text when clicking in / out of the text box
* #7649 - Add --index option to Accessions CSV import CLI task
* #7417 - Hide Archival history in public Advanced search if field is hidden via Visible elements
* #7418 - Fix multiple deletion problem for Alternative identifiers in edit form
* #7419 - Fix public search of Alternative identifier fields
* #7282 - Fix URL export links in EAD when roundtripping
* #7328 - Minor EAD mapping improvements

For a full list of issue tickets related to the 2.1.1 release, please see:
https://projects.artefactual.com/versions/86

TRANSLATORS - Please note, there are NO new translation files included in the 2.1.1 release. Instead, we will be including them in an upcoming translation release, 2.1.2, which we anticipate releasing on February 9, 2015. For more information, please see this User forum post:

https://groups.google.com/d/msg/ica-atom-users/vlnl21EQvOo/ApTpgWg1XSsJ

Meanwhile, we are hard at work on our 2.2 release, which will include a number of exciting new features. We expect the 2.2 release to be available later in Q1, or early Q2, of 2015.

Regards,

Dan Gillean

__NOTOC__

[[Category:Releases]]

Releases/Release announcements/Release 2.1.1

2015-07-01T18:32:24Z

Sromkey: Sromkey moved page Releases/Release announcements/Release 2.1.1 to Releases/Release announcements/Release 2.1.1, 2.0.2, 1.3.2: To prevent making 3 security release pages

#REDIRECT [[Releases/Release announcements/Release 2.1.1, 2.0.2, 1.3.2]]

Releases/Release announcements/Release 2.1.1, 2.0.2, 1.3.2

2015-07-01T18:12:33Z

Sromkey: Created page with "{{#pagetitle: Release 2.1.1, 2.0.1, 1.3.2 }} Main Page > Releases > Releases/Release announcements > Releases/Release announcements/Release 2.1.1, 2.0.1, 1.3.2 '..."

Releases/Release announcements/Release 2.1.2

2015-07-01T17:57:50Z

Sromkey:

{{#pagetitle: Release 2.1.2 }}
[[Main Page]] > [[Releases]] > [[Releases/Release announcements]] > Releases/Release announcements/Release 2.1.2

'''Released February 10, 2015'''

<admonition type="note"> This release announcement was originally posted in the AtoM User Forum, and has been copied here. The original post can be seen at: https://groups.google.com/d/msg/ica-atom-users/Fti1zBJM6pE/YffE4RarPREJ
</admonition>

Greetings AtoM users!

We are pleased to announce the availability of a new 2.1.2 release. You can find it on our Downloads page:

https://accesstomemory.org/download/

This is a minor release that includes a few bug fixes, and updates to the translation files supplied by our volunteer translation community via Transifex. As of this release, here are the current translations available in AtoM:

[[File:Atom-2-1translations-2015-02-09.png]]

Thank you to all of our volunteer translators for helping to make AtoM a truly international application!

At this time, we have locked contributions to the 2.1 translation files in Transifex. We will be porting these files over to the 2.2 project soon, at which point we encourage our translators to help us translate any new strings introduced in AtoM 2.2, or any older strings still awaiting tranlslation. I will make a post in the User Forum letting you know when we've updated the Transifex 2.2 project.

Release 2.1.2 also includes a few minor bug fixes, such as:

* #7818 - An apostrophe in the authorized form of name will break edit page javascript for an authority record
* #7850 - Make sfAPCCache work with php-apcu

For a full list of issue tickets related to the 2.1.2 release, see [https://projects.artefactual.com/projects/atom/issues?fixed_version_id=95&set_filter=1&status_id=* here].

Regards,

Dan Gillean

__NOTOC__

[[Category:Releases]]

File:Atom-2-1translations-2015-02-09.png

2015-07-01T17:51:36Z

Sromkey: AtoM translation progress as of 2.1.2 release

AtoM translation progress as of 2.1.2 release

Releases/Release announcements/Release 2.1.2

2015-07-01T17:46:34Z

Sromkey:

Releases/Release announcements/Release 2.1.2

2015-07-01T17:39:57Z

Sromkey: Created page with ".. note:: note here Greetings AtoM users! We are pleased to announce the availability of a new 2.1.2 release. You can find it on our Downloads page: https://accesstomem..."

.. note::

note here

Greetings AtoM users!

We are pleased to announce the availability of a new 2.1.2 release. You can find it on our Downloads page:

https://accesstomemory.org/download/

This is a minor release that includes a few bug fixes, and updates to the translation files supplied by our volunteer translation community via Transifex. As of this release, here are the current translations available in AtoM:

Community/Users

2014-08-15T21:35:30Z

Sromkey: /* AtoM (2.x) sites */

Resources/Metadata crosswalk

2014-06-23T21:08:02Z

Sromkey: RAD notes

[[Main Page]] > [[Resources]] > [[Resources/Metadata crosswalk]]

AtoM has templates for describing archives using ISAD(G), RAD, MODS, Dublin Core and DACS metadata standards. The table below provides a crosswalk between the standards, using ISAD(G) as its basis. It also displays the EAD export for the fields.

'''3.1 Identity Statement Area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.1 Reference code
|1.8B11. Alpha-numeric designations
|2.1.3 Local identifier
|identifier
|<identifier>
|<unitid>
|-
|3.1.2 Title
|1.1B. Title proper
|2.3 Title
|title
|<titleInfo><title>
|<unittitle>
|-
|3.1.3 Date(s)
|1.4B. Date(s) of creation
|2.4 Date
|coverage (temporal)
date (created)
|<originInfo><dateCreated>
|<unitdate>
|-
|3.1.4 Level of description
|
|
|
|
|<archdesc level="___">
|-
|3.1.5 Extent and medium
|1.5B. Extent of descriptive unit (including specific material designation)
|2.5 Extent
|format (extent)
format (medium)
|<physicalDescription><extent>
<genre>
|<physdesc><extent>
|}

'''3.2 Context area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.2.1 Name of creator(s)
|21.1 Provenance access points
|2.6 Name of creator(s)
|<name type="___"><namePart>
<name><role><roleTerm type="text">creator
|-
|3.2.2 Administrative / biographical history
|1.7B Administrative history / biographical sketch
|2.7 Administrative / biographical history
|-
|3.2.3 Archival history
|1.7C Custodial history
|5.1 Custodial history
|-
|3.2.4 Immediate source of acquisition or transfer
|1.8B12 Immediate source of acquisition
|5.2 Immediate source of acquisition
|

|}

'''3.3 Content and structure area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.3.1 Scope and content
|1.7D Scope and content
|-
|3.3.2 Appraisal, destruction and scheduling information
|
|-
|3.3.3 Accruals
|1.8B19 Accruals
|-
|3.3.4 System of arrangement
|1.8B13 Arrangement

|}

'''3.4 Conditions of access and use area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.4.1 Conditions governing access
|1.8B16a Restrictions on access
|-
|3.4.2 Conditions governing reproduction
|1.8B16c Terms governing use and reproduction
1.8B16d Terms governing publication
|-
|3.4.3 Language / scripts of material
|1.8B14 Language
|-
|3.4.4 Physical characteristics and technical requirements
|1.8B9a Physical condition
|-
|3.4.5 Finding aids
|1.8B17 Finding aids

|}

'''3.5 Allied materials area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.5.1 Existence and location of originals
|1.8B15a Location of originals
|-
|3.5.2 Existence and location of copies
|1.8B15b Availability of other formats
|-
|3.5.3 Related units of description
|1.8B18 Associated material
1.8B20 Related groups of records external to the unit being described
1.8B20a Related groups of records within the same fonds, series or collection
|-
|3.5.4 Publication note
|

|}

'''3.6 Notes area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.6.1 Notes
|-

|}

'''3.7 Description control area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.7.1 Archivist's note
|-
|3.7.2 Rules or conventions
|-
|3.7.3 Date(s) of description
|-

|}

[[Category:Resources]]

Resources/Metadata crosswalk

2014-06-23T20:55:12Z

Sromkey:

[[Main Page]] > [[Resources]] > [[Resources/Metadata crosswalk]]

AtoM has templates for describing archives using ISAD(G), RAD, MODS, Dublin Core and DACS metadata standards. The table below provides a crosswalk between the standards, using ISAD(G) as its basis. It also displays the EAD export for the fields.

'''3.1 Identity Statement Area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.1 Reference code
|1.8B11. Alpha-numeric designations
|2.1.3 Local identifier
|identifier
|<identifier>
|<unitid>
|-
|3.1.2 Title
|1.1B. Title proper
|2.3 Title
|title
|<titleInfo><title>
|<unittitle>
|-
|3.1.3 Date(s)
|1.4B. Date(s) of creation
|2.4 Date
|coverage (temporal)
date (created)
|<originInfo><dateCreated>
|<unitdate>
|-
|3.1.4 Level of description
|
|
|
|
|<archdesc level="___">
|-
|3.1.5 Extent and medium
|1.5B. Extent of descriptive unit (including specific material designation)
|2.5 Extent
|format (extent)
format (medium)
|<physicalDescription><extent>
<genre>
|<physdesc><extent>
|}

'''3.2 Context area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.2.1 Name of creator(s)
|21.1 Provenance access points
|2.6 Name of creator(s)
|<name type="___"><namePart>
<name><role><roleTerm type="text">creator
|-
|3.2.2 Administrative / biographical history
|1.7B Administrative history / biographical sketch
|2.7 Administrative / biographical history
|-
|3.2.3 Archival history
|1.7C Custodial history
|5.1 Custodial history
|-
|3.2.4 Immediate source of acquisition or transfer
|1.8B12 Immediate source of acquisition
|5.2 Immediate source of acquisition
|

|}

'''3.3 Content and structure area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.3.1 Scope and content
|-
|3.3.2 Appraisal, destruction and scheduling information
|-
|3.3.3 Accruals
|-
|3.3.4 System of arrangement

|}

'''3.4 Conditions of access and use area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.4.1 Conditions governing access
|-
|3.4.2 Conditions governing reproduction
|-
|3.4.3 Language / scripts of material
|-
|3.4.4 Physical characteristics and technical requirements
|-
|3.4.5 Finding aids

|}

'''3.5 Allied materials area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.5.1 Existence and location of originals
|-
|3.5.2 Existence and location of copies
|-
|3.5.3 Related units of description
|-
|3.5.4 Publication note

|}

'''3.6 Notes area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.6.1 Notes
|-

|}

'''3.7 Description control area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.7.1 Archivist's note
|-
|3.7.2 Rules or conventions
|-
|3.7.3 Date(s) of description
|-

|}

[[Category:Resources]]

Resources/Metadata crosswalk

2014-06-23T20:53:48Z

Sromkey: DACS- context area

[[Main Page]] > [[Resources]] > [[Resources/Metadata crosswalk]]

AtoM has templates for describing archives using ISAD(G), RAD, MODS, Dublin Core and DACS metadata standards. The table below provides a crosswalk between the standards, using ISAD(G) as its basis. It also displays the EAD export for the fields.

'''3.1 Identity Statement Area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.1 Reference code
|1.8B11. Alpha-numeric designations
|2.1.3 Local identifier
|identifier
|<identifier>
|<unitid>
|-
|3.1.2 Title
|1.1B. Title proper
|2.3 Title
|title
|<titleInfo><title>
|<unittitle>
|-
|3.1.3 Date(s)
|1.4B. Date(s) of creation
|2.4 Date
|coverage (temporal)
date (created)
|<originInfo><dateCreated>
|<unitdate>
|-
|3.1.4 Level of description
|
|
|
|
|<archdesc level="___">
|-
|3.1.5 Extent and medium
|1.5B. Extent of descriptive unit (including specific material designation)
|2.5 Extent
|format (extent)
format (medium)
|<physicalDescription><extent>
<genre>
|<physdesc><extent>
|}

'''3.2 Context area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DACS
! scope="col"|DC
! scope="col"|MODS
! scope="col"|EAD
|-
|3.2.1 Name of creator(s)
|21.1 Provenance access points
|2.6 Name of creator(s)
|<name type="___"><namePart>
<name><role><roleTerm type="text">creator
|-
|3.2.2 Administrative / biographical history
|1.7B Administrative history / biographical sketch
|2.7 Administrative / biographical history
|-
|3.2.3 Archival history
|1.7C Custodial history
|5.1 Custodial history
|-
|3.2.4 Immediate source of acquisition or transfer
|1.8B12 Immediate source of acquisition
|5.2 Immediate source of acquisition
|

|}

'''3.3 Content and structure area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DC
! scope="col"|MODS
! scope="col"|DACS
! scope="col"|EAD
|-
|3.3.1 Scope and content
|-
|3.3.2 Appraisal, destruction and scheduling information
|-
|3.3.3 Accruals
|-
|3.3.4 System of arrangement

|}

'''3.4 Conditions of access and use area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DC
! scope="col"|MODS
! scope="col"|DACS
! scope="col"|EAD
|-
|3.4.1 Conditions governing access
|-
|3.4.2 Conditions governing reproduction
|-
|3.4.3 Language / scripts of material
|-
|3.4.4 Physical characteristics and technical requirements
|-
|3.4.5 Finding aids

|}

'''3.5 Allied materials area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DC
! scope="col"|MODS
! scope="col"|DACS
! scope="col"|EAD
|-
|3.5.1 Existence and location of originals
|-
|3.5.2 Existence and location of copies
|-
|3.5.3 Related units of description
|-
|3.5.4 Publication note

|}

'''3.6 Notes area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DC
! scope="col"|MODS
! scope="col"|DACS
! scope="col"|EAD
|-
|3.6.1 Notes
|-

|}

'''3.7 Description control area'''
{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10"
|-
! scope="col"|ISAD(G)
! scope="col"|RAD
! scope="col"|DC
! scope="col"|MODS
! scope="col"|DACS
! scope="col"|EAD
|-
|3.7.1 Archivist's note
|-
|3.7.2 Rules or conventions
|-
|3.7.3 Date(s) of description
|-

|}

[[Category:Resources]]

Resources/Metadata crosswalk

2014-06-23T19:02:19Z

Sromkey:

Resources/Metadata crosswalk

2014-06-23T17:27:54Z

Sromkey: Context area

Resources/Metadata crosswalk

2014-06-23T16:57:18Z

Sromkey:

Resources/Metadata crosswalk

2014-06-23T16:56:56Z

Sromkey: EAD additions

Resources/Metadata crosswalk

2014-06-23T15:54:02Z

Sromkey:

Resources/Metadata crosswalk

2014-06-23T15:52:59Z

Sromkey: DACS info added

Main Page

2014-06-20T23:53:38Z

Sromkey: /* Resources */

= Welcome to the Access to Memory wiki =

== TO DO ==
* [[Unification]]
* [//www.mediawiki.org/wiki/Localisation#Translation_resources Localise MediaWiki for your language]

== Community ==

* Add Category: community to each page
* Add to URL /Community/...
* '''Have not yet made:''' landing page for "Community" itself

=== Pages already created ===
* [[Community/Users]]
* [[Community/Success stories]]
* [[Community/Community resources]]

== Development documentation ==

* Add to Category: Development documentation
* Add to URL /Development/...
* '''Have not yet made:''' landing page for "Development" itself

=== Pages already created ===
* [[Development/Multilingual]]
* [[Development/Treeview]]
* [[Development/Accession module]]

== Releases ==

* Add all pages to Category: Releases
* Add to URL /Releases/...
* Landing page for [[Releases]] created

===Pages already created ===

* [[Releases]]
* [[Releases/Release announcements]]
* [[Release 1.4]]
* [[Release 1.1]]
* [[Release 1.0-beta]]
* [[Release 1.0.1-beta]]
* [[Release 1.0.2-beta]]
* [[Release 1.0.3-beta]]
* [[Release 1.0.4-beta]]
* [[Release 1.0.5-beta]]
* [[Release 1.0.6-beta]]
* [[Release 1.0.7-beta]]
* [[Release 1.0.8-beta]]
* [[Release 1.0.9-beta]]

== Resources ==

* Add all pages to Category: Resources
* Add to URL /Resources/...
* Landing page for [[Resources]] created

===Pages already created===

* [[Resources]]
* [[Resources/Issue tracker]]
* [[Resources/Presentations]]
* [[Resources/Metadata crosswalk]]

Resources

2014-06-20T23:52:53Z

Sromkey:

This area of the wiki contains resources that may be helpful to the AtoM user community, or those considering AtoM for use at their institution.

'''Papers and Presentations'''

Papers and conference presentations given about AtoM and ICA-AtoM

* [[Resources/Presentations|Papers and Presentations]]

'''Bug/Issue tracking'''

Instructions on how to report bugs or issues while using AtoM.

* [[Resources/Issue tracker|Issue tracker]]

'''Metadata Crosswalk'''

Crosswalk between the metadata standards supported by AtoM.

* [[Resources/Metadata crosswalk|Metadata crosswalk]]

'''Community resources'''

Tutorials, style guides, and helpful examples created by the AtoM community.

* [[Community/Community resources|Community resources]]

'''User forum'''

Still look for help and ideas? Check out our user forum: https://groups.google.com/forum/#!forum/ica-atom-users

[[Category:Resources]]