AtoM wiki - User contributions [en]

User:Kgoetz

2019-12-05T05:51:28Z

Kgoetz: change page text

Staff member of the University of Tasmania; previously employed managing various library systems and now in eResearch.

Sporadically active on Atom and Archivematica user group forums.

To contact me email karl.goetz then add @utas.edu.au.

Development/Archivematica integration

2019-12-05T05:49:04Z

Kgoetz: add more 'see also' links from recent group discussions

{{#pagetitle:Archivematica integration}}
[[Development]] > Development/Archivematica_integration

Access to Memory (AtoM) and [https://www.archivematica.org Archivematica] have a strong connection - they’re developed by the same vendor and Archivematica uses AtoM as its display frontend. Despite that communication is not currently bidirectional and when an AtoM site has data loaded there is currently no mechanism to synchronise the two platforms.

This page attempts to summarise any discoveries from [https://groups.google.com/forum/#!topic/ica-atom-users/TEkIGSUTEvo/overview the discussion started on AtoM-users] and will, in time, start to form a description of what integration might look like.

There is also a similar project for [https://github.com/eprintsug/EPrintsArchivematica integrating EPrints with Archivematica] who’s standards document includes useful information including layout and other legwork required for an export like this.

== Integration should cover ==
* An AtoM instance that contains records to which Archivematica is added as a backend
* An AtoM instance which is installed parallel to Archivematica but which is uploaded to directly

This will mean a variety of factors need to be considered (and questions asked) including
* Data uploaded to AtoM should be exportable to Archivematica for ingest and when the processed data is returned to AtoM no duplicate entries should be recorded.
* Optional support for Metadata only updates
* Should thumbnails generated by AtoM be exported and preserved?
* Does AtoM store revision history? Should we Archive revisions if it does?
* How will metadata only updates be handled
* What to do with external digital objects. ensure they are skipped with warning by export process? Export the thumbnails/metadata? Detection code is available [https://github.com/artefactual/atom/blob/stable/2.4.x/lib/model/QubitDigitalObject.php#L1569 in the repository]

== Options for integration ==

=== Option one: Data exported to directory ===

Export required data+metadata into a single directory which Archivematica imports ; AtoM then imports the export package (DIP) created by Archivematica

==== Export stage ====
Data which we can export (if Archivematica [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#the-transfer-tab transfer] can use it) which would help populate fields in Archivematica and can then be used to re-match with AtoM after processing.

* Transfer type (File type of item)
* Transfer name (current items name or description)
* Accession number (Accession number of item being exported)
* Access system ID (AtoM install performing export)
* Approve automatically (make this optional ; some may want it and some not)

[https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#preparing-digital-objects-for-transfer Export structure is important], as with [https://github.com/eprintsug/EPrintsArchivematica EPrints with Archivematica integration] I suggest this project supplies various bits of data (metadata/ , checksums, documentation, etc) as described on [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#preparing-digital-objects-for-transfer Archivematicas transfer page].

Working draft of filesystem layout
* object_slug-unique_identifer directory (object slug is the slug stored in DB, unique identifier is however AtoM identifies the object internally)
** Metadata directory, files in here are formatted per [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/import-metadata/#import-metadata Archivematica metadata import documentation] and should be populated with data from AtoM
*** metadata.csv (mandatory)
*** rights.csv (mandatory if rights are specified)
*** checksum.md5, checksum.sha1, or checksum.sha256 TBC: check which format/s AtoM uses and restrict output to that/those format/s
*** All other available metadata: optional; if supplied by AtoM and configured to export
** Objects directory, where actual files will go
*** Structure TODO

AtoM uses [https://github.com/artefactual/atom/blob/stable/2.4.x/lib/model/QubitDigitalObject.php#L1796 getAssetPath] to access its objects so we can lean on that to find files we want

==== Reimport ====
Import is currently managed by AtoM already, but will need to be augmented to deal with new issues.

* AtoM will have the original format files and the normalised format following the conversion. A way to remove the original files will need to be included
* AtoM will need to ensure it doesn’t enter a loop of: export files -> import from AM -> update files with those from AM -> trigger export of newly imported files.
* Duplicates in AtoM. Archivematica's DIP upload creates new stub information objects when it deposits the DIP objects - it won't replace existing objects. How manageable this problem is will depend on how much identifying metadata we can feed through Archivematica

Things which might make this easier:
* Using the functions for managing [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#transferring-material-with-preservation-or-access-derivatives-manual-normalization Transfer and Access derivatives] or [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#transferring-material-with-service-mezzanine-files material with service files]

Atom may be extended to use the MDTYPE=OTHER metadata available in the DIP METS file, this would help with the roundtrip and (if it works) might solve all the identification problems.

Atom includes a tool for importing data from a DIP via CSV file. this tool or its backing library might be usable in this context. https://github.com/artefactual/atom/blob/qa/2.5.x/lib/task/import/importDipObjectsTask.class.php

=== Option two: Data exported to SIP/bag ===

Export packages (SIPs, bagit, or similar) which can be imported to Archivematica; AtoM then imports the export package (DIP) created by Archivematica.

This is the same as option one, but includes a packaging step at the end.

=== Option three: Bidirectional communication ===

Atom and Archivematica currently use SWORD1 (a unidirectional protocol) to push data from Archivematica to AtoM. Under this option they would use an existing protocol with bi directional support (eg SWORD2 or the work in progress SWORD3, see http://swordapp.org/).

* Both applications need changes in this instance
** Archivematica doesn't support plugins so its not as easy as a plugin at each end to add support.
* TBD: what would they communicate and how - would a SIP/bag be sent over SWORD or would individual files and associated data be sent and Archivematica packages it up?

== See also ==

* Archivematica's [https://www.archivematica.org/en/docs/archivematica-1.8/getting-started/overview/technical/#format-policies Format Policy] Registry (currently in the process of becoming its own project to be used by multiple digital preservation systems; now known as the Preservation Action Registry) to manage derivatives creation in a more intelligent way.

* TODO describe https://mediaarea.net/Events/2018-10-26_NoTimeToWait3/presentations/07.%20Martin%20Wrigley%20-%20Preservation%20Action%20Registry/OPF%20PAR%20Presentation%20notime%20to%20wait%20event.pdf

* [https://groups.google.com/forum/#!topic/archivematica/Ipvft2zC8ZA Question regarding retroactive archivematica uploads] which links to [https://www.accesstomemory.org/en/docs/2.3/user-manual/add-edit-content/archival-descriptions/#add-alternative-identifiers-to-an-archival-description AtoM documentation on how to Add alternative identifiers to an archival description].

* [https://groups.google.com/forum/#!msg/ica-atom-users/tePqjGpqLdQ/Xe4Lf4W8AAAJ Discussion on Exporting data and metadata from AtoM to archivematica] on the Atom users list.

* [https://groups.google.com/forum/#!msg/archivematica/E-ESb5TiiI4/wwM5Hqj0AwAJ Query around sending DIP to Atom via API] from Archivematica user group

-----

* [[Development|Back to Development]]
* [[Main Page|AtoM wiki home]]

[[Category:Development documentation]]

Development/Archivematica integration

2019-01-10T00:41:08Z

Kgoetz: add links to another stop gap approach for existing data

{{#pagetitle:Archivematica integration}}
[[Development]] > Development/Archivematica_integration

Access to Memory (AtoM) and [https://www.archivematica.org Archivematica] have a strong connection - they’re developed by the same vendor and Archivematica uses AtoM as its display frontend. Despite that communication is not currently bidirectional and when an AtoM site has data loaded there is currently no mechanism to synchronise the two platforms.

This page attempts to summarise any discoveries from [https://groups.google.com/forum/#!topic/ica-atom-users/TEkIGSUTEvo/overview the discussion started on AtoM-users] and will, in time, start to form a description of what integration might look like.

There is also a similar project for [https://github.com/eprintsug/EPrintsArchivematica integrating EPrints with Archivematica] who’s standards document includes useful information including layout and other legwork required for an export like this.

== Integration should cover ==
* An AtoM instance that contains records to which Archivematica is added as a backend
* An AtoM instance which is installed parallel to Archivematica but which is uploaded to directly

This will mean a variety of factors need to be considered (and questions asked) including
* Data uploaded to AtoM should be exportable to Archivematica for ingest and when the processed data is returned to AtoM no duplicate entries should be recorded.
* Optional support for Metadata only updates
* Should thumbnails generated by AtoM be exported and preserved?
* Does AtoM store revision history? Should we Archive revisions if it does?
* How will metadata only updates be handled
* What to do with external digital objects. ensure they are skipped with warning by export process? Export the thumbnails/metadata? Detection code is available [https://github.com/artefactual/atom/blob/stable/2.4.x/lib/model/QubitDigitalObject.php#L1569 in the repository]

== Options for integration ==

=== Option one: Data exported to directory ===

Export required data+metadata into a single directory which Archivematica imports ; AtoM then imports the export package (DIP) created by Archivematica

==== Export stage ====
Data which we can export (if Archivematica [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#the-transfer-tab transfer] can use it) which would help populate fields in Archivematica and can then be used to re-match with AtoM after processing.

* Transfer type (File type of item)
* Transfer name (current items name or description)
* Accession number (Accession number of item being exported)
* Access system ID (AtoM install performing export)
* Approve automatically (make this optional ; some may want it and some not)

[https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#preparing-digital-objects-for-transfer Export structure is important], as with [https://github.com/eprintsug/EPrintsArchivematica EPrints with Archivematica integration] I suggest this project supplies various bits of data (metadata/ , checksums, documentation, etc) as described on [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#preparing-digital-objects-for-transfer Archivematicas transfer page].

Working draft of filesystem layout
* object_slug-unique_identifer directory (object slug is the slug stored in DB, unique identifier is however AtoM identifies the object internally)
** Metadata directory, files in here are formatted per [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/import-metadata/#import-metadata Archivematica metadata import documentation] and should be populated with data from AtoM
*** metadata.csv (mandatory)
*** rights.csv (mandatory if rights are specified)
*** checksum.md5, checksum.sha1, or checksum.sha256 TBC: check which format/s AtoM uses and restrict output to that/those format/s
*** All other available metadata: optional; if supplied by AtoM and configured to export
** Objects directory, where actual files will go
*** Structure TODO

AtoM uses [https://github.com/artefactual/atom/blob/stable/2.4.x/lib/model/QubitDigitalObject.php#L1796 getAssetPath] to access its objects so we can lean on that to find files we want

==== Reimport ====
Import is currently managed by AtoM already, but will need to be augmented to deal with new issues.

* AtoM will have the original format files and the normalised format following the conversion. A way to remove the original files will need to be included
* AtoM will need to ensure it doesn’t enter a loop of: export files -> import from AM -> update files with those from AM -> trigger export of newly imported files.
* Duplicates in AtoM. Archivematica's DIP upload creates new stub information objects when it deposits the DIP objects - it won't replace existing objects. How manageable this problem is will depend on how much identifying metadata we can feed through Archivematica

Things which might make this easier:
* Using the functions for managing [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#transferring-material-with-preservation-or-access-derivatives-manual-normalization Transfer and Access derivatives] or [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#transferring-material-with-service-mezzanine-files material with service files]

Atom may be extended to use the MDTYPE=OTHER metadata available in the DIP METS file, this would help with the roundtrip and (if it works) might solve all the identification problems.

Atom includes a tool for importing data from a DIP via CSV file. this tool or its backing library might be usable in this context. https://github.com/artefactual/atom/blob/qa/2.5.x/lib/task/import/importDipObjectsTask.class.php

=== Option two: Data exported to SIP/bag ===

Export packages (SIPs, bagit, or similar) which can be imported to Archivematica; AtoM then imports the export package (DIP) created by Archivematica.

This is the same as option one, but includes a packaging step at the end.

=== Option three: Bidirectional communication ===

Atom and Archivematica currently use SWORD1 (a unidirectional protocol) to push data from Archivematica to AtoM. Under this option they would use an existing protocol with bi directional support (eg SWORD2 or the work in progress SWORD3, see http://swordapp.org/).

* Both applications need changes in this instance
** Archivematica doesn't support plugins so its not as easy as a plugin at each end to add support.
* TBD: what would they communicate and how - would a SIP/bag be sent over SWORD or would individual files and associated data be sent and Archivematica packages it up?

== See also ==

* Archivematica's [https://www.archivematica.org/en/docs/archivematica-1.8/getting-started/overview/technical/#format-policies Format Policy] Registry (currently in the process of becoming its own project to be used by multiple digital preservation systems; now known as the Preservation Action Registry) to manage derivatives creation in a more intelligent way.

* TODO describe https://mediaarea.net/Events/2018-10-26_NoTimeToWait3/presentations/07.%20Martin%20Wrigley%20-%20Preservation%20Action%20Registry/OPF%20PAR%20Presentation%20notime%20to%20wait%20event.pdf

* [https://groups.google.com/forum/#!topic/archivematica/Ipvft2zC8ZA Question regarding retroactive archivematica uploads] which links to [https://www.accesstomemory.org/en/docs/2.3/user-manual/add-edit-content/archival-descriptions/#add-alternative-identifiers-to-an-archival-description AtoM documentation on how to Add alternative identifiers to an archival description].

-----

* [[Development|Back to Development]]
* [[Main Page|AtoM wiki home]]

[[Category:Development documentation]]

Development/Archivematica integration

2018-12-12T23:09:36Z

Kgoetz: link to atom built in tool for importing objects in a DIP

{{#pagetitle:Archivematica integration}}
[[Development]] > Development/Archivematica_integration

Access to Memory (AtoM) and [https://www.archivematica.org Archivematica] have a strong connection - they’re developed by the same vendor and Archivematica uses AtoM as its display frontend. Despite that communication is not currently bidirectional and when an AtoM site has data loaded there is currently no mechanism to synchronise the two platforms.

This page attempts to summarise any discoveries from [https://groups.google.com/forum/#!topic/ica-atom-users/TEkIGSUTEvo/overview the discussion started on AtoM-users] and will, in time, start to form a description of what integration might look like.

There is also a similar project for [https://github.com/eprintsug/EPrintsArchivematica integrating EPrints with Archivematica] who’s standards document includes useful information including layout and other legwork required for an export like this.

== Integration should cover ==
* An AtoM instance that contains records to which Archivematica is added as a backend
* An AtoM instance which is installed parallel to Archivematica but which is uploaded to directly

This will mean a variety of factors need to be considered (and questions asked) including
* Data uploaded to AtoM should be exportable to Archivematica for ingest and when the processed data is returned to AtoM no duplicate entries should be recorded.
* Optional support for Metadata only updates
* Should thumbnails generated by AtoM be exported and preserved?
* Does AtoM store revision history? Should we Archive revisions if it does?
* How will metadata only updates be handled
* What to do with external digital objects. ensure they are skipped with warning by export process? Export the thumbnails/metadata? Detection code is available [https://github.com/artefactual/atom/blob/stable/2.4.x/lib/model/QubitDigitalObject.php#L1569 in the repository]

== Options for integration ==

=== Option one: Data exported to directory ===

Export required data+metadata into a single directory which Archivematica imports ; AtoM then imports the export package (DIP) created by Archivematica

==== Export stage ====
Data which we can export (if Archivematica [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#the-transfer-tab transfer] can use it) which would help populate fields in Archivematica and can then be used to re-match with AtoM after processing.

* Transfer type (File type of item)
* Transfer name (current items name or description)
* Accession number (Accession number of item being exported)
* Access system ID (AtoM install performing export)
* Approve automatically (make this optional ; some may want it and some not)

[https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#preparing-digital-objects-for-transfer Export structure is important], as with [https://github.com/eprintsug/EPrintsArchivematica EPrints with Archivematica integration] I suggest this project supplies various bits of data (metadata/ , checksums, documentation, etc) as described on [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#preparing-digital-objects-for-transfer Archivematicas transfer page].

Working draft of filesystem layout
* object_slug-unique_identifer directory (object slug is the slug stored in DB, unique identifier is however AtoM identifies the object internally)
** Metadata directory, files in here are formatted per [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/import-metadata/#import-metadata Archivematica metadata import documentation] and should be populated with data from AtoM
*** metadata.csv (mandatory)
*** rights.csv (mandatory if rights are specified)
*** checksum.md5, checksum.sha1, or checksum.sha256 TBC: check which format/s AtoM uses and restrict output to that/those format/s
*** All other available metadata: optional; if supplied by AtoM and configured to export
** Objects directory, where actual files will go
*** Structure TODO

AtoM uses [https://github.com/artefactual/atom/blob/stable/2.4.x/lib/model/QubitDigitalObject.php#L1796 getAssetPath] to access its objects so we can lean on that to find files we want

==== Reimport ====
Import is currently managed by AtoM already, but will need to be augmented to deal with new issues.

* AtoM will have the original format files and the normalised format following the conversion. A way to remove the original files will need to be included
* AtoM will need to ensure it doesn’t enter a loop of: export files -> import from AM -> update files with those from AM -> trigger export of newly imported files.
* Duplicates in AtoM. Archivematica's DIP upload creates new stub information objects when it deposits the DIP objects - it won't replace existing objects. How manageable this problem is will depend on how much identifying metadata we can feed through Archivematica

Things which might make this easier:
* Using the functions for managing [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#transferring-material-with-preservation-or-access-derivatives-manual-normalization Transfer and Access derivatives] or [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#transferring-material-with-service-mezzanine-files material with service files]

Atom may be extended to use the MDTYPE=OTHER metadata available in the DIP METS file, this would help with the roundtrip and (if it works) might solve all the identification problems.

Atom includes a tool for importing data from a DIP via CSV file. this tool or its backing library might be usable in this context. https://github.com/artefactual/atom/blob/qa/2.5.x/lib/task/import/importDipObjectsTask.class.php

=== Option two: Data exported to SIP/bag ===

Export packages (SIPs, bagit, or similar) which can be imported to Archivematica; AtoM then imports the export package (DIP) created by Archivematica.

This is the same as option one, but includes a packaging step at the end.

=== Option three: Bidirectional communication ===

Atom and Archivematica currently use SWORD1 (a unidirectional protocol) to push data from Archivematica to AtoM. Under this option they would use an existing protocol with bi directional support (eg SWORD2 or the work in progress SWORD3, see http://swordapp.org/).

* Both applications need changes in this instance
** Archivematica doesn't support plugins so its not as easy as a plugin at each end to add support.
* TBD: what would they communicate and how - would a SIP/bag be sent over SWORD or would individual files and associated data be sent and Archivematica packages it up?

== See also ==

* Archivematica's [https://www.archivematica.org/en/docs/archivematica-1.8/getting-started/overview/technical/#format-policies Format Policy] Registry (currently in the process of becoming its own project to be used by multiple digital preservation systems; now known as the Preservation Action Registry) to manage derivatives creation in a more intelligent way.

* TODO describe https://mediaarea.net/Events/2018-10-26_NoTimeToWait3/presentations/07.%20Martin%20Wrigley%20-%20Preservation%20Action%20Registry/OPF%20PAR%20Presentation%20notime%20to%20wait%20event.pdf

-----

* [[Development|Back to Development]]
* [[Main Page|AtoM wiki home]]

[[Category:Development documentation]]

Development/Archivematica integration

2018-12-07T03:10:12Z

Kgoetz: mention new information

{{#pagetitle:Archivematica integration}}
[[Development]] > Development/Archivematica_integration

Access to Memory (AtoM) and [https://www.archivematica.org Archivematica] have a strong connection - they’re developed by the same vendor and Archivematica uses AtoM as its display frontend. Despite that communication is not currently bidirectional and when an AtoM site has data loaded there is currently no mechanism to synchronise the two platforms.

This page attempts to summarise any discoveries from [https://groups.google.com/forum/#!topic/ica-atom-users/TEkIGSUTEvo/overview the discussion started on AtoM-users] and will, in time, start to form a description of what integration might look like.

There is also a similar project for [https://github.com/eprintsug/EPrintsArchivematica integrating EPrints with Archivematica] who’s standards document includes useful information including layout and other legwork required for an export like this.

== Integration should cover ==
* An AtoM instance that contains records to which Archivematica is added as a backend
* An AtoM instance which is installed parallel to Archivematica but which is uploaded to directly

This will mean a variety of factors need to be considered (and questions asked) including
* Data uploaded to AtoM should be exportable to Archivematica for ingest and when the processed data is returned to AtoM no duplicate entries should be recorded.
* Optional support for Metadata only updates
* Should thumbnails generated by AtoM be exported and preserved?
* Does AtoM store revision history? Should we Archive revisions if it does?
* How will metadata only updates be handled
* What to do with external digital objects. ensure they are skipped with warning by export process? Export the thumbnails/metadata? Detection code is available [https://github.com/artefactual/atom/blob/stable/2.4.x/lib/model/QubitDigitalObject.php#L1569 in the repository]

== Options for integration ==

=== Option one: Data exported to directory ===

Export required data+metadata into a single directory which Archivematica imports ; AtoM then imports the export package (DIP) created by Archivematica

==== Export stage ====
Data which we can export (if Archivematica [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#the-transfer-tab transfer] can use it) which would help populate fields in Archivematica and can then be used to re-match with AtoM after processing.

* Transfer type (File type of item)
* Transfer name (current items name or description)
* Accession number (Accession number of item being exported)
* Access system ID (AtoM install performing export)
* Approve automatically (make this optional ; some may want it and some not)

[https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#preparing-digital-objects-for-transfer Export structure is important], as with [https://github.com/eprintsug/EPrintsArchivematica EPrints with Archivematica integration] I suggest this project supplies various bits of data (metadata/ , checksums, documentation, etc) as described on [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#preparing-digital-objects-for-transfer Archivematicas transfer page].

Working draft of filesystem layout
* object_slug-unique_identifer directory (object slug is the slug stored in DB, unique identifier is however AtoM identifies the object internally)
** Metadata directory, files in here are formatted per [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/import-metadata/#import-metadata Archivematica metadata import documentation] and should be populated with data from AtoM
*** metadata.csv (mandatory)
*** rights.csv (mandatory if rights are specified)
*** checksum.md5, checksum.sha1, or checksum.sha256 TBC: check which format/s AtoM uses and restrict output to that/those format/s
*** All other available metadata: optional; if supplied by AtoM and configured to export
** Objects directory, where actual files will go
*** Structure TODO

AtoM uses [https://github.com/artefactual/atom/blob/stable/2.4.x/lib/model/QubitDigitalObject.php#L1796 getAssetPath] to access its objects so we can lean on that to find files we want

==== Reimport ====
Import is currently managed by AtoM already, but will need to be augmented to deal with new issues.

* AtoM will have the original format files and the normalised format following the conversion. A way to remove the original files will need to be included
* AtoM will need to ensure it doesn’t enter a loop of: export files -> import from AM -> update files with those from AM -> trigger export of newly imported files.
* Duplicates in AtoM. Archivematica's DIP upload creates new stub information objects when it deposits the DIP objects - it won't replace existing objects. How manageable this problem is will depend on how much identifying metadata we can feed through Archivematica

Things which might make this easier:
* Using the functions for managing [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#transferring-material-with-preservation-or-access-derivatives-manual-normalization Transfer and Access derivatives] or [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#transferring-material-with-service-mezzanine-files material with service files]

Atom may be extended to use the MDTYPE=OTHER metadata available in the DIP METS file, this would help with the roundtrip and (if it works) might solve all the identification problems.

=== Option two: Data exported to SIP/bag ===

Export packages (SIPs, bagit, or similar) which can be imported to Archivematica; AtoM then imports the export package (DIP) created by Archivematica.

This is the same as option one, but includes a packaging step at the end.

=== Option three: Bidirectional communication ===

Atom and Archivematica currently use SWORD1 (a unidirectional protocol) to push data from Archivematica to AtoM. Under this option they would use an existing protocol with bi directional support (eg SWORD2 or the work in progress SWORD3, see http://swordapp.org/).

* Both applications need changes in this instance
** Archivematica doesn't support plugins so its not as easy as a plugin at each end to add support.
* TBD: what would they communicate and how - would a SIP/bag be sent over SWORD or would individual files and associated data be sent and Archivematica packages it up?

== See also ==

* Archivematica's [https://www.archivematica.org/en/docs/archivematica-1.8/getting-started/overview/technical/#format-policies Format Policy] Registry (currently in the process of becoming its own project to be used by multiple digital preservation systems; now known as the Preservation Action Registry) to manage derivatives creation in a more intelligent way.

* TODO describe https://mediaarea.net/Events/2018-10-26_NoTimeToWait3/presentations/07.%20Martin%20Wrigley%20-%20Preservation%20Action%20Registry/OPF%20PAR%20Presentation%20notime%20to%20wait%20event.pdf

-----

* [[Development|Back to Development]]
* [[Main Page|AtoM wiki home]]

[[Category:Development documentation]]

Community/Community resources/Development

2018-11-29T05:38:19Z

Kgoetz: Shibboleth plugins git log indicates it has been updated for 2.3

{{#pagetitle:Community development}}
[[Main Page]] > [[Community]] > [[Community/Community resources]] > Community/Community resources/Development

In this section of the wiki, we'll add links to custom patches, plugins, themes, images, forks, and other resources developed by community users and publicly available for developers to explore and work with.

<admonition type="warning">
'''PLEASE NOTE''': Artefactual does '''not''' test these features and modules developed by AtoM community members. As such, we cannot offer support for them, nor can we speak as to their security, quality, performance, or compatibility with the latest public releases. If you intend to make use of these features, do so at your own risk. We strongly recommend studying the code prior to use, and deploying any community-developed features in a test environment where they can be properly evaluated.
</admonition>

<admonition type="tip">
Have you developed custom code for your AtoM installation? Let us know! And... why not consider contributing your code to the public project? That way, we maintain the code for you through future releases, and the entire community benefits from your work! Here are some links to get you started:
* [[Development/Contribute code]]
* [[Development#Development_resources|Other developer resources in our wiki]]
</admonition>

__TOC__

==CENDARI Shibboleth authentication plugin==

CENDARI (Collaborative European Digital Archive Infrastructure) is a research collaboration aimed at integrating digital archives and resources for research on medieval and modern European history. They have chosen AtoM for use as their collaborative Archival Directory (site [https://archives.cendari.dariah.eu/ here], more information [http://www.cendari.eu/archival-directories/ here]), and have done some custom development, including a plugin to integrate with [http://shibboleth.net/ Shibboleth], an "''open-source project that provides Single Sign-On capabilities and allows sites to make informed authorization decisions for individual access of protected online resources in a privacy-preserving manner.''"

The plugin was developed against AtoM 2.1, and has had compatibility updates for 2.3. Status of testing against other releases is unknown. The development was done by [https://github.com/schildwaechter Carsten Thiel], primarily between November and December of 2014.

* [https://github.com/CENDARI/sfDariahShibUserPlugin GitHub repository]
* [https://int2.cendari.dariah.eu/docs/developer/atom/shibboleth.html Plugin documentation]

==PeaceWorks MAID plugins ==

In March of 2015, [http://www.peaceworks.ca/ PeaceWorks Technology Solutions] helped to launch [http://archives.mhsc.ca/ MAID] (Mennonite Archival Image Database), a collaborative project of the Mennonite Historical Society of Canada which includes Mennonite archival partners in British Columbia, Alberta, Saskatchewan, Manitoba, and Ontario, with images held in a shared AtoM instance. Over the course of the 2-year design and development project leading up to the launch, PeaceWorks developed several custom features and plugins for use in MAID, including an eCommerce "shopping cart" plugin for purchasing prints of images available in MAID; an image carousel for the home page; integration with [http://www.tinymce.com/ TinyMCE] to add a [https://en.wikipedia.org/wiki/WYSIWYG WYSIWYG] set of editing tools to AtoM's static pages (e.g. so users did not need to know basic HTML to style static pages); and the ability to upload a different watermark (to be applied to digital object derivatives) for each institution. More details on each below.

===eCommerce plugin===

This plugin has been developed by [https://github.com/jasonhildebrand Jason Hildebrand] of [http://www.peaceworks.ca/ PeaceWorks Technology Solutions], primarily between July 2014 and March 2015.

According to the [https://github.com/PeaceWorksTechnologySolutions/atom/blob/ecommerce/plugins/sfEcommercePlugin/README.md plugin documentation's] Overview section, "This plugin allows website visitors to select and purchase individual photos. Payment for the photos happens via PayPal. E-Commerce admins review each order and may approve or reject (remove) photos from the order. After approval, the customer is sent an email containing a link which they can use to download the image(s).

Users have access to a shopping cart, where photos from multiple repositories can be added as users browse the site, and then ordered/checked out via PayPal. The shopping cart has basic integration with AtoM's PREMIS rights module, so when Dissemination = "Disallow," a message about restrictions is supplied instead. Other features include a vacation setting and a sales report.

* [https://github.com/PeaceWorksTechnologySolutions/atom/tree/ecommerce PeaceWorks GitHub AtoM fork, eCommerce branch]
* [https://github.com/PeaceWorksTechnologySolutions/atom/blob/ecommerce/plugins/sfEcommercePlugin/README.md eCommerce plugin documentation]
* [http://archives.mhsc.ca/ MAID], where you can see the shopping cart in action

MAID has also prepared a summary description of the plugin's functionality, available as a PDF here:

* [[File:Ecommerce-MAID.pdf]] (PDF, approx 431KB)

===Home page carousel plugin===

This plugin has been developed by [https://github.com/jasonhildebrand Jason Hildebrand] of [http://www.peaceworks.ca/ PeaceWorks Technology Solutions], primarily between July 2014 and March 2015.

This plugin adds a slideshow to the AtoM front page which displays up to 30 of the most recently added/updated photos in the system. Photos and their captions (if available) are displayed. Users may click a photo to go to its archival description.

* [https://github.com/PeaceWorksTechnologySolutions/atom/tree/carousel PeaceWorks GitHub AtoM fork, Carousel branch]
* [https://github.com/PeaceWorksTechnologySolutions/atom/blob/carousel/plugins/sfCarouselPlugin/README.md Carousel README documentation]
* [http://archives.mhsc.ca/ MAID] homepage, where you can see the carousel in action

===TinyMCE visual editor integration for editing static pages===

This integration has been developed by [https://github.com/jasonhildebrand Jason Hildebrand] of [http://www.peaceworks.ca/ PeaceWorks Technology Solutions], primarily between April 2014 and March 2015. At this time, this integration has not been packaged as a plugin.

[http://www.tinymce.com/ TinyMCE] is "'' a platform independent web based Javascript HTML [https://en.wikipedia.org/wiki/WYSIWYG WYSIWYG] editor control released as Open Source under LGPL,''" that, when integrated with AtoM, allows users unfamiliar with HTML to style static pages using familiar buttons, tools, and other user interface elements. The integration includes basic text styling (bold, underline, italics), table creation, image and file upload, link creation, and more.

* [https://github.com/PeaceWorksTechnologySolutions/atom/tree/tinymce PeaceWorks GitHub AtoM fork, TinyMCE branch]
* [https://github.com/PeaceWorksTechnologySolutions/atom/blob/tinymce/README_tinymce.md TinyMCE README documentation]

===Per-institution watermarks for digital objects===

This integration has been developed by [https://github.com/jasonhildebrand Jason Hildebrand] of [http://www.peaceworks.ca/ PeaceWorks Technology Solutions], between July 2014 and March 2015 (primarily in July 2014). At this time, this integration has not been packaged as a plugin.

AtoM includes the little-known ability to apply a watermark to uploaded photos. However there is only provision for a single global watermark which is applied to all images. This customization allows each institution to have their own watermark, and institutional watermarks can be uploaded and managed via the user interface, in the archival institution's "Edit theme" page. Uploading a new watermark will not affect photos already present in AtoM, but AtoM does provide a way for a developer to cause all watermarks to be re-applied (see “Regenerating Derivatives” in the Command-line tools section of the Administrator's Manual of the [https://www.accesstomemory.org/docs AtoM documentation] for more details).

* [https://github.com/PeaceWorksTechnologySolutions/atom/tree/multiple-watermarks Peaceworks GitHub AtoM fork, multiple-watermarks branch]
* [https://github.com/PeaceWorksTechnologySolutions/atom/commit/6e1e9e9e4bd4c2df5bf4da7b4063e4c739031fb4 the commit with the changes]
* [https://github.com/PeaceWorksTechnologySolutions/atom/blob/multiple-watermarks/README_watermarks.md Multiple watermarks README documentation]

==Docker and AtoM==

Two different community developers have combined AtoM with [https://www.docker.com/ Docker], an open, container-based virtualization platform for distributed applications. Details below.

<admonition type="tip">
Did you know? As of the AtoM 2.3 release, the AtoM project now has its own officially supported Docker Compose development environment! See:
* https://www.accesstomemory.org/docs/2.3/dev-manual/env/compose/
</admonition>

===AtoM Docker Files===

Created by Dominic Boisvert ([https://github.com/DominicBoisvert @DominicBoisvert]) for use in an archival course taught at the University of Montreal, Boisvert has shared Docker files for [[Releases/Release_announcements/Release_2.1.2|AtoM 2.1.2]], and [[Releases/Release_announcements/Release_2.1.1,_2.0.2,_1.3.2|ICA-AtoM 1.3.2]], including simple installation instructions (written in French):

* [https://github.com/ARV3054/docker-atom AtoM 2.1.2 Dockerfile]
* [https://github.com/ARV3054/docker-ica-atom ICA-AtoM 1.3.2 Dockerfile]

===AtoM Docker Compose recipe===

Created by John Fink ([https://github.com/jbfink @jbfink]), this is a Docker [https://docs.docker.com/compose/ Compose] recipe for AtoM. Compose is described by Docker as "''a tool for defining and running multi-container applications with Docker. With Compose, you define a multi-container application in a single file, then spin your application up in a single command which does everything that needs to be done to get it running.''"

The developer adds, "''Note that I've only tested this on 64bit native Linux using Docker directly, so if you're using boot2docker or a VM things might be different.''" For further configuration information and troubleshooting, see the [https://github.com/jbfink/docker-atom/blob/master/README.md README] provided.

* [https://github.com/jbfink/docker-atom Docker Compose recipe for AtoM - GitHub repository]
* [https://github.com/jbfink/docker-atom/blob/master/README.md README file for configuration and troubleshooting]
* [https://groups.google.com/d/topic/ica-atom-users/p7ACO3jKZv8/discussion Related AtoM User Forum thread]

===AtoM Docker Image===

Created by the Governo Regional Azores, this is a Docker image of AtoM 2.2.0. Note that it does not currently include the MySQL database required for AtoM, which needs to be linked.

* [https://hub.docker.com/r/governoregionalazores/atom-accesstomemory/ Link on the DockerHub page]
* [https://github.com/GovernoRegionalAcores/atom-accesstomemory GitHub repository for the Docker image]

==AtoM 2.3 ISO image - ready for use on a flash drive==

A group of archivists in Brazil, the [http://documentosarquivisticosdigitais.blogspot.com.br/ Grupo de Pesquisa do CNPq GED/A e Patrimônio Documental Arquivístico], have used the [https://launchpad.net/systemback SystemBack] backup management tool to prepare an ISO image of AtoM 2.3 that can be launched from a flash drive. From the related blog post:

* Operating System: Ubuntu 16.04 - 64 Bits;
* Size: 3GBs
* Format: sblive
* Remasterer: SystemBack

"''After Download, use SystemBack to write to the flash drive, and use with a system boot that is persistent - your data stored in the session will remain available in the next.'"

* Ubuntu Linux password: <code>icaatom99</code>
* User in AtoM (ICA-AtoM): <code>GrupoCNPqDocsDigitais@gmail.com</code>
* Password in AtoM (ICA-AtoM): <code>icaatom</code>

* Blog post here (Brazilian Portuguese): http://documentosarquivisticosdigitais.blogspot.com.br/2017/01/o-grupo-de-pesquisa-cnpq-ufsm-geda.html
* Download link: https://drive.google.com/file/d/0BwBRoubj23bpNWZLSVhtSmNoN00/view?usp=sharing

==HTML scrub scripts for other entities==

<admonition type="important">
'''UPDATE''': Thanks to collaboration among several community members, this work has now been added to the existing AtoM task, and merged into the 2.4 release. For more information, see:

* Issue ticket: https://projects.artefactual.com/issues/11207
* Source pull request: https://github.com/artefactual/atom/pull/568

Thank you to Clara Rosales and Darryl Friesen for this collaborative enhancement!

</admonition>

In [[Releases/Release_announcements/Release_2.2|Release 2.2]] we introduced [[Releases/Release_announcements/Release_2.2#Security_enhancement:_HTML_escaping_to_prevent_XSS_exploits|HTML escaping]] as part of the security enhancements included in the release - the related work is captured in issue ticket #7647. Since some users might have legacy HTML content in their descriptions, we also included a command-line task that would allow users to scrub legacy HTML from archival descriptions - see the related documentation [https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/cli-tools/#remove-html-content-from-archival-description-fields here].

However, this task '''only''' currently works for archival descriptions. To help clean up other entities, users from Brazil have adapted versions of the original task that can be run for actors, notes, repository records, and rights records. These were posted in the AtoM user forum on 2017-05-22:

* https://groups.google.com/d/msg/ica-atom-users/_xdBK0ucegg/RQnNM5DKBAAJ

Artefactual has not tested these scripts. To run them:

# Download and unzip the rar file attached to the message in the user forum.
# Place the scripts in a directory accessible from the root directory of your AtoM instance, such as a new tmp directory.
# Use the tools:run command to run each script. For example, for the script to clean up HTML in authority records:

<pre>
php symfony tools:run tmp/actor_i18n.php
</pre>

The effects will be similar to what is described for the command-line task:

* https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/cli-tools/#remove-html-content-from-archival-description-fields

-----

* [[Community/Community resources|Back to Community resources]]
* [[Community|Back to Community landing page]]
* [[Main Page|Main AtoM wiki page]]

[[Category:Community]]

Development/Archivematica integration

2018-11-28T22:08:55Z

Kgoetz: Start of an atom -> archivematica syncronisation spec

{{#pagetitle:Archivematica integration}}
[[Development]] > Development/Archivematica_integration

Access to Memory (AtoM) and [https://www.archivematica.org Archivematica] have a strong connection - they’re developed by the same vendor and Archivematica uses AtoM as its display frontend. Despite that communication is not currently bidirectional and when an AtoM site has data loaded there is currently no mechanism to synchronise the two platforms.

This page attempts to summarise any discoveries from [https://groups.google.com/forum/#!topic/ica-atom-users/TEkIGSUTEvo/overview the discussion started on AtoM-users] and will, in time, start to form a description of what integration might look like.

There is also a similar project for [https://github.com/eprintsug/EPrintsArchivematica integrating EPrints with Archivematica] who’s standards document includes useful information including layout and other legwork required for an export like this.

== Integration should cover ==
* An AtoM instance that contains records to which Archivematica is added as a backend
* An AtoM instance which is installed parallel to Archivematica but which is uploaded to directly

This will mean a variety of factors need to be considered (and questions asked) including
* Data uploaded to AtoM should be exportable to Archivematica for ingest and when the processed data is returned to AtoM no duplicate entries should be recorded.
* Optional support for Metadata only updates
* Should thumbnails generated by AtoM be exported and preserved?
* Does AtoM store revision history? Should we Archive revisions if it does?
* How will metadata only updates be handled
* What to do with external digital objects. ensure they are skipped with warning by export process? Export the thumbnails/metadata? Detection code is available [https://github.com/artefactual/atom/blob/stable/2.4.x/lib/model/QubitDigitalObject.php#L1569 in the repository]

== Options for integration ==

=== Option one: Data exported to directory ===

Export required data+metadata into a single directory which Archivematica imports ; AtoM then imports the export package (DIP) created by Archivematica

==== Export stage ====
Data which we can export (if Archivematica [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#the-transfer-tab transfer] can use it) which would help populate fields in Archivematica and can then be used to re-match with AtoM after processing.

* Transfer type (File type of item)
* Transfer name (current items name or description)
* Accession number (Accession number of item being exported)
* Access system ID (AtoM install performing export)
* Approve automatically (make this optional ; some may want it and some not)

[https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#preparing-digital-objects-for-transfer Export structure is important], as with [https://github.com/eprintsug/EPrintsArchivematica EPrints with Archivematica integration] I suggest this project supplies various bits of data (metadata/ , checksums, documentation, etc) as described on [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#preparing-digital-objects-for-transfer Archivematicas transfer page].

Working draft of filesystem layout
* object_slug-unique_identifer directory (object slug is the slug stored in DB, unique identifier is however AtoM identifies the object internally)
** Metadata directory, files in here are formatted per [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/import-metadata/#import-metadata Archivematica metadata import documentation] and should be populated with data from AtoM
*** metadata.csv (mandatory)
*** rights.csv (mandatory if rights are specified)
*** checksum.md5, checksum.sha1, or checksum.sha256 TBC: check which format/s AtoM uses and restrict output to that/those format/s
*** All other available metadata: optional, populate if supplied by AtoM and configured to export
** Objects directory, where actual files will go
*** Structure TODO

AtoM uses [https://github.com/artefactual/atom/blob/stable/2.4.x/lib/model/QubitDigitalObject.php#L1796 getAssetPath] to access its objects so we can lean on that to find files we want

==== Reimport ====
Import is currently managed by AtoM already, but will need to be augmented to deal with new issues.

* AtoM will have the original format files and the normalised format following the conversion. A way to remove the original files will need to be included
* AtoM will need to ensure it doesn’t enter a loop of: export files -> import from AM -> update files with those from AM -> trigger export of newly imported files.
* Duplicates in AtoM. Archivematica's DIP upload creates new stub information objects when it deposits the DIP objects - it won't replace existing objects. How manageable this problem is will depend on how much identifying metadata we can feed through Archivematica

=== Option two: Data exported to SIP/bag ===

Export packages (SIPs, bagit, or similar) which can be imported to Archivematica; AtoM then imports the export package (DIP) created by Archivematica.

This is the same as option one, but includes a packaging step at the end.

=== Option three: Bidirectional communication ===

Atom and Archivematica currently use SWORD1 (a unidirectional protocol) to push data from Archivematica to AtoM. Under this option they would use an existing protocol with bi directional support (eg SWORD2 or the work in progress SWORD3, see http://swordapp.org/).

* Both applications need changes in this instance
** Archivematica doesn't support plugins so its not as easy as a plugin at each end to add support.
* TBD: what would they communicate and how - would a SIP/bag be sent over SWORD or would individual files and associated data be sent and Archivematica packages it up?

== See also ==

* Archivematica's [https://www.archivematica.org/en/docs/archivematica-1.8/getting-started/overview/technical/#format-policies Format Policy] Registry (currently in the process of becoming its own project to be used by multiple digital preservation systems; now known as the Preservation Action Registry) to manage derivatives creation in a more intelligent way.

* TODO describe https://mediaarea.net/Events/2018-10-26_NoTimeToWait3/presentations/07.%20Martin%20Wrigley%20-%20Preservation%20Action%20Registry/OPF%20PAR%20Presentation%20notime%20to%20wait%20event.pdf

-----

* [[Development|Back to Development]]
* [[Main Page|AtoM wiki home]]

[[Category:Development documentation]]