Development/Archivematica integration - Revision history

Kgoetz: add more 'see also' links from recent group discussions

2019-12-05T05:49:04Z

add more 'see also' links from recent group discussions

Kgoetz: add links to another stop gap approach for existing data

2019-01-10T00:41:08Z

add links to another stop gap approach for existing data

Kgoetz: link to atom built in tool for importing objects in a DIP

2018-12-12T23:09:36Z

link to atom built in tool for importing objects in a DIP

Kgoetz: mention new information

2018-12-07T03:10:12Z

mention new information

Kgoetz: Start of an atom -> archivematica syncronisation spec

2018-11-28T22:08:55Z

Start of an atom -> archivematica syncronisation spec

Dan: Fix breadcrumb

2018-11-27T22:55:11Z

Fix breadcrumb

Dan: Add basic page structure

2018-11-27T22:54:37Z

Add basic page structure

New page

{{#pagetitle:Archivematica integration}}
[[Development]] > [[Archivematica_integration]]

-----

* [[Development|Back to Development]]
* [[Main Page|AtoM wiki home]]

[[Category:Development documentation]]

← Older revision		Revision as of 05:49, 5 December 2019
Line 90:		Line 90:

	* [https://groups.google.com/forum/#!topic/archivematica/Ipvft2zC8ZA Question regarding retroactive archivematica uploads] which links to [https://www.accesstomemory.org/en/docs/2.3/user-manual/add-edit-content/archival-descriptions/#add-alternative-identifiers-to-an-archival-description AtoM documentation on how to Add alternative identifiers to an archival description].		* [https://groups.google.com/forum/#!topic/archivematica/Ipvft2zC8ZA Question regarding retroactive archivematica uploads] which links to [https://www.accesstomemory.org/en/docs/2.3/user-manual/add-edit-content/archival-descriptions/#add-alternative-identifiers-to-an-archival-description AtoM documentation on how to Add alternative identifiers to an archival description].
		+
		+	* [https://groups.google.com/forum/#!msg/ica-atom-users/tePqjGpqLdQ/Xe4Lf4W8AAAJ Discussion on Exporting data and metadata from AtoM to archivematica] on the Atom users list.
		+
		+	* [https://groups.google.com/forum/#!msg/archivematica/E-ESb5TiiI4/wwM5Hqj0AwAJ Query around sending DIP to Atom via API] from Archivematica user group

	-----		-----

← Older revision		Revision as of 00:41, 10 January 2019
Line 89:		Line 89:
	* TODO describe https://mediaarea.net/Events/2018-10-26_NoTimeToWait3/presentations/07.%20Martin%20Wrigley%20-%20Preservation%20Action%20Registry/OPF%20PAR%20Presentation%20notime%20to%20wait%20event.pdf		* TODO describe https://mediaarea.net/Events/2018-10-26_NoTimeToWait3/presentations/07.%20Martin%20Wrigley%20-%20Preservation%20Action%20Registry/OPF%20PAR%20Presentation%20notime%20to%20wait%20event.pdf

		+	* [https://groups.google.com/forum/#!topic/archivematica/Ipvft2zC8ZA Question regarding retroactive archivematica uploads] which links to [https://www.accesstomemory.org/en/docs/2.3/user-manual/add-edit-content/archival-descriptions/#add-alternative-identifiers-to-an-archival-description AtoM documentation on how to Add alternative identifiers to an archival description].

	-----		-----

← Older revision		Revision as of 23:09, 12 December 2018
Line 64:		Line 64:
	Atom may be extended to use the MDTYPE=OTHER metadata available in the DIP METS file, this would help with the roundtrip and (if it works) might solve all the identification problems.		Atom may be extended to use the MDTYPE=OTHER metadata available in the DIP METS file, this would help with the roundtrip and (if it works) might solve all the identification problems.

		+	Atom includes a tool for importing data from a DIP via CSV file. this tool or its backing library might be usable in this context. https://github.com/artefactual/atom/blob/qa/2.5.x/lib/task/import/importDipObjectsTask.class.php

	=== Option two: Data exported to SIP/bag ===		=== Option two: Data exported to SIP/bag ===

@@ Line 45: / Line 45: @@
 *** rights.csv (mandatory if rights are specified)
 *** checksum.md5, checksum.sha1, or checksum.sha256 TBC: check which format/s AtoM uses and restrict output to that/those format/s
-*** All other available metadata: optional, populate if supplied by AtoM and configured to export
+*** All other available metadata: optional; if supplied by AtoM and configured to export
 ** Objects directory, where actual files will go
 *** Structure TODO
@@ Line 59: / Line 59: @@
 * Duplicates in AtoM. Archivematica's DIP upload creates new stub information objects when it deposits the DIP objects - it won't replace existing objects. How manageable this problem is will depend on how much identifying metadata we can feed through Archivematica

← Older revision	Revision as of 22:08, 28 November 2018
Line 3:	Line 3:


	+	Access to Memory (AtoM) and [https://www.archivematica.org Archivematica] have a strong connection - they’re developed by the same vendor and Archivematica uses AtoM as its display frontend. Despite that communication is not currently bidirectional and when an AtoM site has data loaded there is currently no mechanism to synchronise the two platforms.

	+	This page attempts to summarise any discoveries from [https://groups.google.com/forum/#!topic/ica-atom-users/TEkIGSUTEvo/overview the discussion started on AtoM-users] and will, in time, start to form a description of what integration might look like.

	+	There is also a similar project for [https://github.com/eprintsug/EPrintsArchivematica integrating EPrints with Archivematica] who’s standards document includes useful information including layout and other legwork required for an export like this.


	+	== Integration should cover ==
	+	* An AtoM instance that contains records to which Archivematica is added as a backend
	+	* An AtoM instance which is installed parallel to Archivematica but which is uploaded to directly
	+
	+	This will mean a variety of factors need to be considered (and questions asked) including
	+	* Data uploaded to AtoM should be exportable to Archivematica for ingest and when the processed data is returned to AtoM no duplicate entries should be recorded.
	+	* Optional support for Metadata only updates
	+	* Should thumbnails generated by AtoM be exported and preserved?
	+	* Does AtoM store revision history? Should we Archive revisions if it does?
	+	* How will metadata only updates be handled
	+	* What to do with external digital objects. ensure they are skipped with warning by export process? Export the thumbnails/metadata? Detection code is available [https://github.com/artefactual/atom/blob/stable/2.4.x/lib/model/QubitDigitalObject.php#L1569 in the repository]
	+
	+	== Options for integration ==
	+
	+	=== Option one: Data exported to directory ===
	+
	+	Export required data+metadata into a single directory which Archivematica imports ; AtoM then imports the export package (DIP) created by Archivematica
	+
	+	==== Export stage ====
	+	Data which we can export (if Archivematica [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#the-transfer-tab transfer] can use it) which would help populate fields in Archivematica and can then be used to re-match with AtoM after processing.
	+
	+	* Transfer type (File type of item)
	+	* Transfer name (current items name or description)
	+	* Accession number (Accession number of item being exported)
	+	* Access system ID (AtoM install performing export)
	+	* Approve automatically (make this optional ; some may want it and some not)
	+
	+	[https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#preparing-digital-objects-for-transfer Export structure is important], as with [https://github.com/eprintsug/EPrintsArchivematica EPrints with Archivematica integration] I suggest this project supplies various bits of data (metadata/ , checksums, documentation, etc) as described on [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/transfer/#preparing-digital-objects-for-transfer Archivematicas transfer page].
	+
	+	Working draft of filesystem layout
	+	* object_slug-unique_identifer directory (object slug is the slug stored in DB, unique identifier is however AtoM identifies the object internally)
	+	** Metadata directory, files in here are formatted per [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/import-metadata/#import-metadata Archivematica metadata import documentation] and should be populated with data from AtoM
	+	*** metadata.csv (mandatory)
	+	*** rights.csv (mandatory if rights are specified)
	+	*** checksum.md5, checksum.sha1, or checksum.sha256 TBC: check which format/s AtoM uses and restrict output to that/those format/s
	+	*** All other available metadata: optional, populate if supplied by AtoM and configured to export
	+	** Objects directory, where actual files will go
	+	*** Structure TODO
	+
	+	AtoM uses [https://github.com/artefactual/atom/blob/stable/2.4.x/lib/model/QubitDigitalObject.php#L1796 getAssetPath] to access its objects so we can lean on that to find files we want
	+
	+
	+	==== Reimport ====
	+	Import is currently managed by AtoM already, but will need to be augmented to deal with new issues.
	+
	+	* AtoM will have the original format files and the normalised format following the conversion. A way to remove the original files will need to be included
	+	* AtoM will need to ensure it doesn’t enter a loop of: export files -> import from AM -> update files with those from AM -> trigger export of newly imported files.
	+	* Duplicates in AtoM. Archivematica's DIP upload creates new stub information objects when it deposits the DIP objects - it won't replace existing objects. How manageable this problem is will depend on how much identifying metadata we can feed through Archivematica
	+
	+
	+
	+	=== Option two: Data exported to SIP/bag ===
	+
	+	Export packages (SIPs, bagit, or similar) which can be imported to Archivematica; AtoM then imports the export package (DIP) created by Archivematica.
	+
	+	This is the same as option one, but includes a packaging step at the end.
	+
	+
	+	=== Option three: Bidirectional communication ===
	+
	+	Atom and Archivematica currently use SWORD1 (a unidirectional protocol) to push data from Archivematica to AtoM. Under this option they would use an existing protocol with bi directional support (eg SWORD2 or the work in progress SWORD3, see http://swordapp.org/).
	+
	+	* Both applications need changes in this instance
	+	** Archivematica doesn't support plugins so its not as easy as a plugin at each end to add support.
	+	* TBD: what would they communicate and how - would a SIP/bag be sent over SWORD or would individual files and associated data be sent and Archivematica packages it up?
	+
	+
	+
	+	== See also ==
	+
	+	* Archivematica's [https://www.archivematica.org/en/docs/archivematica-1.8/getting-started/overview/technical/#format-policies Format Policy] Registry (currently in the process of becoming its own project to be used by multiple digital preservation systems; now known as the Preservation Action Registry) to manage derivatives creation in a more intelligent way.
	+
	+	* TODO describe https://mediaarea.net/Events/2018-10-26_NoTimeToWait3/presentations/07.%20Martin%20Wrigley%20-%20Preservation%20Action%20Registry/OPF%20PAR%20Presentation%20notime%20to%20wait%20event.pdf

← Older revision		Revision as of 22:55, 27 November 2018
Line 1:		Line 1:
	{{#pagetitle:Archivematica integration}}		{{#pagetitle:Archivematica integration}}
−	[[Development]] > [[Archivematica_integration]]	+	[[Development]] > Development/Archivematica_integration