Difference between revisions of "Releases/Roadmap"

From AtoM wiki
m (Clipboard)
(Add section on performance)
Line 139: Line 139:
  
 
* #13025 - Convert UTC time to localtime in OAI requests
 
* #13025 - Convert UTC time to localtime in OAI requests
 +
 +
===Performance, Scalability, and Internals===
 +
 +
We've implemented a number of internal enhancements to improve AtoM's overall performance and stability - these changes are predominantly focused on deletions, upgrades, indexing, and CSV imports. One of the biggest changes was upgrading AtoM’s MySQL version to 8.0 using the Percona Monitoring & Management ([https://www.percona.com/software/database-tools/percona-monitoring-and-management PMM]) tool to identify key areas for query optimization. Most significantly, in several areas we have replaced the nested set queries (used to manage hierarchical relations in a flat relational database) with MySQL8’s new Common Table Expressions ([https://dev.mysql.com/doc/refman/8.0/en/with.html CTE]; sometimes referred to as WITH queries). Overall this has led to significant performance gains - details of which have been captured on issue ticket #13224. We've also optimized the search indexing and the CSV import task - and some new options in some of the command line task (captured in other sections of this roadmap) also helped us optimize some of our processes that use these tasks.
 +
 +
* #10306 - Improve foreign key declarations in database schema
 +
* #13220 - Upgrade to MySQL 8
 +
* #13221 - Avoid schema differences between a default install and an upgraded database
 +
* #13224 - Improve hierarchy management queries
 +
* #13237 - Reconsider nested set implementation in some models
 +
* #13239 - Improve deletion of nested set hierarchies (especially terms)
 +
* #13350 - Reduce likelihood of issues when upgrading from 1.x to 2.6.x
 +
* #13354 - Avoid nested set update on descriptions CSV import (reduce DB deadlocks)
 +
* #13356 - Remove lft field from description ES index
 +
* #13261 - Improve Elasticsearch indexing process
 +
* #13362 - Move analyze table calls to the end of the upgrade process
  
 
===Physical storage===
 
===Physical storage===

Revision as of 10:23, 30 June 2020

Main Page > Releases > Releases/Roadmap

On this page, you'll find summaries about upcoming releases, and issue tickets related to them. As the releases are finalized, elements on this page will be used to draft the formal release announcements found here.

Tip

You can also check out the issues associated with upcoming releases by looking at the Roadmap page in our issue tracker.

Release 2.6

Accession records

Sponsored by the Government of the Northwest Territories

  • #13254 - Add the ability to add multiple identifiers to accessions, with type and note values
  • #13265 - Add search support for accession alternative identifiers
  • #13269 - Add the ability to associate accessions with physical storage containers

Analytics

Sponsored by the United Nations Archives and Records Management Section

  • #13284 - Add Google Tag Manager support

Authority records

General bug fixes and enhancements

  • #13157 - Add paging to authority record description relations to avoid editing timeouts

Sponsored by an undisclosed organization

  • #13275 - Display related authority records on Term view and Taxonomy browse pages
  • #13276 - Filter authority record searches by relationship type

Clipboard

Sponsored by the Bernard Becker Medical Library Archives at Washington University in St. Louis

  • #12800 - Add JSON export of AtoM clipboard items for third-party integrations

Command-line tasks

  • #13189 - Add task to move authority record description relations from one actor to another
  • #13257 - Add ES version to search:status task output
  • #13291 - Add task to output search index data for a target description
  • #13358 - Add option to the build nested set task to exclude types

See also the various CLI task enhancements in the Digital objects section.

Digital objects

General bug fixes and enhancements

  • #13196 - CLI regenerate derivatives task produces error when --slug used with --no-overwrite
  • #13243 - Regression: identifier and title context lost on link digital object page

Sponsored by the Oregon Historical Society

  • #12902 - Replace flowplayer with an HTML5 media player and update default derivative generation
  • #12903 - Add new option to Digital object regen-derivatives task to allow regenerating only video file derivatives

Sponsored by the Provincial Archives of Saskatchewan

  • #13103 - Add --attach-only option to digital object load task
  • #13126 - Add --skip-nested-set-build option to digital object load task

Sponsored by the University of Saskatchewan Libraries

  • #13152 - Add --replace option to digital object load task to allow uploads to replace existing digital objects

Sponsored by the Archives Society of Alberta

  • #13288 - Overhaul digitalobject:delete task and add new options to delete all digital objects associated with a repository, and limit to a specific media type

Finding aids

General bug fixes and enhancements

  • #12707 - Finding aids will display custom levels as "otherlevel" instead of using otherlevel value

Collaboratively developed / sponsored by Dalhousie University Library

  • #10591 - Update finding aid XSLTs with formatting improvements

Import/Export

General bug fixes and enhancements

  • #6317 - EAD <repository> will not import properly if it is not nested in a <corpname> element
  • #13130 - DC XML should include inherited creator(s) at lower levels
  • #12281 - Add rountrip option to CLI CSV import for better matching
  • #13204 - An apostrophe in the repository code will cause EAD XML imports to fail
  • #13246 - Add --limit option to task for caching XML representations

Sponsored by the WUSTL Bernard Becker Medical Library Archives

  • #12799 - Add donorNotes column to CSV import template for Accessions

Sponsored by the University of Saskatchewan Libraries

  • #13148 - Add support for command-line bulk authority record CSV export task

Sponsored by the Government of the Northwest Territories

  • #13262 - Add CSV import support for accession alternative identifiers
  • #13310 - Add donorFax and donorContactPerson columns to Accession CSV import

Sponsored by the City of Vancouver Archives

  • #13183 - Add CSV import for physical locations
  • #13185 - Add ability to update physical storage data via CSV import

Sponsored by the University of Strathclyde Archives and Special Collections

  • #13244 - Add setting to determine if identifier or reference code is used in DC XML

Sponsored by an undisclosed organization

  • #13249 - Add support for digital object CSV import/export for authority records
  • #13277: Alias and relations added to CSV GUI import for Authority records
  • #13279: Add relationship type to authority record relations CSV import and export
  • #13280: Create relationships between existing authority records via CSV import
  • #13282: Add Alias fields (parallelNames, standardizedNames, and otherFormNames) to Authority Record CSV export

OAI Repository

General bug fixes and enhancements

  • #12387 - OAI-PMH GetRecord verb should filter Draft descriptions consistent with other verb responses
  • #13163 - OAI identify command base URL does not respect HTTPS; appends index.php unnecessarily

Contributed via community pull request from Rohan Ferris

  • #13025 - Convert UTC time to localtime in OAI requests

Performance, Scalability, and Internals

We've implemented a number of internal enhancements to improve AtoM's overall performance and stability - these changes are predominantly focused on deletions, upgrades, indexing, and CSV imports. One of the biggest changes was upgrading AtoM’s MySQL version to 8.0 using the Percona Monitoring & Management (PMM) tool to identify key areas for query optimization. Most significantly, in several areas we have replaced the nested set queries (used to manage hierarchical relations in a flat relational database) with MySQL8’s new Common Table Expressions (CTE; sometimes referred to as WITH queries). Overall this has led to significant performance gains - details of which have been captured on issue ticket #13224. We've also optimized the search indexing and the CSV import task - and some new options in some of the command line task (captured in other sections of this roadmap) also helped us optimize some of our processes that use these tasks.

  • #10306 - Improve foreign key declarations in database schema
  • #13220 - Upgrade to MySQL 8
  • #13221 - Avoid schema differences between a default install and an upgraded database
  • #13224 - Improve hierarchy management queries
  • #13237 - Reconsider nested set implementation in some models
  • #13239 - Improve deletion of nested set hierarchies (especially terms)
  • #13350 - Reduce likelihood of issues when upgrading from 1.x to 2.6.x
  • #13354 - Avoid nested set update on descriptions CSV import (reduce DB deadlocks)
  • #13356 - Remove lft field from description ES index
  • #13261 - Improve Elasticsearch indexing process
  • #13362 - Move analyze table calls to the end of the upgrade process

Physical storage

General bug fixes and enhancements

  • #13201 - Browse physical storage page does not show values that have not been translated

Sponsored by the Government of the Northwest Territories

  • #13269 - Add new CSV report for physical storage location reporting

Search/Browse

General bug fixes and enhancements

  • #12471 - Hovering over a truncated facet filter doesn't show the full name in simple label tooltip
  • #13186 - Add user-configurable setting for treeview paging limit

Sponsored by the Government of the Northwest Territories

  • #12471 - Hovering over a truncated facet filter doesn't show the full name in simple label tooltip

Sponsored by the Bernard Becker Medical Library Archives at Washington University in St. Louis

  • #12797 - Make "Browse digital objects" link on description view page context sensitive

Sponsored by the Provincial Archives of Sasksatchewan

  • #13175 - Improve information object search indexing speed

Sponsored by the Mills Archive

  • #13230 - Date range sorting is broken following Elasticsearch upgrade in 2.5.x

Collaboratively prepared by community user Monica Wood (University of Tasmania) and Artefactual Systems

  • #13299 - If unique records count matches count of only other language, language facet should not display

Settings

  • #13105 - Add warning message to security settings page

Sponsored by the WUSTL Bernard Becker Medical Library Archives

  • #12798 - Add Visible elements module for DACS area headings
  • #13049 - Add DACS Physical access field to Visible elements
  • #13135 - Add new setting to make full-width treeview collapsible
  • #13320 - Increase treeview pager limit setting from 1,000 to a max value of 10,000

Bug fixes and other tickets

  • #13173 - If identifiers contain syntax that conforms to markdown syntax, the identifier is rendered in markdown
  • #13174 - Jobs can hang if console output is greater than 64KiB
  • #13196 - CLI regenerate derivatives task produces error when --slug used with --no-overwrite

Seealso

For a full list of issues related to the 2.6 release, see the following links to our issue tracker:

Wishlist

This project in our issue tracker is where new unsponsored feature requests for Access to Memory (AtoM) can be added. This allows us to log feature requests, and begin discussion about requirements and design without adding confusion to the main AtoM project's issue tracker as to which features will be included in an upcoming release.

See the AtoM Wishlist project here:

Tip

Once you've followed the link above, you can click on the "Issues" link at the top of the page to browse all issue tickets in the AtoM Wishlist. You can sort the list of issues by clicking on a column header, and use the Filter drop-down menu to the right to customize the query.

How the Wishlist works

When features are sponsored by an institution, or developed and submitted via community pull request, they will be copied to the main AtoM project in our issue tracker (here), and marked verified in the Wishlist project. Developer discussion, quality assurance testing, and other related updates will be posted to the related development ticket in the main AtoM project.

The tested version tag can be used in the Wishlist's issue description to keep track of when the feature was requested (e.g. which version of AtoM was most recent when the feature request came in)

The target version will only be applied before a Wishlist ticket is verified (closed) and copied to the main AtoM project.

Seealso

  • We've got information on our main AtoM issue tracker, and how you can use it to submit a bug report here: Resources/Issue tracker.
  • Got questions? Not sure if something is a bug or not, if a wishlist feature might be of interest to the rest of the community, or if something is already on our upcoming development roadmap? Why not make a post in our AtoM User Forum?