Migration Tools (UXPROD-850)

[UXPROD-2619] Acquisitions API migration enhancements Created: 12/Aug/20  Updated: 23/Jan/23

Status: Open
Project: UX Product
Components: None
Affects versions: None
Fix versions: None
Parent: Migration Tools

Type: New Feature Priority: P2
Reporter: Dennis Bridges Assignee: Dennis Bridges
Resolution: Unresolved Votes: 0
Labels: acq-morningglory-candidate, acquisitions
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Defines
is defined by MODINVOSTO-84 Bulk Create/Update API for invoice lines Open
is defined by MODINVOSTO-85 Bulk DELETE API for invoice lines Open
is defined by MODINVOSTO-86 Bulk Create/Update API for invoices Open
is defined by MODINVOSTO-87 Bulk DELETE API for invoices Open
is defined by MODORDSTOR-178 Bulk Create/Update API for PO Lines Open
is defined by MODORDSTOR-179 Bulk DELETE API for PO Lines Open
is defined by MODORDSTOR-180 Bulk Create/Update API for purchase o... Open
is defined by MODORDSTOR-181 Bulk DELETE API for purchase orders Open
is defined by MODFISTO-134 Can invoice_transaction_summaries and... Closed
is defined by MODORDERS-404 Orders composite crashes API crashes ... Closed
is defined by MODORDERS-473 Memory leak in orders app Closed
Epic Link: Migration Tools
Back End Estimate: XL < 15 days
Development Team: Thunderjet
PO Rank: 55
PO Ranking Note: These enhancements would be helpful in resolving some of the difficulties with migrating existing records into FOLIO.
Rank: Chicago (MVP Sum 2020): R4
Rank: Cornell (Full Sum 2021): R1
Rank: Duke (Full Sum 2021): R3
Rank: 5Colleges (Full Jul 2021): R3
Rank: GBV (MVP Sum 2020): R3
Rank: Mainz (Full TBD): None
Rank: MI State-Lib of MI (Sum 2021): None
Rank: TAMU (MVP Jan 2021): R3
Rank: U of AL (MVP Oct 2020): R4

 Description   

Overview:

Current acquisitions APIs were not specifically designed to be used for migrating data. This feature captures enhancements that have been requested to improve options for migrating acquisitions data into FOLIO. It may include adding a series of APIs or enhancements to existing APIs that facilitate the migration of acquisitions data.

Note: this may be split in order to add small improvements over the span of multiple releases.

Use cases:

See https://folio-org.atlassian.net/wiki/display/DD/Acquisitions+API+migration+enhancements

Approach:

TBD



 Comments   
Comment by Theodor Tolstoy (One-Group.se) [ 19/Feb/21 ]

These are my overall requirements for any (Batch) API. Some could be turned into JIRA, some are more of best practices.

Stability - No memory leaks.

When the library is live, memory leaks can be maintained over time with recurring  module restarts, but in the DM scenario, a restart could fail a whole batch if not handled properly. 

Stability - Batch size

**Instead of promising the world, set a reasonable goal for what batch sizes the modules can handle and make sure the modules can deliver on the promised numbers given it is run/hosted under the minimum requirements. Make this super clear by throwing exceptions for too large batches.

Well crafted API:s - Error reporting 

FOLIO has enough (Batch) APIs that just give you a silent HTTP 500 when the underlying calls fail. Don't join them!
It is a pain having to take a tour of all the sub-module calls and try to find out what went wrong. 

Tell us what went wrong and what record that failed! Every time!

Well crafted API:s - Documentation

The RAML specs allow for a lot of documentation in the relevant areas. Make use of it!

Make sure the examples are up-to-date as well. 

Consistency - Change and improve, but in a controlled fashion

If an API has worked in a certain way, do not make dramatic changes to it. Add another one instead. We have seen APIs that stopped doing certain things. 

Data migration is by nature reverse-engineering, and having to do that for every API for every module update is very painful.

Consistency - Migrators rely on the schemas too

  • Talk to the UI developers and make the "required" properties reflect what is actually required. 
  • Make the descriptions of the properties really good. we show them to our clients/libraries as part of various data mapping activities.
  • Make some kind of indication of what properties are calculated or fetched at display-time. 
  • Personally, I would prefer to have different data models for viewing and writing. 

Data from other modules - If you depend on a module, Depend. On. It.

Take Inventory data for example:

  • If I migrate an order that has an underlying Inventory dependency, I do not want to add the whole thing in once again. An Identifier to the relevant Inventory records should be enough. When the record is fetched, the API should be smart enough to look it up!
  • If I migrate a historical order where the Inventory records have been weeded, I want to add some Inventory metadata to the order for reference. So it is vital that the possibility to add bibliographic data is maintained

So, I am proposing sort of a caching mechanism in the modules, so that we do not have to add everything in. Loading some data from Inventory the first time a migrated order is fetched by the orders module should not be too painful?

BL APIs should work just as well for historical data as for current transactions.

If you want to migrate data using the BL-API route, the APIs should be able to let you replay history from the beginning. 

  • Document things that do not work in the same way when you replay historical transactions as when you work with current data.
  • The advantage of using the BL API strategy is that you do not have to wire everything up, and that your migration process can be more stable as FOLIO evolves. The downside is that some things that are triggered by BL APIs are not wanted and could cause confusion (like sending out EDI invoices, orders, contacting Vendors ETC). These things would ideally be turned off by some API request parameter. There is always the option of turning off the affected modules, but that could lead to other errors.

 

 

 

 

Generated at Fri Feb 09 00:25:28 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.