Acquisitions API migration enhancements
Description
Priority
Fix versions
Development Team
Assignee

Solution Architect
Parent Field Value
Parent Status
is defined by
Checklist
hideTestRail: Results
Activity

Theodor Tolstoy (One-Group.se) February 19, 2021 at 1:48 PM
These are my overall requirements for any (Batch) API. Some could be turned into JIRA, some are more of best practices.
Stability - No memory leaks.
When the library is live, memory leaks can be maintained over time with recurring module restarts, but in the DM scenario, a restart could fail a whole batch if not handled properly.
Stability - Batch size
**Instead of promising the world, set a reasonable goal for what batch sizes the modules can handle and make sure the modules can deliver on the promised numbers given it is run/hosted under the minimum requirements. Make this super clear by throwing exceptions for too large batches.
Well crafted API:s - Error reporting
FOLIO has enough (Batch) APIs that just give you a silent HTTP 500 when the underlying calls fail. Don't join them!
It is a pain having to take a tour of all the sub-module calls and try to find out what went wrong.
Tell us what went wrong and what record that failed! Every time!
Well crafted API:s - Documentation
The RAML specs allow for a lot of documentation in the relevant areas. Make use of it!
Make sure the examples are up-to-date as well.
Consistency - Change and improve, but in a controlled fashion
If an API has worked in a certain way, do not make dramatic changes to it. Add another one instead. We have seen APIs that stopped doing certain things.
Data migration is by nature reverse-engineering, and having to do that for every API for every module update is very painful.
Consistency - Migrators rely on the schemas too
Talk to the UI developers and make the "required" properties reflect what is actually required.
Make the descriptions of the properties really good. we show them to our clients/libraries as part of various data mapping activities.
Make some kind of indication of what properties are calculated or fetched at display-time.
Personally, I would prefer to have different data models for viewing and writing.
Data from other modules - If you depend on a module, Depend. On. It.
Take Inventory data for example:
If I migrate an order that has an underlying Inventory dependency, I do not want to add the whole thing in once again. An Identifier to the relevant Inventory records should be enough. When the record is fetched, the API should be smart enough to look it up!
If I migrate a historical order where the Inventory records have been weeded, I want to add some Inventory metadata to the order for reference. So it is vital that the possibility to add bibliographic data is maintained
So, I am proposing sort of a caching mechanism in the modules, so that we do not have to add everything in. Loading some data from Inventory the first time a migrated order is fetched by the orders module should not be too painful?
BL APIs should work just as well for historical data as for current transactions.
If you want to migrate data using the BL-API route, the APIs should be able to let you replay history from the beginning.
Document things that do not work in the same way when you replay historical transactions as when you work with current data.
The advantage of using the BL API strategy is that you do not have to wire everything up, and that your migration process can be more stable as FOLIO evolves. The downside is that some things that are triggered by BL APIs are not wanted and could cause confusion (like sending out EDI invoices, orders, contacting Vendors ETC). These things would ideally be turned off by some API request parameter. There is always the option of turning off the affected modules, but that could lead to other errors.
Details
Reporter
Dennis BridgesDennis BridgesPO Rank
0PO Ranking Note
These enhancements would be helpful in resolving some of the difficulties with migrating existing records into FOLIO.Back End Estimate
XL < 15 daysRank: 5Colleges (Full Jul 2021)
R3Rank: Cornell (Full Sum 2021)
R1Rank: GBV (MVP Sum 2020)
R3Rank: Mainz (Full TBD)
NoneRank: MI State-Lib of MI (Sum 2021)
NoneRank: TAMU (MVP Jan 2021)
R3Rank: Chicago (MVP Sum 2020)
R4Rank: U of AL (MVP Oct 2020)
R4TestRail: Cases
Open TestRail: CasesTestRail: Runs
Open TestRail: Runs
Details
Details
Reporter

Overview:
Current acquisitions APIs were not specifically designed to be used for migrating data. This feature captures enhancements that have been requested to improve options for migrating acquisitions data into FOLIO. It may include adding a series of APIs or enhancements to existing APIs that facilitate the migration of acquisitions data.
Note: this may be split in order to add small improvements over the span of multiple releases.
Use cases:
See https://folio-org.atlassian.net/wiki/display/DD/Acquisitions+API+migration+enhancements
Approach:
TBD