[UXPROD-47] Batch Importer (Bib/Acq) Created: 18/Jan/18  Updated: 08/Feb/24

Status: Analysis Complete
Project: UX Product
Components: None
Affects versions: None
Fix versions: Q1 2019

Type: Epic Priority: P3
Reporter: Ann-Marie Breaux (Inactive) Assignee: Ryan Taylor
Resolution: Unresolved Votes: 0
Labels: EDI, acquisitions, authority, cornell-priority, data-import, delimited_files, epam-folijet, file-upload, holdings, instances, invoices, items, marcimport, metadatamanagement, migration-load, no-epic-needed, orders, sysops_mgt
Remaining Estimate: Not Specified
Time Spent: 3 hours, 30 minutes
Original estimate: Not Specified

Issue links:
Defines
is defined by UXPROD-2196 NFR: PubSub enhancements (BE) Open
is defined by UXPROD-3969 Improve solution that will refresh In... Open
is defined by UXPROD-4081 Refine and standardize handling of 03... Open
is defined by UXPROD-4348 Data Import: Ability to export Data i... Open
is defined by UXPROD-4444 MARC-Instance default mapping changes... Open
is defined by UXPROD-4584 Enhancements to Instance local fields... Open
is defined by UXPROD-4590 Data import matching by normalized ISBN Open
is defined by UXPROD-4077 NFR: Module releases, RMB/Stripes upd... Open
is defined by UXPROD-4442 Data Import MARC Modification Mapping... Open
is defined by UXPROD-2742 MARC-MARC matching enhancements: Narr... In Progress
is defined by UXPROD-4074 NFR: Data Import/Inventory Support Bu... In Progress
is defined by UXPROD-4075 NFR: Data Import/Inventory Technical,... In Progress
is defined by UXPROD-4076 NFR: UI end-to-end tests tech debt fo... In Progress
is defined by UXPROD-4257 Data Import Architecture Refinements ... In Progress
is defined by UXPROD-4471 Remove step of initial saving of inco... In Progress
is defined by UXPROD-4141 Import and create/update a shared ins... Closed
is defined by UXPROD-4259 DI processing updates for matching an... Closed
is defined by UXPROD-185 Import Orders in MARC format Closed
is defined by UXPROD-658 Import Inventory Items from MARC Bibs... Closed
is defined by UXPROD-659 Import Inventory Holdings from MARC B... Closed
is defined by UXPROD-665 Import Inventory Instances and MARCca... Closed
is defined by UXPROD-1376 CRUD for File Extensions Closed
is defined by UXPROD-1377 CRUD for Job Profiles Closed
is defined by UXPROD-1378 CRUD for Match Profiles Closed
is defined by UXPROD-1379 CRUD for Action Profiles Closed
is defined by UXPROD-1381 CRUD for EDIFACT Invoice Field Mappin... Closed
is defined by UXPROD-1394 ISBN normalization: Refinement Closed
is defined by UXPROD-1447 Instance record with an underlying MA... Closed
is defined by UXPROD-1575 quickMARC | Store and Display MARC Ho... Closed
is defined by UXPROD-1577 Make Inventory Instance rely on SRS M... Closed
is defined by UXPROD-1595 Connect SRS and MARCcat - Folijet wor... Closed
is defined by UXPROD-1769 Remaining Job and Match profile work,... Closed
is defined by UXPROD-1805 SRS MARC-Inventory Instance relations... Closed
is defined by UXPROD-1806 NFR: Data Import Pub-Sub (Event Drive... Closed
is defined by UXPROD-2012 NFR: Data Import Pub-Sub (Event Drive... Closed
is defined by UXPROD-2074 Bring Action Profiles to life: prep work Closed
is defined by UXPROD-2075 MARC Bib Field Mapping Profiles Detai... Closed
is defined by UXPROD-2077 Remaining Job and Match profile work,... Closed
is defined by UXPROD-2078 SRS MARC-Inventory Instance relations... Closed
is defined by UXPROD-2197 NFR: Data Import (Batch Importer for ... Closed
is defined by UXPROD-2208 Remaining Job and Match profile work,... Closed
is defined by UXPROD-2209 Bring Action Profiles to life: next work Closed
is defined by UXPROD-2220 NFR: Data Import (Batch Importer for ... Closed
is defined by UXPROD-2273 Connect SRS and quickMARC for MARC Bi... Closed
is defined by UXPROD-2281 quickMARC | Define human readable ide... Closed
is defined by UXPROD-2282 MARC Field Mapping Profiles Details f... Closed
is defined by UXPROD-2286 MARC Field Mapping Profiles Details f... Closed
is defined by UXPROD-2322 Import MARC authority records - Creat... Closed
is defined by UXPROD-2338 Remaining Job and Match profile refin... Closed
is defined by UXPROD-2351 Import Inventory Holdings from MARC B... Closed
is defined by UXPROD-2353 Import Inventory Items from MARC Bibs... Closed
is defined by UXPROD-2354 Import Inventory Instances from MARC ... Closed
is defined by UXPROD-2355 Import SRS MARC Bib Records: Update Closed
is defined by UXPROD-2363 SRS Performance Improvements Closed
is defined by UXPROD-2446 Finish & cleanup MARC Field Mapping P... Closed
is defined by UXPROD-2550 MARC Bib Field Mapping Profiles Detai... Closed
is defined by UXPROD-2551 NFR: Data Import (Batch Importer for ... Closed
is defined by UXPROD-2614 NFR: Data Import (Batch Importer for ... Closed
is defined by UXPROD-2615 Ability to kill a long-running or err... Closed
is defined by UXPROD-2659 NFR: Refactor data-import flow to inc... Closed
is defined by UXPROD-2697 NFR: Create integration tests for key... Closed
is defined by UXPROD-2741 Import of MARC Bibs to create/update ... Closed
is defined by UXPROD-2753 Data Import Log - "lite version" Closed
is defined by UXPROD-2790 NFR: Make some revisions to Source Re... Closed
is defined by UXPROD-2858 Define human readable identifiers (HR... Closed
is defined by UXPROD-2929 NFR: Increase security of Kafka for D... Closed
is defined by UXPROD-2939 Data Import Log - refine error handling Closed
is defined by UXPROD-2990 Import MARC authority records - Updat... Closed
is defined by UXPROD-3023 NFR: R2 2021 Juniper Data Import Stab... Closed
is defined by UXPROD-3041 NFR: Data Import (Batch Importer for ... Closed
is defined by UXPROD-3074 Create monitoring task for Data Impor... Closed
is defined by UXPROD-3121 NFR: Create integration tests for key... Closed
is defined by UXPROD-3135 NFR: R3 2021 Kiwi Data Import Stabili... Closed
is defined by UXPROD-3173 Folijet support work for Inventory Op... Closed
is defined by UXPROD-3191 NFR: R3 2021 Kiwi Data import perform... Closed
is defined by UXPROD-3193 NFR: R3 2021 Kiwi Data import Stabili... Closed
is defined by UXPROD-3261 NFR: R1 2022 Lotus Data import perfor... Closed
is defined by UXPROD-3262 NFR: Data Import R1 2022 Lotus Techni... Closed
is defined by UXPROD-3263 NFR: UI end-to-end tests for Data Imp... Closed
is defined by UXPROD-3361 Prep for Importing Orders in MARC format Closed
is defined by UXPROD-3419 NFR: UI end-to-end tests for Folijet ... Closed
is defined by UXPROD-3463 NFR: Data Import R1 2022 Lotus Suppor... Closed
is defined by UXPROD-3464 NFR: Data Import R2 2022 Morning Glor... Closed
is defined by UXPROD-3575 NFR: Karate API integration tests for... Closed
is defined by UXPROD-3576 NFR: Data Import Support Bug work (No... Closed
is defined by UXPROD-3623 Refine Data Import Field protection Closed
is defined by UXPROD-3694 NFR: UI end-to-end tests for Folijet ... Closed
is defined by UXPROD-3696 Refine Update individual fields funct... Closed
is defined by UXPROD-3726 NFR: UI end-to-end tests tech debt fo... Closed
is defined by UXPROD-3727 NFR: Data Import Support Bug work (Or... Closed
is defined by UXPROD-3729 Planning for Import of MARC Bibs to c... Closed
is defined by UXPROD-3831 Allow user to assign various job prof... Closed
is defined by UXPROD-3839 NFR: UI end-to-end tests tech debt fo... Closed
is defined by UXPROD-4079 DI Log updates for multiple holdings/... Closed
is defined by UXPROD-130 Parse MARC21 format data records Closed
is defined by UXPROD-145 Batch upload and import of MARC metad... Closed
is defined by UXPROD-149 Create Instance records from a MARC b... Closed
is defined by UXPROD-656 Data Import (Batch Importer for Bib A... Closed
is defined by UXPROD-661 Create interactive log-summary of bat... Closed
is defined by UXPROD-1038 ISBN normalization: Preparation, and ... Closed
is defined by UXPROD-1380 CRUD for MARC Field Mapping Profiles,... Closed
is defined by UXPROD-1448 Connect SRS and Inventory for MARC Ho... Closed
is defined by UXPROD-1576 Connect MARCcat and SRS for MARC Auth... Closed
is defined by UXPROD-1596 SRS-MARCcat integration for Holdings ... Closed
is defined by UXPROD-2076 NFR: Data Import (Batch Importer for ... Closed
is defined by UXPROD-2207 SRS MARC-Inventory Instance relations... Closed
is defined by UXPROD-2372 NFR: Data Import accessibility bugfix... Closed
is defined by UXPROD-2701 Tech Debt: Migrate UI automated unit ... Closed
is defined by UXPROD-2743 NFR: Reuse components from the data-t... Closed
is defined by UXPROD-3001 Data Import: Implement keyboard short... Closed
is defined by UXPROD-3099 Add the ability to print MARC records... Closed
is defined by UXPROD-3105 Tech Debt: Migrate UI unit automated ... Closed
is defined by UXPROD-3106 Tech Debt: Migrate UI automated unit ... Closed
is defined by UXPROD-3120 Folijet - R3 2022 Kiwi Module release... Closed
is defined by UXPROD-3465 NFR: Morning Glory R2 2022 Module rel... Closed
is defined by UXPROD-3577 NFR: Module releases, RMB/Stripes upd... Closed
is defined by UXPROD-3717 Import Invoices in MARC format - DUPL... Closed
is defined by UXPROD-3728 NFR: Module releases, RMB/Stripes upd... Closed
is defined by UXPROD-4095 NFR: Folijet FAT Misc and Karate test... Closed
is defined by UXPROD-663 Import Invoices based on data supplie... Draft
is defined by UXPROD-2802 NFR: Handle implicit/explicit actions... Draft
is defined by UXPROD-3363 Data Import Log - refine error handli... Draft
is defined by UXPROD-3628 Redesign approach for handling extern... Draft
is defined by UXPROD-3667 Change Data Import to use a Saga exec... Draft
is defined by UXPROD-3695 Additional Changes to the Data Import... Draft
is defined by UXPROD-3782 Import of MARC Bibs to create MARC Bi... Draft
is defined by UXPROD-4080 Review and fix MARC Updates for indiv... Draft
is defined by UXPROD-4446 Deleting old versions of records from... Draft
is defined by UXPROD-4529 NFR: Add Jest tests for UIPFIMP Draft
is defined by UXPROD-669 Ability to do a test load (preview), ... Draft
is defined by UXPROD-1382 CRUD for Delimited Data Mapping Profiles Draft
is defined by UXPROD-1383 Ability to roll back a load if problems Draft
is defined by UXPROD-1384 Data Import permissions - Additional ... Draft
is defined by UXPROD-1405 Data Import (Batch Importer for Bib A... Draft
is defined by UXPROD-1479 Simple UI for edit of the default MAR... Draft
is defined by UXPROD-1860 Migration. Simple UI for edit of the ... Draft
is defined by UXPROD-2099 Data Import: Ability to export Data I... Draft
is defined by UXPROD-2152 Additional MARC-Instance relationship... Draft
is defined by UXPROD-2164 Supply default Job, Match, Action, Ma... Draft
is defined by UXPROD-2200 Create interactive log-summary of bat... Draft
is defined by UXPROD-2334 NFR: Data Import & Inventory Not-Yet-... Draft
is defined by UXPROD-2350 Support for the Delete action during ... Draft
is defined by UXPROD-2641 Blocked Field mapping profile updates... Draft
is defined by UXPROD-2661 Load Delimited or MARC records to cre... Draft
is defined by UXPROD-2934 Job profile-level Data import MARC fi... Draft
is defined by UXPROD-2998 Data Import: Add support for Importin... Draft
is defined by UXPROD-2999 Data Import: Add support for Importin... Draft
is defined by UXPROD-3000 Data Import: Add support for tokens i... Draft
is defined by UXPROD-3145 Remove SRS and Data Import Debris Draft
is defined by UXPROD-3146 Add if/then logic to the field mappin... Draft
is defined by UXPROD-655 Import Orders in Delimited format Draft
is defined by UXPROD-657 Import Inventory Items in Delimited f... Draft
is defined by UXPROD-660 Import Inventory Holdings in Delimite... Draft
is defined by UXPROD-662 Automated FTP and Scheduling for batc... Draft
is defined by UXPROD-664 Batch upload and import of delimited ... Draft
is defined by UXPROD-666 Import Inventory Instances (and maybe... Draft
is defined by UXPROD-674 Import Invoices in delimited format Draft
is defined by UXPROD-2098 Add a way to create job profiles on-t... Draft
is defined by UXPROD-3097 Refinements to Data Import MARC Field... Draft
is defined by UXPROD-3194 Assess the work for supporting the UN... Draft
is defined by UXPROD-4440 Control OCLC import or overlay (edge-... Draft
is defined by UXPROD-3838 NFR: Data Import & Inventory Support ... In Review
is defined by UXPROD-3840 NFR: Data Import and Inventory Techni... In Review
is defined by UXPROD-3841 NFR: Module releases, RMB/Stripes upd... In Review
is defined by UXPROD-3174 Analyze Folijet support work for Inve... Blocked
is defined by UXPROD-3814 Import Orders in MARC format - Refine... Analysis Complete
Relates
relates to UXPROD-593 Workflow Services (Back End for Workf... Open
relates to UXPROD-597 Files App Open
relates to UXPROD-652 Metadata Record Export In Progress
relates to UXPROD-868 Bulk Edit In Progress
relates to UXPROD-211 Single MARC Record Integration with OCLC Closed
relates to UXPROD-3081 NFR: UI end-to-end tests for Data Imp... Closed
relates to UXPROD-850 Migration Tools Open
relates to FAT-117 Data Import: Define Application Workf... Closed
relates to UIDATIMP-656 Update stripes to v5 Closed
relates to FAT-27 SPIKE: Investigate if API Karate test... Closed
relates to FAT-65 ui-data-import: UI tests replacement ... Closed
relates to FAT-66 ui-plugin-find-import-profile: UI tes... Closed
relates to UXPROD-559 Migrate Bibliographic & Holdings data Closed
relates to UXPROD-949 Rewrite Cornell's Web based LS-Tools Closed
relates to UXPROD-1220 Ability to assign Tags to Data Import... Closed
relates to UXPROD-600 Workflow App with To-Do App Integration In Review
Epic Name: Batch Importer (Bib/Acq)
Development Team: Folijet
UX Lead: Filip Jakobsen
Score: 47.25
Start date (migrated):
End date:
Epic Color: ghx-label-7

 Description   

Allow for the batch upload and importing of MARC metadata, holdings, and authority records, to affect various records in FOLIO (inventory instance, inventory holdings, inventory items, MARCcat bib record, MARCcat authority record, order record, invoice record). Authority capabilities have dependency on Authority support in general (see UXPROD-787 Open ).

Frontend: mod-acqcat-loader (UIDATIMP)
Backend: ui-acqcat-loader (MODDATAIMP)

Feature from v1 roadmap: Import data: Batch Import of metadata records
Note that holdings data may be embedded in the bibliographic records or separate files.

Older post: https://discuss.folio.org/t/page-facing-up-arrow-right-zap-august-31st-meeting-batch-import-discussion/1166
Workshop #1 (12/18/2018) - see documentation on Google drive
Source: https://discuss.folio.org/t/page-facing-up-zap-importing-marc-approaches-english/1389

[A-M: replace all the following with links to the Migration loader epic and features once created]

The following requirements to become features depending on outcome of meetings in MM and SysOps and are also documented here:
(https://docs.google.com/document/d/1oXbEE48zd889lGD87dP7cF3GfuuKwllI_MDp2zGwTRg/edit):
The loader to handle initial data migration and population of FOLIO for instance, holdings, and item data must meet the following requirements:

  • User needs proper roles and permissions to access needed files and programs.
  • Must be able to accept bibliographic data, possibly with embedded holdings and items, in Standard MARC-21, MARCXML or already mapped JSON file. Flat files for item data must also be accepted. (Common with Batch Loader, need to be able to ingest MARCXML or binary MARC.) - UXPROD-665 Closed
  • The loader must be able to map data from MARC tags and fields correctly to the relevant fields in FOLIO inventory module. In mapping MARC/MARCXML, we should use only one transformation method, common to migration and batch loading. (Common with Batch Loader)
  • When the source data is extracted from the previous system as a separate record, whether in JSON or in the MARC Format for Holdings Data (MFHD), the loader must be able to map the holdings appropriately in the FOLIO system, preserving links among holdings, instances, and items. (Common with Batch Loader-preserve links among bibs, holdings, items) UXPROD-665 Closed , UXPROD-659 Closed , UXPROD-658 Closed
  • We should only need to load records once and the loader should arrange for the data to land in all the right places: inventory module, storage module, and MARCcat.
  • Administrative metadata for bib, holding, and item records must be preserved. This includes but is not limited to information such as staff-only flag, fast-add flag (for bibs or items), create date and operator, last updated date and operator, bibliographic record status, status change date and operator, notes on the different records. These data are not typically contained in the bib/holdings/item records themselves but are associated metadata about those records.
  • The loaders must retain the legacy systems’ req key IDs and map them to the relevant FOLIO UUID. Depending on the legacy system, mapping may be required to link instance, holdings and item records in FOLIO based on the legacy ID numbers.
  • The loader should allow the user the choice of how to create UUIDs: Either they will be transferred from the designated field in the source data and retained in that form; or a new UUID is created in FOLIO and linked back to the legacy ID and creates the corresponding UUIDs. (Common with Batch Loader-those batch operations will always want to preserve the UUIDs, UUIDs must be persistent once established.)
  • If a user chooses to continue to use their source IDs, they have to be able to specify both the pattern and the starting point for the IDs. (Common with Batch Loader)
  • The loaders must create a datastore for both the incoming source data and the FOLIO target data.???? Q: is this only for the inventory module or for all the places the Finventory data is relevant. (Common with Batch Loader-the full MARC record must be preserved when desired.)
  • The loader must perform quickly and efficiently, so that [4,000,000 bibs] can be loaded [per hour] (Common with Batch Loader)
  • The loader requires a CLI. The interface should allow the kick off or scheduling of a load process. (Common with Batch Loader)
  • The loader should also provide User Interface and should allow the kick off or scheduling of a load process, view progress, see status, see errors and issues. (Common with Batch Loader)
  • The loader should allow the ability to truncate the database, and control when indexes get built.
  • Logging and specific error reporting is required. There should be a way to subscribe to these notification. Exception logging should have a way of linking to the source file. (Common with Batch Loader)
    *Exceptions:
  • If the source data fails to load, the record should be written to an exceptions file where it could be corrected and reloaded.


 Comments   
Comment by Hkaplanian [ 17/Jul/18 ]

Connecting these two until it can be determined if there are any diffferences.

Comment by Ann-Marie Breaux (Inactive) [ 19/Sep/19 ]

Jakub Skoczen Did you mean to steal reassign my Data Import epic to yourself? OK if I change it back to me??

Comment by Charlotte Whitt [ 29/Sep/20 ]

Hi Ann-Marie Breaux - looking through the list of features defining UXPROD-47 Analysis Complete , then maybe you want also to add: UXPROD-2659 Closed
NFR: Refactor data-import flow to increase reliability

Comment by Ann-Marie Breaux (Inactive) [ 29/Sep/20 ]

Thanks for catching that, Charlotte Whitt I've added the link, plus checked all the rest of my open features to make sure they are linked properly.

Generated at Fri Feb 09 00:05:23 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.