Import of MARC Bibs to create/update multiple Holdings and Items

Owned by Kateryna Senchenko
Last updated: Oct 08, 2023
5 min read

UXPROD-2741 - Getting issue details... STATUS

1. Create MARC Bib, create Instance and Multiple Holdings, Items

Description

Save MARC_BIB + create Instance + create multiple Holdings, Items.

Multiple Holdings and/or Items can be create out of one MARC Record, which is mapped in a single Instance entity, information for mapping multiple Holdings and/or Items are stored in repeatable MARC fields (MARC fields can be different, but always the same for each record in a given file)

Sample record with multiple holdings/item data:

=LDR 01262nam a2200301Ia 4500
=001 ocm54341618\
=003 OCoLC
=005 20070103101904.0
=008 010330s1798\\\\enkaf\\\\\\\\\000\0\eng\d
=035 \\$a(Sirsi) a551407
=035 \\$a(Sirsi) o54341618
=049 \\$aDRUM
=040 \\$aCUD$beng$cCUD$dDRU$dMvI
=090 \\$aBS2095$b.S33 1798
=130 0\$aBible.$pNew Testament$lEnglish.$sScarlett.$f1798.
=245 12$aA translation of the New Testament from the original Greek /$chumbly attempted by Nathaniel Scarlett, assisted by men of piety & literature ; with notes.
=260 \\$aLondon :$bPrinted by T. Gillet; and sold by Nathaniel Scarlett, No. 349, near Exeter 'Change, Strand; also F. & C. Rivington, St. Paul's Church Yard,$c1798.
=300 \\$axi, 483, vi p., [1] folded leaf of plates :$bill. ;$c19 cm.
=500 \\$aEngraved t.p.
=500 \\$aIncludes Observations on some terms used in this translation: vi p. at end.
=510 3\$aDarlow-Moule-Herbert 1433
=700 1\$aScarlett, Nathaniel,$d1753-1802.
=740 02$aObservations on some terms used in this translation.
=945 \\$a34678234678246423786427$b1$hKU/CC/DI/M <===== $a = item barcode, $b = copy number, $h = holdings permanent location
=945 \\$a34678234678246423786428$b2$hKU/CC/DI/M
=945 \\$$a34678234678246423786429b1$hKU/CC/DI/A


Job Profile is not changing - same Mapping + Action profiles are arranged in the JobProfile. In case of multiple Holdings creation Items should be associated with related Holdings based on the information provided in incoming record (Holdings info stored in the same field)


NB: Save MARC_BIB action is implicit and is executed prior to all other actions specified in the profile.

Flow

  1. [1-4] UploadDefinition is created for importing files in mod-data-import
  2. [2-3] JobExecution is created in mod-srm
  3. [5-7] MARC files are uploaded from WEB client to mod-data-import
  4. [6] Uploaded file is stored in memory of mod-data-import
  5. [8-9] User selects the JobProfile and initiates the processing of the uploaded files
  6. [10-11] Profile ID is set for JobExecution
  7. [12] MARC records are packed into batches and put to Kafka queue DI_RAW_RECORDS_CHUNK_READ
  8. [13-15] mod-srm reads batches from the queue, parses MARC records and passes to mod-srs via Kafka queue DI_RAW_RECORDS_CHUNK_PARSED. JobExecution is set to IN_PROGRESS status on first chunk received.
  9. [16-17] mod-srs stores records into PostgreSQL database and returns the result back via Kafka queue (broken records are also stored as 'error record') - DI_PARSED_RECORDS_CHUNK_SAVED
  10. [18-19] mod-srm reads the profile and creates JSON payload (containing parsed MARC) for processing. Exports it to an appropriate Kafka queue (one message per MARC entry) - DI_SRS_MARC_BIB_RECORD_CREATED
  11. [20-22] mod-inventory reads the message, creates Instance. Stores (via OKAPI HTTP) in mod-inventory-storage. At this point Instance UUID and Instance HRID are generated. Exports message DI_INVENTORY_INSTANCE_CREATED_READY_FOR_POST_PROCESSING
  12. [25-30] The post-processing stage: mod-srs reads message and updates according entry (Instance HRID is set to '001' MARC_BIB field, the value from '001' + '003' is moved to '035', Instance ID is set to '999 ff i' field). Creates new messages with updated payload DI_SRS_MARC_BIB_INSTANCE_HRID_SET. mod-inventory reads message from DI_SRS_MARC_BIB_INSTANCE_HRID_SET and updates Instance with updated identifiers fields - no new messages are sent at this point.
  13. [31] At the same time as in step [27] mod-srs sends DI_INVENTORY_INSTANCE_CREATED event
  14. [32-34] mod-inventory reads the message from DI_INVENTORY_INSTANCE_CREATED, maps (multiple if there is data for multiple Holdings supplied in the MARC record) and creates Holdings. For each permanent location found in the specified field there will be one Holdings created. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_HOLDING_CREATED
  15. [37-40] mod-inventory reads the message from DI_INVENTORY_HOLDING_CREATED, maps and creates multiple Items. Stores (via OKAPI HTTP) in mod-inventory-storage. Exports message to DI_INVENTORY_ITEM_CREATED
  16. [43-44] mod-inventory reads message and exports to DI_COMPLETED
  17. [44-45] mod-srm receives DI_COMPLETED event, marks JobExecution as COMPLETED

Diagram

Source


All mapped entities (Holdings or Items) are created at the same stage - Create Holdings or Create Items respectively. Event payload for DI_INVENTORY_HOLDING_CREATED, DI_INVENTORY_ITEM_CREATED, DI_COMPLETED may contain lists of Holdings/Items in the context that were successfully created at this stage of processing the profile. If at least one of the mapped entities was successfully created, DI process continues, therefore a "success" event is published. For entities that failed to be created - an error message is added to the list of error messages in the payload context of mentioned above events. If none of the mapped entities was created - DI_ERROR event is published to stop further processing of the profile. 

To enable described functionality:

  • data-import-processing-core library should be changed to allow for mapping of multiple entities from a single supplied MARC Bib record
  • mod-inventory Create Holdings and Create Items should be changed to process multiple entities at one stage and fill in relevant information about created/failed entities in the event payload context
  • mod-source-record-manager should contain changes in Journaling and Progress tracking functionality to fill in information about multiple entities and mark record as errored in case at least one of the entities failed to be created
  • mod-orders should be changed to fill in Locations and Material type in POLine based on multiple entities present in the DI_ORDER_CREATED_READY_FOR_POST_PROCESSING event payload  

Notes:

  1. All mapped Holdings from a specific MARC Bib are created on the stage of creating Holdings [32-37], all mapped Items are created on the stage of creating Items [38-43] and each should be associated with a corresponding Holdings. Different Mapping and Action profiles are used for creating Holdings and Items. When Holdings are mapped - a list of Holdings differentiating values (locations) need to be stored in the DataImportEventPayload and used at the stage of Item creation in mod-inventory to link Items to appropriate Holdings before saving in mod-inventory-storage.


2. Match on MARC Bib/Instance and update Multiple Holdings, Items

This flow might change when requirements are finalized. Poppy implementation supports the following flow

Description

Match MARC_BIB/Instance + sub-match on Holdings/Items + update Holdings/Items 

In case multiple Holdings and/or Items linked to one Instance, information for matching multiple Holdings and/or Items can be supplied in repeatable MARC fields and be used in sub-match section. For matched multiple Holdings/Items the same action will be applied. In case profile contains branching (different actions depending on the result of the match) the non-match branch will be followed only if nothing is matched. If at least one entity is matched - the match branch will be followed. Entities that were not matched among the multiple matches will be indicated in the DI log with No Action status.