Data Import FAQ

Owned by Ann-Marie Breaux (Deactivated)
Last updated: Dec 19, 2023
6 min read

Q: What types of records does Data Import work with / support?

The following types of records can be imported via Data Import:

  • MARC Bibliographic records
  • MARC Authority records
  • MARC Holdings records
  • EDIFACT invoice records

The following types of FOLIO records can be acted upon via Data Import:

  • Inventory Instances
  • Inventory Holdings
  • Inventory Items
  • Invoices and Invoice lines (via EDIFACT)
  • Source Record Storage MARC Bibliographic records
  • Source Record Storage MARC Authority records
  • Source Record Storage MARC Holdings records
Q: Where can I find demonstration videos to see how data import works in real time?

A: Yes, videos on various data import topics have been created. You can access them here. A spreadsheet at the top of the folder lists the videos and what they demonstrate. Some sample files are also included. If you have requests for other videos, please contact Ann-Marie Breaux (Deactivated).

Q. Does upper or lower-case matter in a file name or file extension?

A. No, Data import treats upper/lower-case as the same in file names and file extensions

Q. What happens when a MARC field/subfield is mapped to an Inventory record field with authorized values (a validation list), and the incoming data is not valid?

A. Here are a few examples of Inventory fields with authorized values: Permanent or temporary location, Statistical code, Call number type, Material type, Permanent or temporary loan type. If there is a fallback value in the field mapping profile (e.g. 980$a; else "Main (KU/CC/DI/M)"), then the record will be created/updated and the fallback value will be used. If there is not a fallback value, then the record will NOT be created or updated (regardless of whether the field is required or optional), and the user will see an error message in the Data Import log: Provided field name is not a valid value.

Q. Is there a way to remove records from Source Record Storage?

A. Yes, an endpoint was created to allow for the complete emptying of SRS or to remove records associated with a single import job. You can find more details here: How to delete Records in SRS created by specific load and in this readme. Warning: After removal of the target records in SRS, related Instance (from Inventory) doesn't get removed. The Instance remains in Inventory and does not know that the SRS MARC has been removed. The Instance's SOURCE field is still equal to MARC. As a result, the 'View source' and quickMARC actions continue to be displayed. This behavior may be reviewed and refined in the future.

Q: Where can I find documentation for the MARC field mapping syntax?

Documentation is beginning to be gathered here: MARC Mappings Information

Q. In the Source Record Storage data for each MARC record, there's a "rawRecord" which contains the MARC record in blob form. Is that JSON or raw MARC? And if an edit is done in quickMARC, does that affect the rawRecord as well as the parsed JSON record? Why isn't the rawRecord returned in response to queries to /source-storage/source-records?

A. The rawRecords "blob" is not updated via quickMARC. The rawRecords can be in different formats, and are less consistently-formatted than the parsed JSON MARC information. They cannot be retrieved in response to SRS queries. If external systems need to ingest records from FOLIO SRS, then more consistent outputs such as MARC Export data or OAI-PMH extracted data or the SRS parsed records are more reliable than the raw SRS records.

Q. I'm editing a fund distribution in an order field mapping or an adjustment in an EDIFACT invoice field mapping. Why does it look like this? "100"%?

A. In the field mapping create/edit screen, there are separate fields for the Value of the fund distribution or adjustment and the Type (percentage/currency). In the view screen, there are not separate fields. In the view screen, FOLIO displays both of those elements in one field. On the view screen, the field mapping is displayed as the value in double quotations, followed by the type indication outside of the quotation marks. Also note that if FOLIO does not have a symbol for the particular currency, the currency's 3-character code will be displayed.

Create/edit screen:

View screen with percentage:

View screen with currency (symbol known):

View screen with currency (symbol unknown):


Q. When a library upgrades to a new version of Data Import (or more specifically, new versions of the modules that comprise Data Import, does the library need to load the sample data for any of the Data Import modules (e.g. mod-data-import, mod-data-import-converter-storage, mod-data-import-processing-core. mod-source-record-manager, mod-source-record-storage) If not, how do new default profiles get added?

A. There is no need to do anything specific. Default profiles are loaded automatically once a module is enabled for a tenant. New profiles added in a newer release version are also created during the module upgrade with no additional work from the users doing the upgrade.

Q. Data in an Inventory Instance that is based an underlying SRS MARC Bibliographic record is refreshed against the tenant's most recent MARC Bibliographic-to-Inventory Instance map. What triggers the refresh of an Instance with Source = MARC?

A. The following actions trigger such a refresh

  • Saving edits to the SRS MARC Bib made via quickMARC
  • A new or updated MARC Bib is retrieved from a remote source using Inventory Single Record Import
  • An Update SRS MARC Bib action is triggered in a Data Import job profile
  • An Update Instance action is triggered in a Data Import job profile
Q. If a tenant's default MARC Bibliographic-to-Inventory Instance map has been updated, is there a way to refresh all Instances against the updated map? 

A. Yes, as of Kiwi

Q. Are there any best practices for moving SRS bibs from one FOLIO tenant to another?

It is common practice to use an export - edit - reload workflow to maintain metadata with Data Import and Export.

Files may also be exported from one FOLIO tenant (library) and then imported to a different FOLIO tenant. The MARC 999 ff field will contain the UUIDs for the first tenant's SRS MARC record and for the Inventory Instance. Users are encouraged to follow this best practice: manually remove the 999 ff field from the exported file before importing to the second tenant, so that the second tenant can create a new 999 ff with completely unique SRS and Instance UUIDs. If not removed, it will not create a problem within either tenant, but (as of August 2020), the SRS UUID will be the same for both tenants. This can be revisited and refactored if significant issues are identified.

Q. Please explain the Administrative data in source-record-storage for the MARC records (not the MARC data, the non-MARC data associated with the MARC record)

A. See below for each field in orange, sample data in green, and explanation in black.

"id": "908de854-977e-4fc4-b99b-438af3eed2a1"  is the UUID for that particular generation of the record. For generation 0 it matches the 999 ff $s and the matchedId. For every generation after 0, it’s a new UUID (so that the incrementing copies of the SRS MARC can be organized)

"snapshotId": "cf7436ec-fa78-485a-b154-7f1545233d01"  (same as jobExecutionId): UUID is analogous to the Job UUID in the Log UI. 

"matchedId": "908de854-977e-4fc4-b99b-438af3eed2a1" the UUID that’s in 999 ff $s and an id for record with generation 0; it does not change; helps to keep the versions of the same SRS MARC Bib organized and related

"generation": 0 generation: starts at 0 and increments. Generation is set for each record before storing it in the database. A previous version of this record is checked, and if it exists, the generation number is incremented by 1 before being saved to the database.

"recordType": "MARC"

"rawRecord": { "id": "908de854-977e-4fc4-b99b-438af3eed2a1" contains the raw MARC record from the import file, in a blob

"content": ... 

"parsedRecord": { "id": "908de854-977e-4fc4-b99b-438af3eed2a1" contains the parsed MARC record, in MARC fields

"content": "fields":

"001": "in00000000003" Instance HRID; when a MARC record is firs created in SRS, the corresponding Instance is created in Inventory, and then the Inventory Instance HRID and UUID are sent to SRS. SRS moves any existing 001 data to an 035 field, and precedes that number with any 003 data, in parentheses, e.g. 001 12345 and 003 OCoLC become 035 $a (OCoLC)12345. Then the Instance HRID is placed in the (now-empty) 001 field of the SRS MARC record.

All the other MARC data

"999": { "ind1": "f", "ind2": "f", "subfields": [ { "s": "908de854-977e-4fc4-b99b-438af3eed2a1" }, { "i": "5289d4af-d755-47e5-a5fe-e76737163bfa" } ] } } ]

"leader": "02598cam a2200505Ii 4500"  (The leader data is store at the end of the MARC record, instead of the beginning)

"deleted": false

"order": 2

"externalIdsHolder":

"instanceId": "5289d4af-d755-47e5-a5fe-e76737163bfa"

"instanceHrid": "in00000000003" 

"additionalInfo":

"suppressDiscovery": false 

"state": "ACTUAL"

"leaderRecordStatus": "c"

"metadata":

"createdDate": "2021-05-06T19:02:03.338+0000"

"updatedDate": "2021-05-06T19:02:48.885+0000" 


Questions that need updates/answers 

Q: Where can I find documentation for the EDIFACT invoice field mapping syntax?

Q: Are there any recommendations for vendors with regards to EDIFACT invoicing files?

Q. If I create a profile that only is supposed to be updating holdings and/or item records, are the Instance and SRS MARC records affected as well?

Q. In SRS, how are MARC Bibs, Authority, and Holdings records distinguished? Is it in the record's metadata someplace?

Q. If there are multiple versions of a MARC record in SRS, how can I tell which is the canonical/current version of the MARC record?

Q. When creating an EDIFACT invoice field mapping profile, can you have both default data (in double quotes "") and field mapping syntax together in one field? If so, is there a specific character necessary to join them?