Notes on data migration

Content note

This page is intended to share tips and experiences on data migration. Anyone can participate.

This inital description assumes migration from an Excel/CSV source file. There are numerous forms in which the source data can exist, e.g. in a relational database. Therefore there are also numerous ways of data transformation, e.g. via scripts which pull the data directly from a database (like ILS). However, in all cases, knowledge of the interrelationships within OA as attempted to be presented here may be necessary.

Leipzig examples

Notes in green boxes contain examples from the Leipzig implementation, which may not apply everywhere.

Important! UUIDs are very individual. Each Folio instance creates own UUIDs for settings and data. Preparing for migration on a system other than the target system results in the need to recreate the UUID references!

Not all possible relationships are listed here, e.g. links to Agreements are not considered for data migration (Leipzig).

Anyone with a Folio account on the system and the appropriate permissions can access this API endpoint. It is more a question of "how". An GUI API client such as Postman can be used.

Leipzig example

more data ...

{
        "id": "8aaa80fe859fe8ee0185a0ea5d980023",
        "desc": "Work.OaStatus",
        "internal": false,
        "values": [
            {
                "id": "8aaa80fe859fe8ee0185a0ea5d9b0024",    ← UUID for OA status "Gold"
                "value": "gold",
                "label": "Gold"
            },
            {
                "id": "8aaa80fe859fe8ee0185a0ea5da00025",    ← UUID for OA status "Hybrid"
                "value": "hybrid",
                "label": "Hybrid"
            },
            {
                "id": "8aaa80fe859fe8ee0185bac6b6c9004f",    ← UUID for OA status "Subscribe to Open"
                "value": "subscribe_to_open",
                "label": "Subscribe to Open (S2O)"
            }
        ]
    },

... more data

Direct database access is controlled by the system's administrator. A database tool like DBeaver can be used.

Leipzig example: Journal source file

Leipzig example: People

Leipzig example: Publication requests

This table include all data about Publication requests, Charges and Checklist items, including enriched reference values.
But it does not yet include BI:chargeOwnerId, which is the UUID of the Request itself.

Leipzig rexample: lookup table People eMail<->UUID

Leipzig example: Same source file like Publication requests but now including BI:chargeOwnerId

Leipzig example: Journal JSON mapping

{ "title": "SPALTE-A", "instances": [ { "ids": [ { "ns": "issn", "id": "SPALTE-B" } ], "subType": "electronic" }, { "ids": [ { "ns": "issn", "id": "SPALTE-C" } ], "subType": "print" } ], "indexedInDOAJ": "SPALTE-E", "oaStatus": "SPALTE-D", "type": "serial" }

Leipzig example: People JSON mapping

to do

Leipzig example: Request JSON mapping

{"publicationStatuses":[{"publicationStatus":{"id":"Y:publicationStatus1Id"},"statusDate":"Z:statusDate1","statusNote":"AA:statusNote1"},{"publicationStatus":{"id":"AB:publicationStatus2Id"},"statusDate":"AC:statusDate2","statusNote":"AD:statusNote2"},{"publicationStatus":{"id":"AE:publicationStatus3Id"},"statusDate":"AF:statusDate3","statusNote":"AG:statusNote3"},{"publicationStatus":{"id":"AH:publicationStatus4Id"},"statusDate":"AI:statusDate4"}],"fundings":[],"requestStatus":"A:requestStatus","closureReason":null,"requestDate":"B:requestDate","externalRequestIds":[{"externalId":"C:externalId1"},{"externalId":"D:externalId2"}],"correspondingAuthor":{"partyOwner":{"id":"F:correspondingAuthorPartyOwnerId","alternateEmails":[],"institutionLevel1":{"id":"L:correspondingInstitutionLevel1Id"}},"role":"G:correspondingAuthorRole"},"correspondingInstitutionLevel1":{"id":"L:correspondingInstitutionLevel1Id"},"useCorrespondingAuthor":false,"requestContact":{"partyOwner":{"id":"I:requestContactPartyOwnerId","alternateEmails":[]},"role":"J:requestContactRole"},"correspondingInstitutionLevel2":"M:correspondingInstitutionLevel2","doi":"N:doi","license":{"id":"O:licenseId"},"publicationTitle":"P:publicationTitle","publicationType":{"id":"Q:publicationTypeId"},"publisher":{"id":"S:publisherId"},"work":{"id":"U:workId"},"workIndexedInDOAJ":{"id":"X:workIndexedInDOAJId"},"workOAStatus":{"id":"V:workOAStatusId"},"requestNumber":"BH:requestNumber","subtype":null,"bookDateOfPublication":null,"bookPlaceOfPublication":null,"agreement":null}

Charge1 JSON mapping

{"tax":"AR:charge1Tax","discountType":{"id":"8aaa80fe859fe8ee0185a0ea5d5e0017"},"exchangeRate":{"coefficient":1,--->>>type:number"toCurrency":"EUR"},"amount":{"baseCurrency":"AQ:charge1AmountBaseCurrency","value":"AP:charge1AmountValue"},"category":{"id":"AN:charge1CategoryId"},"chargeStatus":{"id":"AO:charge1StatusId"},"description":"OA-Migration","discount":0,--->>>type:number"discountNote":"","paymentPeriod":"2023","payers":[{"payerAmount":AT:charge1PayersPayer1Amount,--->>>type:number"payer":{"id":"AS:charge1PayersPayer1Id"},"payerNote":"AU:charge1PayersPayer1Note"},{"payerAmount":AW:charge1PayersPayer2Amount,--->>>type:number"payer":{"id":"AV:charge1PayersPayer2Id"},"payerNote":"AX:charge1PayersPayer2Note"}],"owner":{"id":"BI:chargeOwnerId"}}

Checklist item JSON mapping

{"id":"BI:chargeOwnerId","checklist":[{"outcome":{"value":"AK:checklist1OutcomeValue"},"notes":[],"definition":{"id":"AJ:checklist1DefinitionId"}},{"outcome":{"value":"AM:checklist2OutcomeValue"},"notes":[],"definition":{"id":"AL:checklist2DefinitionId"}}]}