Feature-level design for data-export Mapping Profiles

Technical solution

1. Retrieving list of field names. Create a specific endpoint in mod-data-export to retrieve a list of properties to display mapping profile transformations. JSON schemas of record types should be requested from mod-inventory-storage module to get a list of record fields through JSON schema API. Necessary schemes: holdingsrecords.json, instance.json, item.json.  To retrieve fields that are represented by keys (settings fields) it's required to send separate requests. For example, all Identifiers values can be retrieved by GET /identifier-types request.
2. Create a new set of entities for mapping profiles and appropriate CRUD endpoints.
3. Verify mapping profile reference to existing job execution profile when delete or update the mapping profile. If reference exists then deletion or updating should not be performed.
4. The mapping profile will be associated with the export job through the job profile. Each job profile can be associated with one mapping profile; each export job can be associated with one job profile. At the same time, one mapping profile can be associated with multiple job profiles.
5. Users should be able to assign mapping profile to job profile before the start of the export flow.
6. Retrieve specific records from mod-inventory module base on mapping profile record types (Instance, Holding Items).
7. To initialize and execute MappingProcessor specific MappingProfileReader and MappingProfileRule should be implemented and prepared base on the list of MappingField from MappingProfile. MappingFiled.path field value should be used to read the field value of retrieved inventory records.  MappingFiled.transformation should be parsed to set values of marc field tag, subfield and indicator. 
8. Specific Writer should initialized base on mapping profile output format to use it with MappingProcessor (JsonRecordWriter, MarcRecordWriter or XmlRecordWriter).
9. If mapping profile is enabled during export flow then default JSON rules will not be used for the mapping process and mapping profile rules will override the default ones using separate MappingProfileReader. Otherwise, default JSON rules and default JPathSyntaxEntityReader will be used for mapping.

Q2 release functionality limitations

• Mapping profile transformation options are not supported for fields of Instance type
• In case if Instance fields are enabled for a certain mapping profile then only default rules form defaultRules.json file are used for generating MARC bib record with Instance fields.
• Mapping profile transformation options are supported only for fields of Holdings and Item fields
• List of supported Holdings fields: 
  • Permanent location
  • Temporary location
  • Electronic access
  • Call number (with prefix and suffix)
• List of supported Item fields:
  • Effective call number
  • Effective location
  • Electronic access
• Multiple subfield transformation values are not supported. For example, like this: 900 $a $b $c
• Editing of mapping profiles is not supported

Reusing of the functionality by other teams

Other teams will be able to reuse mapping profiles functionality partially. Mapping profiles endpoints of mod-data-export can be used to create and receive mapping profile instances by other modules. At the same time, the functionality of mapping (MDEXP-59) will be moved to a sharable jar file to use by other modules. The logic of creation MappingProfileReader and MappingProfileRule can be also added to this jar file. Detailed design how to use MARC records that are generated on the fly from inventory records will be created in the scope of   MDEXP-84 - Getting issue details... STATUS

DB Scheme

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "description": "Mapping Profile schema",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "id": {
      "description": "Unique identifier",
      "$ref": "../common/uuid.json"
    },
    "name": {
      "description": "Mapping Profile name",
      "type": "string"
    },
    "description": {
      "description": "Mapping Profile description",
      "type": "string"
    },
	"recordTypes": {
      "description": "Mapping Profile records type",
	  "type": "array",
      "items": {
        "type": "string",
        "$ref": "mapping-profile/recordTypes.json"
      }
    },
    "mappingFields": {
      "description": "List of maping fields",
      "type": "array",
      "id": "mappingFields",
      "items": {
        "type": "object",
        "$ref": "mapping-profile/mappingField.json"
      }
    },
    "userInfo": {
      "description": "First name, last name and username of the user, who updated the Mapping Profile",
      "type": "object",
      "$ref": "../common/userInfo.json"
    },
    "outputFormat": {
      "description": "Mapping Profile output format",
      "type": "string"
    },
    "metadata": {
      "description": "Metadata provided by the server",
      "type": "object",
      "$ref": "../../raml-util/schemas/metadata.schema",
      "readonly": true
    }
  },
  "required": [
    "name",
    "recordTypes",
	"mappingFields"
  ]
}

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "description": "Mapping Field schema",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "id": {
      "description": "Unique identifier",
      "$ref": "../common/uuid.json"
    },
    "name": {
      "description": "Mapping Field name",
      "type": "string"
    },
    "description": {
      "description": "Mapping Field description",
      "type": "string"
    },
    "path": {
      "description": "JSON path to set mapped value",
      "type": "string"
    },
    "enabled": {
      "description": "Indicates whether field should be mapped",
      "type": "string"
    },
    "transformation": {
      "description": "Mapping expression",
      "type": "string"
    },
	"recordType":{
       "description": "Mapping Profile type",
       "type": "string",
       "$ref": "mapping-profile/recordTypes.json"
	}
  },
  "required": [
    "name",
    "recordTypes",
	"mappingFields"
  ]
}

{
  "id": "3798f546-7afb-11ea-bc55-0242ac130003",
  "name": "Example mapping profile",
  "description": "Example mapping profile description",
  "recordTypes": ["INSTANCE", "HOLDINGS", "ITEM"],
  "outputFormat": "marc",
  "mappingFields": [{
	 "id": "3798f546-7afb-11ea-bc55-0242ac130003",
	 "name": "Example mapping field",
	 "description": "Example mapping field description",
	 "path": "instanceName1.fieldName3",
	 "transformation": "001",

	 "enabled": "true",
	 "recordType": "INSTANCE"
  }],
  "userInfo": {
   "firstName": "User",
   "lastName": "User",
   "userName": "@username"
  },
  "metadata": {
   "createdDate": "2020-04-10T12:41:22.000",
   "createdByUserId": "",
   "createdByUsername": "@username",
   "updatedDate": "2020-04-10T12:41:22.000",
   "updatedByUserId": "",
   "updatedByUsername": "@username"
  }
}

Mapping profile endpoints

Mapping profiles sequence diagram