Async Migration. Guideline and Performance results
- Oleksii Kuzminov
PERF-212 - Getting issue details... STATUS
AsyncMigration API
An asynchronous mechanism has been added to migrate large amounts of data. Additional API has been added to the mod-inventory storage (link).
For now, this API supports "publicationPeriodMigration" and "itemShelvingOrderMigration", but it can be extended to support another kind of migrations very simply.
To retrieve a list of available migrations call GET /inventory-storage/migrations
{
"asyncMigrations": [
{
"migrations": [
"publicationPeriodMigration"
],
"affectedEntities": [
"INSTANCE"
]
},
{
"migrations": [
"itemShelvingOrderMigration"
],
"affectedEntities": [
"ITEM"
]
}
],
"totalRecords": 2
}
To start migration use POST /inventory-storage/migrations/jobs
{"migrations": ["itemShelvingOrderMigration"]}
To start several migrations in the scope of one job it’s possible to run a job with several migrations. In this case, each migration should have its own counter, but status and progress will be tracked on job level.
{"migrations": ["itemShelvingOrderMigration", "publicationPeriodMigration"]}
To check status or cancel job use GET | DELETE /inventory-storage/migrations/jobs/{id}
{
"id": "aa5a11b4-88c9-4372-82fb-7089b8b37c0f",
"migrations": [
"itemShelvingOrderMigration"
],
"affectedEntities": [
"ITEM"
],
"published": [
{
"migrationName": "itemShelvingOrderMigration",
"count": 20
}
],
"processed": [
{
"migrationName": "itemShelvingOrderMigration",
"count": 20
}
],
"jobStatus": "Completed",
"submittedDate": "2022-03-21T11:01:24.890+00:00",
"finishedDate": "2022-03-21T11:01:25.311+00:00"
}
Performance tests results for 1 node
Environment setup
Env: https://falcon.ci.folio.org/
Number of records: 1_000_000
Mod-Inventory-Storage setup: 1 node, resources:
limits: cpu: 192m memory: 864Mi requests: cpu: 128m memory: 536Mi
Results
{
"id": "8202b79d-11eb-44ce-aefa-f4489a20d3fb",
"name": "publicationPeriodMigration",
"affectedEntities": [
"INSTANCE"
],
"published": 1000000,
"processed": 1000000,
"jobStatus": "Completed",
"submittedDate": "2022-02-18T16:47:25.082+00:00",
"finishedDate": "2022-02-18T19:34:22.092+00:00"
}
Result: 2 hours, 47 minutes and 17 seconds
Performance tests results for 2 nodes
Environment setup
Env: https://falcon.ci.folio.org/
Number of records: 1_000_000
Mod-Inventory-Storage setup: 2 nodes, resources:
limits: cpu: 192m memory: 864Mi requests: cpu: 128m memory: 536Mi
Results
{
"id": "cb210eba-4fa3-4d59-8cda-9c15640e4138",
"name": "publicationPeriodMigration",
"affectedEntities": [
"INSTANCE"
],
"published": 1000000,
"processed": 1000000,
"jobStatus": "Completed",
"submittedDate": "2022-02-18T20:01:27.062+00:00",
"finishedDate": "2022-02-18T21:29:54.003+00:00"
}