OAI-PMH data harvesting (Orchid)

Owned by Artem Vronik
Last updated: Apr 24, 2023 by Roman_Fedynyshyn
12 min read

Overview  

The purpose of the OAI-PMH tests is to measure the performance of the Orchid release and to find possible issues, bottlenecks per PERF-410 - Getting issue details... STATUS

Environment


VersionCPU

Memory/MemoryReservation

XmxMaxMetaSpaceSizeMetaspaceSizeTasks CountTask Rev Number
mod-oai-pmh(tick)mod-oai-pmh:3.11.320482048/18451440m512m384m212
edge-oai-pmh(tick)edge-oai-pmh:2.6.110241512/13601440m512m384m25
mod-inventory-storage(tick)mod-inventory-storage:26.0.010242208/19521440m512m384m24
mod-source-record-storage(tick)mod-source-record-storage:5.6.52048

5600/5000


3600m512M384M24
okapi

(error)okapi:5.1.0-SNAPSHOT.1352

okapi:5.0.1

10241684/1440922m512m384m32

Summary

  • Tests were executed by JMeter script in the AWS Carrier-io instance.
  • The average response time per request with a resumption token is 604 ms ( compared to Nolana's 646 ms).
  • 'OutOfMemory' exceptions (OOM) were detected at all times when tests were run without throughput limits (w/o 40 requests/min). Tests with 40 requests/min (which in Morning Glory proved to be successful 100% of the time) had positive results but the integrity of the harvesting was not checked. OOMs seen in Morning Glory and Nolana releases are also seen here.
  • 'Thread blocked' exceptions were detected only during the investigation of a custom branch (https://github.com/folio-org/mod-oai-pmh/tree/vert-timeout-rollback) with vert timeout manipulation.

Recommendations & Jiras

  • for providing additional confirmation, execute OAI-PMH full harvesting by EBSCO Harvester from AWS ec2 instance for avoiding location factors PERF-510 - Getting issue details... STATUS ;
  • PERF-449 - Getting issue details... STATUS potentially, might be closed and re-open in case of similar issues will be detected by EBSCO Harvester testing;
  • create a ticket for the Firebird Team for resolving 'mod-oai-pmh' OOM issues and performance optimization of harvesting functionality.   
  • PERF-509 - Getting issue details... STATUS

Test Results

Test NumberrecordsSourcemaxRecordsPerResponseDurationTotal RequestsRequests/min

Average Response time,

ms

Completed
1Source record storage100~25 h92'04039.5604(warning)No issues 
2Source record storage and Inventory100~9 h19'88938.8488(warning)No issues 

(warning) - the integrity of the harvesting was not checked

Response metadata:

Test #1
{
    "requestMetadataCollection": [
        {
            "requestId": "5457c24a-d3c6-4807-8b9e-9a0123802820",
            "lastUpdatedDate": "2023-04-03T06:12:59.807+00:00",
            "streamEnded": true,
            "downloadedAndSavedInstancesCounter": 10023100,
            "failedToSaveInstancesCounter": 0,
            "returnedInstancesCounter": 9160018,
            "skippedInstancesCounter": 45682,
            "failedInstancesCounter": 0,
            "suppressedInstancesCounter": 0
        }
    ],
    "totalRecords": 1
}


Test #2
{
    "requestMetadataCollection": [
        {
            "requestId": "0c2ef1f4-ab94-450a-b092-723ffc5338d6",
            "lastUpdatedDate": "2023-04-03T18:39:29.738+00:00",
            "streamEnded": true,
            "downloadedAndSavedInstancesCounter": 10023100,
            "failedToSaveInstancesCounter": 0,
            "returnedInstancesCounter": 1957773,
            "skippedInstancesCounter": 33027,
            "failedInstancesCounter": 0,
            "suppressedInstancesCounter": 0
        }
    ],
    "totalRecords": 1
}

For additional investigation of the behaviour of OAI-PMH operations with the throughput limitation of 40 requests/min, a series of test executions have been run on the local machine:

IdrecordsSourceThroughputExecResultComments
1Source records storageNo limitslocalOutOfMemory
240/minlocalconnection lost ~12hNot representative due to the local machine has lost connection
3InventoryNo limitslocal504 Gateway Time-out
5Source record storage and InventoryNo limitslocaljava.net.SocketException: An established connection was aborted by the software in your host machineNot representative due to the local machine has lost connection

Moreover, for additional investigation of the behaviour of OAI-PMH operations, several versions of 'mod-oai-pmh' were tested:

Version of the mod-oai-pmhThroughputrecordsSourceResult
mod-oai-pmh:3.11.0No limitsSRSOutOfMemory
mod-oai-pmh:3.11.1No limitsSRSOutOfMemory
mod-oai-pmh:3.12.0No limitsSRSOutOfMemory
https://github.com/folio-org/mod-oai-pmh/tree/vert-timeout-rollbackNo limitsSRSThread blocked >> OOM

Resources Utilization: Test #1 - SRS > 92'040 requests

Service CPU Utilization

mod-oai-pmh can spike up to over 300% initially.  The other modules barely incur any high CPU utilization above their baselines. 

Service Memory Usage

No memory issues were detected.

Database CPU Utilization

The DB CPU utilization shows spikes in the first 30-60 minutes for transferring instances and is relatively even and smooth for the rest of the harvest. 

ocp2-db-01: in the case of excepting the first 30-60 minutes, we can see a tendency of slow CPU Utilization growth.

Resources Utilization: Test #2 - SRS+Inventory > 19'889 requests

Service CPU Utilization

'mod-oai-pmh' spiked up to less than 80% initially.  The other modules barely incur any high CPU utilization above their baselines. 

Service Memory Usage

No memory issues were detected.


Database CPU Utilization

The DB CPU utilization shows spikes in the first 30-60 minutes for transferring instances and is relatively even and smooth for the rest of the harvest. 


Incremental harvesting


Environment

Incremental harvesting testing were performed on a bit different environment (for most of a modules it's the same except mod-source-record-storage memory configuration for incremental harvesting it's less than for full harvesting env.5600/5000 vs 2048/1440 and xmx 3600 vs 1024)

For env configuration details please see : Appendix section

Summary

  • For all tests all data were harvested successfully;
  • There are a chance to get "504 Gateway Time-out" on initial call. Bug created MODINVSTOR-1052
  • Response times for harvesting calls is in average 650 -750ms;
  • No memory leak suspects were defined;
  • RDS CPU usage is on 12-13% level during harvesting and ±35-40% during data transferring.


Methodology

For testing incremental harvesting PTF team create JMeter script that contains initial oai-pmh call:

/oai?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=<apiKey>&from=<start-date>&until=<end-date>

and subsequent call:

oai/<APIKey>?verb=ListRecords&resumptionToken=<resumptionToken>


Time ranges and data set sizes: 

setfromto num of records
12023-04-072023-04-0725K
22023-04-012023-04-06590K
32023-03-28T18:00:00Z2023-04-06T18:00:00Z1M records




Test Results


test #Data setduration Record sourcestatus
125K3 min 50sSRSSuccess 
2590K1 hr 14minSRS

Success

31M2 hr 1minSRSSuccess
425K4 min 32 sSRS+inventorySuccess
5590K1 hr 7minSRS+inventorySuccess
61M2 hr 21 minSRS+inventorySuccess


Record source SRS resource usage




Record source SRS+inventory resource usage 

Service CPU usage - 25K- 500K instances harvesting

Service CPU usage - 1M instances harvesting

Service Memory usage - 25K- 500K instances harvesting

Service Memory usage - 1M instances harvesting

RDS CPU usage 25K- 500K instances harvesting

RDS CPU usage 1M instances harvesting


ERRORS

Below log from mod-oai-pmh containing 504 Gateway Time-out

 ERROR log mod-oai-pmh

13:07:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:07:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:35168 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:08:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:08:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:50660 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:08:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:08:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:50704 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:08:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:08:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:51752 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:08:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:08:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:51786 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:08:46 [] [] [] [] ERROR OaiPmhJsonParser Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')

at [Source: UNKNOWN; line: 1, column: 2]

13:08:46 [] [] [] [] ERROR OaiPmhJsonParser Error position at error part of json is 2

13:08:46 [] [] [] [] ERROR OaiPmhJsonParser <html><head><title>504 Gateway Time-out</title></head><body><center><h1>504 Gateway Time-out</h1></cen

13:08:46 [] [] [] [] WARN OaiPmhImpl getOaiRecords:: RequestId b3eca50d-eba2-4bfd-b9a4-bad773caa434 completed with error Got error response from inventory-storage, uri: '/inventory-hierarchy/updated-instance-ids?onlyInstanceUpdateDate=false&endDate=2023-04-06T18:00:01Z&deletedRecordSupport=true&startDate=2023-03-28T18:00:00Z&skipSuppressedFromDiscoveryRecords=true' message: Invalid response: Gateway Time-out null

13:08:46 [] [] [] [] INFO LogUtil null GET /oai/records verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=eyJzIjoiVDNUSzAzR2QyViIsInQiOiJmczA5MDAwMDAwIiwidSI6ImZzMDkwMDAwMDAifQ==&from=2023-03-28T18:00:00Z&until=2023-04-06T18:00:00Z HTTP_1_1 500 295 400029 tid=fs09000000 Internal Server Error

13:09:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:09:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:44342 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:09:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:09:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:44360 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:09:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:09:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:55432 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:09:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:09:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:55472 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:10:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:10:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:52508 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:10:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:10:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:52528 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:10:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:10:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:59294 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:10:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:10:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:59350 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:11:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:11:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:37806 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:11:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:11:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:37818 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:11:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:11:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:40848 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:11:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:11:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:40868 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:12:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:12:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:56864 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:12:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:12:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:56888 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:12:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:12:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:55784 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:12:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:12:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:55814 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:13:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:13:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:49750 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:13:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:13:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:49768 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:13:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:13:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:45428 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:13:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:13:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:45448 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:14:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:14:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:44786 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:14:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:14:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:44816 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:14:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:14:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:34882 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:14:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:14:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:34898 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:15:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:15:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:36636 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:15:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:15:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:36660 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:15:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:15:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:38540 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:15:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:15:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:38574 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:16:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:16:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:42914 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:16:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:16:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:42928 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:16:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:16:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:38624 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:16:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:16:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:38650 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:17:13 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:17:13 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:35940 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:17:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:17:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:35968 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:17:43 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:17:43 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:40316 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:17:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:17:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:40356 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:18:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:18:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:46326 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:18:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:18:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:46366 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:18:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:18:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:55804 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:18:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:18:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:55826 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:19:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:19:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:50686 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:19:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:19:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:50708 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:19:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:19:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:60374 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:19:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:19:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:60410 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:20:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:20:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:40474 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:20:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:20:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:40504 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:20:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:20:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:32922 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:20:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:20:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:32958 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:21:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:21:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:37584 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:21:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:21:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:37598 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:21:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:21:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:41464 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:21:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:21:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:41480 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:22:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:22:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:38756 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:22:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:22:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:38786 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:22:18 [529153/oai] [fs09000000] [2ac43d6a-3945-4d99-87ee-7a8281238572] [mod_oai_pmh] INFO RestRouting invoking getOaiRecords

13:22:18 [529153/oai] [fs09000000] [2ac43d6a-3945-4d99-87ee-7a8281238572] [mod_oai_pmh] INFO OaiPmhImpl getOaiRecords:: parameters verb: ListRecords, identifier: null, resumptionToken: null, from: 2023-03-28T18:00:00Z, until: 2023-04-06T18:00:00Z, set: null, metadataPrefix: marc21_withholdings

13:22:18 [529153/oai] [fs09000000] [2ac43d6a-3945-4d99-87ee-7a8281238572] [mod_oai_pmh] INFO OaiPmhImpl getOaiRecords:: Use org.folio.oaipmh.processors.MarcWithHoldingsRequestHelper for requestId 0309d978-374a-44e5-b308-8b9d70b98c51

13:22:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:22:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:49432 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:22:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:22:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:49464 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:23:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:23:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:56854 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:23:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:23:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:56890 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:23:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:23:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:34168 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:23:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:23:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:34184 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:24:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:24:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:56148 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:24:14 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:24:14 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:56176 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:24:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:24:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:47836 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:24:44 [] [] [] [mod_oai_pmh] INFO RestRouting invoking getAdminHealth

13:24:44 [] [] [] [mod_oai_pmh] INFO LogUtil 127.0.0.1:47872 GET /admin/health null HTTP_1_1 200 4 0 tid=null OK

13:25:11 [] [] [] [] INFO LogUtil 127.0.0.1:60144 GET /oai/records verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=eyJzIjoiVDNUSzAzR2QyViIsInQiOiJmczA5MDAwMDAwIiwidSI6ImZzMDkwMDAwMDAifQ==&from=2023-03-28T18:00:00Z&until=2023-04-06T18:00:00Z HTTP_1_1 200 513520 172669 tid=fs09000000 OK

13:25:12 [135497/oai] [fs09000000] [2ac43d6a-3945-4d99-87ee-7a8281238572] [mod_oai_pmh] INFO RestRouting invoking getOaiRecords

13:25:12 [135497/oai] [fs09000000] [2ac43d6a-3945-4d99-87ee-7a8281238572] [mod_oai_pmh] INFO OaiPmhImpl getOaiRecords:: parameters verb: ListRecords, identifier: null, resumptionToken: bWV0YWRhdGFQcmVmaXg9bWFyYzIxX3dpdGhob2xkaW5ncyZmcm9tPTIwMjMtMDMtMjhUMTg6MDA6MDBaJnVudGlsPTIwMjMtMDQtMDZUMTg6MDA6MDBaJm5leHRJbnN0YW5jZVBrVmFsdWU9NzM0MTEyNTA1Jm9mZnNldD04NiZyZXF1ZXN0SWQ9MDMwOWQ5NzgtMzc0YS00NGU1LWIzMDgtOGI5ZDcwYjk4YzUxJm5leHRSZWNvcmRJZD0wMDA0ZGMwMS00YzlkLTQyMGUtYTYzNC05NWZhMzA1YjVhYWQmZXhwaXJhdGlvbkRhdGU9MjAyMy0wNC0wOFQxMzoyNToxMVo, from: null, until: null, set: null, metadataPrefix: null

13:25:12 [135497/oai] [fs09000000] [2ac43d6a-3945-4d99-87ee-7a8281238572] [mod_oai_pmh] INFO OaiPmhImpl getOaiRecords:: Use org.folio.oaipmh.processors.MarcWithHoldingsRequestHelper for requestId 0309d978-374a-44e5-b308-8b9d70b98c51

13:25:12 [] [] [] [] INFO LogUtil 127.0.0.1:57774 GET /oai/records verb=ListRecords&apikey=eyJzIjoiVDNUSzAzR2QyViIsInQiOiJmczA5MDAwMDAwIiwidSI6ImZzMDkwMDAwMDAifQ==&resumptionToken=bWV0YWRhdGFQcmVmaXg9bWFyYzIxX3dpdGhob2xkaW5ncyZmcm9tPTIwMjMtMDMtMjhUMTg6MDA6MDBaJnVudGlsPTIwMjMtMDQtMDZUMTg6MDA6MDBaJm5leHRJbnN0YW5jZVBrVmFsdWU9NzM0MTEyNTA1Jm9mZnNldD04NiZyZXF1ZXN0SWQ9MDMwOWQ5NzgtMzc0YS00NGU1LWIzMDgtOGI5ZDcwYjk4YzUxJm5leHRSZWNvcmRJZD0wMDA0ZGMwMS00YzlkLTQyMGUtYTYzNC05NWZhMzA1YjVhYWQmZXhwaXJhdGlvbkRhdGU9MjAyMy0wNC0wOFQxMzoyNToxMVo HTTP_1_1 200 542951 479 tid=fs09000000 OK

13:25:13 [744834/oai] [fs09000000] [2ac43d6a-3945-4d99-87ee-7a8281238572] [mod_oai_pmh] INFO RestRouting invoking getOaiRecords

13:25:13 [744834/oai] [fs09000000] [2ac43d6a-3945-4d99-87ee-7a8281238572] [mod_oai_pmh] INFO OaiPmhImpl getOaiRecords:: parameters verb: ListRecords, identifier: null, resumptionToken: bWV0YWRhdGFQcmVmaXg9bWFyYzIxX3dpdGhob2xkaW5ncyZmcm9tPTIwMjMtMDMtMjhUMTg6MDA6MDBaJnVudGlsPTIwMjMtMDQtMDZUMTg6MDA6MDBaJm5leHRJbnN0YW5jZVBrVmFsdWU9NzM0MTEyNjA1Jm9mZnNldD0xNzkmcmVxdWVzdElkPTAzMDlkOTc4LTM3NGEtNDRlNS1iMzA4LThiOWQ3MGI5OGM1MSZuZXh0UmVjb3JkSWQ9MDAwOWQ1N2ItMGY0Yy00MzQ0LWI0MDMtYTJiMGRjNjYxN2JiJmV4cGlyYXRpb25EYXRlPTIwMjMtMDQtMDhUMTM6MjU6MTJa, from: null, until: null, set: null, metadataPrefix: null


Appendix

Environment configuration for incremental harvesting:


VersionCPUMemory/MemoryReservationXmxMaxMetaSpaceSizeMetaspaceSizeTasks Count
mod-oai-pmh(tick)mod-oai-pmh:3.11.320482048/20001440m512m384m2
edge-oai-pmh(tick)edge-oai-pmh:2.6.110241512/13601440m512m384m2
mod-inventory-storage(tick)mod-inventory-storage:26.0.010242208/19521440m512m384m2
mod-source-record-storage(tick)mod-source-record-storage:5.6.520482048/14401024m512M384M2
okapi

(tick)okapi:5.0.1

10241684/1440922m512m384m3