In Progress

Overview  

...

• Average response time per request with resumption token 600ms ( compared to Lotus's 850ms).
• Incremental calls performed - 82299  (Bugfest data set 1 user and 20 DB connections)*. 
• OOM happens frequently if followed the recommended setting (soft limit < maxMetaspaceSize + XmX).  Only when changed to soft limit > maxMetaspaceSize + Xmx the harvests completed successfully.Memory doesn't get released after the harvest is over. Potentially problematic if more harvests are run after it. (This issue is to be investigated by PTF.)
• Thread block errors and subsequent OOMs happened about 50% of the time. . This is likely due to a fast rate of incremental calls by the JMeter test script. When changed to 40 requests/min, there were no more errors, but this is a very small rate that would take over 30 hours for the harvest of 8M records to complete.

* Note: Bugfest dataset was used because it has more SRS records than PTF's dataset.

...

• The harvests stopped about an hour in 
• Logs show "Thread blocked" errors which lead to OOM.  The mod-oai-pmh tasks did not crash, however. 
• mod-oai-pmh service's memory and CPU utilization percentage were nominal, at about 70% for memory.
• It's worth noting that the mod-oai-pmh service was not restarted before these tests. Logs are attached. 

Image RemovedImage Added

Test 8

Based on a log of the EBSCO harvester, the rate of the incremental harvest request was anywhere from 9 to 47 requests/min.  In this test we slowed down the request rate to 40/min. This test and subsequent tests did not log any errors for the first couple of hours and even several hours later, which is typically when the thread-block and OOM issues occurred. 

Image Added

Note that the request rate when converted to requests/min equals around 40.02 req/min. This proved to be a point of stability.  This test was stopped short because had it gone on longer, it would have taken about 36 hours to fully harvest all the data at this rate. A Jira was created to improve performance of mod-oai-pmh:

Image Added

Database showing little CPU usage during the harvest.

Image Added

CPU utilization of relevant modules during the harvest.

Image Added

Memory utilization of the modules that involve in the OAI-PMH workflow.