Overview

• The purpose of the OAI-PMH Concurrent Incremental Harvesting tests is to measure performance of Poppy release and to find possible issues, bottlenecks PERF-786 - Getting issue details... STATUS on OCP3 environment.
• The previous results of Incremental OAI-PMH  PERF-660 - Getting issue details... STATUS

Summary

• OAI-PMH - Incremental Harvesting:

  • Three tests have been executed by JMeter script to check performance of harvesting the following number of records 10K, 25K, 50K, 500K and 1 MLN with different OAI-PMH Behaviors :

    • Test 1. Record source set to Source record storage ;

    • Test 2. Record source set to Inventory* (data set limit in OCP3 - 250k) ;

    • Test 3.  Record source set to Source record storage and inventory.
      

  • Number of multiple concurrent harvests:
    • 2 harvests;
    • 4 harvests;
    • 6 harvests.

• CPU utilization during all tests was relevant to number of concurrent harvests. 
  • Test #1 mod-oai-pmh-b: 2 harvests -   5%, 4 harvests - 10%,  6 harvests - 15%
  • Test #2 mod-oai-pmh-b: 2 harvests -   1%, 4 harvests - 3.7%, 6 harvests - 5.5%
  • Test #3 mod-oai-pmh-b: 2 harvests - 10%, 4 harvests - 15%,  6 harvests - 25%
• Memory consumption was stable except of mod-inventory which grew slowly and mod-oai-pmh that grew up from 46% to 56%.  Tests:
  • Tests #1 and #3 mod-oai-pmh-b didn't exceed 40%
  • Test #2 mod-oai-pmh-b achieved 55%
• RDS CPU utilization:
  • The averages CPU usage for  2 harvests - 15%
  • The averages CPU usage for  4 harvests - 20%
  • The averages CPU usage for  6 harvests - 25%

• Durations of harvests differed significantly in tests #1,3 (SRS) and test #2 (Inventory) because of the date creation distribution fromDate and untilDate parameters.
• Durations were not degraded by increased number of concurrent harvests.
• Response times for tests can be found in expanded links in section Test #.  Record source

Improvements that can be noted in Poppy release:
1) Non-ECS environment with Poppy release can handle concurrent OAI-PMH 

Recommendations & Jiras

• To prepare tests it's good point to populate complete_updated_date column in {tenant}_mod_inventory_storage.instance using migration. More info in Appendix section.
• To avoid degradation on OAI-PMH response times check that DB top queries do not have DELETE and INSERT for marc_id values after cluster restart
• To have the same starting conditions before running test with different Record source sets the edge-oai-pmh service was restarted, it was done to return the service memory usage to its starting(after deployment) value;

Test Runs & Results

Incremental harvesting

Incremental harvesting

Test 1.  Record source = Source record storage

Service CPU Utilization

During five harvesting tests with 10K, 25k, 50K, 500K and 1