IN PROGRESS
- PERF-198Getting issue details... STATUS
Overview
The purpose of these set of tests is to measure performance of Kiwi release. Find possible issues, bottlenecks.
Test flow
Test consist of few calls:
Initial call:
- /oai/records?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=[APIKey] - performing only once
Harvesting call:
- /oai/records?verb=ListRecords&apikey=[APIKey]&resumptionToken=[resumptionToken] - performing repeatedly, harvesting 100 records each time until there is no more data in [tenant]_mod_oai_pmh.instances table to harvest.
[resumptionToken] returning in initial call response and in each harvesting call until there is no more records to harvest. When all data has being harvested - resumptionToken will not return with the response.
Software versions
- mod-oai-pmh:3.7.0-SNAPSHOT.188
- edge-oai-pmh:2.4.0
- mod-source-record-manager:3.2.3
- mod-source-record-storage:5.2.1
- mod-inventory-storage:22.0.1
- okapi:4.9.0
Issues detected during testing
List of issues founded, reasons and possible fixes:
1) OutOfMemory exception. fixed in scope of MODOAIPMH-374
2) Thread block issue. fixed in scope of MODOAIPMH-374
3) DB timeout. New issue appearing when we're starting DB transferring and harvesting process at the same time. It's leads to high load on DB and it responding with timeout
2021-12-01T10:02:42,566 ERROR [vert.x-eventloop-thread-0] MarcWithHoldingsRequestHelper Save instance Ids failed: Timeout. |
io.vertx.core.impl.NoStackTraceThrowable: Timeout |
Test results
Test 1
Records transferred . - 4770043 (should be 8415303)
Records harvested - 20618 X 100 = 2 061 800.
Service CPU usage has reached ±200% while data transfer. And it's on 50-60% level during data harvesting. However during "unstable part" of test it has drop down to 20-25%.
- Service memory usage is stable. There is no suspects for memory leak.
Notable observations:
- While data transferring process is going on the background DB CPU usage has reached 70%-75%.
- Data transferring process has failed in 10 minutes and transfer only 4770043 from 8M records.
- Harvesting itself consumes 15% DB CPU.
Test 2
Records transferred . - 4770043 (should be 8415303)
Records harvested - 22305 X 100 = 2 230 500.