| Table of Contents |
|---|
...
- Kiwi release was able ho harvest 7,808,200 records in 19 hr 8 min (1M records per 2 hours and 15 min).
- Average response time per request 0.874s.
- A couple of few issues were found
- OutOfMemory OutOfMemory exception: MODOAIPMH-374
- Thread block issue: MODOAIPMH-374
- When instances didn't have underlying MARC records, multiple repeating calls from mod-edge-oai-pmh to mod-oai-pmh were occurred, resulting in the end-client receiving an timeout, see MODOAIPMH-383
...
- Total Underlying SRS records: 1,212,039
- Duration: 4 hr 57 min
- Records transferred: 4,770,043 (should be 8,415,303)
- Records harvested - 20,618 X 100 = 2,061,800.
...
- While data transferring process is going on the background DB CPU usage has reached 70%-75%.
- Data transferring process has failed in 10 minutes and transfer only 4770043 from 8M records.
- Harvesting itself consumes 15% DB CPU.
Test 2
- Total Underlying SRS records: 1,212,039
- Duration: 4 hr 25 min
- Records transferred: 3,815,867 (should be 8,415,303)
- Records harvested - 22305 X 100 = 2 230 500.
...
We can see here that memory usage is stable and none of containers didn't fail.
Notable observations:
- Unstable OutOfMemory exception MODOAIPMH-374 and Thread block issue MODOAIPMH-374 were found and resolved early on in the testing cycle.
- The remaining issue: unstable parts of first couple of tests were made by data set : MODOAIPMH-383 Jira ticket created to handle client waits: MODOAIPMH-383
- Instances didn't have underlying records and this causes multiple repeating calls from mod-edge-oai-pmh to mod-oai-pmh.
- This leads to end client to wait until oai-pmh will find records with underlying records. And often client fail with 504 getaway timeout (load balancer timeout 400 seconds).
- Timeouts in DB fixed with changing of dataset to "bugfest" like.
- The workaround for this issue was by testing with a dataset that has underlying MARC records, which was Bugfest's. This is an edge case as most systems would have more MARC records than instances. It does not need to be resolved for the Kiwi release.