IN PROGRESS

...

• Unstable parts of first tests was made by data set:
  • instances didn't have underlying records and this causes multiple repeating calls from mod-edge-oai-pmh to mod-oai-pmh.
  • this leads to end client to wait until oai-pmh will find records with underlying records. And often client fail with 504 getaway timeout (load balancer timeout 400 seconds).
• Timeouts in DB fixed with changing of dataset to "bugfest" like.
• Jira ticket created to handle client waits: MODOAIPMH-383

Test flow

Test consist of few calls:

...

Test 1

Records transferred  . - 4770043 (should be 8415303)

Records harvested - 20618 X 100 = 2 061 800.

Underlying records number: 1212039

We can see here unstable part of test. This spikes on chart showing extremely increased response times. which leads to throughput gaps. At this point we still not sure this does it happening we've checked:

...

• While data transferring process is going on the background DB CPU usage has reached 70%-75%.
• Data transferring process has failed in 10 minutes and transfer only 4770043 from 8M records.
• Harvesting itself consumes 15% DB CPU.

Test 2

Records transferred . - 3815867 (should be 8415303)

Records harvested - 22305 X 100 = 2 230 500.

Underlying records number: 1212039

Final test (bugfest data set)

...

Records harvested - 78082 X 100 = 7 808 200

Time spent : 19 hr 8 min

Underlying records number: 8034444

Average response time for call with resumption token 0.874 ms

...