Skip to end of banner
Go to start of banner

OAI-PMH data harvesting [KIWI]

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

IN PROGRESS


PERF-198 - Getting issue details... STATUS



Overview 

The purpose of these set of tests is to measure performance of Kiwi release. Find possible issues, bottlenecks. 


Test flow

Test consist of few calls:

Initial call:

  • /oai/records?verb=ListRecords&metadataPrefix=marc21_withholdings&apikey=[APIKey] - performing only once

Harvesting call:

  • /oai/records?verb=ListRecords&apikey=[APIKey]&resumptionToken=[resumptionToken] - performing repeatedly, harvesting 100 records each time until there is no more data in [tenant]_mod_oai_pmh.instances table to harvest.

[resumptionToken] returning in initial call response and in each harvesting call until there is no more records to harvest. When all data has being harvested - resumptionToken will not return with the response.

Software versions


  • mod-oai-pmh:3.7.0-SNAPSHOT.188
  • edge-oai-pmh:2.4.0
  • mod-source-record-manager:3.2.3
  • mod-source-record-storage:5.2.1
  • mod-inventory-storage:22.0.1
  • okapi:4.9.0


Issues detected during testing

List of issues founded, reasons and possible fixes:

1) OutOfMemory exception.  MODOAIPMH-374


2) Thread block issue 


3) DB timeout. New issue appearing when we're starting DB transferring and harvesting process at the same time. It's leads to high load on DB and it responding with timeout

2021-12-01T10:02:42,566 ERROR [vert.x-eventloop-thread-0] MarcWithHoldingsRequestHelper Save instance Ids failed: Timeout.

io.vertx.core.impl.NoStackTraceThrowable: Timeout








  • No labels