...
We're able to harvest all data set we have, which is 7.2 M records with different "Max records per response" parameter value.
Possibly there's a memory leak on oai-pmh side as we've seen continuously growing memory and CPU usage (see screenshots below).
Tests and results
| test | Max records per response | Time to complete | Result | Issues |
|---|---|---|---|---|
| 1 | 100 | 6 hours 26 minutes | all data harvested | Growing CPU/RAM usage |
| 2 | 300 | 2 hours 31 min | 5.5 M records harvested | connection lost with load generator not an oai-pmh issue |
| 3 | 500 | 2 hour 27 min | all data harvested | Growing CPU/RAM usage |
...