PTF - Performance testing of Data Export (Quesnelia) [ECS]

Overview

  • This document contains the results of Data Export tests on 62 tenants (excluding the central tenant) on the QCPT environment, the previous report is available. 

PERF-888 - Getting issue details... STATUS  

Summary

  • During the Data-Export tests we observe Error "java.lang.OutOfMemoryError: Metaspace", so parameters for data-export module were changed:
    • "memory": 1024 → 2304
    • "memoryReservation": 896 → 2048
    • -XX:MaxMetaspaceSize=128m → 256m
    • -Xmx768m → 1536m
  • After upgrade Data-Export module all Data Export tests finished successfully on qcpt environment using the Default instances, authority, holdings export job profiles and Custom PTF Profiles.
  • No memory leaks, memory consumption was stable during all of the tests.

  • Database CPU usage varied up to a maximum of 26%
  • During the tests we had a some background process on Database side

Test Runs and Results

This table contains Data Export durations for Test Set №1


500
(instance records)
1000
(instance records)
2000
(instance records)
TenantDuration
hh:mm:ss
Duration
hh:mm:ss
Duration
hh:mm:ss
Average0:00:010:00:020:00:04
Member tenants 75pct0:00:010:00:030:00:05
Member tenants MAX0:00:040:00:040:00:08
cs00000001_0042
(maximum instance records)
0:07:550:07:320:07:41
 Full results table for Test Set №1


Tenant50010002000

DurationDurationDuration
cs00000001_00010:00:020:00:030:00:05
cs00000001_00020:00:010:00:020:00:04
cs00000001_00030:00:010:00:030:00:06
cs00000001_00040:00:010:00:020:00:04
cs00000001_00050:00:010:00:020:00:04
cs00000001_00060:00:010:00:020:00:04
cs00000001_00070:00:010:00:030:00:05
cs00000001_00080:00:010:00:020:00:04
cs00000001_00090:00:010:00:020:00:05
cs00000001_00100:00:010:00:030:00:05
cs00000001_00110:00:010:00:020:00:04
cs00000001_00120:00:020:00:030:00:05
cs00000001_00130:00:010:00:020:00:04
cs00000001_00140:00:010:00:020:00:04
cs00000001_00150:00:010:00:020:00:04
cs00000001_00160:00:010:00:030:00:05
cs00000001_00170:00:010:00:020:00:04
cs00000001_00180:00:010:00:030:00:05
cs00000001_00190:00:010:00:020:00:04
cs00000001_00200:00:010:00:030:00:04
cs00000001_00210:00:010:00:020:00:04
cs00000001_00220:00:010:00:020:00:05
cs00000001_00230:00:020:00:030:00:04
cs00000001_00240:00:010:00:020:00:05
cs00000001_00250:00:010:00:020:00:04
cs00000001_00260:00:010:00:020:00:05
cs00000001_00270:00:010:00:020:00:04
cs00000001_00280:00:010:00:020:00:04
cs00000001_00290:00:010:00:020:00:04
cs00000001_00300:00:010:00:020:00:04
cs00000001_00310:00:040:00:020:00:04
cs00000001_00320:00:010:00:020:00:04
cs00000001_00330:00:010:00:020:00:05
cs00000001_00340:00:020:00:030:00:04
cs00000001_00350:00:010:00:020:00:04
cs00000001_00360:00:020:00:020:00:04
cs00000001_00370:00:010:00:020:00:04
cs00000001_00380:00:010:00:020:00:04
cs00000001_00390:00:010:00:020:00:04
cs00000001_00400:00:010:00:030:00:05
cs00000001_00410:00:010:00:030:00:05
cs00000001_00420:07:550:07:320:07:41
cs00000001_00430:00:010:00:020:00:02
cs00000001_00440:00:010:00:030:00:05
cs00000001_00450:00:020:00:040:00:05
cs00000001_00460:00:010:00:020:00:03
cs00000001_00470:00:010:00:020:00:04
cs00000001_00480:00:010:00:020:00:04
cs00000001_00490:00:010:00:020:00:06
cs00000001_00500:00:010:00:030:00:05
cs00000001_00510:00:010:00:030:00:05
cs00000001_00520:00:010:00:020:00:04
cs00000001_00530:00:010:00:030:00:05
cs00000001_00540:00:010:00:020:00:08
cs00000001_00550:00:010:00:020:00:04
cs00000001_00560:00:020:00:040:00:04
cs00000001_00570:00:010:00:030:00:04
cs00000001_00580:00:010:00:030:00:06
cs00000001_00590:00:010:00:020:00:04
cs00000001_00600:00:010:00:020:00:05
cs00000001_00610:00:010:00:030:00:07
cs00000001_00620:00:010:00:020:00:03


This table contains Data Export durations for Test Set №2


500
(instance records)
1000
(instance records)
2000
(instance records)
TenantDuration
hh:mm:ss
Duration
hh:mm:ss
Duration
hh:mm:ss
Average0:00:020:00:040:00:13
Member tenants 75pct0:00:020:00:050:00:16
Member tenants MAX0:00:030:00:050:00:20
cs00000001_0042
(maximum instance records)
0:06:120:06:220:06:09
 Full results table for Test Set №2
50010002000
TenantDurationDurationDuration
cs00000001_00010:00:030:00:040:00:08
cs00000001_00020:00:020:00:040:00:08
cs00000001_00030:00:020:00:050:00:09
cs00000001_00040:00:020:00:040:00:09
cs00000001_00050:00:020:00:040:00:08
cs00000001_00060:00:020:00:040:00:10
cs00000001_00070:00:020:00:040:00:09
cs00000001_00080:00:020:00:040:00:11
cs00000001_00090:00:020:00:040:00:10
cs00000001_00100:00:020:00:040:00:10
cs00000001_00110:00:020:00:040:00:10
cs00000001_00120:00:020:00:040:00:10
cs00000001_00130:00:020:00:040:00:09
cs00000001_00140:00:020:00:040:00:12
cs00000001_00150:00:020:00:040:00:11
cs00000001_00160:00:020:00:040:00:11
cs00000001_00170:00:020:00:040:00:11
cs00000001_00180:00:020:00:050:00:14
cs00000001_00190:00:020:00:040:00:12
cs00000001_00200:00:020:00:040:00:13
cs00000001_00210:00:020:00:050:00:12
cs00000001_00220:00:020:00:050:00:12
cs00000001_00230:00:020:00:040:00:11
cs00000001_00240:00:020:00:040:00:17
cs00000001_00250:00:030:00:050:00:15
cs00000001_00260:00:030:00:050:00:14
cs00000001_00270:00:020:00:040:00:13
cs00000001_00280:00:020:00:040:00:13
cs00000001_00290:00:020:00:040:00:13
cs00000001_00300:00:030:00:050:00:16
cs00000001_00310:00:020:00:030:00:10
cs00000001_00320:00:020:00:040:00:13
cs00000001_00330:00:020:00:040:00:19
cs00000001_00340:00:020:00:040:00:18
cs00000001_00350:00:020:00:040:00:15
cs00000001_00360:00:030:00:040:00:17
cs00000001_00370:00:020:00:050:00:16
cs00000001_00380:00:020:00:040:00:12
cs00000001_00390:00:020:00:040:00:13
cs00000001_00400:00:020:00:040:00:12
cs00000001_00410:00:020:00:050:00:15
cs00000001_00420:06:120:06:220:06:09
cs00000001_00430:00:020:00:030:00:09
cs00000001_00440:00:020:00:040:00:17
cs00000001_00450:00:030:00:050:00:20
cs00000001_00460:00:020:00:040:00:16
cs00000001_00470:00:020:00:040:00:17
cs00000001_00480:00:020:00:040:00:17
cs00000001_00490:00:020:00:040:00:14
cs00000001_00500:00:020:00:050:00:15
cs00000001_00510:00:020:00:050:00:15
cs00000001_00520:00:020:00:050:00:14
cs00000001_00530:00:020:00:040:00:15
cs00000001_00540:00:020:00:050:00:18
cs00000001_00550:00:030:00:050:00:17
cs00000001_00560:00:030:00:050:00:17
cs00000001_00570:00:020:00:050:00:17
cs00000001_00580:00:020:00:050:00:16
cs00000001_00590:00:020:00:040:00:16
cs00000001_00600:00:030:00:050:00:18
cs00000001_00610:00:030:00:050:00:17
cs00000001_00620:00:020:00:040:00:09

This table contains Data Export durations for Test Set №3


500
(instance records)
1000
(instance records)
2000
(instance records)
TenantDuration
hh:mm:ss
Duration
hh:mm:ss
Duration
hh:mm:ss
Average0:00:010:00:020:00:04
Member tenants 75pct0:00:010:00:020:00:04
Member tenants MAX0:00:010:00:020:00:05
 Full results table for Test Set №3
Tenant50010002000

DurationDurationDuration
cs00000001_00010:00:010:00:020:00:04
cs00000001_00020:00:010:00:020:00:04
cs00000001_00030:00:010:00:020:00:04
cs00000001_00040:00:010:00:020:00:04
cs00000001_00050:00:010:00:020:00:04
cs00000001_00060:00:010:00:020:00:03
cs00000001_00070:00:010:00:020:00:04
cs00000001_00080:00:010:00:020:00:03
cs00000001_00090:00:010:00:020:00:04
cs00000001_00100:00:010:00:020:00:04
cs00000001_00110:00:010:00:020:00:03
cs00000001_00120:00:010:00:020:00:04
cs00000001_00130:00:010:00:020:00:04
cs00000001_00140:00:010:00:020:00:04
cs00000001_00150:00:010:00:020:00:04
cs00000001_00160:00:010:00:020:00:04
cs00000001_00170:00:010:00:020:00:04
cs00000001_00180:00:010:00:020:00:05
cs00000001_00190:00:010:00:020:00:04
cs00000001_00200:00:010:00:020:00:04
cs00000001_00210:00:010:00:020:00:04
cs00000001_00220:00:010:00:020:00:04
cs00000001_00230:00:010:00:020:00:04
cs00000001_00240:00:010:00:020:00:04
cs00000001_00250:00:010:00:020:00:04
cs00000001_00260:00:010:00:020:00:04
cs00000001_00270:00:010:00:020:00:04
cs00000001_00280:00:010:00:020:00:04
cs00000001_00290:00:010:00:020:00:04
cs00000001_00300:00:010:00:020:00:04
cs00000001_00310:00:010:00:020:00:03
cs00000001_00320:00:010:00:020:00:03
cs00000001_00330:00:010:00:020:00:04
cs00000001_00340:00:010:00:020:00:04
cs00000001_00350:00:010:00:020:00:03
cs00000001_00360:00:010:00:020:00:04
cs00000001_00370:00:010:00:020:00:04
cs00000001_00380:00:010:00:020:00:03
cs00000001_00390:00:010:00:020:00:04
cs00000001_00400:00:010:00:020:00:04
cs00000001_00410:00:010:00:020:00:04
cs00000001_00420:00:010:00:020:00:04
cs00000001_00430:00:010:00:020:00:02
cs00000001_00440:00:010:00:020:00:04
cs00000001_00450:00:010:00:020:00:05
cs00000001_00460:00:010:00:020:00:05
cs00000001_00470:00:010:00:020:00:04
cs00000001_00480:00:010:00:020:00:04
cs00000001_00490:00:010:00:020:00:04
cs00000001_00500:00:010:00:020:00:04
cs00000001_00510:00:010:00:020:00:04
cs00000001_00520:00:010:00:020:00:04
cs00000001_00530:00:010:00:020:00:04
cs00000001_00540:00:010:00:020:00:04
cs00000001_00550:00:010:00:020:00:04
cs00000001_00560:00:010:00:020:00:04
cs00000001_00570:00:010:00:020:00:04
cs00000001_00580:00:010:00:020:00:04
cs00000001_00590:00:010:00:020:00:04
cs00000001_00600:00:010:00:020:00:05
cs00000001_00610:00:010:00:020:00:04
cs00000001_00620:00:020:00:020:00:03

This table contains Data Export durations for Test Set №4


500
(instance records)
1000
(instance records)
2000
(instance records)
TenantDuration
hh:mm:ss
Duration
hh:mm:ss
Duration
hh:mm:ss
Average0:00:010:00:020:00:04
Member tenants 75pct0:00:010:00:020:00:04
Member tenants MAX0:00:020:00:030:00:05
cs00000001
(maximum authority records)
0:01:360:01:440:02:05
 Full results table for Test Set №4
Tenant50010002000

DurationDurationDuration
cs000000010:01:360:01:440:02:05
cs00000001_00010:00:020:00:030:00:05
cs00000001_00020:00:010:00:020:00:05
cs00000001_00030:00:010:00:030:00:05
cs00000001_00040:00:010:00:020:00:04
cs00000001_00050:00:010:00:020:00:04
cs00000001_00060:00:010:00:020:00:04
cs00000001_00070:00:010:00:020:00:04
cs00000001_00080:00:010:00:020:00:04
cs00000001_00090:00:010:00:020:00:04
cs00000001_00100:00:010:00:020:00:04
cs00000001_00110:00:010:00:020:00:04
cs00000001_00120:00:010:00:020:00:04
cs00000001_00130:00:010:00:020:00:04
cs00000001_00140:00:010:00:020:00:05
cs00000001_00150:00:010:00:020:00:04
cs00000001_00160:00:010:00:020:00:04
cs00000001_00170:00:010:00:020:00:04
cs00000001_00180:00:010:00:030:00:05
cs00000001_00190:00:010:00:020:00:05
cs00000001_00200:00:010:00:020:00:04
cs00000001_00210:00:010:00:020:00:05
cs00000001_00220:00:010:00:030:00:05
cs00000001_00230:00:010:00:020:00:04
cs00000001_00240:00:010:00:020:00:04
cs00000001_00250:00:010:00:030:00:05
cs00000001_00260:00:010:00:020:00:04
cs00000001_00270:00:010:00:020:00:04
cs00000001_00280:00:010:00:020:00:04
cs00000001_00290:00:010:00:020:00:04
cs00000001_00300:00:010:00:020:00:04
cs00000001_00310:00:010:00:020:00:03
cs00000001_00320:00:010:00:020:00:04
cs00000001_00330:00:010:00:020:00:04
cs00000001_00340:00:010:00:020:00:04
cs00000001_00350:00:010:00:020:00:04
cs00000001_00360:00:010:00:020:00:04
cs00000001_00370:00:010:00:030:00:05
cs00000001_00380:00:010:00:020:00:04
cs00000001_00390:00:010:00:020:00:04
cs00000001_00400:00:010:00:020:00:04
cs00000001_00410:00:010:00:020:00:04
cs00000001_00420:00:010:00:030:00:04
cs00000001_00430:00:010:00:020:00:02
cs00000001_00440:00:010:00:020:00:04
cs00000001_00450:00:010:00:020:00:05
cs00000001_00460:00:010:00:020:00:04
cs00000001_00470:00:010:00:020:00:04
cs00000001_00480:00:010:00:020:00:04
cs00000001_00490:00:010:00:020:00:04
cs00000001_00500:00:010:00:020:00:04
cs00000001_00510:00:020:00:020:00:04
cs00000001_00520:00:010:00:020:00:04
cs00000001_00530:00:010:00:020:00:04
cs00000001_00540:00:010:00:020:00:05
cs00000001_00550:00:010:00:020:00:04
cs00000001_00560:00:010:00:020:00:04
cs00000001_00570:00:020:00:020:00:04
cs00000001_00580:00:020:00:020:00:04
cs00000001_00590:00:010:00:020:00:04
cs00000001_00600:00:020:00:020:00:05
cs00000001_00610:00:010:00:020:00:04
cs00000001_00620:00:020:00:020:00:02

This table contains Data Export durations for Test Set №5


500
(records)
1000
(records)
2000
(records)
TenantDuration
hh:mm:ss
Duration
hh:mm:ss
Duration
hh:mm:ss
cs00000001_0003
(instance records)
0:00:030:00:040:00:07

cs00000001_0003
(holdings records)

0:00:010:00:010:00:02
cs00000001
(maximum authority records)
0:02:180:01:490:01:44
cs00000001_0003
(maximum holdings records)
0:02:110:01:130:01:14

This table contains Data Export durations for Test Set №6


500
(instance records)
1000
(instance records)
2000
(instance records)
TenantDuration
hh:mm:ss
Duration
hh:mm:ss
Duration
hh:mm:ss
cs00000001_00420:00:080:00:070:00:12
cs00000001_0042
(maximum instance records)
0:06:210:05:410:05:41

Resource utilization for Test Set №1

 Resource utilization table
CPURAM
mod-data-export-b572%mod-inventory-b88%
mod-inventory-b10%mod-data-export-b87%
okapi-b3.50%okapi-b56%
mod-source-record-manager-b2.10%mod-source-record-storage-b42%
mod-authtoken-b1.70%mod-source-record-manager-b41%
mod-source-record-storage-b1.50%mod-users-bl-b23%
mod-users-bl-b0.50%mod-authtoken-b23%
mod-inventory-storage-b0.30%mod-inventory-storage-b13%
nginx-okapi0.20%nginx-okapi4%
pub-okapi0.10%pub-okapi4%

Service CPU Utilization

Here we can see that mod-data-export used 575% CPU in spike.

Service Memory Utilization

Here we can see that data-export module used 88% memory.


Kafka metrics



DB CPU Utilization

DB CPU was MAX 20-22% for all tests.

DB Connections

Max number of DB connections was 1500 in avarage.

DB load

                                                                                                                     

Top SQL-queries


Resource utilization for Test Set №2

 Resource utilization table
CPURAM
mod-data-export-b565%mod-data-export-b106%
mod-inventory-b14%okapi-b56%
okapi-b2.50%mod-source-record-manager-b53%
mod-source-record-manager-b1.80%mod-source-record-storage-b47%
mod-source-record-storage-b1.60%mod-inventory-b36%
mod-authtoken-b1.40%mod-users-bl-b24%
mod-users-bl-b0.50%mod-authtoken-b22%
mod-inventory-storage-b0.30%mod-inventory-storage-b14%
nginx-okapi0.10%nginx-okapi4.90%
pub-okapi0.10%pub-okapi4.50%

Service CPU Utilization

Here we can see that mod-data-export used 570% CPU in spike.

Service Memory Utilization

Here we can see that data-export module used 110% memory.


Kafka metrics



DB CPU Utilization

DB CPU was MAX 24-26% for Test 1 and 2, and MAX 16% for Test 2.

DB Connections

Max number of DB connections was 2230.

DB load

                                                                                                                     

Top SQL-queries


Resource utilization for Test Set №3

 Resource utilization table
CPU
RAM
mod-data-export-b230%mod-data-export-b107%
mod-inventory-b14%okapi-b56%
mod-authtoken-b7.70%mod-source-record-manager-b53%
okapi-b4.30%mod-source-record-storage-b47%
mod-source-record-manager-b1.80%mod-inventory-b36%
mod-source-record-storage-b1.60%mod-users-bl-b25%
pub-okapi1.20%mod-authtoken-b22%
mod-users-bl-b0.80%mod-inventory-storage-b14%
mod-inventory-storage-b0.30%nginx-okapi5%
nginx-okapi0.30%pub-okapi4%

Service CPU Utilization

Here we can see that mod-data-export used 230% CPU.

Service Memory Utilization

Here we can see that mod-data-export used 108% memory .


Kafka metrics




DB CPU Utilization

DB CPU was MAX 20% for Test 3.

DB Connections

Max number of DB connections was 2040.

DB load

                                                                                                                    

Top SQL-queries


Resource utilization for Test Set №4

 Resource utilization table
CPURAM
mod-data-export-b324%mod-data-export-b72%
mod-inventory-b12%okapi-b56%
mod-authtoken-b6.1mod-source-record-manager-b53%
okapi-b4mod-source-record-storage-b47%
mod-source-record-manager-b1.8mod-inventory-b46%
mod-source-record-storage-b1.5mod-users-bl-b25%
pub-okapi0.9mod-authtoken-b22%
mod-users-bl-b0.6mod-inventory-storage-b14%
mod-inventory-storage-b0.3nginx-okapi5%
nginx-okapi0.1pub-okapi4%

Service CPU Utilization

Here we can see that mod-data-export used 325% CPU.

Service Memory Utilization

Here we can see that mod-data-export used 73% memory .


Kafka metrics




DB CPU Utilization

DB CPU was 22%.

DB Connections

Max number of DB connections was 2630.

DB load

                                                                                                                    

Top SQL-queries


Resource utilization for Test Set №5

 Resource utilization table
CPURAM
mod-data-export-b129%mod-data-export-b82%
mod-inventory-b13%okapi-b57%
mod-source-record-manager-b1.90%mod-source-record-manager-b54%
mod-source-record-storage-b1.70%mod-inventory-b38%
okapi-b1.60%mod-source-record-storage-b37%
mod-authtoken-b1.10%mod-users-bl-b23%
mod-users-bl-b0.50%mod-authtoken-b19%
mod-inventory-storage-b0.50%mod-inventory-storage-b11%
pub-okapi0.20%nginx-okapi5%
nginx-okapi0.00%pub-okapi4.50%

Service CPU Utilization

Here we can see that mod-data-export used 130% CPU.

Service Memory Utilization

Here we can see that mod-data-export used 82% memory .


Kafka metrics



DB CPU Utilization

DB CPU was 25%.

DB Connections

Max number of DB connections was 1850.

DB load

                                                                                                                   

Top SQL-queries


Resource utilization for Test Set №6

 Resource utilization table
CPURAM
mod-data-export-b589%mod-data-export-b80%
mod-inventory-b13%okapi-b57%
mod-source-record-manager-b1.90%mod-source-record-manager-b54%
okapi-b1.60%mod-inventory-b44%
mod-source-record-storage-b1.50%mod-source-record-storage-b37%
mod-authtoken-b1.20%mod-users-bl-b23%
mod-inventory-storage-b0.60%mod-authtoken-b19%
mod-users-bl-b0.50%mod-inventory-storage-b11%
pub-okapi0.10%nginx-okapi5%
nginx-okapi0.00%pub-okapi4.50%

Service CPU Utilization

Here we can see that mod-data-export used 590% CPU.

Service Memory Utilization

Here we can see that mod-data-export used 82% memory .


Kafka metrics



DB CPU Utilization

DB CPU was 25%.

DB Connections

Max number of DB connections was 3050.

DB load

                                                                                                                     

Top SQL-queries


Appendix

Infrastructure

PTF -environment qcpt

  • 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 1 database  instance, writer

    NameMemory GIBvCPUs

    db.r6g.4xlarge

    128 GiB16 vCPU
  • Number of records in DB:
    •  cs00000001_0042
      • instances - 637243
    •  cs00000001
      • authorities - 55000
    •  cs00000001_0003
      • holdings  - 146704
      • instances - 148713
  • Open Search ptf-test
    • Data nodes
      • Instance type - r6g.2xlarge.search
      • Number of nodes - 4
      • Version: OpenSearch_2_7_R20240502
    • Dedicated master nodes
      • Instance type - r6g.large.search
      • Number of nodes - 3
  • MSK tenant
    • 4 m5.2xlarge brokers in 2 zones
    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


 qcpt modules memory and CPU parameters

Cluster Resources - qcpt-pvt (Fri Jul 26 09:55:21 UTC 2024)

ModuleTask Definition RevisionModule VersionTask CountMem Hard LimitMem Soft LimitCPU UnitsXmxMetaspace SizeMax Metaspace Size
mod-remote-storage1mod-remote-storage:3.2.024920447210243960512512
mod-finance-storage1mod-finance-storage:8.6.121024896102470088128
mod-ncip1mod-ncip:1.14.52102489612876888128
mod-agreements5mod-agreements:7.0.52409635841024000
mod-ebsconet1mod-ebsconet:2.2.0212481024128700128256
mod-organizations1mod-organizations:1.9.22102489612876888128
mod-consortia1mod-consortia:1.1.025136477610244416384512
edge-sip21edge-sip2:3.2.42102489612876888128
mod-settings1mod-settings:1.0.32102489620076888128
mod-serials-management1mod-serials-management:1.0.32248023121281792384512
mod-data-import1mod-data-import:3.1.11204818442561292384512
edge-dematic1edge-dematic:2.2.21102489612876888128
mod-search1mod-search:3.2.6225922480204814405121024
mod-tags1mod-tags:2.2.02102489612876888128
mod-authtoken1mod-authtoken:2.15.121440115251292288128
edge-courses1edge-courses:1.4.12102489612876888128
edge-inventory1edge-inventory:1.4.02102489612876888128
mod-inventory-update1mod-inventory-update:3.3.12102489612876888128
mod-notify1mod-notify:3.2.02102489612876888128
mod-configuration1mod-configuration:5.10.02102489612876888128
mod-orders-storage1mod-orders-storage:13.7.22102489651270088128
edge-caiasoft1edge-caiasoft:2.2.22102489612876888128
mod-login-saml1mod-login-saml:2.8.12102489612876888128
mod-gobi1mod-gobi:2.8.12102489612876888128
mod-licenses1mod-licenses:6.0.22248023125121792384512
mod-password-validator1mod-password-validator:3.2.0214401298128768384512
edge-dcb1edge-dcb:1.1.0-SNAPSHOT.152102489612876888128
mod-bulk-operations1mod-bulk-operations:2.0.223072260010241536384512
mod-fqm-manager1mod-fqm-manager:2.0.42102489612876888128
mod-graphql2mod-graphql:1.12.12102489612876888128
mod-finance1mod-finance:4.9.02102489612876888128
mod-batch-print1mod-batch-print:1.1.02102489612876888128
mod-lists1mod-lists:2.0.62102489612876888128
mod-copycat1mod-copycat:1.6.02102489612876888128
mod-entities-links1mod-entities-links:3.0.1225922480400144001024
mod-permissions2mod-permissions:6.5.02168415445121024384512
pub-edge1pub-edge:2023.06.142102489612876800
mod-orders1mod-orders:12.8.822048174010241024384512
edge-patron2edge-patron:5.1.12102489625676888128
edge-ncip1edge-ncip:1.10.02102489612876888128
mod-users-bl1mod-users-bl:7.7.321440115251292288128
mod-invoice1mod-invoice:5.8.221440115251292288128
mod-inventory-storage1mod-inventory-storage:27.1.324096369020483076384512
edge-ea-data-export1edge-ea-data-export:4.2.02102489612876888128
mod-user-import1mod-user-import:3.8.02102489612876888128
mod-sender1mod-sender:1.12.02102489612876888128
edge-oai-pmh1edge-oai-pmh:2.9.121512136010241440384512
mod-data-export-worker1mod-data-export-worker:3.2.423072280010242048384512
mod-rtac1mod-rtac:3.6.02102489612876888128
mod-task-list1mod-task-list:1.9.22102489612876888128
mod-circulation-storage1mod-circulation-storage:17.2.122880259215361814384512
mod-calendar1mod-calendar:3.1.02102489612876888128
mod-source-record-storage1mod-source-record-storage:5.8.525600500020483500384512
mod-event-config1mod-event-config:2.7.12102489612876888128
mod-courses1mod-courses:1.4.102102489612876888128
mod-circulation-item3mod-circulation-item:1.0.0-SNAPSHOT.1221024896128000
mod-inventory1mod-inventory:20.2.622880259210241814384512
mod-email1mod-email:1.17.02280025505121800384512
mod-di-converter-storage1mod-di-converter-storage:2.2.22102489612876888128
mod-circulation1mod-circulation:24.2.522880259215361814384512
mod-pubsub1mod-pubsub:2.13.12153614401024922384512
edge-rtac1edge-rtac:2.7.22102489612876888128
edge-orders1edge-orders:3.0.22102489612876888128
mod-template-engine1mod-template-engine:1.20.02102489612876888128
mod-users1mod-users:19.3.12102489612876888128
mod-patron-blocks1mod-patron-blocks:1.10.021024896102476888128
edge-fqm1edge-fqm:2.0.12102489612876888128
mod-audit1mod-audit:2.9.02102489612876888128
mod-source-record-manager1mod-source-record-manager:3.8.525600500020483500384512
nginx-edge1nginx-edge:2023.06.1421024896128000
mod-quick-marc1mod-quick-marc:5.1.11228821761281664384512
nginx-okapi1nginx-okapi:2023.06.14210248961024000
okapi-b1okapi:5.3.03168414401024922384512
mod-feesfines1mod-feesfines:19.1.02102489612876888128
mod-invoice-storage1mod-invoice-storage:5.8.121872153610241024384512
edge-users1edge-users:1.2.02102489612876888128
mod-service-interaction1mod-service-interaction:4.0.22204818442561290384512
mod-dcb1mod-dcb:1.1.0-SNAPSHOT.172102489612876888128
mod-data-export3mod-data-export:5.0.41230420481024153688256
mod-patron1mod-patron:6.1.02102489612876888128
mod-oai-pmh1mod-oai-pmh:3.13.224096369020483076384512
edge-connexion1edge-connexion:1.3.02102489612876888128
mod-notes1mod-notes:5.2.021024896128952384512
mod-kb-ebsco-java1mod-kb-ebsco-java:4.0.02102489612876888128
mod-login1mod-login:7.11.12144012981024768384512
mod-data-export-spring1mod-data-export-spring:3.2.21204818442561536384512
mod-organizations-storage1mod-organizations-storage:4.7.02102489612876888128
pub-okapi1pub-okapi:2023.06.142102489612876800
mod-eusage-reports1mod-eusage-reports:2.0.02102489612876888128
edge-erm1edge-erm:1.0.02102489612876888128


Methodology/Approach

Test set №1:

  • Test 1: Concurrently run Data Export using 500 instance records file with the Default instances export job profile on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter
  • Test 2: Concurrently run Data Export using 1000 instance records file with the Default instances export job profile on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter.
  • Test 3: Concurrently run Data Export using 200 instance records file with the Default instances export job profile on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter.

Test set №2:

  • Test 4: Concurrently run Data Export using 500 instance records file with the Custom PTF Profiles on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter.
  • Test 5: Concurrently run Data Export using 1000 instance records file with the Custom PTF Profiles on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter.
  • Test 6: Concurrently run Data Export using 2000 instance records file with the Custom PTF Profiles on 61 tenants, and run Data Export for all instances with the Default instances export job profile on one tenant (cs00000001_0042), started by JMeter.

Test set №3:

  • Test 7: Concurrently run Data Export using 500 instance records file with the Default instances export job profile on 62 tenants, started by JMeter.
  • Test 8: Concurrently run Data Export using 1000 instance records file with the Default instances export job profile on 62 tenants, started by JMeter.
  • Test 9: Concurrently run Data Export using 2000 instance records file with the Default instances export job profile on 62 tenants, started by JMeter.

Test set №4:

  • Test 10: Concurrently run Data Export using 500 instance records file with the Default instances export job profile on 62 tenants, and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started by JMeter.
  • Test 11: Concurrently run Data Export using 1000 instance records file with the Default instances export job profile on 62 tenants, and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started by JMeter.
  • Test 12: Concurrently run Data Export using 2000 instance records file with the Default instances export job profile on 62 tenants, and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started by JMeter.

Test set №5:

  • Test 13: Concurrently run Data Export using 500 instance records file with the Default instances export job profile, 500 instance records file with the Default holdings export job profile, Data Export for all holdings with the Default holdings export job profile on One tenant (cs00000001_0003) and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started manually.
  • Test 14: Concurrently run Data Export using 1000 instance records file with the Default instances export job profile, 1000 instance records file with the Default holdings export job profile, Data Export for all holdings with the Default holdings export job profile on One tenant (cs00000001_0003) and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started manually.
  • Test 15: Concurrently run Data Export using 2000 instance records file with the Default instances export job profile, 2000 instance records file with the Default holdings export job profile, Data Export for all holdings with the Default holdings export job profile on One tenant (cs00000001_0003) and run Data Export for all authority with the Default authority export job profile on Central tenant (cs00000001), started manually.

Test set №6:

  • Test 16: Concurrently run Data Export using 500 instance records file with the Custom PTF Profiles and Data Export for all instances with the Default instances export job profile on One tenant (cs00000001_0003), started manually.
  • Test 17: Concurrently run Data Export using 1000 instance records file with the Custom PTF Profiles and Data Export for all instances with the Default instances export job profile on One tenant (cs00000001_0003), started manually.
  • Test 18: Concurrently run Data Export using 2000 instance records file with the Custom PTF Profiles and Data Export for all instances with the Default instances export job profile on One tenant (cs00000001_0003), started manually.

To get status and time range for import jobs the query used: 

 SQL Query
SQL Query
 SELECT 'cs00000001' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL
    SELECT 'cs00000001_0001' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0001_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0002' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0002_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0003' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0003_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0004' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0004_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0005' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0005_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0006' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0006_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0007' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0007_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0008' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0008_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0009' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0009_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0010' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0010_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0011' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0011_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0012' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0012_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0013' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0013_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0014' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0014_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0015' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0015_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0016' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0016_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0017' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0017_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0018' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0018_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0019' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0019_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0020' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0020_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0021' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0021_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0022' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0022_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0023' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0023_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0024' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0024_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0025' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0025_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0026' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0026_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0027' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0027_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0028' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0028_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0029' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0029_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0030' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0030_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0031' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0031_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0032' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0032_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0033' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0033_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0034' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0034_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0035' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0035_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0036' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0036_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0037' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0037_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0038' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0038_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0039' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0039_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0040' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0040_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0041' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0041_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0042' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0042_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0043' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0043_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0044' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0044_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0045' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0045_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0046' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0046_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0047' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0047_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0048' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0048_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0049' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0049_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0050' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0050_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0051' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0051_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0052' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0052_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0053' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0053_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0054' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0054_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0055' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0055_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0056' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0056_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0057' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0057_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0058' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0058_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0059' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0059_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0060' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0060_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
 UNION ALL

    SELECT 'cs00000001_0061' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0061_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
UNION ALL

    SELECT 'cs00000001_0062' AS schema_name,
           jsonb->'exportedFiles'->0->>'fileName' AS fileName,
           job_profile_name,
           exported,
           started_date,
           completed_date,
           completed_date - started_date AS duration,
           CAST(status AS text) AS status
    FROM cs00000001_0062_mod_data_export.job_executions
    WHERE started_date > '2024-07-25 07:43:00'
ORDER BY started_date DESC
LIMIT 1000;