The following resources are used:

...

Previous Lotus testing performance results:

Lotus Snapshot Performance testing

Modules:

Data Import Module (mod-data-import-2.5.0-SNAPSHOT.231)

...

Performance-optimized configuration:

Folio

...

MAX_REQUEST_SIZE = 4000000 (for all modules)

Kafka

...

2

...

Tasks for all

...

DI Modules (except mod-data-import)

2

...

Partition for all DI

...

Kafka topics

Please Notice: an environment should be configured in such a way that for every Kafka topic there are as many partitions as many instances created for a module connected to that topic

Examples:

...

...

./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --delete --topic perf-eks-folijet.Default.fs09000000.DI_ERROR

...

./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --create --topic perf-eks-folijet.Default.fs09000000.DI_ERROR --partitions 2 --replication-factor 1

...

...

./kafka-topics.sh --bootstrap-server=<kafka-ip>:9092 --describe --topic perf-eks-folijet.Default.fs09000000.DI_ERROR

JVM

...

mod-data-import: -XX:MaxRAMPercentage=85.0 -XX:+UseG1GC / cpu: 128m/192m | memory: 1Gi/1Gi

...

mod-inventory-storage: -XX:MaxRAMPercentage=80 -XX:MetaspaceSize=120M -XX:+UseG1GC / DB_MAXPOOLSIZE = 15 / cpu: 512m/512m 1024m | memory: 778Mi1024Mi/846Mi1200Mi

Tests:

NEW

-Ddi.flow.control.enable=true
-Ddi.flow.control.max.simultaneous.records=100 
-Ddi.flow.control.records.threshold=50

60 errors (500K - PTF Create - 2):

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.1.0-SNAPSHOT.692 http://mod-inventory-storage: Connection was closed: POST /holdings-storage/holdings

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.1.0-SNAPSHOT.692 http://mod-inventory-storage: Connection was closed: POST /instance-storage/instances

io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"must not be null","type":"1","code":"javax.validation.constraints.NotNull.message","parameters":[{"key":"contributors[0].name","value":"null"}]}]}

io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"must not be null","type":"1","code":"javax.validation.constraints.NotNull.message","parameters":[{"key":"contributors[2].name","value":"null"}]}]}

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.1.0-SNAPSHOT.692 http://mod-inventory-storage: finishConnect(..) failed: Connection refused: mod-inventory-storage.folijet.svc.cluster.local/172.20.250.48:80: POST /holdings-storage/holdings

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.1.0-SNAPSHOT.692 http://mod-inventory-storage: finishConnect(..) failed: Connection refused: mod-inventory-storage.folijet.svc.cluster.local/172.20.250.48:80: POST /instance-storage/instances

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.1.0-SNAPSHOT.692 http://mod-inventory-storage: readAddress(..) failed: Connection reset by peer: POST /holdings-storage/holdings

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.1.0-SNAPSHOT.692 http://mod-inventory-storage: readAddress(..) failed: Connection reset by peer: POST /instance-storage/instances

...

Results before flow control fix: MODSOURMAN-811

60 errors (500K - PTF Create - 2):

Almost all errors with mod-inventory storage related to not having enough memory for instances (memory: 778Mi/846Mi).  Instances of mod-inventory-storage were restarted 2 times.

Image Added

io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"must not be null","type":"1","code":"javax.validation.constraints.NotNull.message","parameters":[{"key":"contributors[0].name","value":"null"}]}]}

io.vertx.core.impl.NoStackTraceThrowable: {"errors":[{"message":"must not be null","type":"1","code":"javax.validation.constraints.NotNull.message","parameters":[{"key":"contributors[2].name","value":"null"}]}]}

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.1.0-SNAPSHOT.692 http://mod-inventory-storage: Connection was closed: POST /holdings-storage/holdings

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.1.0-SNAPSHOT.692 http://mod-inventory-storage: Connection was closed: POST /instance-storage/instances

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.1.0-SNAPSHOT.692 http://mod-inventory-storage: finishConnect(..) failed: Connection refused: mod-inventory-storage.folijet.svc.cluster.local/172.20.250.48:80: POST /holdings-storage/holdings

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.1.0-SNAPSHOT.692 http://mod-inventory-storage: finishConnect(..) failed: Connection refused: mod-inventory-storage.folijet.svc.cluster.local/172.20.250.48:80: POST /instance-storage/instances

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.1.0-SNAPSHOT.692 http://mod-inventory-storage: readAddress(..) failed: Connection reset by peer: POST /holdings-storage/holdings

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.1.0-SNAPSHOT.692 http://mod-inventory-storage: readAddress(..) failed: Connection reset by peer: POST /instance-storage/instances

io.vertx.core.impl.NoStackTraceThrowable: proxyClient failure: mod-inventory-storage-23.1.0-SNAPSHOT.692 http://mod-inventory-storage: readAddress(..) failed: Connection reset by peer: POST /item-storage/items

Performance testing of update item scenario after removal of index (MODDATAIMP-697):

In the scope of FOLIO-3388 "item_status_name_idx_gin" index for the Item status field was removed from mod-inventory-storage. However, this index potentially can be used for matching by Item status during Item update.

Job profile structure to update item and matching item by status field:

• Job profile
• Match profile (902$a to Item HRID)
  • For matches: sub match profile (Static match of "Available" to Item Loan and Availability Status)
    • For matches: action profile (action = update; Folio record type = Item)
      • Mapping profile (Folio record type = Item)

Test results:

Analysis of the query for matching item by status field:

Example of the CQL query built while processing the sub-match profile for matching by status: 

status.name == "Available" AND id == "4ae2603d-1f71-457f-b69a-3eed820d6cfb"

This CQL query is translated by mod-inventory-storage to the following SQL:

SELECT id, jsonb, creation_date, created_by, holdingsrecordid, permanentloantypeid, temporaryloantypeid, materialtypeid, permanentlocationid, temporarylocationid, effectivelocationid
FROM fs09000000_mod_inventory_storage.item
WHERE (
	CASE WHEN length(lower(f_unaccent('Available'))) <= 600 
		 THEN left(lower(f_unaccent(item.jsonb->'status'->>'name')),600) LIKE lower(f_unaccent('Available')) 
		 ELSE left(lower(f_unaccent(item.jsonb->'status'->>'name')),600) LIKE left(lower(f_unaccent('Available')),600) AND lower(f_unaccent(item.jsonb->'status'->>'name')) LIKE lower(f_unaccent('Available')) 
	END	
) AND lower(f_unaccent(item.jsonb->'status'->>'name')) LIKE lower(f_unaccent('Available')) END) AND (id='4ae2603d-1f71-457f-b69a-3eed820d6cfb')
LIMIT 2 OFFSET 0

For the particular case when matching by item status is used as sub match profile no indexes of the Item status field are used. Instead, a more efficient algorithm is applied to perform data lookup using the index for the id field.

"Limit  (cost=0.56..8.84 rows=1 width=1156) (actual time=0.038..0.040 rows=1 loops=1)"
"  ->  Index Scan using item_pkey on item  (cost=0.56..8.84 rows=1 width=1156) (actual time=0.038..0.039 rows=1 loops=1)"
"        Index Cond: (id = '4ae2603d-1f71-457f-b69a-3eed820d6cfb'::uuid)"
"        Filter: ("left"(lower(f_unaccent(((jsonb -> 'status'::text) ->> 'name'::text))), 600) ~~ 'available'::text)"
"Planning Time: 0.195 ms"
"Execution Time: 0.053 ms"

During the testing item update scenario it was observed that the "item_status_name_idx_gin" index deletion does not impact the performance of matching Item by status. According to the results of analysis, this index is not used for matching Item by status field during data import.