...

• folio_circulation-2.0.0
• Item Check-in (folio_checkin-2.0.1)
• Item Check-out (folio_checkout-3.0.2)

Folio build was deployed with 50+ ECS services distributed randomly across four Environment:

• 55 back-end modules deployed in 110 ECS services
• 3 okapi ECS services
• 8 m5.large  EC2

...

• instances
• 1  db.r5.xlarge AWS RDS instance

...

• INFO logging level

High Level Summary

1. Overall check in, check out time in seconds

   	Average (seconds)		50th %tile (seconds)		75th %tile (seconds)		95th %tile  (seconds)
   	Check-in	Check-out	Check-in	Check-out	Check-in	Check-out	Check-in	Check-out
   1 user	1.015	1.234	0.96	1.277	1.071	1.409	1.322	1.653
   5 users	1.236	1.488	1.156	1.393	1.464	1.869	1.704	2.219
   8 users	1.512	1.751	1.403	1.852	1.741	2.031	2.02	2.274
   20 users	1.649	1.898	1.535	1.996	1.896	2.211	2.252	2.539

   
   

2. Slow APIs taking more than 100 ms to return

   API	1 user (75th %tile)	5 users (75th %tile)	8 users (75th %tile)	20 Users (75th %tile)
   POST checkout-by-barcode	615 ms	905 ms	906 ms	988 ms
   POST checkin-by-barcode	548 ms	830 ms	1053 ms	1137 ms
   Get circulation/loans	283 ms	346 ms	449 ms	479 ms
   Get inventory/items	217 ms	232 ms	237 ms	281 ms

   
   

3. Excess logging of missing indexes - 64K lines in  45 minutes run. Logging level could be reduced to WARNING or INFO, but at the cost of having less data to work with should there be a need to troubleshoot.

   
   

4. JVM profiling shows JSON de/serialization operations one of the slowest operations.

...

• • Can drill down these methods for more information at : http://ec2-3-93-19-104.compute-1.amazonaws.com/grafana/d/b2iP-9hiz/java-profiling-folio?orgId=1&from=1589940195166&to=1589943457693&var-service_tag=mod-circulation&var-aggregation=10s
    • Areas of code that serializes and deserializes JSON objects
      • org.folio.inventory.storage.external.ExternalStorageModuleCollection
      • org.forlio.inventory.resources.Items$$Lambdas...apply()
      • org.folio.okapi.util.LockedStringMap.lambda$getString()
      • org.folio.okapi.util.LockedTypedMap.lambda...handle()
      • org.folio.okapi.managers..proxy
      • org.folio.cql2pgjson.CQL2PgJSON.loadDbSchema
      • org.folio.rest.persist.PostgresClient.deserializeRow
    • Logging
      • org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled
      • org.apache.logging.slf4j.Log4jLogger.info
    • org.joda.time.DateTime.parse
      • org.folio.circulation.domain.Loan.getReturnDate
      • org.folio.circulation.domain.Loan.getDueDate

Database

Database does not show much CPU usage for 1, 5, 8 and 20 users runs.  At maximum only 25% CPU usage for the high case of 20 users.

...