Sunflower CSP1 Verifications and Recommendations

Sunflower CSP1 Verifications and Recommendations

Overview

The PTF verified several Jiras for Sunflower CSP1 release. Here are results and setting configurations

Issue

JIRA

Description/Resolution

Recommendations

Issue

JIRA

Description/Resolution

Recommendations

Investigate mod-source-record-manager scaling out slowness

https://folio-org.atlassian.net/browse/PERF-1156

For more details see https://folio-org.atlassian.net/browse/MODSOURMAN-1330 . It has been deprioritized due to not many clusters in prod require scaling up DI modules.

Await the fix for MODSOURMAN-1330, which is scheduled for possible Umbrella Leaf.

Run CICO and DI tests with new value for random_page_cost

https://folio-org.atlassian.net/browse/PERF-1157

New value for random_page_cost encourages the query planner to use the index more

Set random_page_cost value in DB Parameter Group to 1.1

Optimizing Eureka Deployment

https://folio-org.atlassian.net/browse/PERF-1144

Performance can best be achieved by configuring the modules and sidecars to the following values →

  • All modules having CPU = 0 except for:

    • folio-keycloak: 1536

    • mod-consortia-keycloak: 512

    • mod-login-keycloak: 128

    • mod-roles-keycloak: 128

    • mod-users-keycloak: 128

  • Sidecars' CPU = 0 except for: mod-inventory and mod-inventory-storage: 512

  • All modules' task placement strategies: Spread by Availability Zone and InstanceId

Check and Test KAFKA_CONSUMER_MAX_POLL_RECORDS settings for mod-search | rsam

https://rally1.rallydev.com/#/?detail=/userstory/828791776143&fdp=true

mod-search’s KAFKA_CONSUMER_MAX_POLL_RECORDS value of 600 or 200 don’t affect performance of DI or Reindexing

Safe to set KAFKA_CONSUMER_MAX_POLL_RECORDS to 200

Perform A|B tests with Shared DB connection Pool

https://folio-org.atlassian.net/browse/PERF-1133

In Progress, blocked in favor of tackling other higher priorities

 

Test deploying Eureka components with two tasks for HA purpose

https://folio-org.atlassian.net/browse/PERF-1160

All components are HA capable, can be deployed with multiple copies, except for folio-keycloak, which needs special configurations

  • Deploy folio-keycloak in cluster mode

  • keycloak cluster should have 3 nodes

  • folio-keycloak configuration:

    • CPU: 1536

    • memory: 2750

    • memory-reservation: 2250

    • Xms:768m

    • Xmx:1536M

  • Additional keycloak env variables:

    • {"name": "KC_CACHE_EMBEDDED_REALMS_MAX_COUNT", "value": "300000"},

    • {"name": "KC_HOSTNAME_BACKCHANNEL_DYNAMIC", "value": "true"}, 

    • {"name":"KC_CACHE_METRICS_HISTOGRAMS_ENABLED", "value": "true"},

    • {"name": "KC_CACHE_EMBEDDED_USERS_MAX_COUNT", "value": "50000"},

    • {"name": "KC_CACHE_EMBEDDED_AUTHORIZATION_MAX_COUNT", "value":"300000" }

Test Eureka cluster's performance without NGINX

https://folio-org.atlassian.net/browse/PERF-1158

Without NGINX tests are more stable, response times are 5-15 % better than with it. Report

Deploy without NGINX

Test performance pf FOLIO with Binary Native sidecars

https://folio-org.atlassian.net/browse/PERF-1159

  • Response times ~20% better than with JVM based sidecars (v3.0.1).

  • No crashes, leaks or issues found.

  • Report

Deploy with native binary sidecars to save memory and potentially reducing ec2 instances

Sunflower: performance testing for different values of new ENV variables

https://folio-org.atlassian.net/browse/PERF-1141

 

See Jira for links to the reports and a summary table of recommended values for the new ENV variables

mod-data-export

-

In the context of Galileo, mod-dcb neeeds the following memory values.

 

{ "name": "JAVA_OPTS",
"value":-XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=192m -Xmx768m"
},