Versions Compared

Old Version 6

changes.mady.by.user Olga Kondratenko

Saved on

compared with

New Version 7

changes.mady.by.user Olga Kondratenko

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Table of Contents

...

Test #

Start time

End time

Instances number

Test Conditions

reindexing on Poppy release, consortium environment

Duration *

Notes


1

2024-10-17T12:41:14

2024-10-17T14:06:46

10,099,620

Sequential: fs07000001

1 hour 25 min

  • mod-search:

  1. task count = 4

  2. Mem Hard Limit = 2592

  3. Mem Soft Limit = 2480

  4. Xmx = -XX:MaxRAMPercentage=85.0

  • mod-inventory-storage task count = 4

  • open search Data nodes instance scaled up to r6g.4xlarge.search

2

2024-10-17T14:35:59

2024-10-17T19:49:26

27,957,839

Sequential: fs09000000

5 hours 14 min

3

2024-10-17T19:58:07

2024-10-17T20:12:24

1,210,000

Sequential: fs07000002

14 min

4

2024-10-17T20:21:23

2024-10-17T22:46:34

In parallel: 3 tenants

All tenants reindex FAILED in
2 hours 25 min "MERGE_FAILED"

5

2024-10-16T14:38:08

2024-10-16T15:53:37

10,099,620

Sequential: fs07000001

1 hour 15 min

  • mod-search:

  1. task count = 4

  2. Mem Hard Limit = 4592

  3. Mem Soft Limit = 4480

  4. Xmx = -XX:MaxRAMPercentage=85.0

  • mod-inventory-storage task count = 4

  • open search Data nodes instance scaled up to r6g.4xlarge.search

6

2024-10-16T16:11:53

2024-10-16T21:08:00

27,957,839

Sequential: fs09000000

4 hours 57 min

7

2024-10-17T06:20:22

2024-10-17T06:34:04

1,210,000

Sequential: fs07000002

14 min

8

2024-10-17T06:40:22

2024-10-17T09:12:00

In parallel: 3 tenants

reindex FAILED for 1 tenant [tenantId: fs09000000, error: java.util.concurrent.CompletionException: org.folio.search.exception.FolioIntegrationException: Failed to publish reindex records range]

9

2024-10-15T10:16:50

2024-10-15T12:04:51

10,099,620

Sequential: fs07000001

1 hour 48 min

  • mod-search:4.0.0-SNAPSHOT.278

10

2024-10-16T08:01:14

2024-10-16T12:32:19

10,099,620

Sequential: fs07000001

4 hours 31 min

  • Data nodes instance scaled down to r6g.large.search

...

Configuration

Ramsons

Poppy

Database

1 instance of db.r6g.8xlarge database instances, writer

2 instances of db.r6g.xlarge database instances, one reader, and one writer

mod-search version

mod-search:4.0.0-SNAPSHOT.281

3.0.0-SNAPSHOT.151

mod-search task count

4

8

Data node instances count (r6g.4xlarge.search)

4

6

Dedicated master nodes

No

3 of r6g.large.search instances

Duration comparison:

Ramsons

Poppy

Delta absolut

Delta

1 hour 25 min

9 hours 38 min

8 hours 13 min

6.8 times

...

Database use the same average amount of connections

...

Open Search metrics

CPU utilization percentage for all data nodes (Average).

...

Memory usage percentage for all data nodes (Average).

...

chrome_tuP3wGC9cE.png

Appendix

Infrastructure

PTF-environment pconrcp1

  • 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 

  • 1 instances of db.r6g.8xlarge database instances. Engine version 16.1

  • MSK 
    4 m5kafka.m7g.2xlarge xlarge brokers in 2 zones

    • Apache Kafka version 23.87.0x

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

  • OpenSearchcluster

    • OpenSearch version 2.713;

    • Data nodes

      • Availability Zone(s) - 2-AZ without standby

      • Instance type - r6g.4xlarge.search

      • Number of nodes - 64

      • EBS volume size (GiB) - 500300

      • Provisioned IOPS - 3000IOPS

      • Provisioned Throughput (MiB/s) - 250 MiB/s

    • Dedicated master nodes

    • Instance type - r6g.large.search

      • Number of nodes - 3

      Enabled - No

Data structure

Module
Cluster Resources -

Task Definition Revision

Module Version

Task Count

Mem Hard Limit

Mem Soft Limit

CPU Units

Xmx

Metaspace Size

Max Metaspace Size

mod-search

10

mod-search:4.0.0-SNAPSHOT.281

4

2592

2480

2048

MaxRAMPercentage=85.0

512

1024

mod-authtoken

1

mod-authtoken:2.16.0-SNAPSHOT.303

2

1440

1152

512

922

88

128

mod-inventory-storage

4

mod-inventory-storage:27.2.0-SNAPSHOT.773

4

4096

3690

2048

3076

512

1024

mod-inventory

1

mod-inventory:20.3.0-SNAPSHOT.546

2

2880

2592

1024

1814

384

512

mod-users

1

mod-users:19.3.2-SNAPSHOT.696

2

1024

896

0

768

88

128

nginx-okapi

1

nginx-okapi:2023.06.14

2

1024

896

0

0

0

0

okapi-b

1

okapi:5.3.0

3

1684

1440

1024

922

384

512

pub-okapi

1

pub-okapi:2023.06.14

2

1024

896

0

768

0

0

...