[PERF-173] Create AWS instances for Elasticsearch perf testing Created: 23/Dec/20  Updated: 29/Apr/21  Resolved: 29/Apr/21

Status: Closed
Project: perf-testing
Components: None
Affects versions: None
Fix versions: None

Type: Story Priority: P2
Reporter: Mikhail Fokanov Assignee: Mykola Borchuk
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Gantt End to Start
has to be done before MSEARCH-68 SPIKE - Investigate impact of impleme... Closed
Relates
relates to FOLIO-2973 Include mod-search into reference env... Closed
relates to FOLIO-2997 provision ES for reference envs Closed
Sprint: DevOps Sprint 110, DevOps Sprint 111, DevOps Sprint 112, DevOps Sprint 113
Development Team: FOLIO DevOps

 Description   

Purpose/Overview:

To make appropriate testing of required features on real-like data volume in cluster mode , there should be 2 AWS instances for Elasticsearch with the following configuration:

  1. Type: m5.xlarge
  2. 500 Gb EBS type GP-3 for storage

There should be (temporary for testing purposes):

  1. Public IP
  2. Access by SSH (e.g. with publickey) provided for Falcon developers (Bohdan Suprun, Pavel Filippov)

Note: these instances can be turned off from 18:00 to 06:00 UTC.



 Comments   
Comment by Marc Johnson [ 26/Jan/21 ]

Are 2 instances needed for each environment e.g. folio-snapshot, folio-testing, folio-snapshot-core, folio-testing-core?

Is there an issue for also adding elastic search to the scratch environments?

Is elastic search also going to be added to the Vagrant images?

Comment by Jakub Skoczen [ 02/Feb/21 ]

Mikhail Fokanov we've have discussed this during the DevOps planning meeting and there are some open questions:

1. does this ticket request an ES instance for the purpose of performance testing, if so this is not the same as providing instances for the sole purpose of running reference environments (which have very little data) and there should be a separate ticket created as a blocker for FOLIO-2973 Closed .

Mikhail Fokanov it might make sense for you to join the DevOps meeting at 15:30 CET to clarify?

Comment by Mikhail Fokanov [ 02/Feb/21 ]

Jakub Skoczen, Unfortunetly, I wasn't available at 15:30 CET today and I also will have another meeting at this time tommorrow.

First of all, these instances are needed for testing on the real data volume before the production. Also they could be used for: folio-snapshot, folio-testing, folio-snapshot-core, folio-testing-core . But if there is a small ammount of data on the mentioned envs, and you can provide another solution for the envs I would vote not to use these AWS instances for these envs.
On our rancher env (https://falcon.ci.folio.org/) we just deploy pod with Elasticsearch image and it works well with data less than 1 mln (we haven't tested it with bigger dataset yet).

Comment by Marc Johnson [ 02/Feb/21 ]

Mikhail Fokanov

Also they could be used for: folio-snapshot, folio-testing, folio-snapshot-core, folio-testing-core. But if there is a small ammount of data on the mentioned envs, and you can provide another solution for the envs I would vote not to use these AWS instances for these envs.

This issue blocks FOLIO-2973 Closed . That implied to me that these were intended to be used for the reference environments. Was / is that not the case?

Comment by Jakub Skoczen [ 03/Feb/21 ]

Mikhail Fokanov John Malconian In this case we will create a new ticket for creating ES instances to support reference envs and Vagrant boxes and block FOLIO-2973 Closed on the new ticket instead. ES for these will be deployed as a container and will be sufficient for small data sets. I will leave this ticket open.

Comment by John Malconian [ 16/Mar/21 ]

It's not entirely clear what resources are needed here, but If you are using the us-east-1 region, create the instance(s) in 'FOLIO VPC' in subnet 'ID Public East 1a' and/or 'ID Public East 1b'. If you are providing developers SSH access to the system, please create a unique ssh key pair. Store the private key in Secrets Manager so we can access if needed.

Comment by Mykola Borchuk [ 19/Apr/21 ]

For this task we use AWS ElasticSearch Service in us-west-2 region with Rancher. Amazon Elasticsearch Service (ES) uses private subnet and has own SG. Access to ES is from Rancher and containers. Snapshots for ES will be implemented (if possible) in other task. Also we changed DB to RDS for using DB snapshots with actual data.

Comment by Jakub Skoczen [ 27/Apr/21 ]

Stanislav Miroshnichenko have you reviewed this ticket? If so, can it be closed?

Comment by Stanislav Miroshnichenko [ 27/Apr/21 ]

Jakub Skoczen Yes, we can close it

Generated at Thu Feb 08 23:24:17 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.