[FOLIO-3590] Set-up multi-node elasticsearch cluster for snapshot environment Created: 23/Sep/22 Updated: 10/May/23 Resolved: 26/Sep/22 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | TBD |
| Reporter: | Viacheslav Kolesnyk | Assignee: | Unassigned |
| Resolution: | Won't Do | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue links: |
|
||||||||
| Sprint: | DevOps Sprint 160 | ||||||||
| Development Team: | FOLIO DevOps | ||||||||
| RCA Group: | TBD | ||||||||
| Description |
|
Overview: Snapshot environment is expected to have the same configuration as other reference environments. The bug (https://folio-org.atlassian.net/browse/MSEARCH-419) that could be reproduced only on multi-node Elasticsearch cluster was discovered recently. According to curl -X GET "localhost:9200/_cat/shards?v=true&pretty" command output for snapshot (see attached screenshot) there're no active replica shards that usually means that cluster is "single-node". In a scope of this story we need to configure elasticsearch cluster to have the same configuration as on bugfest environments. We need this update in order to test fix for
|
| Comments |
| Comment by jroot [ 23/Sep/22 ] |
|
For what it's worth, we use the Bitnami HA Elasticsearch Helm chart at Tamu. Here is our Answer config in Rancher: coordinating.heapsize = 256m coordinating.livenessProbe.enabled = false coordinating.readinessProbe.enabled = false coordinating.replicas = 2 coordinating.securityContext.enabled = true coordinating.securityContext.fsGroup = 1001 coordinating.securityContext.runAsUser = 1001 coordinating.service.port = 9200 coordinating.serviceAccount.create = true coordinating.startupProbe.enabled = false data.heapSize = 2048m data.livenessProbe.enabled = false data.persistence.enabled = true data.persistence.size = 200Gi data.readinessProbe.enabled = false data.replicas = 2 data.securityContext.enabled = true data.securityContext.fsGroup = 1001 data.securityContext.runAsUser = 1001 data.serviceAccount.create = true data.startupProbe.enabled = false global.coordinating.name = es-conn global.kibanaEnabled = true global.storageClass = "vsphere-kube-datastore" image.debug = false ingest.service.port = 9300 ingest.service.type = ClusterIP kibana.elasticsearch.hosts = {elasticsearch-prod-es-conn} kibana.elasticsearch.port = 9200 master.heapSize = 512m master.livenessProbe.enabled = false master.persistence.enabled = true master.persistence.size = 20Gi master.readinessProbe.enabled = false master.replicas = 3 master.securityContext.enabled = true master.securityContext.fsGroup = 1001 master.securityContext.runAsUser = 1001 master.service.port = 9300 master.service.type = ClusterIP master.serviceAccount.create = true master.startupProbe.enabled = false plugins = analysis-icu,analysis-kuromoji,analysis-smartcn,analysis-nori,analysis-phonetic |
| Comment by jroot [ 23/Sep/22 ] |
|
Here is our output of
curl -X GET "http://elasticsearch-prod-es-conn:9200/_cat/shards?v=true&pretty"
: index shard prirep state docs store ip node .kibana-event-log-7.14.2-000009 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.14.2-000009 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana_task_manager_7.17.1_001 0 p STARTED 17 163.7mb 10.42.12.32 elasticsearch-prod-data-0 .kibana_task_manager_7.17.1_001 0 r STARTED 17 163.7mb 10.42.11.3 elasticsearch-prod-data-1 .kibana_task_manager_7.14.1_001 0 r STARTED 14 77.6kb 10.42.12.32 elasticsearch-prod-data-0 .kibana_task_manager_7.14.1_001 0 p STARTED 14 77.6kb 10.42.11.3 elasticsearch-prod-data-1 .ds-ilm-history-5-2022.09.22-000022 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-ilm-history-5-2022.09.22-000022 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.16.1-000010 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.16.1-000010 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.16.3-000009 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.16.3-000009 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.15.2-000010 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.15.2-000010 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.09.06-000020 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.09.06-000020 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2021.12.27-000002 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2021.12.27-000002 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.17.1-000004 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.17.1-000004 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.14.1-000011 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.14.1-000011 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-ilm-history-5-2022.06.24-000016 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-ilm-history-5-2022.06.24-000016 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana_task_manager_7.16.3_001 0 p STARTED 17 3.6mb 10.42.12.32 elasticsearch-prod-data-0 .kibana_task_manager_7.16.3_001 0 r STARTED 17 3.6mb 10.42.11.3 elasticsearch-prod-data-1 folio-prod_instance_subject_tamu 2 p STARTED 0 226b 10.42.12.32 elasticsearch-prod-data-0 folio-prod_instance_subject_tamu 2 r STARTED 0 226b 10.42.11.3 elasticsearch-prod-data-1 folio-prod_instance_subject_tamu 2 r UNASSIGNED folio-prod_instance_subject_tamu 1 r STARTED 2208454 331mb 10.42.12.32 elasticsearch-prod-data-0 folio-prod_instance_subject_tamu 1 p STARTED 2208454 374.4mb 10.42.11.3 elasticsearch-prod-data-1 folio-prod_instance_subject_tamu 1 r UNASSIGNED folio-prod_instance_subject_tamu 3 r STARTED 0 226b 10.42.12.32 elasticsearch-prod-data-0 folio-prod_instance_subject_tamu 3 p STARTED 0 226b 10.42.11.3 elasticsearch-prod-data-1 folio-prod_instance_subject_tamu 3 r UNASSIGNED folio-prod_instance_subject_tamu 0 p STARTED 0 226b 10.42.12.32 elasticsearch-prod-data-0 folio-prod_instance_subject_tamu 0 r STARTED 0 226b 10.42.11.3 elasticsearch-prod-data-1 folio-prod_instance_subject_tamu 0 r UNASSIGNED .ds-.logs-deprecation.elasticsearch-default-2022.04.04-000009 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.04.04-000009 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.03.21-000008 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.03.21-000008 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 folio-prod_instance_tamu 2 p STARTED 0 226b 10.42.12.32 elasticsearch-prod-data-0 folio-prod_instance_tamu 2 r STARTED 0 226b 10.42.11.3 elasticsearch-prod-data-1 folio-prod_instance_tamu 2 r UNASSIGNED folio-prod_instance_tamu 1 r STARTED 4594902 14.7gb 10.42.12.32 elasticsearch-prod-data-0 folio-prod_instance_tamu 1 p STARTED 4594902 14.8gb 10.42.11.3 elasticsearch-prod-data-1 folio-prod_instance_tamu 1 r UNASSIGNED folio-prod_instance_tamu 3 r STARTED 0 226b 10.42.12.32 elasticsearch-prod-data-0 folio-prod_instance_tamu 3 p STARTED 0 226b 10.42.11.3 elasticsearch-prod-data-1 folio-prod_instance_tamu 3 r UNASSIGNED folio-prod_instance_tamu 0 p STARTED 0 226b 10.42.12.32 elasticsearch-prod-data-0 folio-prod_instance_tamu 0 r STARTED 0 226b 10.42.11.3 elasticsearch-prod-data-1 folio-prod_instance_tamu 0 r UNASSIGNED .kibana-event-log-7.15.2-000009 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.15.2-000009 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.09.20-000021 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.09.20-000021 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2021.12.13-000001 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2021.12.13-000001 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.05.02-000011 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.05.02-000011 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.08.09-000018 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.08.09-000018 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.05.30-000013 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.05.30-000013 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana_task_manager_7.16.1_001 0 r STARTED 17 90.4mb 10.42.12.32 elasticsearch-prod-data-0 .kibana_task_manager_7.16.1_001 0 p STARTED 17 90.4mb 10.42.11.3 elasticsearch-prod-data-1 .kibana_7.17.1_001 0 p STARTED 24 2.1mb 10.42.12.32 elasticsearch-prod-data-0 .kibana_7.17.1_001 0 r STARTED 24 2.1mb 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.05.16-000012 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.05.16-000012 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.14.2-000011 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.14.2-000011 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.17.1-000007 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.17.1-000007 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-ilm-history-5-2022.08.23-000020 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-ilm-history-5-2022.08.23-000020 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.17.1-000006 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.17.1-000006 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana_task_manager_7.14.2_001 0 p STARTED 14 54.9kb 10.42.12.32 elasticsearch-prod-data-0 .kibana_task_manager_7.14.2_001 0 r STARTED 14 54.9kb 10.42.11.3 elasticsearch-prod-data-1 .apm-agent-configuration 0 r STARTED 0 208b 10.42.12.32 elasticsearch-prod-data-0 .apm-agent-configuration 0 p STARTED 0 226b 10.42.11.3 elasticsearch-prod-data-1 .kibana_7.15.2_001 0 r STARTED 21 2.1mb 10.42.12.32 elasticsearch-prod-data-0 .kibana_7.15.2_001 0 p STARTED 21 2.1mb 10.42.11.3 elasticsearch-prod-data-1 .kibana_7.16.3_001 0 r STARTED 22 2.1mb 10.42.12.32 elasticsearch-prod-data-0 .kibana_7.16.3_001 0 p STARTED 22 2.1mb 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.14.2-000010 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.14.2-000010 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.16.1-000009 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.16.1-000009 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-ilm-history-5-2022.07.24-000017 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-ilm-history-5-2022.07.24-000017 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.14.2-000012 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.14.2-000012 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.02.21-000006 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.02.21-000006 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.14.1-000012 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.14.1-000012 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.01.10-000003 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.01.10-000003 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.15.2-000008 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.15.2-000008 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.06.13-000014 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.06.13-000014 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.03.07-000007 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.03.07-000007 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .geoip_databases 0 p STARTED 41 38.8mb 10.42.12.32 elasticsearch-prod-data-0 .geoip_databases 0 r STARTED 41 38.8mb 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.15.2-000011 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.15.2-000011 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .tasks 0 p STARTED 30 43.8kb 10.42.12.32 elasticsearch-prod-data-0 .tasks 0 r STARTED 30 49.9kb 10.42.11.3 elasticsearch-prod-data-1 .kibana_7.14.1_001 0 r STARTED 18 2.1mb 10.42.12.32 elasticsearch-prod-data-0 .kibana_7.14.1_001 0 p STARTED 18 2.1mb 10.42.11.3 elasticsearch-prod-data-1 folio-prod_authority_tamu 2 p STARTED 0 226b 10.42.12.32 elasticsearch-prod-data-0 folio-prod_authority_tamu 2 r STARTED 0 226b 10.42.11.3 elasticsearch-prod-data-1 folio-prod_authority_tamu 2 r UNASSIGNED folio-prod_authority_tamu 1 r STARTED 0 226b 10.42.12.32 elasticsearch-prod-data-0 folio-prod_authority_tamu 1 p STARTED 0 226b 10.42.11.3 elasticsearch-prod-data-1 folio-prod_authority_tamu 1 r UNASSIGNED folio-prod_authority_tamu 3 r STARTED 0 226b 10.42.12.32 elasticsearch-prod-data-0 folio-prod_authority_tamu 3 p STARTED 0 226b 10.42.11.3 elasticsearch-prod-data-1 folio-prod_authority_tamu 3 r UNASSIGNED folio-prod_authority_tamu 0 p STARTED 0 226b 10.42.12.32 elasticsearch-prod-data-0 folio-prod_authority_tamu 0 r STARTED 0 226b 10.42.11.3 elasticsearch-prod-data-1 folio-prod_authority_tamu 0 r UNASSIGNED .ds-.logs-deprecation.elasticsearch-default-2022.06.28-000015 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.06.28-000015 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .apm-custom-link 0 p STARTED 0 208b 10.42.12.32 elasticsearch-prod-data-0 .apm-custom-link 0 r STARTED 0 226b 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.16.3-000007 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.16.3-000007 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.16.3-000008 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.16.3-000008 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.07.26-000017 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.07.26-000017 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.16.3-000006 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.16.3-000006 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.14.1-000013 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.14.1-000013 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.17.1-000005 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.17.1-000005 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.16.1-000008 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.16.1-000008 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.08.23-000019 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.08.23-000019 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.14.1-000010 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.14.1-000010 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana_task_manager_7.15.2_001 0 p STARTED 15 50.4mb 10.42.12.32 elasticsearch-prod-data-0 .kibana_task_manager_7.15.2_001 0 r STARTED 15 50.4mb 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.04.18-000010 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.04.18-000010 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.02.07-000005 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.02.07-000005 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana-event-log-7.16.1-000007 0 p STARTED 10.42.12.32 elasticsearch-prod-data-0 .kibana-event-log-7.16.1-000007 0 r STARTED 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.01.24-000004 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.01.24-000004 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana_7.14.2_001 0 p STARTED 18 2.1mb 10.42.12.32 elasticsearch-prod-data-0 .kibana_7.14.2_001 0 r STARTED 18 2.1mb 10.42.11.3 elasticsearch-prod-data-1 .ds-.logs-deprecation.elasticsearch-default-2022.07.12-000016 0 r STARTED 10.42.12.32 elasticsearch-prod-data-0 .ds-.logs-deprecation.elasticsearch-default-2022.07.12-000016 0 p STARTED 10.42.11.3 elasticsearch-prod-data-1 .kibana_7.16.1_001 0 r STARTED 22 2.1mb 10.42.12.32 elasticsearch-prod-data-0 .kibana_7.16.1_001 0 p STARTED 22 2.1mb 10.42.11.3 elasticsearch-prod-data-1 |
| Comment by Jakub Skoczen [ 26/Sep/22 ] |
|
Viacheslav Kolesnyk there's no expectation that the snapshot env will match a "production" env setup and e.g run certain infrastructure components in a clustered/multi-node setup. I assume the original bug has been reproduced on bugfest and the subsequent fix has been verified there? If that's the case you should be able to close the
|
| Comment by Viacheslav Kolesnyk [ 27/Sep/22 ] |
|
Jakub Skoczen Ok, thank you for explanation |