[FOLIO-2973] Include mod-search into reference environments Created: 21/Jan/21  Updated: 03/Mar/21  Resolved: 02/Mar/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Story Priority: P3
Reporter: Magda Zacharska Assignee: Ian Hardy
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File ftt-110-mod-search.txt.gz     PNG File image-2021-02-22-15-27-05-596.png     File mod-search-22-feb.log     File mod-search.log    
Issue links:
Blocks
is blocked by FOLIO-2997 provision ES for reference envs Closed
Gantt End to Start
has to be done before FOLIO-2981 Include ui-inventory-es into referenc... Closed
Relates
relates to PERF-173 Create AWS instances for Elasticsearc... Closed
relates to UISEES-39 Missing dependency in package.json Closed
relates to FOLIO-2863 pipeline changes required for Spring ... Blocked
Sprint: DevOps Sprint 108
Development Team: FOLIO DevOps

 Description   

To allow integration the module mod-search needs to be added into folio-snapshot/testing environments:

GitHub:
https://github.com/folio-org/mod-search

The mod-search requires Elasticsearch engine and Kafka to be run before start the module.

Following properties are have to be propagated to the container:
KAFKA_HOST
KAFKA_PORT
ELASTICSEARCH_HOST
ELASTICSEARCH_PORT

And all standard DB properties.



 Comments   
Comment by Jakub Skoczen [ 26/Jan/21 ]

We'll discuss this request for next sprint.

Comment by Jakub Skoczen [ 03/Feb/21 ]

The DevOps team discussed that on Tuesday but first we need to address the Elastic search dependency ( FOLIO-2997 Closed ).

Comment by Ian Hardy [ 17/Feb/21 ]

Note that elasticsearch is running on a non-standard port (9301) in the reference builds (See FOLIO-2997 Closed ) so the ops group will need to set ELASTICSEARCH_PORT appropriately when this issue is done.

Comment by David Crossley [ 18/Feb/21 ]

To prepare adding for real, i have configured the folio-testing-test reference environment.

It does deploy successfully, see Jenkins output folio-testing-test/110

However doing ssh to the server, shows the docker logs spewing errors.
Attached to this ticket: ftt-110-mod-search.txt.gz

Comment by Bohdan Suprun (Inactive) [ 18/Feb/21 ]

Hi David Crossley,

There is:

Caused by: java.net.UnknownHostException: elasticsearch: System error

Is it proper ES instance host name elasticsearch? Have you set ELASTICSEARCH_HOST and ELASTICSEARCH_PORT env variables?

The errors at the top, saying org.postgresql.util.PSQLException: ERROR: relation "language_config" does not exist most likely because /_/tenant init has not been called.

Comment by Bohdan Suprun (Inactive) [ 18/Feb/21 ]

David Crossley,

Ok as per Ian's comment above, ES has non standard port, please specify ELASTICSEARCH_PORT=9301

Comment by Ian Hardy [ 19/Feb/21 ]

Hi Bohdan Suprun I've added mod-search to a test build, it gets deployed OK now and finds elasticsearch but the log doesn't look too happy. Can you have a look at this (mod-search.log)?

Comment by Bohdan Suprun (Inactive) [ 19/Feb/21 ]

Hi Ian Hardy,

I believe it something expected.

Most likely that when mod-search is started it starts immediately receive events from Kafka topics (because they are accumulated), however this process requires some interaction with DB (but DB is set up only when tenant init is called, what happens only at 15:49:06.731 - liquibase attempts to create schemas here).

And then this attempt of events processing should be successful:

15:49:06.863 [mod-search-listener-0-C-1] INFO  org.folio.search.integration.KafkaMessageListener - Processing resource events from kafka [number of events: 29]
Comment by Ian Hardy [ 19/Feb/21 ]

OK thanks Bohdan Suprun. Would it be possible to modify at some point so it doesn't start to receive events/talk to the DB until after tenant init? This might be a bit confusing for operators.

I added configuration for mod-search to the build scripts. I should show up on folio-testing tomorrow. It won't show up on snapshot until it's required as a dependency presumably by inventory-es. I did not add it to the *-core builds.

 

Comment by Ian Hardy [ 20/Feb/21 ]

looks like mod-search was OOM killed by Docker 4 hours after being deployed on folio testing for exceeding its configured memory limit. The memory limit is configured in the launch descriptor and is currently set at 357913941 bytes.

Comment by Bohdan Suprun (Inactive) [ 22/Feb/21 ]

Hi Ian Hardy,

Is there a dump file available?

We have few more thinks remaining to configure the module (e.g. MSEARCH-50 Closed ). It is probably related to this.

Comment by Ian Hardy [ 22/Feb/21 ]

Hi Bohdan Suprun There's no dump as everything in the module is operating normally, it just gets killed by docker for using more memory than the container is allowed. FWIW, when I restart the container with 512MB it's staying up, so you might try setting that in your launch descriptor (thats 536870912 bytes) unless you have reason to suspect it should not be using that much memory.

After giving it more memory and starting it back up, I do get some errors in the UI diong a keyword search.

As far as I can tell, this is just the error elasticsearch throws when you're operating an a field that does not exist (or not ignoring unmapped fields) Here's what the mod-search log says

 {"error":{"root_cause":[{"type":"query_shard_exception","reason":"No mapping found for [sort_title] in order to sort on","index_uuid":"HcTIcGznTROQI-uC_SVIxA","index":"instance_diku"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"instance_diku","node":"ZtBo_nlHQiK0OSTqzNs1Xw","reason":{"type":"query_shard_exception","reason":"No mapping found for [sort_title] in order to sort on","index_uuid":"HcTIcGznTROQI-uC_SVIxA","index":"instance_diku"}}]},"status":400}

I'll attach the complete log here as well. Let me know if there's anything I can do to help. mod-search-22-feb.log

Comment by Dmytro Melnyshyn [ 23/Feb/21 ]

Hi Ian Hardy  Dependency "search": "0.2" added to package.json in ui-inventory-es.

Comment by Ian Hardy [ 23/Feb/21 ]

Great, thanks Dmytro Melnyshyn, I see mod-search showing up on folio-snapshot now. I'll close FOLIO-2981 Closed (add ui-inventory-es to snapshot/testing). I'll leave this one open for troubleshooting mod-search.

Comment by Ian Hardy [ 01/Mar/21 ]

Bohdan Suprun have you had a chance to look at mod-search/ui-inventory-es on folio-snapshot? It available now but still reports the Elasticsearch error described above. Is there some Elasticsearch setup that needs to be happening but is not?

Comment by Bohdan Suprun (Inactive) [ 01/Mar/21 ]

Hi Ian Hardy, We have merged a required fix today, I'll check after redeploy.

Comment by Bohdan Suprun (Inactive) [ 02/Mar/21 ]

Hi Ian Hardy, verified and works as expected. Thank you!

Comment by Ian Hardy [ 02/Mar/21 ]

Awesome, looks good Bohdan Suprun! One final nitpick, would it be possible to handle the errors that fill the log up before the module is enabled for a tenant? You can see an example in the attached mod-search.log. Let me know what you think. If you prefer I can open a new issue for this.

Comment by Bohdan Suprun (Inactive) [ 02/Mar/21 ]

Hi Ian Hardy,

Is it still the issue? I thought Pavel Filippov, committed some changes few days ago. If it is the case please open a new issue, we will take a look at this (maybe next sprint, because the team is fully committed now).

Comment by Ian Hardy [ 02/Mar/21 ]

Thanks, closing this one now then. This is still happening as of today's snapshot build. I opened MSEARCH-72 Closed to address this when the team has time. 

Generated at Thu Feb 08 23:24:38 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.