[FOLIO-2973] Include mod-search into reference environments Created: 21/Jan/21 Updated: 03/Mar/21 Resolved: 02/Mar/21 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Story | Priority: | P3 |
| Reporter: | Magda Zacharska | Assignee: | Ian Hardy |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||||||
| Issue links: |
|
||||||||||||||||||||||||||||||||
| Sprint: | DevOps Sprint 108 | ||||||||||||||||||||||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||||||||||||||||||||||
| Description |
|
To allow integration the module mod-search needs to be added into folio-snapshot/testing environments: GitHub: The mod-search requires Elasticsearch engine and Kafka to be run before start the module. Following properties are have to be propagated to the container: And all standard DB properties. |
| Comments |
| Comment by Jakub Skoczen [ 26/Jan/21 ] |
|
We'll discuss this request for next sprint. |
| Comment by Jakub Skoczen [ 03/Feb/21 ] |
|
The DevOps team discussed that on Tuesday but first we need to address the Elastic search dependency (
|
| Comment by Ian Hardy [ 17/Feb/21 ] |
|
Note that elasticsearch is running on a non-standard port (9301) in the reference builds (See
|
| Comment by David Crossley [ 18/Feb/21 ] |
|
To prepare adding for real, i have configured the folio-testing-test reference environment. It does deploy successfully, see Jenkins output folio-testing-test/110 However doing ssh to the server, shows the docker logs spewing errors. |
| Comment by Bohdan Suprun (Inactive) [ 18/Feb/21 ] |
|
Hi David Crossley, There is: Caused by: java.net.UnknownHostException: elasticsearch: System error Is it proper ES instance host name elasticsearch? Have you set ELASTICSEARCH_HOST and ELASTICSEARCH_PORT env variables? The errors at the top, saying org.postgresql.util.PSQLException: ERROR: relation "language_config" does not exist most likely because /_/tenant init has not been called. |
| Comment by Bohdan Suprun (Inactive) [ 18/Feb/21 ] |
|
Ok as per Ian's comment above, ES has non standard port, please specify ELASTICSEARCH_PORT=9301 |
| Comment by Ian Hardy [ 19/Feb/21 ] |
|
Hi Bohdan Suprun I've added mod-search to a test build, it gets deployed OK now and finds elasticsearch but the log doesn't look too happy. Can you have a look at this (mod-search.log)? |
| Comment by Bohdan Suprun (Inactive) [ 19/Feb/21 ] |
|
Hi Ian Hardy, I believe it something expected. Most likely that when mod-search is started it starts immediately receive events from Kafka topics (because they are accumulated), however this process requires some interaction with DB (but DB is set up only when tenant init is called, what happens only at 15:49:06.731 - liquibase attempts to create schemas here). And then this attempt of events processing should be successful: 15:49:06.863 [mod-search-listener-0-C-1] INFO org.folio.search.integration.KafkaMessageListener - Processing resource events from kafka [number of events: 29] |
| Comment by Ian Hardy [ 19/Feb/21 ] |
|
OK thanks Bohdan Suprun. Would it be possible to modify at some point so it doesn't start to receive events/talk to the DB until after tenant init? This might be a bit confusing for operators. I added configuration for mod-search to the build scripts. I should show up on folio-testing tomorrow. It won't show up on snapshot until it's required as a dependency presumably by inventory-es. I did not add it to the *-core builds.
|
| Comment by Ian Hardy [ 20/Feb/21 ] |
|
looks like mod-search was OOM killed by Docker 4 hours after being deployed on folio testing for exceeding its configured memory limit. The memory limit is configured in the launch descriptor and is currently set at 357913941 bytes. |
| Comment by Bohdan Suprun (Inactive) [ 22/Feb/21 ] |
|
Hi Ian Hardy, Is there a dump file available? We have few more thinks remaining to configure the module (e.g.
|
| Comment by Ian Hardy [ 22/Feb/21 ] |
|
Hi Bohdan Suprun There's no dump as everything in the module is operating normally, it just gets killed by docker for using more memory than the container is allowed. FWIW, when I restart the container with 512MB it's staying up, so you might try setting that in your launch descriptor (thats 536870912 bytes) unless you have reason to suspect it should not be using that much memory. After giving it more memory and starting it back up, I do get some errors in the UI diong a keyword search. As far as I can tell, this is just the error elasticsearch throws when you're operating an a field that does not exist (or not ignoring unmapped fields) Here's what the mod-search log says {"error":{"root_cause":[{"type":"query_shard_exception","reason":"No mapping found for [sort_title] in order to sort on","index_uuid":"HcTIcGznTROQI-uC_SVIxA","index":"instance_diku"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"instance_diku","node":"ZtBo_nlHQiK0OSTqzNs1Xw","reason":{"type":"query_shard_exception","reason":"No mapping found for [sort_title] in order to sort on","index_uuid":"HcTIcGznTROQI-uC_SVIxA","index":"instance_diku"}}]},"status":400}
I'll attach the complete log here as well. Let me know if there's anything I can do to help. mod-search-22-feb.log |
| Comment by Dmytro Melnyshyn [ 23/Feb/21 ] |
|
Hi Ian Hardy Dependency "search": "0.2" added to package.json in ui-inventory-es. |
| Comment by Ian Hardy [ 23/Feb/21 ] |
|
Great, thanks Dmytro Melnyshyn, I see mod-search showing up on folio-snapshot now. I'll close
|
| Comment by Ian Hardy [ 01/Mar/21 ] |
|
Bohdan Suprun have you had a chance to look at mod-search/ui-inventory-es on folio-snapshot? It available now but still reports the Elasticsearch error described above. Is there some Elasticsearch setup that needs to be happening but is not? |
| Comment by Bohdan Suprun (Inactive) [ 01/Mar/21 ] |
|
Hi Ian Hardy, We have merged a required fix today, I'll check after redeploy. |
| Comment by Bohdan Suprun (Inactive) [ 02/Mar/21 ] |
|
Hi Ian Hardy, verified and works as expected. Thank you! |
| Comment by Ian Hardy [ 02/Mar/21 ] |
|
Awesome, looks good Bohdan Suprun! One final nitpick, would it be possible to handle the errors that fill the log up before the module is enabled for a tenant? You can see an example in the attached mod-search.log. Let me know what you think. If you prefer I can open a new issue for this. |
| Comment by Bohdan Suprun (Inactive) [ 02/Mar/21 ] |
|
Hi Ian Hardy, Is it still the issue? I thought Pavel Filippov, committed some changes few days ago. If it is the case please open a new issue, we will take a look at this (maybe next sprint, because the team is fully committed now). |
| Comment by Ian Hardy [ 02/Mar/21 ] |
|
Thanks, closing this one now then. This is still happening as of today's snapshot build. I opened
|