Elasticsearch (UXPROD-2591)

[UXPROD-2592] ElasticSearch - APIs for search and data import Created: 17/Jul/20  Updated: 13/Oct/20  Resolved: 30/Sep/20

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: None
Parent: Elasticsearch

Type: New Feature Priority: TBD
Reporter: Cate Boerema (Inactive) Assignee: twliu
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Epic Link: Elasticsearch
Development Team: 仁者无敌 "Benevolence"
Rank: Cornell (Full Sum 2021): R2
Rank: GBV (MVP Sum 2020): R2
Rank: MO State (MVP June 2020): R4

 Description   

*High level feature description:* Provide an API for search and data import.

Search will slow down with the increasing number of data in the databases. The ES search engine will help increase search efficiency when data volume is big and data relationship is complicated.

Provide an API for search and data import. The ES component includes three modules: mod-search-es, mod-data-loader-es and raml-module-builder. The mod-search-es provides an interface for business modules to call. The mod-search-es is the backend to convert the database data into index data. The raml-module-builder will monitor the changes in the database and send data to be indexed to the mod-data-loader-es.

Each individual business module decides whether to introduce the ES and where to introduce it. In order to connect to the ES modules, 1.Database query code should be changed to call the mod-search-es; 2. Integrate the raml-module-builder with the mod-inventory-storage.

*Detailed plans:*

1. mod-search-es
1) This module is used for checking ES data.
2) Configure some configuration information of import ES data, such as database connection configuration, database driver, database user name, database password, SQL statement of database query, configuration of source data fields and target fields, as well as the configuration content of converter used.

2. mod-data-loader-es
1) This module is used to parse the read configuration information and import the data into ES.
2) Listen to the message content in Kafka and parse the data into ES after transformation.

3. raml-module-builder
Modify THE RMB module to listen for HTTP requests under the configured route, forward the requested content to Kafka, and then synchronize the data to ES through MOd-data-Loader-ES.



 Comments   
Comment by Nick Cappadona [ 09/Oct/20 ]

Cate Boerema twliu – Any details you can share since this was closed and marked as resolved last week?

Comment by twliu [ 09/Oct/20 ]

Nick CappadonaThe Benevolence team has completed the three modules and will demo in the next sprint review. Thank you for asking!

Comment by Nick Cappadona [ 09/Oct/20 ]

Thanks twliu. Looking forward to it

Comment by Massoud Alshareef [ 11/Oct/20 ]

Thank you twliu. Can't wait to see it.

Comment by Cate Boerema (Inactive) [ 12/Oct/20 ]

Glad to hear you'll have something to demo next sprint review, twliu! Will this work be merged and available in the Honeysuckle release? If not, please remove the Q3 2020 fix version from this issue, as that indicates the release in which a feature will become generally available.

Comment by Jenn Colt [ 13/Oct/20 ]

We looked for these in the FOLIO github and could not find them. If they are ready to be demo'd would you be able to point us to where they are? Maybe we just didn't know the right names for the modules, the ones listed here aren't in github.

Comment by twliu [ 13/Oct/20 ]

Jenn ColtThank you for asking! The code was not submitted to Github. The community has a plan to adjust the code to make it cover more business logics and usable in a larger scope. The plan will be demoed in tomorrow's TC meeting.

Generated at Fri Feb 09 00:25:14 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.