[FOLIO-2260] Include LDP in reference environments Created: 12/Sep/19  Updated: 06/Oct/20  Resolved: 06/Oct/20

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: TBD
Reporter: Wayne Schneider Assignee: Ian Hardy
Resolution: Done Votes: 0
Labels: devops
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks FOLIO-2711 Deploy mod-ldp Closed
Sprint: DevOps: Sprint 97, DevOps: Sprint 99, DevOps: Sprint 95, DevOps: sprint 94, DevOps: Sprint 96
Development Team: FOLIO DevOps

 Description   

The LDP is ready for inclusion in the FOLIO reference environments. It is not a FOLIO module, per se, but a Postgres or Redshift database and an ETL data loader that harvests FOLIO data from Okapi.

Information from Nassib Nassar:

For this purpose, it's basically a Postgres database - could be simply an RDS instance - and a command line utility that needs to run periodically to update the data from a FOLIO instance.

One unusual thing is that the public user would be connecting directly to the Postgres database.

The database currently uses about 20 MB to store data from 85 interfaces in folio-snapshot. It will grow with the addition of more interfaces and the potential accumulation of history data (it keeps a history of any changes in the FOLIO instance). The current ETL/loading time is about 24 seconds.

It would be ideal to leave the database running for the most part so that it can build up some history of changes (not that there are many changes in folio-snapshot), and then once per evening just update the loader software and run it, and only rarely rebuild the entire database.



 Comments   
Comment by Ian Hardy [ 15/Sep/20 ]

Approach:

  • folio-ansible will set up an additional database for ldp while provisioning the reference environment server. This is the same instance that theFOLIO db for the environment is. This provides easy access for mod-ldp to the ldp db. Note that this is an ephemeral environment.
  • The LDP software will be deployed as a docker container. This is to avoid building it from source as part of the reference builds. This keeps concerns separate (ability to build reference environment does not depend on ability to build LDP) folio-ansible will provision the config file and docker container likely using docker-compose for orchestration.

Tasks that need to be completed to finish this

  • ansible role for deploying LDP. Has two main parts. 1. creating the LDP db. use the existing create-database role and adds the extra users for mod-ldp (see branch folio-2260-ldp-user) 2. template the configfile and start the LDP container with docker-compose
  • Update the jenkins build agent image so it has the prerequisites required to compile the LDP software from source
  • Add a Jenkinsfile for ldp that will publish a container with each commit to master. Make sure the server is logging to stdout so logs can be grabbed the normal docker way.
Comment by Ian Hardy [ 28/Sep/20 ]

Status update.

Done:
We've updated the jenkins build agent to be able to build LDP (needed to install requirements like cmake g++ etc). We've also created a Jenkinsfile to publish a docker image representing the tip of master with each commit to LDP.

Still to do:

  • complete ansible role in folio-ansible to set up ldp database when system is built and deploy ldp container.
Comment by Ian Hardy [ 06/Oct/20 ]

Added playbooks to folio-ansible to install and run LDP on folio-snapshot and folio-testing (also *-core builds).

Generated at Thu Feb 08 23:19:22 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.