2018-01-18 - System Operations and Management SIG Notes

Date

18 Jan 2018

Attendees

Goals

Discussion items

Time	Item	Who	Notes
5	Welcome	Ingolf	Welcome new SIG member Lina Lakhia Assign today's note taker - Chris Manly
10	Review last week's action items	Ingolf	List of Integrations review by Tod and Chris is done. Discussion of user management issues with Kate and Katalin hasn't happened UUID issue and API consistency issue has been sent to Sharon and Holly to be shared in Madrid. Holly will not be in Madrid. (Follow up discussion on the issue of potential inherent performance issue related to UUIDs, as was discussed in e-mail)
20	Conceptual Architectural Diagram	Wayne	Diagrams loaded as attachments (Simple conceptual.pdf) (Vagrant box (folio-ansible).pdf) (Cloud.pdf).
20	Data migration	Ingolf / group	Data migration is a big load of work to the SIG. what aspects of data migration will the SIG be responsible for ? what should be delivered by the SIG ? how will the SIG involve other Subject Matter Experts (SMEs) from other SIGs to work on data migration ? (in what form and when will that happen ?) It seems to me (Ingolf) that just one hour per week is not enough to discuss this (data migration) and work on this further. Can we choose one of the following solutions: build a subgroup. A limited number of experts meet in addition to the regular SIG meetings, on a regular basis, and is concerned with questions of data migration. The entire SIG meets more often, say, twice a week. The SIG meetings take place weekly and cover data migration issues, but the SIG meetings will be longer (90 minutes).
5	Next Meeting	Ingolf	topics for next meeting

Notes From Discussion

Architectural Diagrams and Explanation from Wayne Schneider

FOLIO Architecture is a microservices model with an API gateway
User connects to webserver which delivers a Stripes bundle to the browser (i.e. essentially a javascript application running in the browser) in a single transaction.
From then on the browser is running the js app, which is communicating with the gateway (OKAPI), essentially a proxy gateway for web services with some additional services such as permission enforcement.
Okapi can have its own datastore to persist informtion on how it is configured (technically optional, but needed for any real-world use)
When a request comes in from the browser, say, for inventory, Okapi knows the route to that endpoint and passese the request through.
There are business logic modules and storage modules (Codex might be a third kind depending on how you look at it).
If a BLM needs data from storage, that request will go through OKAPI, and then back to the BLM before it goes to the user.
OKAPI is central, but it's fairly simple (one of the smallest pieces of code)
It's possible to have different pieces of code implementing the same interface - OKAPI doesn't care what the code is behind an API interface, it only cares about the interface definition.
Three things OKAPI cares about:

1. Interface definition, as captured in the module descriptor for the module implementation. There is a desire/plan to make the interface definition be a first order thing (but it is currently embedded in the module implementation).
2. Service discovery: Okapi can be told that certain modules lives in a certain place. Since the address of a module is a URL, different modules can live on different servers, etc.
3. Service deployment: Okapi has its own method for launching services (an executable or a docker container) and contains some primitive lifecycle management for the modules it deploys itself.

The datastore for every storage module is completely independent from all the others. You can put it all in one databse, or not, as you see fit. Different tenants are handled via different Postgres schemas to segregate tenants.

Question: Where is the cross-module business logic?

A: The UI modules have some cross-module business logic. The circ module has dependency on circ storage and on inventory storage. So the circ BLM talks to both the circ and inventory storage modules. Also looking at GraphQL as a way of joining across independent data sets (at the UI level). This may be a topic of significant discussion in Madrid.

Second Diagram

This shows how things are arranged on the Vagrant boxes.
Okapi is packaged as a Debian package, uses Java 8. Set up to deploy modules itself -- service runs on startup that posts curl commands to okapi that deploys the modules that are needed for a given deployment. Currently OKAPI doesn't persist what it has deployed (that's being fixed) though it does persist the module descriptors. Currently using Postgres 9.x, possibly some testing on postgres 10.
Docker engine is running, Okapi communicates with it to deploy moudles locally. Web server is done as an nginx docker image (though it doesn't have to be this way).
Question: Currently Okapi is a deb package, but everything else is a docker image. Is there a reason Okapi isn't a docker image? Okapi is available as a docker image -- the configuration is a bit more tricky because you have to bake the configs.
Using docker for nginx is clunky because deploying a new Stripes bundle means rebuilding the docker image (so that might not actually make sense).
Memory requirements are pretty substantial - an all-in-one install won't run well without 8GB of RAM. 15 back-end modules, mostly Java. Knowledge base -- mod_kbebsco -- in ruby. Mod_circ might be groovy.
RAML module builder is java-based, which encourages modules to be built in java

Question: Is it efficient to run each app in a separate JVM in separate docker containers?

Answer:

We are using vert.x, don't know if that would allow co-habitation
Can modern JVMs give memory back effectively when it's not needed? Wayne says it seems to be so, yes.
Starting to work with Docker containers make by Fabric8 that offer better memory management, monitoring, and analysis of performance.

How many containers do you need? between storage and UI modules, 15 right now.

Back to OKAPI - it's a big jar at the moment. OKAPI isn't running in docker so it can be reconfigured easily.

Where does OKAPI store it's config - depends.

Startup config, using standard systemctl conventions (.conf file)
The interfaces it knows about are stored in the Postgres database (can be segregated or not)

Is the web server doing anything that requires nginx? No, it's pretty much a static site with a little bit of redirect logic. Nginx is lightweight, but there's not a lot of load on it anyway.

Tod Olson - How does authentication work? The Stripes core knows where the login endpoint is in OKAPI. It would pass credentials to OKAPI for validation. There are 3-4 modules right now that can validate tokens or talk to login modules.

What auth tokens are passed around? A session token is generated for the user and passed back in a header; Stripes passes that in for subsequent transactions.

Third diagram : imagined implementation in the cloud

Nginx in ECS redirects user to Stripes bundle in S3, then Stripes talks to Okapi in ECS. OKAPI doesn't manage deployment, it has URLs talking to other clusters that are deployed in Amazon, Postgres database is in RDS.
Why triples? To show that they are HA clusters
Is Okapi the only thing that needs to be aware of their fellow modules? Yes. You could code a module that talked direclty to another, but that would break the design.
Scalability mindset is in the design. It is as stateless as it can be, and should support horizontal scaling. In theory, there shouldn't be table locking issues, long running transactions, etc.

Intended deployment environments: Cornell will be AWS. Chicago is considering local VMs or AWS. TAMU is looking at a containerized deployment and looking at an on-premise VMware based container environment (VMWARE integrated containers).

Other Discussion

We decided to defer the data migration discussion until next time.

Next week? Wayne will be in Madrid; but the rest of us will be around. Wayne can probably make it.

Prep for PC meeting