2023-09-15 Sys Ops & Management SIG Agenda and Meeting notes

Session of the SysOps SIG at WolfCon: WOLFcon 2023: Platform Decisions for FOLIO Self-Deploy... (sched.com)

Ingolf: A concern about the (proposed) changing notion of a Platform. If there will be not just one big platform (platform-complete), as it is now, but several smaller platforms, maybe specialized to ERM, circulation, other specific purposes:  All modules which one might possible install must go through thorough buckfest testing (or some QA testing of similar quality). Having just a smaller set of modules undergo the QA testing workflows does not seem acceptable.

----------------------------------------------------------------------------------------------------------------------------

Topics for this Meeting:

• Microservice boundaries, Application Descriptors

Meeting Notes 2023-09-08:

Microservice Boundaries
- example of a large, monolithic module: Inventory & Inventory Storage
 - avoid circular dependencies
 - another example: edge-rtac & mod-rtac; they are useless without each other
    - at the moment, each of it would be considered a microservice
    - it makes sense for some API calls that other modules make use of them,
      but not all of them
    - join databases:
       - cons: even more dependencies
       - pros: smaller data sets via APIs
     - the boundaries of the modules need to be redefined
     - direct database access for other modules should be restricred. This is dangerous.
     - modules within an application are typically being developed by the same developer team
      - loan, users ,   probably not
      - calendar & circ rules could be coupled together
       - define what are really closely related modules
         Julian: Okapi is already an Application Manager
    Nils: If one team changes the database model, the other team needs
           to chnage their module as well
    Nils/Julian: Applications need to be small enought that ONE team can handle this.

    Ingolf: I don't agree with this. It should be possible for any team to contribute to any application. I can assume that teams talk to each other.
     Not related to optimistic locking
    Julian: Needed in some case. But should not be handled in this discussion. Should be restricted to cases where it is needed.
            Discussion should be among the Tech Leads / Dev Leads.
    Tod: It is going to be case by case.
         Check out a single item takes more than 30 disctinct API calls.
     Tod: You cannnot do queriying at a large scale, because you can not do the database joins.
   Julian: Queriyng mod-fqm could produce queries that could take a very long time. You have additinal http request if you do not have database joins.
      You don't have database indices. People could run any query that they want.   A way out could be to allow only specific joins, and dissallow other joins.
   Tod: I think this is the way how the list application is set up.
   Tod: Tod will find some context to the fqm app.

   There is general agreement that it is useful to have database joins.

    Nils: Applications should not expose storage modules APIs to the outside.
    This helps to create a translation how users and admins think of an application.
   Tobias: a platform descripor only shows interface versions.

Discussion about ui boundaries.

Julian, Florian, Tobias: One does not need applications in order to decouple modules, backend or frontend. It could be done with the existing concept. Could be decided on a case by case basis by the dev leads. Introducing Applications will introduce unneccesarry complexity.

Ingolf: But Applications would make it much more clearly structured and visible to the operator what has been deployed (and for what tenant). Applications will be an incentive for dev teams to bundle some modules closer together while decoupling them from the dependencies to other modules. Having 135+ modules to install is a nightmare to an operator. Would be much more clear to have to administrate just ca. 20 Applications. Don't want to go into the details for what happends on the module level, want to hand that over to the Application Manager. 

Others says that this could all be done with the existing structure, using Okapi as the application manager.

Comparison with linux system which has just one level of complexity.

Agreement, that not having the domain explicitly been baked into the UI bundles will be an improvement.

There is no agreement about introducing the notion of an application descriptor. The majority of the group is against having application descriptors.

https://folio-org.atlassian.net/wiki/pages/viewpage.action?pageId=2098208

Julian Ladisch  an  Alle 16:38
https://folio-org.atlassian.net/wiki/display/~marcjohnson/Check+Out+Performance
However, that check out performance analysis had been done before the optimization that greatly reduced the calls.

his is the PC meeting where the Lists App was presented: https://folio-org.atlassian.net/wiki/display/PC/2023-06-08+Product+Council+Meeting+notes
The meeting recording would have more detail.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Meeting notes 2023-09-15 :

Tod: TC and PC surfacing both technical and non-technical questions. Start this meeting with basic temperature check for SysOps , what is perspective you're seeing this from? Hosting small/large number of sites? Responsible for development? Positive/Negative formalizing application?

Ingolf: System administrator, working part-time on this. Concern that FOLIO is getting more and more complicate to maintain/install/upgrade system to automatize. At his institution, small business, one or two tenants. GVB collaborative, large number of tenants 200. In favor of applications because it will make more convenient to install and maintain a system.

Florian Glexaier: Starting to host a number of FOLIO instances, first library is productive ERM module. In future host FOLIO 50+ libraries, system administrator at Super computing center, run FOLIO on K8s. Application formalization specs that can make life easier, especially in the UI, but not clear if it will work to separate the modules to application or if this is the right now. Future will show us. Not really clear if this a step in the right direction. Not against, vote for it and try it.

Florian Kreft: With Florian Glexair, host multiple dozen tenant, currently one productive tenant. Needs to host quite a few tenants. In general in favor for application, make system more manageable and maintainable. Separate. two main concerns, not clear how will the testing and upgrades cycles will be done. If applications change how will different applications be work together. Additional complexity on the module descriptors currently have dependencies negative of another layer, who is responsible for applications. Details may need to worked out later. Too earlier

Jeremy: I am on the application side. Microservices are too tightly coupled across FOLIO. It is really important to make FOLIO easy to install and maintain.

Olof Paulsson: System administrator at the University not library. Trying to get FOLIO up and running. Application good for getting all modules are a kind of mess, looks for Applications make requirements, three or four applications up and run easier trying to get the entire platform up and running. Production maintenance, debugging, and troubleshooting, small islands to see what is happening. Reverse grep to logs to try to figure out the errors, not fun.

Jason Root: Multiple instances single tenant, trying to get FOLIO up and running since 2017. Bug reporting or general reporting. Two big problems. FOLIO is big and huge to deploy distributed ball-of-mud, hard for developer to get up to run and slows developer. Hard for developer and insures a huge amount of overhead and expensive and time-consuming to run. Trouble For Chambers running ERM but run entire FOLIO in backend. If the idea of FOLIO is live up to microservices, we haven't lived up to the ideal. Applications formalization to run as selected services the idea.  Deploying, running, and maintain. FOLIO most complicated system to run at TAMU. What we have isn't sustainable in the long run. Particular concerns of proposal? Application descriptor growing and adding more layers, place for it, go to one place to see what the dependencies and what the apps is at least manageable, right now there are dependencies that aren't in the descriptor (like Kafka) just another endpoint for running FOLIO.

Chris Rutledge: Mostly here to listen, defer to Jason as an institution.

Tobias Stumpp: System administrator, we are data center 3 year exploration phase for SW German network. As part of evaluation, check and grasp FOLIO for 60 tenants. Per application proposal, agree with principle. As currently defined, oppose proposal. Interface versions managed through Okapi and repeating Julian, application bundles may already exist in platform-complete, platform-minimal. Application formalization highlight and explain application dependencies not hidden in Okapi and descriptor. Application formalization is required for this., can be done in other ways like visualization  found through module repositories and building a graph for, calculating dependency graphs with Okapi. Fusing and tightening modules, make several smaller modules into more monolithic module. Agree with intention, oppose proposal if binary decision.

Tod: Manage a group that use hosting provider, fare amount of programming around the edges, involved in identifying the pain points in the FOLIO, concerns about being stand-up FOLIO, being able to stand-up a smaller version of FOLIO for development. Don't do much application development or system administration directly, collect information and provide background. In favor of application formalization, defining an application at an operational and deploy-able level, manage at a higher level think about the interactions and scope, boundaries more at the functional level. Being able to distinguish how application within itself verses crossing the boundaries with other applications. Several reasons, not a solution in itself, but makes it easier to talk about and operate at that level. Modules are like cells, ion exchange between , useful to think at the "organ" level. Hearing about the raft of concerns, articulate a vision about possible ways of getting there.

Board agreement and a set of goals, difference in how we achieve those goals. More to be done on this topic today?

Ingolf: We are not supposed to give answers (from TC) but surface questions.

Tod: SysOps provide feedback.

Jason Root: Question about QA/Formalization spreadsheet?

Ingolf: SysOps go through the spreadsheet to now or have individuals in SysOps.

Jason Root: We all go through the spreadsheet and evaluate for future SysOps meeting. Kafka dependencies, modules dependent on other modules topics.

Ingolf: Dependencies still hidden, don't deal with interfaces but with  modules versions, not information about dependencies between modules. Not obvious.

Tod: If we feel the document is useful, each individual has a column to make comments. What would we expect the spreadsheet be used for?

Ingolf: Give feedback to TC, TC make RFC based on spreadsheets.

Tod: RFC go to through review cycle, comments taken and responded to before vote to adopt/reject.

Olof: RFC 1149 as an example

Tod: SysOps provide technical feedback/concerns comments to RFC, TC would have that information as part of the write-up. Spreadsheet much more granular feedback.

Need to pick up topics from the Application Formalization proposal in further sessions:

• Application store or marketplace?

• Overview: List of Integrations - System Operations and Management SIG - FOLIO Wik