2020-09-18 - System Operations and Management SIG Agenda and Notes

Exchange experiences with your installation -

Experiences with Q2 release

• Ingolf - Circulation didn't work because mod-pub-sub was not connected to Kafka.  Had to fix by un-deploying, fix the launch descriptor and disable all modules and re-deploy.  Dale ran into this as well.  Ingolf was able to fix this error in the single Server Installation: org.folio.util.pubsub.exceptions.EventSendingException: Error during publishing EventMessage in PubSub: Status code 500 . Status message: Internal Server Error . Response body: Internal Server Error Error during publishing Event Message in PubSub. Status code 500.  And this is what is in Okapi.log: [vert.x-kafka-consumer-thread-21] WARN NetworkClient [2747347616eqId] [Consumer clientId=consumer-22, groupId=folio.pub-sub.diku.DI_ERROR] Connection to node -1 (/10.0.2.15:9092) could not be established. Broker may not be available.  it is due to trying to use the Vagrant host instead of the real (virtual) host.
  Ingolf will put a documentation on how to fix this to the SysOps wiki.
• Jason - 3 Rounds of hot fixes doesn't bode well.  We tried to make too many platform changes to this release.  Will try with the latest fixes.
• Documentation that is out there now only works for a vagrant setup.  But it's lacking details such as environment variable.  Documentation needs to be improved for installation and configuration.  Lacking test coverage.  PO's were testing with superuser permissions, librarians were not.  Some of this goes along with the large number of features we are trying to build and release.  Improvements in testing are being made to deal with permissions.
• We lack a good way to migrate permissions.  During migrations, permissions are being migrated that no longer work or collide name wise with newer permissions.  Useless permissions need to be dropped.  
• Permissions are finally biting us back....
• Test environments are migrated and carried forward to help find upgrade issues.  Also use large data sets.
• New tech writer needs a setup on AWS based on Rancher or else he will need Vagrant on Asure.
• What is the smallest foot print for a FOLIO install?  Need at least 16gb ram.
• One Rancher server and 3 VM's seem to make a good minimum systems.  Stanislov's script will deploy across this easily.
• 2 of every module will run on that cluster with a tiny amount of HA.
• What is the state of migration? Is it working now?  Mostly working now.  There are some bugs.  The last with Mod-courses.  If you have an existing instance, TAMU has had several successes.  Robbie is trying to get FOLIO running aligned with FOLIO snapshot.  Taking a snapshot and restoring is completely broken.  The upgrade path is currently brittle.  TAMU needed to migrate in a day.  SO they dropped API's and Camunda.  They are pulling logic out of the API's and building their own.  They are using the API's for loans migration due to the excessive business logic.  There is a migration wiki for all this.  It's slow.  It takes them 12 hours to migrate from Voyager to FOLIO.  They have also tested FOLIO to FOLIO migration.  TAMU is not doing any database to database migration.  Not acceptable or sustainable. 
• In data migration they are starting to organize a development effort for bulk API migration.  Will be filing requirements. 
• Developers will be required to build a test for every bug.
• Developers are no longer using a shared reference environment.  They have an isolated environment now.  They can zap the entire installation and build a new one from Master.  The developer can deploy multiple times per day and test.  They can rebuild to sync with other teams and their modules as well.