2019-07-15 - Data Migration Subgroup Agenda and Notes

Jeremy will review some of the work he has been doing in data migration using work flows at TAMU. Discussion to follow.

PowerPoint Jeremy reviewed

Document

Exploring process of using a workflow engine for the migration process (Voyager to FOLIO)

• Proof of Concept using "Organizations"
  
  
• Architecture
  • mod-data-extractor - persist a custom query, starting point in the process - execute query and stream response of query
  • mod-workflow - provides an abstraction for the workflow engine.  can persist a workflow, dynamically generates xml needed to execute(?)
  • mod-camunda - service wrapper around Camunda workflow engine.  Event listening to trigger workflows registered with it
  • mod-organization-storage - destination for all migrated data for this POC
    
    
• Workflow tasks (e.g. ExtractorTask, ProcessorTask, CreateForEachTask, AccumulatorTask)
  • Support for multiple scripting languages.  They used JS.
  • Used APIs
    
    
• Results
  • slightly disappointing
  • created 7,294 organizations, 1,577 contacts...this took 45 minutes.  At 3-4,000 it would slow and stop....would stop for 35 minutes, resume and finish...maybe a database indexing issue?
  • They will be investigating
  • If indexing is the problem...a batch loading process may help
  • They may investigate the way the DB is setup via RAML module builder.  Maybe turn off indexing during the load?
  • The process was run on a dev machine (40GB RAM) pointing to FOLIO/Cloud
  • Getting this running in the same ecosystem will be a next step
    
    
• Conclusions
  • Has potential, capable of orchestrating complex migration....
  • Figure out slowdown
  • Move dynamic workflow creation from mod-workflow into mod-camunda
  • Explore batch create endpoints
  • ...see further details in the powerpoint & document
    
    

Chicago has experience investigating indexing issues.  May be able to help.

Q: What determines which scripting languages are supported?  → JVM support of various scripting languages.  Several enabled at this time – JS, Ruby, Perl, Python...5-6....more could be added to the that.  Javax script package.  Future plan – UI interface to pick scripting language.

Q: How do you handle reference data?  Lookup Tables?  → For this POCs...they statically expressed them in the script.  Created manually ahead of time.  Feasible for categories - small set a data.  Future – some sort of delegate that does lookup.  Chicago generates the IDs ahead of time in their migration process.

Q: Advantages over a scripting language?  Using the workflow doesn't offer as much as it seems to...scripts can solve the same problem.  (e.g. they had a set of Perl scripts that accomplished some data migration).  However...

• Workflow - gave them a chance to explore using workflows against FOLIO API
• They've created a reusable framework for interacting with FOLIO...supporting different processes that will vary from institution to institution
  1. Extracting the data 
  2. Processing of the data
• This could also be used to migrate data from FOLIO to a new version of FOLIO

Link to public repo. in PowerPoint.  

Possible agenda item for next week - more questions about this POC.

Discussion

Tabled for lack of time