2019-12-16 - Data Migration Subgroup Agenda and Notes

• Find a note taker. (Jackie)

Checkin on WolfCon planning. Any issues that people want to bring up?

Data Migration Sessions:  (Anne:: we have been given 2 of the 3 sessions for 1 hour. data migration with RM was mentioned. Texas/chicago each get 20min. no one voiced an issue with the time reduction. Bulk APIs dropped. Might be able to speak about Bulk APIs during one of the other session.

Data Migration Problems and Solutions (best practices) Session: 
what else to talk about in that session? Are we looking for short presentations or just get together and wing it?

Dale: envisioned it is about tips and tricks for those needing to get up to speed but wasn’t necessarily about reference data. Open ended to discussion contributions. so if want to do something about reference data that is okay. Could be just an open floor discussion or should it be more pl­­­anned. For those who are looking to migration who haven’t done so yet.“

Anne wants to also talk about data clean up.

Tod: We need to be clear on the purpose of the session and let people know what it is to make a decision. Patty: we shouldn’t leave it up for grabs. lightning talks is an option. should not be left up to chance.

Dale: lots of opinions on what is best practices.

Tod: what does this group need out of this session?

Patty: are we going to help this group or are there things we want to get answered? Find a ‘buddy’ to collaborate with? We are a bit ahead of the migraters. Germans have a lot of data. what we want to get out of it besides friends is that it is possible to migrate data into Folio--tell people that.

Dale: legacy data mapping is a hurdle and even as bulk API comes about, it will still be a challenge.

Patty: steps of data migration we don’t necessarily want to tell them how but do tell them it is possible.

Tod: group want to spend time going into details? probably not. Patty: know who your peers are, see what others have done, and persuade a diverse group to learn how viable Folio is.

Dale: hopefully this session would come after presentations and use this session as more of question/answer.

Patty: not individually with people but rather be at a bar and be available to answer questions.

Dale: we have diverse collection of practices not a documented standard. Let’s bring this up next meeting. talk about actual problems with their migrations and see if we can decide on an approach for this meeting.

Ingolf- need to prepare--best practices (list methods to load data and modules) + follow with discussion (address questions)

??? - Hoping case studies in the other session would have the best practices.

Dale: Mapping relationships. particularly because folks are coming from few legacy systems. perhaps models in place can look at. how to manage UUIDs. need to iterate your migrations. plus many other things to consider with a migration

Ingolf: what can folio offer? APIs? maybe present some slides; methods for loading data. Think attendees won’t know this and need to provide an overview. wouldn’t expect everyone is on the same page.

Dale: important to orient folks; discussing particular tools could be part of it but need a more general orientation.

Tod: need a beginners orientation and a separate session to get into more detail

Ingolf--so will this session be for the more experienced population?

Dale: thought we were intending to make it more inclusive for this session as opposed to other sessions geared towards those already doing the work.

Tod: what is the most important stuff for this group at WolfCon?

Dale: intention behind breakdown of submitted sessions: indepth case studies, bulk api’s (major area of migration), provide overview about what we learned and best practices for migrating. Since we don’t have a lot of oppty to present, we have to do some summary work about tools and approaches for migrations, docs, resources, channels, etc.

Ian: Not sure we will come out of best practices out of this session. not sure how many are coming to folio who are exploring folio as opposed to in the trenches now. Who is the audience?

Ingolf: core team, POs, ??

Ann-Marie: people who go to the implementers group. folks trying to get existing data moved over.

Dale: given these are unknowns, we should address the needs of the audience

Tod: we need a default and basic plan. going in unstructured will not be a good use of time.

Ann-Marie--need seed questions.

Dale: Reference data and data clean up suggestions by Ann-marie,

Ann: what type of documentation  and where (wiki) as a topic.

Dale: what other things should be brought up? We should think about what we can contribute to this meeting for those who are attending and offer it at our next session.

Jason may have an update for us on work he has been doing on conguration and deployment of Postgresql

Jason and Brandon are working on high availability postgres solutions. 10node HSA postgres, vagrant running locally on Brandon's machine and will implement on the server this week. goal=run infrastructure (2 data centers) of Folio on both with little to no downtime. been working on Bibs loaded --1.6 million in with failures and currently analyzing it.

LDP? haven’t gotten to that yet. plan is to run it with the rest of it. looking at using postgres for it as well. not as mission critical so won’t run it in HSA way. When it gets more mature, around Feb will approach it then.

Port folio to posgres 12? Jason is on postgres 10. part of the problem with 12, is the java 11 embedded postgres wasn’t working. need embedded postgres 12 into the java modules.

Jon: The LDP requires PostgreSQL 12 as far as I know

Dale: some code writing is currently only in 12 and that would mean keeping up with 2 versions

Jason: yes, he had to deal with that. operational concern and annoyance.

hoping to have it deployed in infrastructure and database dump this week. expect network latency to be impacted/cost.

Crunchy data being used and has a support contract for consultation. (a vendor who contribute to the postgres community--the container solution they are using)

Dale: Are you willing to do a presentation for sysops ?

Jason: once make more progress, yes, will make a presentation (once live). soon. and expect to participate in infrastructure overview at Folio

We had a useful discussion last time on work on-going from different institutions in migrating their data to Folio. Let's continue that at this meeting.

Jason--loading all bibs into folio = it dies.

Patty: reached out to Theo?

Jason: not yet, using the APIs right now.

Patty: Theo has found volume is an issue. a bug about SRS (3-4 min to show the source record). it has been addressed. continues to be an issue.

Jason: just trying to get that amount of data into.

Patty: what will help the situation?

Jason: people are complaining about modules dying all the time. trying to figure out the requirements depending on the scale. lies in the SysOps court.

Ann-Marie: Gotten it down but not enough. so talking about other methods. will need to continue in SRS and other modules to improve the time. newest fix for this will be out Wed Dec 18. Bugfest finishes on Dec 17. Raised some alarm= demo at last sprint demo, querying in inventory to make up for filtering short comings--great but it is resource intensive and slows things down. if trying to get full data out, need to do a query that won’t time out.

Dale: have search types don’t work…times out.

Ann-Marie: most testing environments aren’t scaled enough to come across this performance issues

Dale: scale issue rather than a resource issue?

Ann-Marie: need the schools currently doing this large scaled data migration need to voice their experiences.

Ann will get in touch with Laura Wright

Dale: inventory records from SRS issues. can’t have unrestricted reference data and expect it to work.