ERM Sprint 185

Sprint Goal / Focus 

  •  

Sprint Schedule

  • Sprint: 185
  • Release: 2024.1 Quesnalia 
    • Sprint 10 of 11 since Release Development Cycle Starts (Sprint 176) to Feature Freeze (Sprint 186: 15 Mar 2024)
    • Sprint 187: Module Release Deadline (22 Mar 2024) [FOLREL-565]
    • Sprint 189: Bugfix Release Deadline (19 Apr 2024) [FOLREL-581]

Development Meetings

  • Wed 21 Feb - 
  • Mon 26 Feb - backlog review
  • Wed 28 Feb - dev update

Sprint Capacity

Team AvailabilitySchedule | Calendar

Notes / Exceptions:

Lead Roles:

QA Environment: 

  • folio-snapshot, folio-snapshot-2

Present

Planning Questions

  • Does the issue meet the criteria for Definition of Ready?
  • What front and back end components are affected?
  • What changes need to be made? (additions, removals or modifications)
  • What development tests need to be written?  
  • What data does the developer need to verify their work?
  • What are the known unknowns? 
  • What is needed to QA? (environment, data, scripts)

Navigation

 Sprint Planning Agenda
  1. Sprint Goal / Focus
  2. Sprint Capacity
  3. Review sprint candidates 
  4. Agree technical approach / define key implementation tasks
  5. Finalise estimates / costings
  6. Confirm sprint scope
  7. Confirm first actions





Sprint Planning  

(error) - not in sprint

(tick) or @ - in sprint 

(warning) - not ready

(question) - pending triage / planning

Sprint Focus

Planning Notes Template

  • Triage

  • Approach
  • Components and Changes

    • Frontend

    • Backend

  • Tests / Data / Dependencies

  • Known Unknowns

  • QA: snapshot | local | testing | other

  • Release Target: 
  • Development Estimate


Issue ID

Sprint Backlog?

Notes / Estimates / Actions

Carried Over

ERM-1910 - Getting issue details... STATUS


TESTRAILS

Sprint 180

ERM-3063 - Getting issue details... STATUS


QA

Sprint 176

ERM-2792 - Getting issue details... STATUS

OPEN

Sprint 178

ERM-2793 - Getting issue details... STATUS

IN PROGRESS

Sprint 178

ERM-3090 - Getting issue details... STATUS

(question) Steve Osguthorpe

CODE REVIEW

(warning) Sprint 180

ERM-3111 - Getting issue details... STATUS


READY TO PROGESS 

  • Triage

    • previously created new Grails 6 project and incrementally added components
    • once pitfalls identified, each module took very little time
  • Approach
    • will starting up a new grails project with the CLI even work?
  • Components and Changes

    • Frontend

    • Backend

  • Tests / Data / Dependencies

    • gradle upgrade is necessary
  • Known Unknowns

    • will starting up a new grails project with the CLI even work?
  • QA: snapshot | local | testing | other

  • Release Target: 
  • Development Estimate

  • Current:
    • grails and gradle versions determined
    • web-toolkit passing tests
    • smoother process than 4 to 5, but integration is inevitably 
  • TODO
    • service-interaction failing
    • need to build into okapi environment
      • may need to rebuild rancher, which could be risky at the moment
    • review migrations, but difficult to triage atm


 

  • Release candidate for Grails 6 and okapi ready
  • Should be ready to move forward
  • Once merged, cannot backport to Poppy

Sprint 182

ERM-3078 - Getting issue details... STATUS


IN PROGRESS

  • enforceVersionCheck is taken from grails default rest controller - not sure of impact to changing from false to true
  • can add to to yaml without changing default to false
  • would need to make changes to KIWT restful controller
  • need to push version numbers
  • likely not to be rolled in until Grails 6
  • TODO: 
    • Jack to write up findings 
    • Owen to review with users

(warning) Sprint 178

ERM-3089 - Getting issue details... STATUS

(minus)


(warning) Sprint 178

SI-34 - Getting issue details... STATUS


(warning) Sprint 173

SI-38 - Getting issue details... STATUS


(warning) Sprint 179

ERM-3133 - Getting issue details... STATUS

  • Triage

    • FOLIO session-level
  • Approach
    • review implementation across other apps
      • eg, Orders, 
      • eg, stripes-smart-components useColumnManager 
    • can this be used as-is within ERM or does it need streamlining?
  • Components and Changes

    • Frontend: TBC

    • Backend: none

  • Tests / Data / Dependencies: ??

  • Known Unknowns: ??

  • QA: snapshot | local | testing | other

  • Release Target: Quesnelia
  • Development Estimate: ??


ERM-3118 - Getting issue details... STATUS

Triage

    • NamedThreadFactory is used in string templating service
    • out of end-of-life support, so should find a  way to upgrade/remove
  • Approach
    • need to review any documented upgrade path - is there a straightforward path to go from 2 to 3
    • are we vulnerable to the jetty-io vulnerabilty?
      • influences backport requirements (esp Orchid which is on Grails 4)
  • Components and Changes

    • Frontend: NA

    • Backend: 

  • Tests / Data / Dependencies

  • Known Unknowns

    • whether we are affected by virtue of using netty
  • QA: snapshot | local | testing | other

  • Release Target: 
  • Development Estimate

Current: 

  • upgrade path documentation doesn't help
  • looks like (via grep) this is the only use of ehcache

Sprint 182

  • need to decide whether to bail ehcache to use another NamedFactory provider
  • potentially could do this internally, without another dependency
  • run gradle clean  and check dependencies to see if ehcache/jetty-io are brought in transitively - CONFIRMED on v2

Sprint 183

  • Current
    • NamedThreadFactory doesn't seem critical or fancy
    • and it looks like it's only in mod-agreements
    • inclination is to create own version
  • TODO: 
    • check with Steve
      • is ehcache being used elsewhere?
      • can we sue our own thread factory
    • try ehcache 3
  • Decision on way forward by Wed 24 Jan

Sprint 182

ERM-3125 - Getting issue details... STATUS

(error) Not vulnerable

  • Triage

    • need to revisit what choice we made for Poppy
    • need to check whether Orchid/Poppy have the same vulnerability to determine backport requirement
    • otherwise to be dealt with (for Quesnalia) by Grails 6 upgrade
  • Approach
    • Prefer not to patch grails, but should be able to bump data-binding 
    • Try 4.1.3, then 4.1.2, then 4.1.1
    • Otherwise will need to change databindings specifically - which will be messy
    • Owen: release plan
  • Components and Changes

    • Frontend: NA

    • Backend: mod-agreements, mod-licenses, mod-service-interaction

      • shouldn't need grails-okapi or KIWT
  • Tests / Data / Dependencies

  • Known Unknowns

    • what's the minimum version that has the fix?
    • confirm that  grails-okapi and KIWT do not need bumping
  • QA: snapshot | local | testing | other

    • dev: does it compile and run as expected?
    • regression testing
  • Release Target: NA
  • Development Estimate: NA

Sprint 182

ERM-3130 - Getting issue details... STATUS


  • Triage

    • fetch of loadMore is causing cache of what's opened to refresh
      • fetch is at rootLevel, so redraw pushes a different set of lines
    • which may be because of batch fetching PO lines 
    • redraw closes and remounts 
  • Approach
    • needs investigation to understand chain of events leading to invalidation or refresh
      • what's the trigger?
      • is there a quick fix? 
      • if not will need a refactor of PO line fetch handling
    • Owen to revisit whether this is an issue in current versions (tick)
  • Components and Changes

    • Frontend: ui-agreements

    • Backend: NA

  • Tests / Data / Dependencies: 

    • none for now, potentially add to testrails catalog
  • Known Unknowns

    • then need to decide what's desirable behaviour - do we need to retain the user's view or reset it post-redraw?
      • if we don't then any data changes won't be reflected
      • if we do, we need to develop a reusable component for consistency across apps
  • QA: snapshot 

  • Release Target: Quesnelia (no backport)
  • Development Estimate

Sprint 182

ERM-3131 - Getting issue details... STATUS

  

  • Not clear how to verify
  • Need to follow up with Molly

Sprint 182

ERM-3118 - Getting issue details... STATUS



ERM-2631 - Getting issue details... STATUS

  • Approach

    • set up new open source project in K-Int gitlab
    • new micronaut service required (not using okapi or grails)
    • set up a test instance
    • will need a different dev workflow (probably using local copy of service)
    • working title: pushkb
    • avoid storing caches of transient data
  • Components and Changes

    • Frontend: na

    • Backend: new

  • Tests / Data / Dependencies

    • Align to when GBV start testing and using pushkb endpoint
    • ERM-3048 Possible package schema changes  with endpoint changes
  • Known Unknowns

    • should be (but not clear yet) about TitleInstance Resolver
    • packageSchemaUpdater: potential conflicts when there are multiple copies of schemas in play where backwards compatibility is broken
  • QA: snapshot | local | testing | other

  • Release Target:  Quesnalia
  • Development Estimate


  • Current State
    • pushKb can make a scheduled call (1s, 1h) to Gokb
    • will fetch roughly ~700k TIPPS 
    • Stores these in a local postgres db
    • proteus json transform implemented with proof of concept schema
  • TODO
    • scaling
    • extensive transformation testing
    • write fetch chunks and push to FOLIO,
      • implementing algorithm to ensure no data gaps 
      • deciding what to do with the data volume (700k feels beyond FOLIO's performance capability)
      • at a minimum: need to move the FOLIO side to being not an open http request per chunk (would be killed by  okapi), by, eg, 
        • small batch sizes? would take full day for initial ingest
        • speeding up FOLIO? (not clear how this could be done)
        • move pushKb to background thread (like jobs, but on a much smaller scale)? 
  • Constraints
    • want to avoid jobs being badly interrupted by a docker container being restarted
    • tradeoff is that we have a slower but more stable process 
  • Development Estimate


Sprint 183

  • Current State
    • modelling in place for setting up destinations, and links to
    • improved bootstrapping
    • working on algorithm to extend source record handling from single to process queue of source records without gaps and error handling
  • TODO
    • scaling
    • extensive transformation testing
    • write fetch chunks and push to FOLIO,
      • implementing algorithm to ensure no data gaps 
      • deciding what to do with the data volume (700k feels beyond FOLIO's performance capability)
      • at a minimum: need to move the FOLIO side to being not an open http request per chunk (would be killed by  okapi), by, eg, 
        • small batch sizes? would take full day for initial ingest
        • speeding up FOLIO? (not clear how this could be done)
        • move pushKb to background thread (like jobs, but on a much smaller scale)? 
      • set up api rather than write straight to db
    • scheduling
    • cleanup
    • then: devops work to make releaseable
    • then: reciprocal changes to mod-agreements
  • Constraints
    • want to avoid jobs being badly interrupted by a docker container being restarted
    • tradeoff is that we have a slower but more stable process 
  • Development Estimate (to minimum): 
    • Functional: end of Sprint 184
    • Agreements Work: 
    • DevOps: need input from Steve and/or Ian (from start of Sprint 185)
  • Release:
    • Quesnalia - balance pushKb with mod-agreements interop
  • Unknown:
    • can we use snapshot to test this>?

ERM-3048 - Getting issue details... STATUS

  • Triage

    • needs some investigation work
    • priority is for GOKb
  • Approach
  • Components and Changes

  • Known Unknowns

    • how do we want to track (count) the external (remote) counts? 
      • In the XML for the package rather than database
      • once the package XML is in hand, count all TIPPs with status=current 
    • where do we want to populate from in other cases?
  • QA: snapshot | local | testing | other

  • Release Target: 
  • Development Estimate

Sprint 184

ERM-3063 - Getting issue details... STATUS (error) Owen Stephens

QA

Deferred to BugFest environment setup (week of 25-29 Mar)

Sprint 177
ERM-3133 - Getting issue details... STATUS (tick) Owen StephensQASprint 182
ERM-2981 - Getting issue details... STATUS (tick) Owen StephensQASprint 183
ERM-3041 - Getting issue details... STATUS (error) Claudia Malzer

Deferred to Sprint 186. No progress expected. Dev on vacation until Sprint 186. 

  • Approach

  • Components and Changes
    • Frontend: stripes-erm-components

  • Tests / Data / Dependencies

    • new unit test to set up form and presents close as expected
  • Known Unknowns: NA
  • QA: snapshot 

  • Release Target: Quesnelia 
  • Development Estimate: <2d

Sprint 183
ERM-3130 - Getting issue details... STATUS (tick) Monireh Rasouli

Issues with frontend delaying start.

Cannot see changes on the frontend. 

Sprint 182
ERM-3132 - Getting issue details... STATUS (tick) Jack Golding

IN PROGRESS

  • Triage

    • is this the kind of issue that pr707 is meant to have fixed?
    • review log file against pr707 zombie job
    • check that jobs are not stuck, just queued?
  • Approach: subject to investigation findings
  • Components and Changes

    • Frontend

    • Backend

  • Tests / Data / Dependencies: NA

  • Known Unknowns: check with GBV if restarting mod-agreements is viable workaround

  • QA: NA

  • Release Target: Quesnalia 
    • not expecting to backport, as restart may be viable workaround
  • Development Estimate

 

  • Issues tracking this back to federation service
  • Need to revisit the pulse after zombie job changes
Sprint 183
ERM-3140 - Getting issue details... STATUS (tick) Jack Golding

IN PROGRESS

 

  • Does title count need to be fetched also?
    • NO see ERM-3141
    • BUT test that it is fillable with a dummy number
  • Does this need to be Nullable? YES 
  • Update integration tests: glob this to related package schema tests
  • Then ready for Code Review
Sprint 184

Ethan Freestone

Optimisation on ingest

  • Triage

  • Approach
    • Review which queries are affected to confirm that the branch is correct and complete
    • All ingest queries on branch are changed, but need to be refactored
  • Components and Changes

    • Frontend

    • Backend

  • Tests / Data / Dependencies

  • Known Unknowns

  • QA: snapshot | local | testing | other

  • Release Target: 
  • Development Estimate


Performance fixesEthan Freestone

ERM-3148 - Getting issue details... STATUS

  •  
    • Bug: clean up situation where log error causes downstream null pointer exception 
    • clean up for query optimisation 
    • title rename

ERM-3151 - Getting issue details... STATUS

ERM-3153 - Getting issue details... STATUS

  • Triage

  • Approach
    • revert locked changes and add fallback (ERM-3151) 
    • add query (ERM-3148)
    • investigate triplicate error message
    • remove the match key modelling (ERM-3153)
  • Components and Changes

    • Frontend

    • Backend

  • Tests / Data / Dependencies

  • Known Unknowns

  • QA: 

    • harvesting on snapshot and snapshot-2
    • initially all on live. then review 
  • Release Target: Quesnalia
  • Development Estimate


ERM-3149 - Getting issue details... STATUS (question) Owen Stephens

NEEDS UI

Needs mockup

  • Triage

  • Approach
  • Components and Changes

    • Frontend: ui-agreements

    • Backend: NA

  • Tests / Data / Dependencies

  • Known Unknowns

  • QA: snapshot 

  • Release Target: Quesnalia
  • Development Estimate


ERM-3150 - Getting issue details... STATUS (question) Owen Stephens

NEEDS ANALYSIS

  • Triage

    • Query is slow because it has to count all resources every time we display agreement line, which is every time we display agreement
    • May also affect title lookup 
  • Approach
  • Components and Changes

    • Frontend: NA

    • Backend: mod-agreements

  • Tests / Data / Dependencies

  • Known Unknowns

  • QA: snapshot

  • Release Target: Quesnalia
  • Development Estimate 


ERM-3152 - Getting issue details... STATUS Ethan Freestone
  • Triage

  • Approach
    • Extends the work on marking a job as interrupted
    • Closes loop on job interruption to mark data source as released ("idle")
  • Components and Changes

    • Frontend: NA

    • Backend: mod-agreements

  • Tests / Data / Dependencies: NA

  • Known Unknowns: NA

  • QA: dev

  • Release Target: Quesnalia
  • Development Estimate: Wed

 Closed as tested by dev. Not reproduceable for QA.



ERM-3142 - Getting issue details... STATUS NA

AWAITING DEPLOYMENT

Fixed by  STCOM-1262 - Getting issue details... STATUS

Pending release of STCOM


ERM-3124 - Getting issue details... STATUS
Moved to QA for review
ERM-3155 - Getting issue details... STATUS Ethan Freestone

ERM-3154 - Getting issue details... STATUS (plus) Ethan FreestoneCan be backported
ERM-3141 - Getting issue details... STATUS (plus) Jack Golding
  • Triage

  • Approach
    • changes needed to OAI-PMH adaptor
    • need to parse xml to count states
    • to test: 
      • run harvest locally from gokb-live until seeing a couple of the big packages (should take ~30 minutes)
      • compare with old to make sure consistently handled
  • Components and Changes

    • Frontend: NA

    • Backend: mod-agreements

  • Tests / Data / Dependencies: NA

  • Known Unknowns

    • best way to count in xml? needs some research by dev
    • is there a cursor to point to on gokb-live
      • OS to add example to ticket
  • QA: snapshot

  • Release Target: Quesnelia
  • Development Estimate


ERM-3157 - Getting issue details... STATUS (question) Jag Goraya 

 

  • confer with II about deprecation

Tests

(minus) Cypress test development stalled pending implementation of STCOR-484 - Getting issue details... STATUS

Issue ID

Sprint Backlog?

Notes / Estimates / Actions





Release Tasks (
Non-Flower Release)

Issue ID

Sprint Backlog?

Notes / Estimates / Actions




Release Tasks (Orchid CSP)

Issue ID

Sprint Backlog?

Notes / Estimates / Actions

NA

Release Tasks (Poppy Bug Fix)

Issue ID

Sprint Backlog?

Notes / Estimates / Actions

NA



Maintenance

Issue ID

Sprint Backlog?

Notes / Estimates / Actions

NA




Changes

(plus) Added

(minus) Removed: note whether rescheduled or deferred

Feature ID

Issue ID

Change

Notes


Add column select functionality to selected MCL cases

(question)

Example: ui-orders

Approach: 

  1. investigate whether native stripes component integrates cleanly and easily with ERM applications
  2. apply stripes or bespoke component to selected MCLs






Sprint Summary

ERM


key summary type assignee status components
Loading...
Refresh

Dashboard

key summary type assignee status components
Loading...
Refresh

Other

key summary type assignee status components labels
Loading...
Refresh

No Label

key summary type assignee status components
Loading...
Refresh