Hot Fix Process Today

• Issues are earmarked for HFs by Community and EBSCO Product Owners based on their sense of priority
• Past Flower releases were followed by multiple Hotfixes
  • Lotus 3 HFs (54 issues total)
    • 12 P1
    • 26 P2
    • 11 P3 and unprioritized
    • 5 stories
  • Morning Glory 1 HF (39 issues total)
    • 7 P1
    • 25 P2
    • 3 unprioritized
    • 4 stories
  • Nolana 1 HF so far (8 issues total)
• Each large Hotfix could potentially contain fixes that aren’t critical in nature but could introduce regressions
• Each Hotfix release causes disruption for development teams due to context switches

Going Forward

• Hot Fix -> Critical Service Patch (CSP), a small and targeted software patch to address CRITICAL issues, i.e. issues that significantly disrupt customers’ workflow
  • CSP Requestor: Product Owners. Requestor’s onus is to justify CSP initiation and inclusion of targeted issues into it.
  • CSP Approver: Release Management Stakeholders (RMS) who make Go/No Go decision
    • Release Management Stakeholders Group information and Meeting Notes
      
• What Issues Can be Included in CSP
  • P1 bugs
  • P2 bugs without reasonable workarounds (for example, Budgets beyond 10 are not visible )
• What Issues Cannot be included into CSP
  • P2 bugs with reasonable workarounds
  • P3, P4 and unprioritized bugs
  • User Stories and Tasks

Example of Critical Service Patch Process

FOLIO library reports an issue that cannot be addressed by system administrator or hosting provider. The issue is created in JIRA and assigned to Product Owner (PO). PO reviews and prioritizes the issue as a P1/P2 (without reasonable workaround)

• Step 1. The PO works with her team to investigate the issue. The team determines fix strategy, effort, risk, and a test plan
• Step 2. PO prepares the Jira ticket (see ProcessLogistics and Requestor will present these items in meeting with RMS panel). When the ticket is populated, the PO sends a message to #release_bug_triage channel. The RMS Panel members, who monitor the channel vigilantly, immediately review the provided material. If the case is compelling enough, they issue their approval right away. If the case is so not clear-cut, the Release Coordinator schedules a meeting for the Requestor PO with the RMS Panel members. 
  • Approvers: D Howell (Support rep), K Martin (PC rep), Y Kumar (QA), L Braginski (Dev), J Skoczen, M Veksler, M Gorrell (CC rep), H Kaplanian, K Gambrell. Need 5 approvals, including 2 technical approvers (Mark V, Lee, Mike G, Jakub)
• Step 3. If the RMS decision is a Go, the appropriate development team is tasked to go ahead with the fix.
  • If the decision is a NoGo, then PO will make sure to add a workaround to the JIRA issue and additional details to release notes (see ProcessLogistics)
• Step 4. In the case of a Go, the CSP is released as soon as the work completed

Process Logistics

• The Requestor PO submits Jira issue using existing Slack channel to request CSP and provides the following information:

1. The value in “Release” field is set to <Target release> Service Patch <Number>, e.g. “Orchid (R1 2023) Service Patch #1”
2. “RCA” field is populated
3. Test cases in TestRail linked
4. “CSP Request Details” field is populated (see Requestor will present these items in meeting with RMS pane)

• RMS reviews and provides the disposition on the request in a meeting (see Requestor will present these items in meeting with RMS pane)
  • If approved, the Requestor must set the field “CSP Approved” = Yes
  • If not approved, the Requestor must set the field “CSP Approved” = No and populate the field “CSP Rejection Details”

Note: these fields will be created in Jira shortly

• Proceed to implementation, testing and module release
• When the fix has been certified in Bugfest environment, the Requestor PO should change the Jira Status to Closed
• Add information about the CSP to the particular release’s Release Notes page on the Wiki in the Post-release CSP table
• DevOps team to add tag to platform-complete branch for the particular release
• Release Coordinator announces CSP availability using the regular Slack channels

Requestor will present these items in meeting with RMS panel

1. Describe issue impact on business
2. What institutions are affected? (field “Effected Institutions” in Jira to be populated)
3. What is the workaround if exists?
4. What areas will be impacted by fix (i.e. what areas need to be retested)
5. Brief explanation of technical implementation and the level of effort (in workdays) and technical risk (low/medium/high)
6. Brief explanation of testing required and level of effort (in workdays). Provide test plan agreed with by QA Manager and PO.
7. What is the roll back plan in case the fix does not work?

Clarification on the processing of security related issues within CSP workflow

Workflow #1: Treatment of CVE requires work in module(s) of one Development team

If Security team considers the vulnerability (CVE) as required for inclusion in CSP and CVE impacts only module(s) of one development team, security team need to create a jira with the following data:

• label “security-reviewed”
• priority
• recommended release/CSP
• information on the CVE and its impact
• dependent tickets if any
• Security team has to add clarification to the ticket as much as possible

PO of the development team who is responsible for the module(s), will consult with TL/SA/QA/SM and fill in justification in the CSP Request Details field according to current process.

Security team will need to fully support the team (consult) on CVE's specifics (impact and treatment).

All the process logistics stays intact, that is it is PO who will need to drive request via approval and jira completion.

Workflow #2: Treatment of CVE requires work in modules of several Development teams

If Security team considers the CVE as required for inclusion in CSP and CVE impacts modules of several development teams, Security team needs to create a jira(s) with the required per current process data and seek approval from RMS panel according to the process.

In case Security team creates one “umbrella” ticket for the specific CVE and gets RMS' approval for it, they need to prepare child jiras for the impacted modules and link those with the "umbrella" one. Security team needs to notify POs/SMs regarding the scope of work and provide necessary advice/recommendations on a need basis.