Terminology

• A Critical Service Patch (CSP), is a small and targeted software patch to address CRITICAL issues, i.e. issues that significantly disrupt customers’ workflow

  • CSP Requestor: Product Owners. Requestor’s onus is to justify CSP initiation and inclusion of targeted issues into it.

  • CSP Approver: Release Management Stakeholders (RMS) group who make Go/No Go decision

    • Release Management Stakeholders Group, list of current members, information and meeting notes

  • Technical Approvers: Specific members of the RMS group who can review the request from a technical perspective

• What Issues Can be Included in CSP

  • P1 bugs

  • P2 bugs without reasonable workarounds (for example, Budgets beyond 10 are not visible )

• What Issues Cannot be included into CSP

  • P2 bugs with reasonable workarounds

  • P3, P4 and unprioritized bugs

  • User Stories and Tasks

Process Logistics

1. The Requestor PO submits a request for a CSP to the #release_bug_triage channel, tagging @folio_rms, and providing a link to a Jira ticket which describes the issue and has:

   • The “Release” field set to <Target release> Service Patch <Number> for the targeted CSP release e.g. “Quesnelia (R1 2024) Service Patch #8”

   • The “RCA” field populated

   • Test cases in TestRail linked

   • The “CSP Request Details” field populated with the Required information for a CSP request

2. The RMS panel members review and provide a thumbs up (approved) or thumbs down (not approved) on the request in the #release_bug_triage channel. For approval a request must receive at least 5 approvals (thumbs up) including at least 2 approvals from technical approvers. If approval cannot be agreed immediately in the channel, then the Requestor PO will be invited to present the request at a RMS group meeting.

   • If the request is approved, the Requestor must set the field “CSP Approved” = Yes

   • If the request is not approved, the Requestor must set the field “CSP Approved” = No and populate the field “CSP Rejection Details”

3. Proceed to implementation, testing and module release

4. When the fix has been certified in Bugfest environment, the Requestor PO should change the Jira Status to Closed

5. Add information about the CSP to the particular release’s Release Notes page on the Wiki in the Post-release CSP table

6. DevOps team to add tag to platform-complete branch for the particular release

7. Release Coordinator announces CSP availability using the regular Slack channels

Required information for a CSP request

1. Describe issue impact on business

2. What institutions are affected? (field “Affected Institutions” in Jira to be populated)

3. What is the workaround if exists?

4. What areas will be impacted by fix (i.e. what areas need to be retested)

5. Brief explanation of technical implementation and the level of effort (in workdays) and technical risk (low/medium/high)

6. Brief explanation of testing required and level of effort (in workdays). Provide test plan agreed with by QA Manager and PO.

7. What is the roll back plan in case the fix does not work?

Example of Critical Service Patch Process

FOLIO library reports an issue that cannot be addressed by system administrator or hosting provider. The issue is created in JIRA and assigned to Product Owner (PO). PO reviews and prioritizes the issue as a P1/P2 (without reasonable workaround)

1. The PO works with their team to investigate the issue. The team determines fix strategy, effort, risk, and a test plan

2. PO prepares the Jira ticket (see Process Logistics and Required information for a CSP request). When the ticket is populated, the PO sends a message to #release_bug_triage channel, tagging in @folio_rms which will alert all members of the RMS group of the request. The RMS group members will review the provided material. If the case is compelling enough, they issue their approval right away. If the case is so not clear-cut, the Release Coordinator schedules a meeting for the Requestor PO with the RMS group members. 

   1. Approvers: D Howell (Support rep), K Martin (PC rep), Y Kumar (QA), L Braginski (Dev), J Skoczen, M Veksler, M Gorrell (CC rep), H Kaplanian, K Gambrell. Need 5 approvals, including 2 technical approvers (Mark V, Lee, Mike G, Jakub)

3. If the RMS decision is a Go, the appropriate development team is tasked to go ahead with the fix.

   • If the decision is a No Go, then PO will make sure to add a workaround to the JIRA issue and additional details to release notes (see Process Logistics)

   • In the case of a Go decision, the CSP is released as soon as the work completed

Clarification on the processing of security related issues within CSP workflow

Workflow #1: Treatment of CVE requires work in module(s) of one Development team

If Security team considers the vulnerability (CVE) as required for inclusion in CSP and CVE impacts only module(s) of one development team, security team need to create a jira with the following data:

• label “security-reviewed”

• priority

• recommended release/CSP

• information on the CVE and its impact

• dependent tickets if any

• Security team has to add clarification to the ticket as much as possible

PO of the development team who is responsible for the module(s), will consult with TL/SA/QA/SM and fill in justification in the CSP Request Details field according to current process.

Security team will need to fully support the team (consult) on CVE's specifics (impact and treatment).

All the process logistics stays intact, that is it is PO who will need to drive request via approval and jira completion.

Workflow #2: Treatment of CVE requires work in modules of several Development teams

If Security team considers the CVE as required for inclusion in CSP and CVE impacts modules of several development teams, Security team needs to create a jira(s) with the required per current process data and seek approval from RMS group according to the process.

In case Security team creates one “umbrella” ticket for the specific CVE and gets RMS' approval for it, they need to prepare child jiras for the impacted modules and link those with the "umbrella" one. Security team needs to notify POs/SMs regarding the scope of work and provide necessary advice/recommendations on a need basis.