Versions Compared

Old Version 1

changes.mady.by.user Jennifer Eustis

Saved on

compared with

New Version Current

changes.mady.by.user Jennifer Eustis

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Table of Contents
stylenone

...

Overview

This document outlines the processes for product evaluation of new functionality for inclusion in a FOLIO release. The following activities are covered:

  1. Before Development

  2. Submission

  3. Evaluation

  4. Review

  5. Feedback

  6. Approval/Rejection

Supporting documentation

...

Details of the JIRA workflow are captured in a separate document.

...

Product Council reviews new functionality at the earliest possible stages in terms of the idea of that new functionality in order to consider it in relation to the FOLIO as a whole product.

Before Development

  • Please review the Evaluation Process for new FOLIO Functionality and Evaluation FAQ Frequently Asked Questions before (or as early as possible during) development, to avoid any issues which would lead to delay or rejection of the module.

  • The Product Council welcomes questions. In particular, if you anticipate that the functionality will introduce any cross-app changes or other innovations that might conflict with existing PCR criteria, or would imply significant changes to the hosting environment or required infrastructurefunctionality, please do reach out to the Product Council ASAP.

...

Submit a Review Using JIRA

  1. The Submitter fills out a form in JIRA (using a template) and submits it to the PCR Board for review.

    • See JIRA workflow for details.

    • An Evaluator will only respond to submissions from the official Submitters Submitter as defined in the 'Roles' section of this document. The JIRA ticket is set to SUBMITTED.

    The

JIRA Template To Be Answered For the Product Council

The Template will includes these prompts. The Submitter should provides answers to and documentation for the following criteria. The answers to these questions can be done in collaboration with the appropriate Special Interest Group and can reference any presentations done in a Special Interest Group meeting.

  • Functionality Name

  • Functionality Description

  • Desired release date

  • Expected TC review period

  • Describe how this functionality is unique

  • Describe how this functionality impacts or compliments existing functionality

  • Describe how this functionality fits into the roadmap

  • High level description of how the functionality is being built (back and front end aspects, interactions with existing functionality, etc.)

  • Describe any dependencies

  • Describe your user acceptance test plans

  • Links to documentation (technical and users) or describe documentation plans

  • Link(s) to presentation materials

Review

  1. The Evaluator sets the JIRA issue from SUBMITTED to REVIEW.

    1. The Evaluator will add a statement to the ticket on the criteria provided and any other information needed for a successful endorsement.

  2. The Evaluator will update the JIRA within 1 week (See JIRA workflow for details) with the following information:

    • Confirm receipt of the submissionWho will be performing the role of Evaluator, as defined in the 'Roles' section of this document

    • Provide a rough estimate for the initial evaluation to be completed

      • Should be 3 weeks or less.

    • Request additional information

    • If there are missing points of contact, the Product Council will be contacted

      • Any questions about the module will be directed to the points of contact.

      such as a presentation of the what the new functionality

    • Request a demonstration of the module

      • In some cases the PC may request a demonstration of the module

      • In such instances the The Evaluator will contact the Submitter with a request for a demonstration, and a mutually suitable time for the demonstration will be scheduled.

Submission form

Information gathered includes:

  • Functionality name

  • Description of functionality

  • Link to source - a tag/commit in a publicly accessible git (preferably GitHub) repository

  • List of related functionality (e.g. is this a Frontend/backend pair? Business logic/storage pair? ePC.)

  • Points of contact (Subject Matter / Domain / Technical)

    • Each point of contact's role should be clearly identified.

    • A single, main point of contact must have an account in http://issues.folio.org JIRA.

  • Free form section for notes, clarifications, background, links to supporting documentation.

  • Link to self evaluation results

  • Informational Section - provides info, instead of eliciting it.

    • A link/pointer to the most recent version of the Acceptance Criteria.

    • The PC has a maximum duration of 3 weeks to perform the initial review.

      • The evaluator may have between 2 and 3 weeks, depending when the PC meeting happens during that first week.

      • This does not include activities that happen after feedback is given, ePC.

    • The applicable deadline for submissions in the release timeline (not the date, but the name of the milestone/deadline e.g. "deadline for new modules")

Evaluation

  1. The Product Council reviews a JIRA board to see if there are any new submissions, and check on the status of evaluations in progress.

  2. If there’s a new submission, the Product Council finds one or more people to perform the role of Evaluator.

    • The review team will be formed in a way to allow for a fair review and avoid either perceived or actual conflicts of interest. No members should have participated in the module's development, and at least one member should be independent of the submitter's organization. The lead evaluator will be assigned in JIRA; if the submitting team has concerns about the chosen evaluator they should contact the PC chairs within one week of the assignment.

    • JIRA workflow: the ticket is transitioned from SUBMITTED to UNDER EVALUATION

    • In order to improve notifications when the Jira is updated, the primary reviewer should be the Assignee and other members of the review team should be issue watchers.

  3. The module is evaluated.

    • If the Evaluator(s) have questions, they may reach out to the Point of Contact for clarification.

      • This may be an iterative process: the evaluator and submitter may choose to iterate on the submission, making adjustment to address the questions raised.

      • Whenever changes are made, the submitter should update the commit hash in the Jira.

    • If answers aren't provided in a timely manner, the criteria in question may be failed.

  4. Upon completion of the evaluation the JIRA will be updated to indicate this.

    • JIRA workflow: the ticket is transitioned from UNDER EVALUATION to PC REVIEW

Review

  1. The Product Council reviews the evaluation results.

  2. The evaluators present the evaluation results, and optionally reccommends a PC decision.

  3. Asks any questions they have for the evaluators.

  4. Weigh in on any subjective criteria.

  5. The Product Council may ask the Evaluator(s) to make changes to the evaluation.

    6. This review may happen either out of band, during the regularly scheduled Product Council meeting, or a separate meeting scheduled specifically for this purpose

    The Evaluator will work collaboratively with the Submitted to ensure all criteria has been answered in the JIRA issue and that the submission is set up for a successful review.

  6. Timeline: JIRA issues needs to pass to the next step within one week after submission. If that deadline falls on a holiday or weekend, then the week deadline is the next business day.

Endorsement

  1. The Submitter presents at the schedule time to the Product Council.

    1. This presentation is a time for Product Council members to become familiar with the new functionality, understand how it works, and how it impacts current functionality. Product Council members can also review the recording and materials asynchronously.

    2. The Product Council members will strive to see how this new functionality fits into FOLIO as a whole and as an international product.

  2. The Product Council decides, using its standard protocols, to accept, provisionally accept with a follow-up, or reject the module, or else to defer a decision pending other action.the new functionality under review within 2 weeks after the presentation. The following statuses can be assigned:

    1. An evaluation where all criteria are met will typically result in the

    acceptance of a module by the Product Council

    1. Product Council’s Endorsement of the new functionality.

      1. The JIRA issue transitions from UNDER REVIEW to ENDORSED.

    2. An evaluation where

    not all

    1. some criteria are met will typically result in

    the rejection of a module by the Product Council. However:

    • If some of the failures are due to proposed architectural (or other cross-module) changes, the PC may request that Submitter first propose those changes via the RFC process to get sufficient community input. In that situation the PC may defer its decision pending the resolution of the RFC. (See Before Development.)

    • If PC and Submitter unanimously agree on module changes that would resolve any failures, the PC may decide to provisionally accept the module with an agreed-upon timeline for the changes to be completed. When Submitter notifies PC that the changes are complete, the reviewer(s) may update the evaluation and PC may adjust its decision.

    • If the PC determines that some failed criteria would be resolved by non-controversial changes to the criteria themselves (or referenced requirements like the Officially Supported Technologies), PC may decide to accept the module and make the agreed-upon changes.

  3. When the Product Council's decision is contrary to the evaluation, or after any provisional acceptance or deferral, the Product Council is required to provide written justification for their decision.

  4. Upon completion of the review, the JIRA will be updated to indicate this.

    • JIRA workflow: the ticket is transitioned from PC REVIEW to APROVED or REJECTED

Feedback & Acceptance/Rejection

  1. Evaluation results are published (in markdown format) to the tech-council GitHub repository and linked to the JIRA ticket.

    An Evaluation Results Template will be used for this

    1. an accept with a follow-up:

      1. Because Product Council reviews should occur early in the development process, not all intersections might have been seen before the review. A follow-up allows the Product Council to reach out to additional stakeholders and work collaboratively with the Submitter on questions that arose during the review.

      2. The JIRA issue transitions from UNDER REVIEW to ENDORSED WITH FOLLOW-UP

    2. An evaluation where none of the criteria even after the presentation are met will typically result in a status of cancelled, blocked, or deferred. The Product Council will work with the Submitter to understand how to ensure all criteria are met and which status is appropriate.

      1. The JIRA issue transitions from UNDER REVIEW to CANCELLED, BLOCKED, or DEFERRED.

        1. The Evaluator will add a statement to the ticket on the reasons for the status change.

  2. Timeline: JIRA issues needs to pass to the next step within two weeks after submission. If that deadline falls on a holiday or weekend, then the week deadline is the next business day.

Documenting Endorsements

  • Endorsement results are published in the Functionality Evaluated by the Product Council within 2 weeks after a endorsement is made.

  • Interested parties (e.g. release coordinator, contributor points of contact, Product Council members,

...

  • SIG convenors) are notified, and provided a link to the results.

...

JIRA workflow: notification is in the form of transition from PC REVIEW to APROVED or REJECTED

  • Timeline: Documentation and communication occurs within two weeks after submission. If that deadline falls on a holiday or weekend, then the week deadline is the next business day.

Follow-Up Status

  • If there is a follow-up, the Evaluator will contact the Submitter, and a mutually suitable time for the follow-up will be scheduled.

    • For a follow-up, the Jira Issue is transitioned from ENDORSED WITH FOLLOW-UP to REVIEW.

  • Timeline: JIRA issues needs to pass to the next step within two weeks after submission. If that deadline falls on a holiday or weekend, then the week deadline is the next business day.

Resubmission, Blocked, Deferred, or Cancelled Status

The Product Council plays a role in the decisions about what is included in the “Flower release” of FOLIO. This role doesn’t dictate what can or cannot be developed. This roles also doesn’t dictate what can or cannot be included in a FOLIO release supported by any particular service provider.

  • If none of the criteria are met even after the presentation, the Product Council will work with the Submitter to understand what is needed to continue the endorsement process. The ticket should be updated with decisions and any information relative to the successful endorsement of the new functionality.

  • The Submitter can also elect to cancel the submission.

    • The comments section of the JIRA may be used for this purpose.

    • If a meeting is required or desired, it's the responsibility of the Point of Contact to find a time that works and set the meeting up with the

...

    • Product Council

...

    • .

  • If

...

  • the Submitter can elect to resubmit, then all issues are resolved and

...

  • the Submitter requests a re-review.

    • JIRA workflow: the ticket is transitioned from

...

    • REVIEW to SUBMITTED.

Process Feedback

...

  • The Submitted can also elect to withdraw their submission.

    • JIRA workflow: the ticket is transitioned from REVIEW to CANCELLED.

  • The Submitted can also elect to defer their submission.

    • JIRA workflow: the ticket is transitioned from REVIEW to DEFERRED.

  • The Submitted and Product Council see that the new functionality is blocked.

    • JIRA workflow: the ticket is transitioned from REVIEW to BLOCKED.

  • Timeline: JIRA issues needs to pass to the next step within two weeks after submission. If that deadline falls on a holiday or weekend, then the week deadline is the next business day.

Process Feedback

  1. The Evaluator asks the Submitter if they have any feedback on the New Module Technical Evaluation process.

    • PC considers that feedback as part of the next process evaluation improvement round.

Roles

  1. Submitter

    • Creates the JIRA ticket on the PCR board.

  2. Evaluator

    • Processes a PCR ticket through the JIRA workflow

    • Can be a member of the PC or a delegate appointed by the PC

    • Must not be a member of the development team seeking approvalendorsement.

  3. Point of Contact

    • Communicates relevent relevant module information through updates to the PCR Issue in jiraJIRA, once it has been created by the submitter.

    • Includes developers, product owners and other team members.