2023-03-02 Meeting notes

Date

Attendees

Discussion items

TimeItemWhoNotes
1 minKafka security Team

The topic of Kafka security was raised as part of a conversation at the TC yesterday.  

The Security Team should be aware of this and probably should weigh in on the topic, or even generate proposals if we have ideas for how to solve the problem.


Today:

5-10 minKeeping Dependencies Up to dateJulian/Team

MODEXPS-186 "Describe way how to check dependency vulnerabilities during RMB versions upgrade" reads:

Purpose/Overview: We need to increase version of dependencies that have vulnerabilities during RMB versions upgrade
Approach:
Contact with Julian Ladisch how to run vulnerabilies scanner to identify list of versions that need to be increased. Add step into acceptance criteria of RMB upgrade stories to run this scanner and update versions of these dependencies.
Update Jira template of RMB upgrade story to include this new step to check vulnerable versions.


Is there some misunderstanding?
For each flower release development teams should upgrade all third party dependencies. Either to the latest production ready version, or to an older version that is supported by the third party for the complete period of the FOLIO flower release (if the development team uses some unsupported version it must do security monitoring and prepare for backporting security fixes and for writing security fixes if no fixes are provided by the third party).
These upgrades should be made for all dependencies, regardless of any known vulnerabilities. This is needed to keep hot fixes as small as possible if a vulnerability needs to be back-ported later on.

  • If teams commit to updating all of their dependencies on a per-flower-release basis, the need for teams to use a vulnerability scanner is much less.
    • When communicating the revised security team processes (see above) to the product owners (via Khalilah), we should also strongly recommend that teams do this.
    • Maybe we can get the TC to endorse this and make it official policy?  
  • Update from slack:

    @here I just had a conversation with Khalilah, she has a plan for asking teams to regularly update their dependencies each release cycle and will be communicating it out to teams on March 1.  Leading up to that we both took some action items:

    1. Khalilah will create user stories for individual teams/modules to check their dependencies and upgrade them during the bugfix phase of each release cycle (starting with Orchid).
    2. She will also create user stories for teams to do these checks ~1/2 way through each release cycle to get a better idea for what's coming, and identify inter-team dependencies (e.g. for shared libraries, etc.), and help with planning.
    3. I will discuss with the Security Team what guidance and/or tool recommendations we can make to help teams identify dependencies which can be updated...  I'm thinking there must be a maven plugin or something for backend... and for frontend something like npm outdated?  We can discuss at our meeting on Thursday.

    John Coburn have raised the issue there are so many dependencies which need to be upgraded.  All we can do is chip away at this in Orchid and Poppy, the expectation being that we'd be caught up by Quesnellia and from there on it should be more doable to keep dependencies up to date on a per-release basis.

    Doing the dependency upgrades in the "bugfix" phase of the release cycle is too late.  We need to do this prior to the module release deadline.  Craig McNally will circle back with Khalilah on this. Maybe there was a misunderstanding.  It seems like we don't have enough time to start doing this work in Orchid.  Let's plan it out and give teams a more reasonable amount of time, guidance, etc. 

    As far as providing guidance to teams...

    • Julian Ladisch proposed the idea of introducing BOMs for sets of folio dependencies.  He will write it up with an example so we can share this with dev teams.
    • For frontend yarn outdated should do the trick to identify what needs to be updated.
    • For backend we need to find something that performs the same task (maven plugin?) Craig McNally will investigate and provide options.
    • Dependabot may be another option.  If this is the direction we want to head in, we need to check which repos it's enabled for, etc.
  • Julian Ladisch will write it up with an example of the BOM for folio dependencies so we can share this with dev teams.
  • For frontend modules "yarn outdated --all" gives you the complete picture. 
    • Would we be running this at the platform level?  Module level?  Both?
      • Dev teams should run this against their own modules
      • It will also be run at the platform level (by stripes architecture?)  
    • We also have dependabot running in some places - running weekly
    • How do we prioritize what get's updated?
      • Dependencies with known vulnerabilities should be updated first
      • Support periods should be considered as well, but understanding what's supported, for how long may be challenging in itself
    • It's very difficult to stay on top of all these dependencies, it may not be realistic to expect to be able to update all outdated dependencies each release cycle
  • For backend modules maybe this is what we're looking for: https://www.mojohaus.org/versions/versions-maven-plugin/examples/display-dependency-updates.html 

Today:

1 minOWASP/SNYKTeam
  • Epic: FOLIO-3582 OWASP checks, reviews, and fitness functions 
  • Feature: FOLIO-3583 OWASP Zed Attack Proxy (ZAP)
  • User Story: FOLIO-3584 SPIKE - investigate OWASP Zed Attack Proxy (ZAP)
  • Progress on FOLIO-3584?
    • Skott ran a scan against MG Bugfest
    • Skott Klebe  and Craig McNally to review results... weed out false positives, then bring results to next week's meeting.
    • Scan stopped at 95% (OOM issue in Firefox)
  • Any other features/stories we want to create?
    • Not related to OWASP, but I think Skott Klebe suggested that our SNYK could be tuned/adjusted to better suit our needs.
    • Maybe make the Epic more generic – "Security Fitness Functions" or Create another epic to track the non-OWASP stuff?
  • SNYK
    • Should define JIRAs
      • Spike to investigate how we can tune snyk to better suit our needs – Craig McNally  to create this.
    • Provided some background/history about SNYK in the FOLIO community
    • Stripes architecture reviews dependencies every other flower release (1-2x/year)
    • Currently only looking at platform level (with package.json/lock files)
      • Avoids a lot of the dev/peer dependency false alarms
  • Skott Klebe will have limited time to spend on SG activities but will have some time to spend on the TBD Snyk spike in mid-December.
  • It sounds like he's seeking help from someone more familiar with snyk ... either to have a conversation or provide documentation links.
    • All this means is the spike JIRA should contain documentation links
    • Also need to clarify which repos are integrated with Snyk, and what the process looks like for managing this.
    • TODO:  look at the FOLIO documentation on Snyk.
  • Craig McNally to touch base with Skott Klebe and try to get back on top of this effort.
  • Spike has been created:   FOLIO-3709 - Getting issue details... STATUS
  • Skott Klebe will find some time next week to work on this.
  • We discussed the possibility to scan release branches → could be the subject of a later spike
  • We discussed a workaround for getting snyk to scan non-master branches - e.g. temporarily change the default branch, run the snyk scan, then change it back.
    • Let's hold off on using this for now unless absolutely necessary - For the most part teams merge changes to the master (default) branch anyway.

Today:

1 minCumulative upload problemTeam
  • Regarding file upload size issues (See FOLIO-3317 - Spike - investigate possible file upload vulnerability OPEN ), let's brainstorm ideas for mitigating the cumulative upload problem, not just the large file upload size problem.  
    • Some APIs are more vulnerable to this than others, such as those not protected by permissions - e.g. mod-login, edge APIs, etc.
  • Axel provided some background/context.  We still need to give this some thought and possibly suggest a solution
  • Use case 1: Some script unintentionally sends endless data to some API. This is caught by a maximum upload size.
  • Use case 2: Denial of service. Difficult to address in Okapi. Might be better handled in other tools like nginx or firewalls that can limit requests. Unlikely that a denial of service attack has a valid login / access token.
  • TODO: For use case 2: Only add documentation that implementers should use an external firewall (or external nginx) to limit requests.
  • Some investigation is required, let's capture this in a spike (JIRA).  
    • Axel Dörrer to help define this.  – Started, not finished yet.
    • We can review together and find someone to work on this... maybe have a champion on this team work with someone in the Sys-ops SIG/community.
  • Created FOLIO-3615 - Getting issue details... STATUS Chris Rutledge and Axel Dörrer to look at it and ask sysops folks to chime in
  • Let's create a wiki page to capture ideas/feedback - Axel Dörrer → done
  • Raised at the SysOps meeting
  • A small working group was formed, but has not yet met.  
  • Group is meeting on Fridays... See link above for meeting notes, etc.
  • Will be creating a test env. to aid in the investigation
  • A similar issue was discovered in folio-vertx-lib... linked the issue to FOLIO_3317
  • Wrapping up definition of test cases.
  • Investigating available pentest tools.
  • Test system/honeypot is in place
  • preparing to create scripts for test cases
  • SysOps group is working with a developer from Frankfurt
  • Working towards running the pentest tools → utilizing information from RAML/OpenAPI for each module.
  • Which environment is being targeted?  
    • A dedicated one provided by Leipizig

Today:

1 minSTCOR-395 "refactor login form to avoid using any form framework whatsoever"

All

  • Create a spike to investigate what's involved in divorcing the login page from the NPM ecosystem.  Will reach out to John and Ryan as needed. 
  • Reopen STCOR-395 and block on the spike. – Done.
  • Where does this stand?  Get an update from Ryan Berger / John Coburn
  • Waiting for the spike ( STCOR-651 - Getting issue details... STATUS ) to be completed. – currently in the Open state.
  • John to reach out to Skott to discuss what the level of risk associated with this.  – Still needs to happen.
  • John Coburn pulled together two PoCs.  See comments in STCOR-651 - Getting issue details... STATUS
  • How do we want to move forward?
    • Solutions need to be reviewed and discussed.
    • Sounds like the iframe approach is a non-starter... actually a step in the wrong direction security-wise
  • John Coburn (and others) to read up on browser CSPs
  • John Coburn has made some progress on investing CSPs
    • Will share some draft guidance we may want to include into the installation documentation (via slack)
    • SG will review and provide feedback.  Skott Klebe please take a look too
  • Next up:  John to work on some spike work - focused on introducing CSPs on the folio-snapshot site
    • can serve as a reference impl of the guidance we'll be adding to the install docs
  • Not much progress since last week, but hopefully get some movement on this soon.
  • FOLIO-3691 - Getting issue details... STATUS Spike
  • John Coburn trying to get this lined up for next sprint

Today:

1 minSpring versions and the Official Supported Technologies pages

Julian Ladisch  3:51 AM

I've added versions for the Spring components to the Officially Supported Technologies list.  For Orchid it's easy: https://folio-org.atlassian.net/wiki/display/TC/OrchidFor Nolana it's more complicated: https://folio-org.atlassian.net/wiki/display/TC/NolanaThe OSS support of several Spring components ends in May 2023. This is more than three months before the end of the Nolana support period that end in August or September 2023. The next minor version of those Spring components will be released in November 2022, this is during Nolana bug fest. We need to bump to this next minor version and release a bug fix for all affected FOLIO Spring libraries and FOLIO modules in November 2022.Running unsupported Spring components is a security risk.  If there is a security issue in a supported Spring component we simply can bump the patch version, no risk.  If there is a security issue in an unsupported Spring component we don't get notified because it's unsupported, and if we know of an issue we need to bump the minor version, this comes with some risk. Bumping the minor version as early as possible gives more testing time and is less risky.  We can discuss this today.

  • How do we want to approach this?

    • Consult the Spring Force team... Is this feasible?  Do you have better suggestions?

      • Craig McNally  to post in #folio-spring-base, at-mentioning Petrenko.

  • How do we communicate this out?  

    • Coordinate with the release manager (Oleksii Petrenko)

  • How much work is expected to be required?

  • Craig McNally  to follow up with Oleksii P. again to see if the team has discussed this yet.

  • Met with Oleksii P. and Spring-force.  They're onboard with this proposal and will take it from here.

  • There are complications with this... conversations in #folio-spring-base:

    • For details see Nolana#Frameworks.1

    • Spring Boot 2.7 has extended OSS support until 2023-11-18, this completely covers the Nolana support period (~ August 2023), no need to switch to Spring Boot 3, but see below

    • However, folio-spring-base also uses Spring Cloud, and other Spring way modules also use other Spring components without extended OSS support

    • Spring Cloud 3.1 support ends 2023-05-18, this is at least three months before the end of the Nolana support period around August 2023.

    • Spring Cloud 4.0 GA release will be in November 2022.

    • Spring Force team has agreed that Milestone versions can be used. These need to be replaced by the GA version. Nolana GA release will be in December 2022.

    • Only recently (2022-10-04) Spring Force noticed that Spring Cloud 4 requires Java 17 and only (officially) supports Spring Boot 3.

  • Also introduced the topic at the Technical Council meeting yesterday but ran out of time.  The TC will discuss more in slack and if necessary at next week's TC meeting.

  • Update from Julian Ladisch / Craig McNally...  Summary:

  • Last week it was decided that the Spring Upgrade (Spring Boot from 2.7 to 3, Spring Cloud from 3.1 to 4) and the Java Upgrade from 11 to 17 is postponed to Orchid; and:

Two options that need to be discussed in January 2023

  • Limit support of Nolana release up to May 2023

  • Backport of Orchid spring-based changes to Nolana HF

Today:

type key summary assignee reporter priority status resolution created updated due
Loading...
Refresh

shows good progress to be in time for Orchid and Nolana backport:

type key summary assignee reporter priority status resolution created updated due
Loading...
Refresh

1 minLogging & Personal DataCraig/Team

A developer recently reached to me asking if the security team or TC has guidance or rules in place for logging of personal data.  Some guidelines are documented on the wiki, but I'm wondering if it's worth making some clarifications and creating a draft decision record for the TC to formally endorse

Is this even in our purview?  Should we seek input from the Privacy SIG?  Should I raise this with the TC first?

Next steps:

  • For now, put this on hold.  See how the Draft DR approach works for the periodic dependency updates (see above).  If that goes smoothly, we'll take this on next.  Otherwise we'll consider other approaches.

Today:

*

Review the Kanban boardTeam

Today:

  • Where do these things stand?
    • Julian Ladisch to discuss the outstanding ERM issues (no movement since Nov).
      • Some progress, no need to dig into it during today's meeting.
    • MODLDP-44 "Upgrade snakeyaml and postgresql fixing vulns" In code review since December.
      • PR has been merged and JIRA is closed
    • MODGQL-160 "Upgrade apollo-server-express" (EOL within Orchid support period)
      • Seems Mike T. is inclined to NOT do this... who if anyone will?  What are our options?
      • Maybe we need to raise this general question to the councils... what happens when modules are no longer maintained?
        • Do they get removed from platform-complete/flower releases?
        • Do we find someone else to do the maintenance?
        • NOTE:  in this particular case most backend teams will not have the skill set to update mod-graphql since it's written on nodejs
    • https://folio-org.atlassian.net/browse/FOLS3CL-5 "Make aws-sdk-java optional (maven scope: provided)"
      • https://github.com/folio-org/mod-bulk-operations has just been released with the folio-s3-client including the aws stuff.
        mod-bulk-operations has 589 dependencies, 356 of them (60 %) are aws-sdk-java dependencies.
        Should we request that the default release of mod-bulk-operations should ship without the aws for security reasons?
      • Craig McNally to ping Slava K. about incompatibilities between min.io client and S3, then report back to the Security team.
Time permittingCyber Resilience ActTeam

Action items

  •