[FOLIO-3368] SPIKE: Review platform-complete update pipelines and procedures Created: 14/Dec/21  Updated: 10/May/23  Resolved: 25/Jul/22

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P3
Reporter: John Malconian Assignee: John Malconian
Resolution: Won't Do Votes: 0
Labels: security, security-reviewed
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Sprint: DevOps Sprint 160
Development Team: FOLIO DevOps

 Description   

yarn.lock is frequently updated on the release branches even when there are no FOLIO NPM updates. This is less than ideal and can lead to broken builds down the line - particularly over time. yarn.lock should be modified only when there are relevant FOLIO NPM packages that need to be updated and the scope of the modifications should ideally be limited to intended updates.



 Comments   
Comment by Julian Ladisch [ 21/Dec/21 ]

Modules indicate for each dependency whether they allow patch version or minor version upgrades.

That way security and functional fixes automatically get included.

Disadvantage is that a patch version or minor version upgrade might be not as compatible as intended.

Is it difficult to revert such an upgrade?

Comment by Ryan Berger [ 17/Feb/22 ]

I do question if this issue is necessary as there are legitimate updates made to 3rd-party dependencies in a formal way which does result in an update to `yarn.lock`. See https://github.com/folio-org/platform-complete/pull/1799 for an example.

Comment by Julian Ladisch [ 19/Feb/22 ]

Running yarn install works exactly as desired and required:

{{}}

 

yarn install https://classic.yarnpkg.com/en/docs/cli/install#toc-yarn-install

Install all the dependencies listed within package.json in the local node_modules folder.

The yarn.lock file is utilized as follows:

  • If yarn.lock is present and is enough to satisfy all the dependencies listed in package.json, the exact versions recorded in yarn.lock are installed, and yarn.lock will be unchanged. Yarn will not check for newer versions.
  • If yarn.lock is absent, or is not enough to satisfy all the dependencies listed in package.json (for example, if you manually add a dependency to package.json), Yarn looks for the newest versions available that satisfy the constraints in package.json. The results are written to yarn.lock.

 

If one change to yarn.lock is needed then all dependencies are bumped if the version spec set by the developers allow it. This is fine, installs bug fixes and security fixes, and is stored in the git repository so that you can always check out a specific commit for a reproducible build.

Comment by Julian Ladisch [ 22/Feb/22 ]

What others say:

https://classic.yarnpkg.com/blog/2016/11/24/lockfiles-for-all/

https://11sigma.com/blog/2021/09/03/yarn-lock-how-it-works-and-what-you-risk-without-maintaining-yarn-dependencies-deep-dive/

Comment by Craig McNally [ 24/Feb/22 ]

The Security Team has reviewed this. We aren't going to stop you from doing this spike, but our preference at this time is to keep things as they currently stand. Please share your spike findings with us prior to making any decisions

Comment by Jakub Skoczen [ 24/Feb/22 ]

John Malconian What's the configuration for these updates? Is it patch-releases only?

Comment by John Malconian [ 25/Feb/22 ]

I think there may be a misunderstanding of the issue that prompted this Jira issue. My feeling is that yarn.lock should only be updated when there is a specific NPM update to be made. Currently, the CI pipeline that is run for PRs in this repo, ignores the yarn.lock file and generates an updated yarn.lock even when the PR contains a change that is not even related to an NPM update. This is the issue I'd like to review and address.

Comment by Ryan Berger [ 01/Mar/22 ]

Sounds like we need to make sure the --frozen-lockfile flag get passed when running yarn install in CI builds. See https://classic.yarnpkg.com/lang/en/docs/cli/install/ (4th paragraph): 

If you need reproducible dependencies, which is usually the case with the continuous integration systems, you should pass --frozen-lockfile flag.
Comment by John Malconian [ 07/Apr/22 ]

Yes something like that. Possibly need to add some logic to the pipeline that only updates yarn.lock when PR includes NPM-related updates.

Generated at Thu Feb 08 23:27:35 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.