[FOLIO-2863] pipeline changes required for Spring Way projects Created: 05/Nov/20  Updated: 02/Feb/21

Status: Blocked
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: New Feature Priority: P3
Reporter: Jakub Skoczen Assignee: David Crossley
Resolution: Unresolved Votes: 0
Labels: devops-backlog, springway
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Relates
relates to FOLSPRINGS-3 Spring Way: Provide shared versioned ... Open
relates to FOLSPRINGS-6 Spring Way: Describe what metrics and... Open
relates to FOLSPRINGS-7 Spring Way: Document consistent termi... Open
relates to FOLSPRINGS-10 Spring Way: Explain database schema, ... Open
relates to FOLSPRINGS-2 Reinstate "USER folio" in Dockerfile Closed
relates to FOLIO-2893 Create folio-tools/api-lint to verify... Closed
relates to FOLIO-2898 Create folio-tools/api-doc, from RAML... Closed
relates to FOLIO-2916 Modify CI job buildDocker to publish ... Closed
relates to FOLIO-2917 Develop CI job to ensure API JSON Sch... Closed
relates to FOLSPRINGS-4 Spring Way: Fix Maven replacement tok... Closed
relates to FOLSPRINGS-5 Spring Way: Add documentation for con... Closed
relates to FOLSPRINGS-8 Spring Way: Provide Dockerfile for st... Closed
relates to FOLSPRINGS-9 Spring Way: Disable old healthChkCmd ... Closed
relates to FOLIO-2793 Investigate AMF "translate" facility ... Open
relates to FOLIO-2792 Investigate AMF "validate" facility t... Closed
relates to FOLIO-2973 Include mod-search into reference env... Closed
Sprint: DevOps: Sprint 103, DevOps: Sprint 102, DevOps: Sprint 104
Development Team: FOLIO DevOps

 Description   

Example projects:

https://github.com/folio-org/mod-password-validator
https://github.com/MikhailFokanov/folio-mod-users
https://github.com/folio-org/folio-spring-base
https://github.com/folio-org/mod-spring-template
https://github.com/folio-org/mod-remote-storage
https://github.com/folio-org/mod-search

Existing pipelines likely need to be updated with regards to:

  • RAML limiting/validation in favour of OpenAPI validation
  • HTML API docs publication from RAML in favour of OpenAPI
  • Operational concerns:
    1. DB credentials (for RMB these are defined through envvars in MD)
    2. JVM options – should these differ from RMB configuration
    3. Command line options – RMB has extensive docs for things like how to configure logging, instrumentation, JMX, etc?
    4. Any other JVM system properties and evironment variable supported
  • Metrics/Instrumentation – what e.g connection pool metrics are supported?
  • Anything else?

See Gdoc: Summary Spring Way CI FOLIO-2863



 Comments   
Comment by David Crossley [ 17/Nov/20 ]

Define each needed piece in the following Comments.

Not in any particular order.

When ready, then create Jira tickets for each.

Comment by David Crossley [ 17/Nov/20 ]

A) api-lint : FOLIO-2893 Closed

Develop application to discover and process API description files. Report non-conformance.

The most likely tool for this is AMF https://github.com/aml-org/amf (see initial investigation in FOLIO-2792 Closed FOLIO-2793 Open ). These tools can process both RAML and OAS files. So the same application would handle both RAML-using and OAS-using FOLIO projects.

The application would be able to be run locally, and during CI on any branch.

Update: 2020-12-11: This part is done. See linked ticket.

Comment by David Crossley [ 17/Nov/20 ]

B) api-doc : FOLIO-2898 Closed

Develop an application to discover and process API description files (OpenAPI OAS) in a module repository, and generate standalone API documentation.

The generated documentation would be published by CI and integrated to the "API documentation" at https://dev.folio.org/reference/api/

Use https://github.com/Redocly/redoc

The application would be able to be run locally, and during CI on merge to main branch. And on finalising a release.

Comment by David Crossley [ 17/Nov/20 ]

C) API configuration
FOLIO-2893

Each module declare necessary items to enable other tools, e.g.

  • Where are the api descriptions in the repo filesystem
  • What type RAML/OAS
  • Other
Comment by David Crossley [ 17/Nov/20 ]

D) Provide shared versioned API definitions and schemas
FOLIO-2959

The RAML-based modules use "ramls/raml-util" as a git submodule of the shared "raml" repository. This avoids proliferation and inconsistent versions of these shared files.

How will Spring-based OpenAPI-based modules handle this?

Comment by David Crossley [ 17/Nov/20 ]

E) Terminology
FOLIO-2963

Define consistent terms.

Not old terms like "Swagger" but instead "OpenAPI" "OAS"

Comment by David Crossley [ 17/Nov/20 ]

F) New Jenkinsfile properties and related jenkins-pipeline-libs scripts
FOLIO-2893 and FOLIO-2898 and FOLIO-2917

Define new properties in module Jenkinsfile to declare the stages for "doApiLint" and "doApiDoc", to hook up the new jenkins-pipeline-libs scripts.

These might need extra parameters.

Document their use.

Comment by David Crossley [ 17/Nov/20 ]

G) New healthChkCmd in Jenkinsfile
FOLIO-2869

The RMB-based modules use the local "apidocs" embedded application.

The SpringWay modules will need something else.

Better to disable it until there is something relevant.

See also item "S) Define standard healthChkCmd for liveness check"

Comment by David Crossley [ 24/Nov/20 ]

H) FOLIO-2854 Open – Add folio-tools repo as git submodule of folio-infrastructure; adjust CI jobs lint-raml and generate-api-docs

Enable the folio-tools repo as git submodule of folio-infrastructure. Reconfigure the existing tools jenkins-pipeline-libs jobs to use that.

The new tools "api-lint" and "api-doc" will be similarly configured.

Update: 2020-12-16: Not at this stage

Comment by David Crossley [ 24/Nov/20 ]

I) Restore processing of "./descriptors/ModuleDescriptor-template.json" etc.

For the current Maven-based modules, it was decided to store Descriptors in a top-level directory at "./descriptors".

See FOLIO-701 Closed and documentation.

The maven build then transforms the replacement tokens (for "artifactId" and "version") in those template descriptors, and the resultant final Descriptors are generated to "./target/ModuleDescriptor.json" etc.

Various CI jobs then utilise those files at those locations:

The "jenkins-pipeline-libs/resources/org/folio/dockerHubPublishMetadata.sh" uses the template Descriptor directly.

The "jenkins-pipeline-libs/vars/buildMvn.groovy" does various different operations using the transformed file, expected to be at "./target/ModuleDescriptor.json".

The current example SpringWay project moves those Descriptors somewhere else, and does not transform them.

Is there reason to change those default locations?

Compare with other projects which have separate maven POM modules (e.g. mod-configuration, mod-data-import-converter-storage). Their files remain in the standard locations.

If the change is necessary, then the SpringWay projects will need special configuration (e.g. task "C") to declare the filesystem location of both the template and the transformed Descriptors.

Update: 2020-12-15: This part is completed. However see item J below.

Comment by Jakub Skoczen [ 08/Dec/20 ]

David Crossley mentioned there are other things that needs discussing like handling the envvars. We're going to discuss this on Monday.

Comment by David Crossley [ 15/Dec/20 ]

J) transform "./descriptors/ModuleDescriptor-template.json" etc.

FOLIO-2960

The replacement tokens for Spring-based modules are different to the normal form.

Instead they require tokens of the form: "@artifactId@-@version@"

The maven build then transforms the replacement tokens (for "artifactId" and "version") in those template descriptors, and the resultant final Descriptors are generated to "./target/ModuleDescriptor.json" etc.

See documentation.

Ensure that all Spring-based modules are using this form. For example, when last inspected, at least mod-spring-template and mod-search and mod-password-validator are still using the old form.

Comment by David Crossley [ 15/Dec/20 ]

K) Use 'buildDocker' in Jenkinsfile 'doDocker' section.

Use this instead of the 'buildJavaDocker'.

See documentation.

Comment by David Crossley [ 15/Dec/20 ]

L) Remove "USER root" from Dockerfile
MODPWD-52

The Dockerfile of the new mod-password-validator branch, uses the "folioci/alpine-jre-openjdk11:latest" from "folio-java-docker".

That provides the "USER folio", so use that.

Comment by David Crossley [ 16/Dec/20 ]

M) Align environment variable definitions in LaunchDescriptor with ./run-env.sh etc.

FOLIO modules define environment variables and Java options in the LaunchDescriptor portion of their ModuleDescriptor (documentation). These are utilised by folio-ansible. They also provide documentation for other DevOps operators, and are automatically published as part of the module's Docker Hub README.

The initial Spring Way modules seem to not use some of the variables that are defined in their LaunchDescriptor.

Comment by David Crossley [ 16/Dec/20 ]

N) Add documentation for configuration options
FOLIO-2961

The RMB has extensive documentation to enable DevOps operators to understand how to configure logging, instrumentation, JMX, etc.

The Spring Way group of modules need to provide such documentation.

Comment by David Crossley [ 16/Dec/20 ]

O) Metrics and instrumentation
FOLIO-2962

Describe what metrics are supported (e.g. connection pool metrics, etc.)

Comment by David Crossley [ 16/Dec/20 ]

P) Modify jenkins-pipeline-libs/vars/buildDocker.groovy to use dockerHubPublishMetadata
FOLIO-2916 Closed

The existing modules Jenkinsfiles use "buildJavaDocker", which utilises dockerHubPublishMetadata.sh to gather operational metadata from the module's LaunchDescriptor and publish it as part of the module's Docker Hub README.

The Spring Way modules are instead using the simpler "buildDocker" which does not yet utilise that facility.

Comment by David Crossley [ 16/Dec/20 ]

Q) Develop new CI tool to ensure descriptions of JSON schema properties
FOLIO-2917 Closed

The current lint-raml CI tool assesses the JSON schemas to ensure that every property is well-described. See the "Lint raml schema" stage of the existing pipeline.

Now we need a new tool that assesses schemas which are associated with either RAML or OpenAPI descriptions.

Comment by David Crossley [ 16/Dec/20 ]

Taras Spashchenko We think that we have now defined most of the items that are required.

Please coordinate with your teams that will be the initial Spring Way implementers. Ensure that mod-spring-template and folio-spring-base are ready, and that the early-starter projects are aligned.

The main items listed above that need attention by developers are:
D, E, G, J, K, L, M, N, O, R, T, U

The FOLIO DevOps team will attend to the others.

Comment by David Crossley [ 18/Dec/20 ]

R) Base Docker image
FOLIO-2966

The DevOps and Spring Way people need a discussion about using the provided folio-java-docker (e.g. currently non-standard are: mod-remote-storage and mod-search).

Comment by Marc Johnson [ 07/Jan/21 ]

David Crossley

The DevOps and Spring Way people need a discussion about using the provided folio-java-docker (e.g. current mod-remote-storage).

Does that mean that Spring Way modules are using a non-standard base docker image at the moment?

Comment by David Crossley [ 08/Jan/21 ]

Marc Johnson As indicated at item "R".

Today investigated again the set of initial modules listed in this ticket Description:
mod-remote-storage is the only non-standard base docker image.

Comment by David Crossley [ 08/Jan/21 ]

Jakub Skoczen I summarised this ticket to date, and added Gdoc link from this ticket's Description.

See Gdoc: Summary Spring Way CI FOLIO-2863

Comment by Marc Johnson [ 08/Jan/21 ]

David Crossley

As indicated at item "R".

Apologies, I was reacting to your comment, I didn't notice this in the definition of part R at the time

Comment by David Crossley [ 10/Jan/21 ]

S) Define standard healthChkCmd for liveness check
FOLIO-2968

Define new way of doing this for all modules.

The RMB-based modules use the local "apidocs" embedded application.
The SpringWay modules will need something else. (See item "G".)

Marc Johnson commented:

Might now be a good time to have a FOLIO standard for health checks that are independent of an implementation technology?

Status: 2021-01-11: TODO: tech-leads

Comment by David Crossley [ 13/Jan/21 ]

T) Using $schema keyword in JSON Schema

For the RAML-based modules, most schema files do have the "$schema" keyword:
e.g. "$schema": "http://json-schema.org/draft-04/schema#"

The current Springway modules do not have any such.

Comment by David Crossley [ 19/Jan/21 ]

U) Explain database schema, indexes, etc.
FOLIO-2970

For RMB-based modules The Post Tenant API consults the database definitions at "src/main/resources/templates/db_scripts/schema.json".

Document how Spring Way modules handle that.

Generated at Thu Feb 08 23:23:48 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.