[FOLIO-2863] pipeline changes required for Spring Way projects Created: 05/Nov/20 Updated: 02/Feb/21 |
|
| Status: | Blocked |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | New Feature | Priority: | P3 |
| Reporter: | Jakub Skoczen | Assignee: | David Crossley |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | devops-backlog, springway | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | DevOps: Sprint 103, DevOps: Sprint 102, DevOps: Sprint 104 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
Example projects: https://github.com/folio-org/mod-password-validator Existing pipelines likely need to be updated with regards to:
See Gdoc: Summary Spring Way CI FOLIO-2863 |
| Comments |
| Comment by David Crossley [ 17/Nov/20 ] |
|
Define each needed piece in the following Comments. Not in any particular order. When ready, then create Jira tickets for each. |
| Comment by David Crossley [ 17/Nov/20 ] |
|
A) api-lint :
Develop application to discover and process API description files. Report non-conformance. The most likely tool for this is AMF https://github.com/aml-org/amf (see initial investigation in
The application would be able to be run locally, and during CI on any branch. Update: 2020-12-11: This part is done. See linked ticket. |
| Comment by David Crossley [ 17/Nov/20 ] |
|
B) api-doc :
Develop an application to discover and process API description files (OpenAPI OAS) in a module repository, and generate standalone API documentation. The generated documentation would be published by CI and integrated to the "API documentation" at https://dev.folio.org/reference/api/ Use https://github.com/Redocly/redoc The application would be able to be run locally, and during CI on merge to main branch. And on finalising a release. |
| Comment by David Crossley [ 17/Nov/20 ] |
|
C) API configuration Each module declare necessary items to enable other tools, e.g.
|
| Comment by David Crossley [ 17/Nov/20 ] |
|
D) Provide shared versioned API definitions and schemas The RAML-based modules use "ramls/raml-util" as a git submodule of the shared "raml" repository. This avoids proliferation and inconsistent versions of these shared files. How will Spring-based OpenAPI-based modules handle this? |
| Comment by David Crossley [ 17/Nov/20 ] |
|
E) Terminology Define consistent terms. Not old terms like "Swagger" but instead "OpenAPI" "OAS" |
| Comment by David Crossley [ 17/Nov/20 ] |
|
F) New Jenkinsfile properties and related jenkins-pipeline-libs scripts Define new properties in module Jenkinsfile to declare the stages for "doApiLint" and "doApiDoc", to hook up the new jenkins-pipeline-libs scripts. These might need extra parameters. Document their use. |
| Comment by David Crossley [ 17/Nov/20 ] |
|
G) New healthChkCmd in Jenkinsfile The RMB-based modules use the local "apidocs" embedded application. The SpringWay modules will need something else. Better to disable it until there is something relevant. See also item "S) Define standard healthChkCmd for liveness check" |
| Comment by David Crossley [ 24/Nov/20 ] |
|
H)
Enable the folio-tools repo as git submodule of folio-infrastructure. Reconfigure the existing tools jenkins-pipeline-libs jobs to use that. The new tools "api-lint" and "api-doc" will be similarly configured. Update: 2020-12-16: Not at this stage |
| Comment by David Crossley [ 24/Nov/20 ] |
|
I) Restore processing of "./descriptors/ModuleDescriptor-template.json" etc. For the current Maven-based modules, it was decided to store Descriptors in a top-level directory at "./descriptors". See
The maven build then transforms the replacement tokens (for "artifactId" and "version") in those template descriptors, and the resultant final Descriptors are generated to "./target/ModuleDescriptor.json" etc. Various CI jobs then utilise those files at those locations: The "jenkins-pipeline-libs/resources/org/folio/dockerHubPublishMetadata.sh" uses the template Descriptor directly. The "jenkins-pipeline-libs/vars/buildMvn.groovy" does various different operations using the transformed file, expected to be at "./target/ModuleDescriptor.json". The current example SpringWay project moves those Descriptors somewhere else, and does not transform them. Is there reason to change those default locations? Compare with other projects which have separate maven POM modules (e.g. mod-configuration, mod-data-import-converter-storage). Their files remain in the standard locations. If the change is necessary, then the SpringWay projects will need special configuration (e.g. task "C") to declare the filesystem location of both the template and the transformed Descriptors. Update: 2020-12-15: This part is completed. However see item J below. |
| Comment by Jakub Skoczen [ 08/Dec/20 ] |
|
David Crossley mentioned there are other things that needs discussing like handling the envvars. We're going to discuss this on Monday. |
| Comment by David Crossley [ 15/Dec/20 ] |
|
J) transform "./descriptors/ModuleDescriptor-template.json" etc. The replacement tokens for Spring-based modules are different to the normal form. Instead they require tokens of the form: "@artifactId@-@version@" The maven build then transforms the replacement tokens (for "artifactId" and "version") in those template descriptors, and the resultant final Descriptors are generated to "./target/ModuleDescriptor.json" etc. See documentation. Ensure that all Spring-based modules are using this form. For example, when last inspected, at least mod-spring-template and mod-search and mod-password-validator are still using the old form. |
| Comment by David Crossley [ 15/Dec/20 ] |
|
K) Use 'buildDocker' in Jenkinsfile 'doDocker' section. Use this instead of the 'buildJavaDocker'. See documentation. |
| Comment by David Crossley [ 15/Dec/20 ] |
|
L) Remove "USER root" from Dockerfile The Dockerfile of the new mod-password-validator branch, uses the "folioci/alpine-jre-openjdk11:latest" from "folio-java-docker". That provides the "USER folio", so use that. |
| Comment by David Crossley [ 16/Dec/20 ] |
|
M) Align environment variable definitions in LaunchDescriptor with ./run-env.sh etc. FOLIO modules define environment variables and Java options in the LaunchDescriptor portion of their ModuleDescriptor (documentation). These are utilised by folio-ansible. They also provide documentation for other DevOps operators, and are automatically published as part of the module's Docker Hub README. The initial Spring Way modules seem to not use some of the variables that are defined in their LaunchDescriptor. |
| Comment by David Crossley [ 16/Dec/20 ] |
|
N) Add documentation for configuration options The RMB has extensive documentation to enable DevOps operators to understand how to configure logging, instrumentation, JMX, etc. The Spring Way group of modules need to provide such documentation. |
| Comment by David Crossley [ 16/Dec/20 ] |
|
O) Metrics and instrumentation Describe what metrics are supported (e.g. connection pool metrics, etc.) |
| Comment by David Crossley [ 16/Dec/20 ] |
|
P) Modify jenkins-pipeline-libs/vars/buildDocker.groovy to use dockerHubPublishMetadata The existing modules Jenkinsfiles use "buildJavaDocker", which utilises dockerHubPublishMetadata.sh to gather operational metadata from the module's LaunchDescriptor and publish it as part of the module's Docker Hub README. The Spring Way modules are instead using the simpler "buildDocker" which does not yet utilise that facility. |
| Comment by David Crossley [ 16/Dec/20 ] |
|
Q) Develop new CI tool to ensure descriptions of JSON schema properties The current lint-raml CI tool assesses the JSON schemas to ensure that every property is well-described. See the "Lint raml schema" stage of the existing pipeline. Now we need a new tool that assesses schemas which are associated with either RAML or OpenAPI descriptions. |
| Comment by David Crossley [ 16/Dec/20 ] |
|
Taras Spashchenko We think that we have now defined most of the items that are required. Please coordinate with your teams that will be the initial Spring Way implementers. Ensure that mod-spring-template and folio-spring-base are ready, and that the early-starter projects are aligned. The main items listed above that need attention by developers are: The FOLIO DevOps team will attend to the others. |
| Comment by David Crossley [ 18/Dec/20 ] |
|
R) Base Docker image The DevOps and Spring Way people need a discussion about using the provided folio-java-docker (e.g. currently non-standard are: mod-remote-storage and mod-search). |
| Comment by Marc Johnson [ 07/Jan/21 ] |
Does that mean that Spring Way modules are using a non-standard base docker image at the moment? |
| Comment by David Crossley [ 08/Jan/21 ] |
|
Marc Johnson As indicated at item "R". Today investigated again the set of initial modules listed in this ticket Description: |
| Comment by David Crossley [ 08/Jan/21 ] |
|
Jakub Skoczen I summarised this ticket to date, and added Gdoc link from this ticket's Description. See Gdoc: Summary Spring Way CI FOLIO-2863 |
| Comment by Marc Johnson [ 08/Jan/21 ] |
Apologies, I was reacting to your comment, I didn't notice this in the definition of part R at the time |
| Comment by David Crossley [ 10/Jan/21 ] |
|
S) Define standard healthChkCmd for liveness check Define new way of doing this for all modules. The RMB-based modules use the local "apidocs" embedded application. Marc Johnson commented:
Status: 2021-01-11: TODO: tech-leads |
| Comment by David Crossley [ 13/Jan/21 ] |
|
T) Using $schema keyword in JSON Schema For the RAML-based modules, most schema files do have the "$schema" keyword: The current Springway modules do not have any such. |
| Comment by David Crossley [ 19/Jan/21 ] |
|
U) Explain database schema, indexes, etc. For RMB-based modules The Post Tenant API consults the database definitions at "src/main/resources/templates/db_scripts/schema.json". Document how Spring Way modules handle that. |