[FOLIO-3238] Add new mod-oa and ui-oa modules to FOLIO CI pipeline and platform-complete Created: 15/Jul/21  Updated: 13/Sep/21  Resolved: 13/Sep/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: TBD
Reporter: John Malconian Assignee: David Crossley
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File testing-test-155.tar.gz    
Sprint: DevOps Sprint 121, DevOps Sprint 122
Development Team: FOLIO DevOps

 Description   

Prepare new modules, mod-oa (https://github.com/folio-org/mod-oa) and ui-oa (https://github.com/folio-org/ui-oa) for inclusion into FOLIO CI pipeline and platform-complete. The modules are targeted for the Kiwi release.



 Comments   
Comment by John Malconian [ 15/Jul/21 ]

Before these modules can be added to platform-complete and be included in daily snapshot reference builds, they need to be added to the FOLIO CI pipeline so that, at a minimum, module descriptors are posted to the FOLIO MD registry and Docker and NPM artifacts are generated and uploaded to the usual FOLIO Docker and NPM registries with artifact versioning consistent with other folio-org modules. Runtime configuration options should also be documented in their respective repos as needed. Additional info here: https://dev.folio.org/guidelines/create-new-repo/

Comment by Jakub Skoczen [ 03/Aug/21 ]

Owen Stephens can you please see John's comment above about missing stuff that we need in order to add these module into ref envs. Thanks!

Comment by Ian Ibbotson (Use this one) [ 03/Aug/21 ]

Added FOLIO Jenkinsfiles following the same pattern as mod-agreements and ui-agreements with appropriate changes for mod-oa.

Bounced back to JM - please bat back to me any problems.

Comment by David Crossley [ 04/Aug/21 ]

With mod-oa, your mainline build failed after adding Jenkinsfile.

I do not know much about Gradle, but ...

diff mod-oa/service/build.gradle mod-agreements/service/build.gradle
shows many differences. One stands out:

<   from 'folioci/alpine-jre-openjdk8:latest'
---
>   from 'folioci/alpine-jre-openjdk11:latest'
Comment by David Crossley [ 04/Aug/21 ]

Our small FOLIO DevOps team does not have time to spare.

I have done some of the fiddly bits, following https://dev.folio.org/guidelines/create-new-repo/ and the linked docs.

Comment by David Crossley [ 04/Aug/21 ]

At the mod-oa ModuleDescriptor-template.json it is missing the "memory" and "JAVA_OPTIONS" for the "launchDescriptor".

Also does it use a database (like does mod-agreements)? If so, then you need to add DB properties as explained at the above-mentioned doc.

Comment by Ian Ibbotson (Use this one) [ 04/Aug/21 ]

Thanks david - corrected jar filename in docker build step, bumped JDK to 11 just for compliance with folio regs and added some launch descriptor props as requested. Seems to be building OK now. Is that the end of the process and will the builds start to appear now?

Comment by Ian Ibbotson (Use this one) [ 04/Aug/21 ]

Master built and docker image published - word of warning the module descriptors are likely ending up in one of your okapi registries before this build runs as our local CI also builds and publishes a docker image to nexus and then publishes the module descriptors to the okapi being used for reshare - same build script.

Comment by David Crossley [ 05/Aug/21 ]

Regarding mod-oa:

I verified that the latest ModuleDescriptor is at the folio-registry and the Docker image is at folioci, with the correct tags/ids. So good.

As John mentioned above, if there are any additional runtime configuration options for mod-oa, then please add to its README.

Comment by David Crossley [ 05/Aug/21 ]

Regarding mod-oa:

FOLIO has a requirement for a PERSONAL_DATA_DISCLOSURE form. See notes at https://dev.folio.org/guidelines/create-new-repo/

Comment by David Crossley [ 05/Aug/21 ]

Regarding ui-oa: Please tell us when its build gets past the tests stage, and then we can verify the remainder.

Comment by David Crossley [ 05/Aug/21 ]

The mod-oa LaunchDescriptor has high "memory" allocation and high DB_MAXPOOLSIZE, compared to the doc "launchDescriptor" and other modules. Wondering if that is due to copying from mod-agreements. But if mod-oa does need that, then fine.

Comment by Jakub Skoczen [ 10/Aug/21 ]

Ian Ibbotson (Use this one) Just a friendly reminder that the DevOps teams is waiting for fixes to the ui- module build so it passes its tests.

Comment by Jakub Skoczen [ 19/Aug/21 ]

Ian Ibbotson (Use this one) ping?

Comment by Jakub Skoczen [ 31/Aug/21 ]

Ian Ibbotson (Use this one) David mentioned that he can see the ui-oa build succeeeding now so he will try to include the modules in the reference environments. We will still need some improvements to the repo – these are listed by David above. Will you guys be able to handle these?

Comment by Ian Ibbotson (Use this one) [ 31/Aug/21 ]

Sorry - I thought we had done everything requested - can you be specific which we missed?

Comment by David Crossley [ 02/Sep/21 ]

You recently added some instructions to the mod-oa readme as requested, so i suppose that that part is now done.

There is this unanswered query about LaunchDescriptor settings.

Okapi is warning about the ModuleDescriptor:

WARN  ModuleDescriptor Module 'mod-oa-1.0.0-SNAPSHOT.11' has no Requires section. If the module really does not require any other interfaces, provide an empty array to be explicit about it.

It is not up to DevOps to be gatekeeper on this one (and not needed for this current ticket) but it may be a later holdup. The personal data disclosure form.

Comment by David Crossley [ 02/Sep/21 ]

Today i attempted to add the back-end mod-oa to the "folio-testing-test" reference environment. However it failed. Okapi tries the full 90 cycles to wait for the module to respond, but it does not, after 15 minutes.

...
00:27:57 [] [] [] [] INFO  TcpPortWaiting       Try connect to service mod-oa-1.0.0-SNAPSHOT.11 at 10.36.1.96:9187 count 89
00:28:15 [] [] [] [] INFO  TcpPortWaiting       Try connect to service mod-oa-1.0.0-SNAPSHOT.11 at 10.36.1.96:9187 count 90
00:28:15 [] [] [] [] INFO  DockerModuleHandle   stop container 36e27737637b1e48cab9f1e34d0dbec67ab32fde32332b06689296c2be091c35 image docker.ci.folio.org/folioci/mod-oa:1.0.0-SNAPSHOT.11
00:28:15 [] [] [] [] WARN  DeploymentManager    Deploying mod-oa-1.0.0-SNAPSHOT.11 failed
00:28:15 [405101/proxy] [supertenant] [] [] INFO  TenantManager        job complete
00:28:15 [405101/proxy] [supertenant] [] [] WARN  TenantManager        job failed
org.folio.okapi.util.OkapiError: Deployment failed for service mod-oa-1.0.0-SNAPSHOT.11. Could not connect to 10.36.1.96:9187: finishConnect(..) failed: Connection refused: /10.36.1.96:9187
...

Please follow these instructions https://dev.folio.org/guides/install-backend-module/#ensure-recent-local-vm
and tell us when it is successful.

Comment by Owen Stephens [ 02/Sep/21 ]

I've created tickets for the personal data disclosure form MODOA-1 Closed and UIOA-6 Closed

Comment by David Crossley [ 09/Sep/21 ]

I tried again today with folio-testing-test. It failed in the same way. This time i got the docker logs before okapi removed the container (see attached). I note that the logs for mod-licenses are similar, but mod-licenses does finish its deployment.

...
00:16:40 [] [] [] [] INFO  DockerModuleHandle   mod-oa-1.0.0-SNAPSHOT.11 Grails application running at http://localhost:8080 in environment: production
00:16:40 [] [] [] [] INFO  DockerModuleHandle   mod-agreements-5.0.0-SNAPSHOT.390 Application Initialization: Runtime memory reported 545.25 mb
00:16:40 [] [] [] [] INFO  DockerModuleHandle   mod-agreements-5.0.0-SNAPSHOT.390 Application Initialization: Runtime CPUs reported 8
00:16:40 [] [] [] [] INFO  DockerModuleHandle   mod-agreements-5.0.0-SNAPSHOT.390 Application Initialization: Allocated 4 IO Threads
00:16:40 [] [] [] [] INFO  DockerModuleHandle   mod-agreements-5.0.0-SNAPSHOT.390 Application Initialization: Allocated 24 worker threads
00:16:42 [] [] [] [] INFO  DockerModuleHandle   mod-licenses-4.0.0-SNAPSHOT.183 Grails application running at http://localhost:8080 in environment: production
00:16:44 [] [] [] [] INFO  TcpPortWaiting       Try connect to service mod-agreements-5.0.0-SNAPSHOT.390 at 10.36.1.16:9189 count 22
00:16:44 [] [] [] [] INFO  TcpPortWaiting       Try connect to service mod-oa-1.0.0-SNAPSHOT.11 at 10.36.1.16:9187 count 22
00:16:44 [] [] [] [] INFO  TcpPortWaiting       Try connect to service mod-licenses-4.0.0-SNAPSHOT.183 at 10.36.1.16:9190 count 22
00:16:45 [] [] [] [] INFO  TcpPortWaiting       Connected to service mod-licenses-4.0.0-SNAPSHOT.183 at 10.36.1.16:9190 count 22
00:16:45 [] [] [] [] INFO  DockerModuleHandle   mod-agreements-5.0.0-SNAPSHOT.390 Grails application running at http://localhost:8080 in environment: production
00:16:49 [] [] [] [] INFO  TcpPortWaiting       Try connect to service mod-oa-1.0.0-SNAPSHOT.11 at 10.36.1.16:9187 count 23
00:16:49 [] [] [] [] INFO  TcpPortWaiting       Connected to service mod-agreements-5.0.0-SNAPSHOT.390 at 10.36.1.16:9189 count 23
...
Comment by Ian Ibbotson (Use this one) [ 09/Sep/21 ]

Hey David - thanks for the suggestions, but we do already run this module through vagrant - it's how the devs work mostly. We also CD/CD deploy this module to kifull.libsdev.k-int.com where it seems to work without issue. What I've realised however is that we manually override the service port number running locally, it had --8079 in the descriptor file but the service actually runs on 8080. I've asked Ethan to update the deployment descriptor so the port number there matches the default.

Please do retry.

Comment by David Crossley [ 10/Sep/21 ]

I started my day with such high hopes, but no, same problem.

I do see that the port number was changed in the LaunchDescriptor of the ModuleDescriptor.

So i did grep the source code, and found that there are still mentions of "8079". Notably in "service/build.gradle" where it constructs the Dockerfile. See "EXPOSE 8079" in the Jenkins log.

As explained in this documentation, it is not required to be 8080, but does of course need to be consistent.

Comment by Ian Ibbotson (Use this one) [ 10/Sep/21 ]

Have updated the EXPOSE port to 8080, tho my understanding is that it has no FUNCTIONAL impact - it's only there as a form of documentation

Would it be possible for me to see the module logs please - just to see if there is any reason the module is not starting up as expected? There should be useful info output at module boot time.

Comment by David Crossley [ 10/Sep/21 ]

As in my Jira comment from yesterday explaining the startup and behaviour description, please see the module logs for both mod-oa and mod-licenses previously attached to this Jira.

Comment by David Crossley [ 10/Sep/21 ]

Hooray. With your recent change to the generated Dockerfile, we now have greenness at Jenkins folio-testing-test.

That Dockerfile is actually the real thing. As i noted in the previous comment, the image is built and deployed during the mod-oa FOLIO CI run.

The "documentation" that you refer to, i wonder if that is the Docker Hub doc that is generated from the module's LaunchDescriptor (e.g. for mod-notes). However our CI does not do that for Groovy-based modules.

Comment by David Crossley [ 13/Sep/21 ]

The mod-oa and ui-oa have now been added to platform-complete snapshot and to reference environments folio-snapshot and folio-testing.

 

Generated at Thu Feb 08 23:26:38 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.