[FOLIO-701] generate Module and Deployment Descriptors from POM files Created: 28/Jun/17  Updated: 12/Nov/18  Resolved: 06/Sep/17

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: New Feature Priority: P2
Reporter: Jakub Skoczen Assignee: David Crossley
Resolution: Done Votes: 0
Labels: ci, core, sprint19, sprint20, sprint21, sprint22
Remaining Estimate: Not Specified
Time Spent: 1 hour, 35 minutes
Original estimate: Not Specified

Issue links:
Blocks
blocks MODCONF-4 include version string in the module ... Closed
is blocked by FOLIO-793 Use mod descriptors from repository i... Closed
is blocked by FOLIO-768 Fix the Descriptor discrepancies Closed
is blocked by FOLIO-777 Jenkins needs 'gradle generateDescrip... Closed
Relates
relates to FOLIO-787 Some backend repositories are missing... Open
relates to FOLIO-728 publish ModuleDescriptors to the cent... Closed
relates to FOLIO-786 Update folio-test-env to use generate... Closed
relates to FOLIO-813 rename artifacts of circulation and i... Closed
relates to OKAPI-415 Update folio-sample-modules for Okapi... Closed
Sprint:

 Description   

1. Drop ModuleDescriptor.json from each repo and instead add ModuleDescriptor.template.json with values to be substituted
2. Replace id fields with the expression ${artifactId}-${version} from the pom.xml during generate-resources phase

Do the same for the DeploymentDescriptor if any such is generated/kept in Git.



 Comments   
Comment by David Crossley [ 26/Jul/17 ]

I do have a working solution here, with both the input file and the output file at the top-level of the repository.

Is that where the output file should be generated to?

Or should it be to "target/classes/ModuleDescriptor.json" so that it gets packed into the top-level of the jar file?

Or both?

Comment by David Crossley [ 01/Aug/17 ]

Wayne Schneider: How will we co-ordinate this with folio-ansible and its use of the ModuleDescriptor.json files? I am almost ready to apply my changes to each repository, probably during my tomorrow. If it would make it easier, then i could leave each current mod-*/ModuleDescriptor.json in place, and when folio-ansible is eventually using the generated ones, then remove each.

Comment by David Crossley [ 01/Aug/17 ]

In each mod-* repository, the output files will go into ./target for the Maven ones and to ./build for the Gradle ones.

Comment by Wayne Schneider [ 01/Aug/17 ]

David Crossley – please do leave the current ModuleDescriptor.json files in place. We will have to make this change fairly quickly, so that the static files don't get out of date. Is your work available on a branch somewhere, so I can preview it?

Comment by David Crossley [ 01/Aug/17 ]

Wayne Schneider: See branch "folio-701-generate-descriptors" in each of "mod-configuration" and "mod-users" and "mod-circulation".

Comment by Wayne Schneider [ 02/Aug/17 ]

What if...we used the generated module descriptor as metadata (a label) for the Docker image? Then the JSON would be available to the build process via docker inspect.

Comment by David Crossley [ 02/Aug/17 ]

Done mod-configuration. So that is ready now for the FOLIO-728 Closed to publish to the central registry.

Comment by David Crossley [ 03/Aug/17 ]

I would like clarification about how to proceed.

Because we are temporarily retaining each repository's current ModuleDescriptor.json, and working with a copy in src/main/resources/ModuleDescriptor-template.json instead, then it should be okay for me to proceed to apply this to each of the other mod-* repositories.

I will ensure that any changes to the active MD are reflected in the other, but that is not critical as the latter is not yet in production.

Comment by Hongwei Ji [ 03/Aug/17 ]

Hi David Crossley, I noticed that not every project has consistent project/src/main/resources layout (for example mod-configuration). What about putting descriptor templates under a separate folder like project/descriptor to stand it out? Just a thought.

Comment by David Crossley [ 04/Aug/17 ]

Thanks for your suggestion. I was trying to keep resources that will be processed in that standard project/src/main/resources location. Yes, the mod-configuration is different (being two sub-projects), so had to create the path just for that. Also, in the other projects maven is being too helpful and automatically processing it again unfiltered (although i have solved that in my mod-users branch).

I will re-do them following your suggestion.

Comment by David Crossley [ 04/Aug/17 ]

Done mod-configuration and mod-users.

Comment by David Crossley [ 07/Aug/17 ]

Now completed most of the Maven-based ones (still mod-acquisitions* and folio-sample-modules).
So 'mvn install' uses the templates in the ./descriptors directory, and generates into ./target/*Descriptor.json

Also done the Gradle-based ones. Ditto, but into its ./build directory.
John Malconian: Jenkins can now also call 'gradle generateDescriptors' (or just 'gradle' which has defaultTasks 'build', 'generateDescriptors').

Comment by David Crossley [ 07/Aug/17 ]

So for each repository, the Descriptors at the top-level are still the ones that are in production.

When we make changes to them, it is ideal to commit to both the top-level file and the file in the ./descriptors directory.
I do have a local shell script to assist with keeping them synchronised.

Comment by David Crossley [ 07/Aug/17 ]

On the way i found some discrepancies between the "id" and "version" in the POM, and that in the Descriptors.

See those using `diff ModuleDescriptor.json target` etc.

For example, no differences in mod-configuration and mod-circulation-storage. However in mod-users-bl there is different id and different version ...

diff ModuleDescriptor.json target
2c2
<   "id": "mod-users-bl-2.0.0",
---
>   "id": "users_bl-2.0.2-SNAPSHOT",

Soon i will make a list of remaining discrepancies.

Comment by Marc Johnson [ 07/Aug/17 ]

David Crossley Is my understanding correct that for the time being, any changes to either descriptor means changing the static files in the root and the templates in the descriptors directory (until such time as we drop the static files)?

Comment by David Crossley [ 07/Aug/17 ]

Yes. Thanks for emphasising.
As explained above i will also monitor and fix.

Comment by Jakub Skoczen [ 09/Aug/17 ]

David, please establish the necessary changes to pom.xml (e.g artifact name) to make sure the generated MDs are aligned with the MDs currently stored in Git (apart from version number which is okay to change). Please, validate with John Malconian if those changes to pom.xml files can be performed immediately without breaking the existing builds.

Comment by David Crossley [ 10/Aug/17 ]

The discrepancies are listed in FOLIO-768 Closed and each related one.

Comment by David Crossley [ 10/Aug/17 ]

We discussed this. Proposing to attend to it together on Monday.

Comment by Heikki Levanto [ 11/Aug/17 ]

I have one reservation with this method. The descriptors in the main directory will now be machine-generated code, and as such, should in principle not be committed in git. Maybe we don't need them at all? Would it not be enough to take the generated descriptors from the target directory, when we need one to push to the central repo, pack into the docker image, etc.

Comment by Marc Johnson [ 11/Aug/17 ]

Heikki Levanto

I believe the descriptors in root of the repository are intended to be the static ones (from before the generation work was started).

I think these have been left (and need to be updated as usual), for compatibility, until the build / release processes use the newly generated ones, at which point they will be removed.

Comment by David Crossley [ 14/Aug/17 ]

That is correct. Wayne will no doubt be busy on return this week, but we hope to co-ordinate (as there are side-effects) and finalise this task early.

Comment by David Crossley [ 15/Aug/17 ]

Some repositories have no DeploymentDescriptor. Will add those as a separate task.

Comment by David Crossley [ 16/Aug/17 ]

The FOLIO-768 Closed discrepancies are now fixed. Wayne is proceeding to adjust CI/Ansible and to publish ModuleDescriptors to Okapi.

Comment by David Crossley [ 18/Aug/17 ]

The folio-test-env needs to be adjusted to use these generated ModuleDescriptors FOLIO-786 Closed .

Comment by David Crossley [ 23/Aug/17 ]

The FOLIO-728 Closed and FOLIO-793 Closed are now completed, so followed up to remove all old top-level Descriptors.

Now being generated from ./descriptors/*-template.json

Comment by David Crossley [ 29/Aug/17 ]

I needed to fix some test scripts in mod-circulation etc.
Those are ready to merge now.

Generated at Thu Feb 08 23:07:43 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.