[RMB-30] Trouble with a schema file referring to another schema in raml-util Created: 31/May/17  Updated: 05/Sep/17  Resolved: 09/Aug/17

Status: Closed
Project: RAML Module Builder
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P3
Reporter: David Crossley Assignee: David Crossley
Resolution: Done Votes: 0
Labels: raml, sprint19
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks FOLIO-592 meta-data section in FOLIO records (c... Closed
Relates
relates to FOLIO-614 Be able to share JSON schema definiti... Open
relates to FOLIO-573 Use a consistent method for RAML to r... Closed
relates to RMB-50 Clean the temp space after build to r... Closed
Sprint:

 Description   

Using relative links between schema for RAML does work okay for both sets of tools ( FOLIO-573 Closed ), i.e. use the relative pathname in the schema files, and as the schema key name in the RAML file.

However, when attempting to refer to a schema file in the shared "raml-util" then there can be a problem.

(Note that some early comments below would need some adjustment.
See the diagnosis at 21/Jun/17
and the solution at 22/Jun/17 onwards.)



 Comments   
Comment by David Crossley [ 31/May/17 ]

So for example, for the raml-module-builder repository, if it was like this then it is processed okay:

raml-module-builder/ramls/schemas/
raml-module-builder/raml-util/schemas/

in raml-module-builder/ramls/schemas/job.schema
  "$ref" : "../../raml-util/schemas/parameters.schema"

in raml-module-builder/ramls/jobs.raml
  - ../../raml-util/schemas/parameters.schema: !include ../raml-util/schemas/parameters.schema

However with the current location (raml-module-builder/domain-models-api-interfaces/src/main/resources/raml/schemas) or any other place, then raml-for-jax-rs in RMB cannot handle it.

Comment by Kurt Nordstrom [ 01/Jun/17 ]

Isn't this more specifically an RMB issue? When I check these files with raml-cop, they validate fine.

Comment by David Crossley [ 01/Jun/17 ]

The raml-module-builder repo is the only one at the moment that uses $ref in a schema to refer to another schema in the "raml-util" area. When we try to use the relative pathname as the schema key, then raml-cop is fine because its underlying parser does not utilise the schema key name (just the "!include" path). The raml-for-jax-rs (underlying the RMB) then chokes.

However all is okay in other repos where the raml-util directory is at the same directory tree level as its "ramls" directory.

So yes it is "only" the RMB that has a problem if the directories are not like that.

The complicated issues with raml-for-jax-rs are linked in FOLIO-573 Closed . This seems like another wrinkle.

Comment by shale99 [ 05/Jun/17 ]

hi David Crossley
creating relative paths entries works with a limited amount of ../../
can we set this as an open issue that the /raml and /raml-util should both be root directories in an rmb project?
i dont really know how to fix this

Comment by David Crossley [ 07/Jun/17 ]

I enhanced the note in raml-module-builder/README.md to advise about the placement of those directories.

Our other RMB-using modules do already have such an arrangement.

Comment by David Crossley [ 07/Jun/17 ]

Regarding actually "fixing", there are various similar open issues at raml-for-jax-rs. It might assist to add another bug report for this specific case.

Comment by shale99 [ 07/Jun/17 ]

I enhanced the note in raml-module-builder/README.md to advise about the placement of those directories.

thanks!

Comment by David Crossley [ 07/Jun/17 ]

For "raml-module-builder" i have done this:

 
git mv domain-models-api-interfaces/src/main/resources/raml domain-models-api-interfaces/raml
git mv raml-util domain-models-api-interfaces/raml-util

The RMB does build with success. Using that snapshot in mod-inventory-storage runs its tests with success.

Now ready in branch folio-573-rmb-30-filename-schema-key

shale99: Are you happy with that directory arrangement?

Comment by shale99 [ 07/Jun/17 ]

yea, that works for me

Comment by David Crossley [ 07/Jun/17 ]

Beaut. I will go ahead.

Comment by David Crossley [ 07/Jun/17 ]

Drat. It was fine here on macOS with its "tmp" filesystem. However following the merge of pull/85 it failed on the build on the server.

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java (generate_interfaces) on project domain-models-api-interfaces: An exception occured while executing the Java class. null: InvocationTargetException: Directory '/tmp/1496818545170-0/../../raml-util/schemas' could not be created

I reverted the PR merge.

Comment by David Crossley [ 09/Jun/17 ]

Beware too many dotdots.

Putting the "raml-util" git submodule directly inside the "raml" directory does work for the "raml-module-builder" repo, i.e. at raml-module-builder/domain-models-api-interfaces/src/main/resources/raml/raml-util/

See branch folio-573-rmb-30-filename-schema-key-2

The build is successful on both macOS and on Ubuntu Linux.

Comment by David Crossley [ 16/Jun/17 ]

I did various testing for the location of the "raml-util" directory, for RMB's own RAMLs and for mod-configuration. For some it did work on macOS but not on Linux. (See attached results.txt)

Summary: If there are schema files that use $ref to another schema in raml-util, then that "raml-util" is best placed inside the "ramls" directory. In the RAML file, use the filename as the schema key name (not relative pathname).

Comment by David Crossley [ 19/Jun/17 ]

After that "careful" testing (working on both macOS and Ubuntu Linux) when i merged the branch (pull/87), it then failed on our Jenkins server:

java.lang.IllegalArgumentException: Unrecognised URI, can't resolve this: file:/tmp/raml-util/schemas/parameters.schema

The reason that it seemed to work locally on Linux, was that there was a left-over old "/tmp/raml-util" from a previous failed build.

Sorry for the noise.

Comment by David Crossley [ 19/Jun/17 ]

After a tweak similar to what was suggested above, the branch now does work:

Use relative pathname as schema key name in RAML. 
This is the relative pathname that is used as the $ref in the parent schema file.

raml-module-builder/domain-models-api-interfaces/raml/schemas/
raml-module-builder/domain-models-api-interfaces/raml/raml-util/schemas/

in raml-module-builder/domain-models-api-interfaces/raml/schemas/job.schema
  "$ref" : "../raml-util/schemas/parameters.schema"

in raml-module-builder/domain-models-api-interfaces/raml/jobs.raml
  - ../raml-util/schemas/parameters.schema: !include raml-util/schemas/parameters.schema

During the compile phase, on Linux that is created at "/tmp/raml-util".

Comment by David Crossley [ 21/Jun/17 ]

It works in that specific situation. During the compilation phase, either raml-for-jax-rs (or perhaps jsonschema2pojo) creates a temporary directory and copies the schema files to it. If a schema has $ref with relative path to another schema (e.g. ../raml-util/schema/...) with only one set of dot-dots then they are copied to "/tmp/raml-util" and the build is successful. Of course more than one set, and it will not.

(Perhaps raml-for-jax-rs version 2 has better handling. FOLIO-523 Closed )

As our raml-util now has schema files which refer to other files in that area, then compiling say mod-users-bl puts those schema in "/tmp/mod-login" and friends.

A successful build does remove its temp space. However those extraneous directories do remain.

That seems fragile, e.g. perhaps that could get confused by a build process of another module.

Comment by David Crossley [ 21/Jun/17 ]

A potential solution could be a shared "schema-util" inside each "schemas" directory.

However that will be tortuous in the raml-util/schemas/mod-* directories.

Comment by David Crossley [ 22/Jun/17 ]

Ah, of course, there is a solution. For any schema file that needs to refer to another schema file that is outside of its space, then move that parent schema file up to a higher level directory. This does work for both of our current complex ones, i.e. raml-module-builder own ramls and for mod-users-bl.

Comment by David Crossley [ 30/Jun/17 ]

Added documentation to RMB to explain the best location of the shared raml-util and using $ref in schema.

Followed that for the raml-module-builder's own docs and for mod-configuration.
(Now moving on to the others.)

For the shared ones in raml-util mod-users-bl (not yet done) its schema files need to move up one level, i.e.

cd folio-org/raml
mv schemas/mod-users-bl/*.json schemas
and adjust paths in ramls/schema.

(Kurt Nordstrom, shale99)

Comment by David Crossley [ 17/Jul/17 ]

There is pull/22 in folio-org/raml and i have success building mod-users-bl with that.

Comment by David Crossley [ 09/Aug/17 ]

Closed.

All necessary mod-* repositories now have "raml-util" inside the "ramls" directory. See RMB README.

(As a separate issue, the user-related mod-* repositories had their RAMLs moved to raml-util proper.)

Comment by David Crossley [ 22/Aug/17 ]

There is an example for mod-circulation-storage needing to link to shared schema in raml-util.
Do:

cd ramls; grep \$ref *.json

and see the corresponding RAML files.

Generated at Thu Feb 08 23:07:17 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.