[RMB-30] Trouble with a schema file referring to another schema in raml-util Created: 31/May/17 Updated: 05/Sep/17 Resolved: 09/Aug/17 |
|
| Status: | Closed |
| Project: | RAML Module Builder |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | P3 |
| Reporter: | David Crossley | Assignee: | David Crossley |
| Resolution: | Done | Votes: | 0 |
| Labels: | raml, sprint19 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||||||||||
| Sprint: | |||||||||||||||||||||||||
| Description |
|
Using relative links between schema for RAML does work okay for both sets of tools (
However, when attempting to refer to a schema file in the shared "raml-util" then there can be a problem. (Note that some early comments below would need some adjustment. |
| Comments |
| Comment by David Crossley [ 31/May/17 ] |
|
So for example, for the raml-module-builder repository, if it was like this then it is processed okay: raml-module-builder/ramls/schemas/ raml-module-builder/raml-util/schemas/ in raml-module-builder/ramls/schemas/job.schema "$ref" : "../../raml-util/schemas/parameters.schema" in raml-module-builder/ramls/jobs.raml - ../../raml-util/schemas/parameters.schema: !include ../raml-util/schemas/parameters.schema However with the current location (raml-module-builder/domain-models-api-interfaces/src/main/resources/raml/schemas) or any other place, then raml-for-jax-rs in RMB cannot handle it. |
| Comment by Kurt Nordstrom [ 01/Jun/17 ] |
|
Isn't this more specifically an RMB issue? When I check these files with raml-cop, they validate fine. |
| Comment by David Crossley [ 01/Jun/17 ] |
|
The raml-module-builder repo is the only one at the moment that uses $ref in a schema to refer to another schema in the "raml-util" area. When we try to use the relative pathname as the schema key, then raml-cop is fine because its underlying parser does not utilise the schema key name (just the "!include" path). The raml-for-jax-rs (underlying the RMB) then chokes. However all is okay in other repos where the raml-util directory is at the same directory tree level as its "ramls" directory. So yes it is "only" the RMB that has a problem if the directories are not like that. The complicated issues with raml-for-jax-rs are linked in
|
| Comment by shale99 [ 05/Jun/17 ] |
|
hi David Crossley |
| Comment by David Crossley [ 07/Jun/17 ] |
|
I enhanced the note in raml-module-builder/README.md to advise about the placement of those directories. Our other RMB-using modules do already have such an arrangement. |
| Comment by David Crossley [ 07/Jun/17 ] |
|
Regarding actually "fixing", there are various similar open issues at raml-for-jax-rs. It might assist to add another bug report for this specific case. |
| Comment by shale99 [ 07/Jun/17 ] |
thanks! |
| Comment by David Crossley [ 07/Jun/17 ] |
|
For "raml-module-builder" i have done this: git mv domain-models-api-interfaces/src/main/resources/raml domain-models-api-interfaces/raml git mv raml-util domain-models-api-interfaces/raml-util The RMB does build with success. Using that snapshot in mod-inventory-storage runs its tests with success. Now ready in branch folio-573-rmb-30-filename-schema-key shale99: Are you happy with that directory arrangement? |
| Comment by shale99 [ 07/Jun/17 ] |
|
yea, that works for me |
| Comment by David Crossley [ 07/Jun/17 ] |
|
Beaut. I will go ahead. |
| Comment by David Crossley [ 07/Jun/17 ] |
|
Drat. It was fine here on macOS with its "tmp" filesystem. However following the merge of pull/85 it failed on the build on the server. [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java (generate_interfaces) on project domain-models-api-interfaces: An exception occured while executing the Java class. null: InvocationTargetException: Directory '/tmp/1496818545170-0/../../raml-util/schemas' could not be created I reverted the PR merge. |
| Comment by David Crossley [ 09/Jun/17 ] |
|
Beware too many dotdots. Putting the "raml-util" git submodule directly inside the "raml" directory does work for the "raml-module-builder" repo, i.e. at raml-module-builder/domain-models-api-interfaces/src/main/resources/raml/raml-util/ See branch folio-573-rmb-30-filename-schema-key-2 The build is successful on both macOS and on Ubuntu Linux. |
| Comment by David Crossley [ 16/Jun/17 ] |
|
I did various testing for the location of the "raml-util" directory, for RMB's own RAMLs and for mod-configuration. For some it did work on macOS but not on Linux. (See attached results.txt) Summary: If there are schema files that use $ref to another schema in raml-util, then that "raml-util" is best placed inside the "ramls" directory. In the RAML file, use the filename as the schema key name (not relative pathname). |
| Comment by David Crossley [ 19/Jun/17 ] |
|
After that "careful" testing (working on both macOS and Ubuntu Linux) when i merged the branch (pull/87), it then failed on our Jenkins server:
java.lang.IllegalArgumentException: Unrecognised URI, can't resolve this: file:/tmp/raml-util/schemas/parameters.schema
The reason that it seemed to work locally on Linux, was that there was a left-over old "/tmp/raml-util" from a previous failed build. Sorry for the noise. |
| Comment by David Crossley [ 19/Jun/17 ] |
|
After a tweak similar to what was suggested above, the branch now does work: Use relative pathname as schema key name in RAML. 
This is the relative pathname that is used as the $ref in the parent schema file. raml-module-builder/domain-models-api-interfaces/raml/schemas/ raml-module-builder/domain-models-api-interfaces/raml/raml-util/schemas/ in raml-module-builder/domain-models-api-interfaces/raml/schemas/job.schema "$ref" : "../raml-util/schemas/parameters.schema" in raml-module-builder/domain-models-api-interfaces/raml/jobs.raml - ../raml-util/schemas/parameters.schema: !include raml-util/schemas/parameters.schema During the compile phase, on Linux that is created at "/tmp/raml-util". |
| Comment by David Crossley [ 21/Jun/17 ] |
|
It works in that specific situation. During the compilation phase, either raml-for-jax-rs (or perhaps jsonschema2pojo) creates a temporary directory and copies the schema files to it. If a schema has $ref with relative path to another schema (e.g. ../raml-util/schema/...) with only one set of dot-dots then they are copied to "/tmp/raml-util" and the build is successful. Of course more than one set, and it will not. (Perhaps raml-for-jax-rs version 2 has better handling.
As our raml-util now has schema files which refer to other files in that area, then compiling say mod-users-bl puts those schema in "/tmp/mod-login" and friends. A successful build does remove its temp space. However those extraneous directories do remain. That seems fragile, e.g. perhaps that could get confused by a build process of another module. |
| Comment by David Crossley [ 21/Jun/17 ] |
|
A potential solution could be a shared "schema-util" inside each "schemas" directory. However that will be tortuous in the raml-util/schemas/mod-* directories. |
| Comment by David Crossley [ 22/Jun/17 ] |
|
Ah, of course, there is a solution. For any schema file that needs to refer to another schema file that is outside of its space, then move that parent schema file up to a higher level directory. This does work for both of our current complex ones, i.e. raml-module-builder own ramls and for mod-users-bl. |
| Comment by David Crossley [ 30/Jun/17 ] |
|
Added documentation to RMB to explain the best location of the shared raml-util and using $ref in schema. Followed that for the raml-module-builder's own docs and for mod-configuration. For the shared ones in raml-util mod-users-bl (not yet done) its schema files need to move up one level, i.e. cd folio-org/raml mv schemas/mod-users-bl/*.json schemas and adjust paths in ramls/schema. |
| Comment by David Crossley [ 17/Jul/17 ] |
|
There is pull/22 in folio-org/raml and i have success building mod-users-bl with that. |
| Comment by David Crossley [ 09/Aug/17 ] |
|
Closed. All necessary mod-* repositories now have "raml-util" inside the "ramls" directory. See RMB README. (As a separate issue, the user-related mod-* repositories had their RAMLs moved to raml-util proper.) |
| Comment by David Crossley [ 22/Aug/17 ] |
|
There is an example for mod-circulation-storage needing to link to shared schema in raml-util. cd ramls; grep \$ref *.json and see the corresponding RAML files. |