[FOLIO-1490] Enable various tools for validation and editing of RAML files and examples Created: 12/Dec/16  Updated: 12/Nov/18  Resolved: 12/Sep/18

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: New Feature Priority: P3
Reporter: David Crossley Assignee: Unassigned
Resolution: Done Votes: 0
Labels: lint, raml
Remaining Estimate: Not Specified
Time Spent: 3 weeks, 2 days, 2 hours, 45 minutes
Original estimate: Not Specified

Issue links:
Relates
relates to FOLIO-1213 Using an "id" property in JSON Schema... Closed
relates to RMB-56 Dereference $ref in RAML schema Closed
relates to FOLIO-573 Use a consistent method for RAML to r... Closed
Sprint:

 Description   

While doing DMOD-88, some issues arose with the RAML and associated JSON files.
These were difficult to determine, and additional warning messages from some tools made that harder.

Also a pre-processing workaround is currently required, to replace the $ref which links parent and child JSON schema files.



 Comments   
Comment by David Crossley [ 12/Dec/16 ]

Add .eslintrc to relevant modules to declare the plugins eslint-plugin-raml and eslint-plugin-json. (Not automating via build for the first phase.)

Comment by David Crossley [ 12/Dec/16 ]

Some of the error and warning messages are obscure. Will list some here with their fix.

Comment by David Crossley [ 12/Dec/16 ]

eslint mod-metadata/doc/api/catalogue.raml
...
18:17 warning Example does not conform to schema: Content is not valid according to schema: Expected type object but found type array object,array bad-raml

Interpretation:
The "18:17" refers to the line in types/collection.raml "example: <<exampleCollection>>"
In doc/api/catalogue.raml the "schemaItem" refers to a collection items schema, whereas "schemaCollection" refers to a single item schema. Should be the other way around.

Fixed in 6a685b8

Comment by Marc Johnson [ 12/Dec/16 ]

Well that is embarrassing, I don't know how those got switched around

The knowledge base core and catalogue core modules will be deprecated pretty soon. How does es-lint determine which RAML / JSON.Schema files are checked? Is it run on every build?

Comment by David Crossley [ 13/Dec/16 ]

It is not automated to run for every build (as noted above). I should have explained better how to use it manually.

Install the plugins (eslint-plugin-raml and eslint-plugin-json).

The schema and examples and other RAML files are linked from each top-most raml file. So for example do 'eslint mod-metadata/doc/api/catalogue.raml' – the eslint-plugin-raml does find and process all of its pieces.

The eslint-plugin-raml will also notify when a linked JSON file has issues. Then use the eslint-plugin-json to report the detail, e.g. 'eslint schema/instance.json' (a real example is upcoming).

Comment by David Crossley [ 13/Dec/16 ]

Now some examples from mod-users. These occur in other modules too, but mod-users is the smallest so easier to analyse.

Comment by David Crossley [ 13/Dec/16 ]

eslint mod-users/ramls/users.raml
...
35:13 warning Can not parse JSON: Unexpected token } in JSON at position 426 bad-raml
44:17 warning Can not parse JSON: Unexpected token } in JSON at position 426 bad-raml
13:15 warning Can not parse JSON: Unexpected token } in JSON at position 426 bad-raml
56:13 warning Can not parse JSON: Unexpected token } in JSON at position 426 bad-raml

Interpretation:
The single issue is a trailing comma in JSON at examples/user.sample

The same warning is reported four times. The first two: "35:13 and 44:17" are where "exampleItem" is referenced in raml-util/rtypes/collection.raml file. The second two: "13:15 and 56:13" are where "exampleItem" is referenced in raml-util/rtypes/item-collection.raml file.

Fixed in 8d315b2

To see the detail of the JSON issue: The eslint-plugin-json wants files with *.json extension.
So 'cp examples/user.sample examples/user.json; eslint examples/user.json'

Comment by David Crossley [ 13/Dec/16 ]

eslint mod-users/ramls/users.raml
...
5:11 warning string is expected bad-raml

Interpretation:
Tracked this to raml-util/traits/searchable.raml which expects the default of string for the "example" but is a JSON object.

Fix: Not yet applied.

5,6c5,6
<           example: |
<             <<example>>
---
>           example: <<example>>
>             type: "object"
Comment by David Crossley [ 13/Dec/16 ]

eslint mod-users/ramls/users.raml
...
18:17 warning Example does not conform to schema: Content is not valid according to schema: Reference could not be resolved: .../mod-users/ramls/schemas/userdata# .../mod-users/ramls/schemas/userdata# bad-raml

Interpretation:
Cannot resolve the reference in schemas/userdataCollection.json to the "userdata" schema.
Perhaps we could be using JSON Schema "id", but we are not yet.

Fix: Not yet applied.
Refer to the actual filename:

10c10
<                 "$ref" : "userdata"
---
>                 "$ref" : "userdata.json"
Comment by Marc Johnson [ 02/Jan/17 ]

Hi David,

How is eslint run for the modules?

Is it intended to be part of the build process during the work on this issue?

Comment by David Crossley [ 03/Jan/17 ]

See above for temporary instructions for how to run eslint manually.
See Comment A.

And note the need to apply a temporary workaround for "Reference could not be resolved".
See Comment B.
Also need to undo that after running eslint. That workaround does not sit well with the raml-module-builder. Might need proper JSON Schema "id", which i am not familiar with.

No, not yet intending to add this eslint to the build process. At the moment just using it manually to tidy up all our RAML files.

I have also been exploring using "raml-tester", which is used in the "okapi/okapi-core" build process.

Comment by Marc Johnson [ 03/Jan/17 ]

Thanks David,

I thought I'd been through the comments, sorry.

I thought I'd have a quick go at scripting this, mod-metadata now has a lint.sh script to run eslint (and it replaces the references so the checks pass currently).

Comment by Adam Dickmeiss [ 03/Jan/17 ]

The RAML parser of RMB does not like the latest commit ( FOLIO-1490 Closed ) and says:

java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Invalid RAML definition:
	ERROR mapping values are not allowed here (line 5, col 16)
	at org.raml.jaxrs.codegen.core.AbstractGenerator.run(AbstractGenerator.java:792)
	at org.raml.jaxrs.codegen.core.GeneratorProxy.run(GeneratorProxy.java:40)
	at org.folio.rest.tools.GenerateRunner.main(GenerateRunner.java)
	... 6 more

I suggest we revert it. Besides.. The query is supposedly going to be string anyway - because it's eventually CQL.

Comment by David Crossley [ 04/Jan/17 ]

Reverted in commit 4731f00. (Oops – a lesson in testing.)

So users of eslint-raml, ignore the warning described at comment-12923 above.

Comment by David Crossley [ 11/Jan/17 ]

Added a Python script to folio-infrastructure/run-raml-tools.

Different tools are used to process a set of RAML files for each module repository. Each tool has certain capabilities. None seem to report all issues. So it is useful to run a number of them, and review their output.

As explained above in this FOLIO-1490 Closed , some schema files use "$ref" references to other schema. This does work for some RAML processing tools, but not for others. So we temporarily replace its value with the actual filename.

Currently applying "eslint-raml", "ramlev", "ramlfications".

The outputs are proving useful for tidying our RAML files, and enhancing the api docs (DMOD-88).

Comment by Marc Johnson [ 12/Jan/17 ]

David David Crossley, would it be useful to use that Python script (using a submodule) in mod-metadata?

Comment by David Crossley [ 16/Jan/17 ]

I had only intended it to be used manually, external to any particular repository, and then review the output. It does install heaps of stuff in NPM local "./node_modules".

Also currently it is in a private repo, as i do not yet know where to put such.

Comment by David Crossley [ 08/Feb/17 ]

Improved the run-raml-tools script:

Now also apply "raml-tester" via its local proxy.
Now also generate local api docs using "raml2html" and "raml-fleece".

Comment by David Crossley [ 09/Mar/17 ]

As explained above, some processing tools handle schema references differently.
The issue is wider than just eslint-raml and raml2html. This umbrella issue might help to keep in touch:
raml2html/issues/327 "Schemas and $ref".

Comment by David Crossley [ 16/Mar/17 ]

Improved the run-raml-tools script:

Added 'raml-cop' which keeps well up-to-date. It is now my favourite tool.

Comment by David Crossley [ 12/Sep/18 ]

Now we have consistent $ref, and the dereferencer in RMB.

Generated at Thu Feb 08 23:13:43 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.