[FOLIO-3014] Add edge-dematic module to testing env/testing and snapshot VMs Created: 14/Feb/21 Updated: 10/Mar/21 Resolved: 10/Mar/21 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | P2 |
| Reporter: | Viachaslau Khandramai (Inactive) | Assignee: | Wayne Schneider |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||||||||||
| Issue links: |
|
||||||||||||||||||||||||||||||||||||
| Sprint: | DevOps Sprint 109, DevOps Sprint 108 | ||||||||||||||||||||||||||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||||||||||||||||||||||||||
| Description |
|
A new back-end edge module edge-dematic should be added to the folio-testing reference environment as well as for the Vagrant VM boxes for folio-testing and folio-snapshot for testing and demo purposes. This module depends on interfaces from mod-remote-storage. |
| Comments |
| Comment by Viachaslau Khandramai (Inactive) [ 18/Feb/21 ] |
|
Hello Jakub Skoczen , John Malconian, I have created this story four days ago according instructions https://dev.folio.org/guides/install-backend-module/ #Configure folio-ansible, but there is no any activity on it up to now. Maybe I missed something from procedure? If no, could someone from DevOps assist us with it because we need to have this story completed for Remote Storage feature verification. |
| Comment by Jakub Skoczen [ 18/Feb/21 ] |
|
Viachaslau Khandramai It's in our backlog, waiting for devops capacity. |
| Comment by Viachaslau Khandramai (Inactive) [ 18/Feb/21 ] |
|
thank you! |
| Comment by Victoria_Smelova [ 22/Feb/21 ] |
|
Jakub Skoczen, is it possible to speed up the work on this item, please? |
| Comment by Wayne Schneider [ 23/Feb/21 ] |
|
Is this an edge module like other edge modules in the project (e.g edge-rtac)? Or is this just another backend module that is proxied by Okapi? I note that it does not include the edge-common library – is access to the endpoints not controlled by an API key? The repository looks like any other backend module. I note that it has a module descriptor with endpoints to proxy through Okapi, and that it defines the tenant interface, so apparently there is tenant initialization as well. It appears that it also requires database access (at least, both the run.sh and the launch descriptor use the database connection environment variables). If this is simply a backend module to be proxied by Okapi, I highly recommend that it be renamed (perhaps to "mod-dematic"). If it is an edge module (that is, an Okapi client), then it is really not clear how to deploy it, nor how to manage access control. I am blocking this issue until there is more clarity from the development team. |
| Comment by Viachaslau Khandramai (Inactive) [ 23/Feb/21 ] |
|
Hi Wayne Schneider, we are expecting that this should be edge module like edge-tac, edge-orders, etc. Now we are working on authorization mechanism a la edge-common https://folio-org.atlassian.net/browse/EDGDEMATIC-10. So it will be nice to have this module enabled to have possibility for verification this approach and all the functionality that we have already implemented.
Actually, yes, this module has a DB but for internal purposes only (for creation system-user) and Tenant API. But we are going to refactor this mechanism when this module will be available as edge module.
Thanks, Slava
|
| Comment by Wayne Schneider [ 23/Feb/21 ] |
|
If I set this up as an edge module, how is the system user configured? Edge modules are generally not "enabled" by Okapi, at least as they are currently architected. Or does this need to be set up both as an Okapi module (registered in the Okapi module descriptor registry and with the Okapi discovery service, and enabled for a tenant so that the tenant API is invoked by Okapi) and as an endpoint outside of the Okapi proxy server and its authn/z system? |
| Comment by Viachaslau Khandramai (Inactive) [ 24/Feb/21 ] |
|
Hi Wayne Schneider, could you please enable this module only as EDGE (like edge-oai-pmh, etc.). Mentioned by you Tenant API will be deleted and logic will be changed in scope of EDGDEMATIC-9 (it is in progress now) to correspond edge modules specification https://github.com/folio-org/edge-common. We don't need to have it as okapi module. Let me know any blockers or reasons why we can't do this now. The only one case that module is in development now can't be blocker from my perspective. We'd like to have this module available on the envs asap to be able to verify and close stories that are in review/in progress at the moment as they are critical for R1 release and any delays in verification might result in our team missing the deadlines.
Mikhail Fokanov please confirm or decline.
Thanks |
| Comment by Mikhail Fokanov [ 24/Feb/21 ] |
|
Viachaslau Khandramai I confirm. |
| Comment by Wayne Schneider [ 24/Feb/21 ] |
|
Viachaslau Khandramai in order to deploy this as an "edge" module, we need the following things changed:
|
| Comment by Jakub Skoczen [ 25/Feb/21 ] |
|
Mikhail Fokanov Viachaslau Khandramai Guys, please see Wayne's requests above. I have an additional question: are external systems communicating with the edge dematic module via non-standard APIs? If not, it would be simpler to turn it into a regular FOLIO module. If yes, it might be worth consider splitting the module into an edge- and mod- modules, with the edge- module handling non-standard communication and mod- module handling persistency. A similar approach has been used in edge-connexion and mod-copycat for single record import. |
| Comment by Viachaslau Khandramai (Inactive) [ 25/Feb/21 ] |
|
Jakub Skoczen Wayne Schneider, thank you for your help. Looks like we'll have to wait until
Jakub Skoczen, yes, edge-dematic communicating with non-standard API. So now we are working on adaptation of spring-based approach a la standard FOLIO edge modules. Thanks, Slava |
| Comment by Viachaslau Khandramai (Inactive) [ 01/Mar/21 ] |
|
Hi Wayne Schneider , as far as we completed and merged
CC Jakub Skoczen Stephanie Buck Former user Thank you! |
| Comment by Victoria_Smelova [ 03/Mar/21 ] |
|
Hi Wayne Schneider, Jakub Skoczen, As this module is required for R1 feature and we have a feature freeze in two days, it would be really great if you can prioritize this task in the current sprint of your team, so that we can get it available this week and Firebird can verify and close their stories before feature freeze. Thanks for your assistance. |
| Comment by Jakub Skoczen [ 03/Mar/21 ] |
|
Wayne Schneider have you had a chance to verify this since Viachaslau Khandramai updated the code? |
| Comment by Wayne Schneider [ 03/Mar/21 ] |
|
Unfortunately, since edge-dematic depends on mod-remote-storage, this ticket is blocked until mod-remote-storage can be deployed again (
|
| Comment by Viachaslau Khandramai (Inactive) [ 04/Mar/21 ] |
|
Hi Wayne Schneider, as far as
Thanks, Slava |
| Comment by Wayne Schneider [ 04/Mar/21 ] |
|
edge-dematic will be deployed in the next round of reference builds for folio-testing, folio-snapshot, and folio-snapshot load. The module launches and responds to requests (on port 8000 of the system it is installed on)....however, it doesn't appear to work. When I try to hit the API with what should be a valid API key, I get a 403 response with the body Cannot get system connection properties for: diku (diku is the name of the institutional user embedded in the API key). The edge-dematic container log is attached: edge-dematic.log This may be speculation, but hopefully it is helpful...When we launch edge modules that use the edge-common library, we pass command line arguments for configuration: -Dokapi_url=http://10.36.1.139:9130 -Dsecure_store_props=/mnt/edge-dematic-ephemeral.properties. Looking at the container java command line, it doesn't look like these arguments are getting passed along by the run.sh script – assuming they are supported by edge-dematic at all. If they are not supported, please let us know how to configure. Thanks! |
| Comment by Viachaslau Khandramai (Inactive) [ 04/Mar/21 ] |
|
Hi Wayne Schneider, thank you very much! Module working! Error in the logs is a result of using fake user (test_user/password) usage from default ephemeral.properties. Regarding to configuration: -Dokapi_url = http: //10.36.1.139: 9130 -Dsecure_store_props = / mnt / edge-dematic-ephemeral.properties. Edge-dematic should support these command-line arguments. At least I verified this locally. I expect that you pass correct okapi_url and secure_store_props path from your side as a part of APP_OPTS= (like this is done for other edge modules). So, could you pass these arguments for edge-dematic.
Thanks, Slava |
| Comment by Viachaslau Khandramai (Inactive) [ 05/Mar/21 ] |
|
Hi Wayne Schneider, is it possible to start edge-dematic with the following VM configuration:
exec java -Dokapi_url=http://10.36.1.208:9130 -Dsecure_store_props=/Users/viachaslau_khandramai/Documents/test-ephemeral.properties -cp . -jar /Users/viachaslau_khandramai/FOLIO/edge-dematic/target/edge-dematic-1.0.0-SNAPSHOT.jar
I successfully tested application on my local environment with these command-line arguments. I have opened PR for changes in edge-dematic. Could you please assist in reviewing of this PR? And, probably, you can provide useful remarks how to provide support of $JAVA_OPTIONS on our ens. UPD:
exec java -XX:+ExitOnOutOfMemoryError -cp . -jar /usr/verticles/edge-oai-pmh-fat.jar -Dokapi_url=http://10.36.1.208:9130 -Dsecure_store_props=/mnt/edge-oai-pmh-ephemeral.properties
It will be nice if edge-dematic starts with the same command. Thanks,
|
| Comment by Wayne Schneider [ 05/Mar/21 ] |
|
Hi, Slava. I believe with the changes to the Dockerfile, you should get the same command-line as above. I will test.
The base container image you are using should support the JAVA_OPTIONS environment variable. Do you need options passed to the container? We can set environment variables if needed. |
| Comment by Wayne Schneider [ 05/Mar/21 ] |
|
In our testing, we found that edge-dematic requires the -D options to be before the -jar option, rather than processing them as command arguments after -jar. But otherwise the new Dockerfile seems to work well. We should be able to accommodate this by putting the -D options in the JAVA_OPTIONS environment variable. Once the PR is merged, we can build a test environment. |
| Comment by Viachaslau Khandramai (Inactive) [ 06/Mar/21 ] |
|
Hi Wayne Schneider, now it works with VM options (before -jar):
-Dokapi_url=http://localhost:2222 -Dsecure_store_props=/Users/viachaslau_khandramai/Documents/test-ephemeral.properties
as well as with Program arguments (after -jar):
--Dokapi_url=http://localhost:2222 --Dsecure_store_props=/Users/viachaslau_khandramai/Documents/test-ephemeral.properties
or
--okapi_url=http://localhost:2222 --secure_store_props=/Users/viachaslau_khandramai/Documents/test-ephemeral.properties
The only one difference - two dashes in case of Program arguments. Thanks, Slava |
| Comment by Wayne Schneider [ 09/Mar/21 ] |
|
Thanks, Slava. That does not work the same as the other edge modules, however (which accept the single dash -D options as program arguments), so it still requires special configuration. But I've updated the build script to allow for more flexibility. In my testing on folio-snapshot-test, I get the following response to the request https://folio-snapshot-test.dev.folio.org:8000/asrService/asr/lookupNewAsrItems/12345?apikey=eyJzIjoidDJFZTU0a2s5bCIsInQiOiJkaWt1IiwidSI6ImRpa3UifQ==: <asrItems/> So it seems like at least the module is responding as expected. The Docker container log shows that the institutional user is logged in and able to complete requests. Just a note – you may want to tune down the default logging from DEBUG to INFO. Putting this issue in review. After the overnight reference environment builds, the module should be available in folio-testing and folio-snapshot. |
| Comment by Wayne Schneider [ 10/Mar/21 ] |
|
Based on comments in
|