[FOLIO-3014] Add edge-dematic module to testing env/testing and snapshot VMs Created: 14/Feb/21  Updated: 10/Mar/21  Resolved: 10/Mar/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Viachaslau Khandramai (Inactive) Assignee: Wayne Schneider
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: Text File edge-dematic.log    
Issue links:
Blocks
blocks EDGDEMATIC-1 Dematic EMS integration Closed
blocks EDGDEMATIC-2 Dematic Staging Director integration Closed
blocks EDGDEMATIC-8 Handle dematic status IN Closed
is blocked by EDGDEMATIC-9 Parsing API token information Closed
is blocked by MODRS-21 mod-remote-storage tenant initializat... Closed
Relates
relates to EDGDEMATIC-5 Project Setup: edge-dematic Closed
relates to FOLIO-2892 Create edge-dematic GitHub repository Closed
Sprint: DevOps Sprint 109, DevOps Sprint 108
Development Team: FOLIO DevOps

 Description   

A new back-end edge module edge-dematic should be added to the folio-testing reference environment as well as for the Vagrant VM boxes for folio-testing and folio-snapshot for testing and demo purposes. This module depends on interfaces from mod-remote-storage.



 Comments   
Comment by Viachaslau Khandramai (Inactive) [ 18/Feb/21 ]

Hello Jakub Skoczen , John Malconian, I have created this story four days ago according instructions https://dev.folio.org/guides/install-backend-module/ #Configure folio-ansible, but there is no any activity on it up to now. Maybe I missed something from procedure? If no, could someone from DevOps assist us with it because we need to have this story completed for Remote Storage feature verification.

Comment by Jakub Skoczen [ 18/Feb/21 ]

Viachaslau Khandramai It's in our backlog, waiting for devops capacity.

Comment by Viachaslau Khandramai (Inactive) [ 18/Feb/21 ]

Jakub Skoczen,

thank you!

Comment by Victoria_Smelova [ 22/Feb/21 ]

Jakub Skoczen, is it possible to speed up the work on this item, please?
It is blocking the team to close some R1 related stories before feature freeze.

Comment by Wayne Schneider [ 23/Feb/21 ]

Is this an edge module like other edge modules in the project (e.g edge-rtac)? Or is this just another backend module that is proxied by Okapi? I note that it does not include the edge-common library – is access to the endpoints not controlled by an API key?

The repository looks like any other backend module. I note that it has a module descriptor with endpoints to proxy through Okapi, and that it defines the tenant interface, so apparently there is tenant initialization as well. It appears that it also requires database access (at least, both the run.sh and the launch descriptor use the database connection environment variables).

If this is simply a backend module to be proxied by Okapi, I highly recommend that it be renamed (perhaps to "mod-dematic"). If it is an edge module (that is, an Okapi client), then it is really not clear how to deploy it, nor how to manage access control.

I am blocking this issue until there is more clarity from the development team.

Comment by Viachaslau Khandramai (Inactive) [ 23/Feb/21 ]

Hi Wayne Schneider,

we are expecting that this should be edge module like edge-tac, edge-orders, etc. Now we are working on authorization mechanism a la edge-common https://folio-org.atlassian.net/browse/EDGDEMATIC-10. So it will be nice to have this module enabled to have possibility for verification this approach and all the functionality that we have already implemented.

 

Actually, yes, this module has a DB but for internal purposes only (for creation system-user) and Tenant API. But we are going to refactor this mechanism when this module will be available as edge module.

 

CC Mikhail Fokanov

 

Thanks,

Slava

 

Comment by Wayne Schneider [ 23/Feb/21 ]

If I set this up as an edge module, how is the system user configured? Edge modules are generally not "enabled" by Okapi, at least as they are currently architected.

Or does this need to be set up both as an Okapi module (registered in the Okapi module descriptor registry and with the Okapi discovery service, and enabled for a tenant so that the tenant API is invoked by Okapi) and as an endpoint outside of the Okapi proxy server and its authn/z system?

Comment by Viachaslau Khandramai (Inactive) [ 24/Feb/21 ]

Hi Wayne Schneider,

could you please enable this module only as EDGE (like edge-oai-pmh, etc.). Mentioned by you Tenant API will be deleted and logic will be changed in scope of EDGDEMATIC-9 (it is in progress now) to correspond edge modules specification https://github.com/folio-org/edge-common. We don't need to have it as okapi module. Let me know any blockers or reasons why we can't do this now. The only one case that module is in development now can't be blocker from my perspective. We'd like to have this module available on the envs asap to be able to verify and close stories that are in review/in progress at the moment as they are critical for R1 release and any delays in verification might result in our team missing the deadlines.

 

 

Mikhail Fokanov please confirm or decline. 

 

Thanks

Comment by Mikhail Fokanov [ 24/Feb/21 ]

Viachaslau Khandramai I confirm.

Comment by Wayne Schneider [ 24/Feb/21 ]

Viachaslau Khandramai in order to deploy this as an "edge" module, we need the following things changed:

  • The module descriptor needs to have all provided interfaces removed, including the _tenant interface
  • The requirement for database storage must be removed (preferably) or database setup needs to be documented, since Okapi will not invoke the _tenant interface to tell the module to set up the database.
Comment by Jakub Skoczen [ 25/Feb/21 ]

Mikhail Fokanov Viachaslau Khandramai Guys, please see Wayne's requests above. I have an additional question: are external systems communicating with the edge dematic module via non-standard APIs? If not, it would be simpler to turn it into a regular FOLIO module. If yes, it might be worth consider splitting the module into an edge- and mod- modules, with the edge- module handling non-standard communication and mod- module handling persistency. A similar approach has been used in edge-connexion and mod-copycat for single record import.

Comment by Viachaslau Khandramai (Inactive) [ 25/Feb/21 ]

Jakub Skoczen Wayne Schneider,

thank you for your help.

Looks like we'll have to wait until EDGDEMATIC-9 Closed is done and all the required changes are available. We have prioritized that story and we'll notify as soon as it is ready.

Jakub Skoczen, yes, edge-dematic communicating with non-standard API. So now we are working on adaptation of spring-based approach a la standard FOLIO edge modules.

Thanks,

Slava

Comment by Viachaslau Khandramai (Inactive) [ 01/Mar/21 ]

Hi Wayne Schneider ,

as far as we completed and merged EDGDEMATIC-9 Closed we are unblocked now for testing edge-dematic. Could you please review and make some kind dry run just to be sure that module successfully enabled and doesn't destroy our ref envs? As you know this is new spring-based edge module so let me know in case of any deployment issues (please, attach docker logs if smth goes wrong).

CC Jakub Skoczen Stephanie Buck Former user

Thank you!

Comment by Victoria_Smelova [ 03/Mar/21 ]

Hi Wayne Schneider, Jakub Skoczen,

As this module is required for R1 feature and we have a feature freeze in two days, it would be really great if you can prioritize this task in the current sprint of your team, so that we can get it available this week and Firebird can verify and close their stories before feature freeze.

Thanks for your assistance.

CC Stephanie Buck

Comment by Jakub Skoczen [ 03/Mar/21 ]

Wayne Schneider have you had a chance to verify this since Viachaslau Khandramai updated the code?

Comment by Wayne Schneider [ 03/Mar/21 ]

Unfortunately, since edge-dematic depends on mod-remote-storage, this ticket is blocked until mod-remote-storage can be deployed again ( MODRS-21 Closed ) Stephanie Buck Viachaslau Khandramai

Comment by Viachaslau Khandramai (Inactive) [ 04/Mar/21 ]

Hi Wayne Schneider,

as far as MODRS-21 Closed doesn't block this issue now (we have reverted some changes), could you please proceed with this?

 

Thanks,

Slava

Comment by Wayne Schneider [ 04/Mar/21 ]

edge-dematic will be deployed in the next round of reference builds for folio-testing, folio-snapshot, and folio-snapshot load. The module launches and responds to requests (on port 8000 of the system it is installed on)....however, it doesn't appear to work. When I try to hit the API with what should be a valid API key, I get a 403 response with the body Cannot get system connection properties for: diku (diku is the name of the institutional user embedded in the API key).

The edge-dematic container log is attached: edge-dematic.log

This may be speculation, but hopefully it is helpful...When we launch edge modules that use the edge-common library, we pass command line arguments for configuration: -Dokapi_url=http://10.36.1.139:9130 -Dsecure_store_props=/mnt/edge-dematic-ephemeral.properties. Looking at the container java command line, it doesn't look like these arguments are getting passed along by the run.sh script – assuming they are supported by edge-dematic at all. If they are not supported, please let us know how to configure. Thanks!

Comment by Viachaslau Khandramai (Inactive) [ 04/Mar/21 ]

Hi Wayne Schneider,

thank you very much! Module working! Error in the logs is a result of using fake user (test_user/password) usage from default ephemeral.properties.

Regarding to configuration: -Dokapi_url = http: //10.36.1.139: 9130 -Dsecure_store_props = / mnt / edge-dematic-ephemeral.properties. Edge-dematic should support these command-line arguments. At least I verified this locally. I expect that you pass correct okapi_url and secure_store_props path from your side as a part of APP_OPTS= (like this is done for other edge modules). So, could you pass these arguments for edge-dematic.

 

Thanks,

Slava

Comment by Viachaslau Khandramai (Inactive) [ 05/Mar/21 ]

Hi Wayne Schneider, is it possible to start edge-dematic with the following VM configuration:

exec java -Dokapi_url=http://10.36.1.208:9130 -Dsecure_store_props=/Users/viachaslau_khandramai/Documents/test-ephemeral.properties -cp . -jar /Users/viachaslau_khandramai/FOLIO/edge-dematic/target/edge-dematic-1.0.0-SNAPSHOT.jar

I successfully tested application on my local environment with these command-line arguments. I have opened PR for changes in edge-dematic. Could you please assist in reviewing of this PR? And, probably, you can provide useful remarks how to provide support of $JAVA_OPTIONS on our ens.

UPD:
As I saw from edge-tai-pmh it started from the following command:

exec java -XX:+ExitOnOutOfMemoryError -cp . -jar /usr/verticles/edge-oai-pmh-fat.jar -Dokapi_url=http://10.36.1.208:9130 -Dsecure_store_props=/mnt/edge-oai-pmh-ephemeral.properties

It will be nice if edge-dematic starts with the same command.

Thanks,
Slava

 

Comment by Wayne Schneider [ 05/Mar/21 ]

Hi, Slava. I believe with the changes to the Dockerfile, you should get the same command-line as above. I will test.

And, probably, you can provide useful remarks how to provide support of $JAVA_OPTIONS on our ens.

The base container image you are using should support the JAVA_OPTIONS environment variable. Do you need options passed to the container? We can set environment variables if needed.

Comment by Wayne Schneider [ 05/Mar/21 ]

In our testing, we found that edge-dematic requires the -D options to be before the -jar option, rather than processing them as command arguments after -jar. But otherwise the new Dockerfile seems to work well.

We should be able to accommodate this by putting the -D options in the JAVA_OPTIONS environment variable. Once the PR is merged, we can build a test environment.

Comment by Viachaslau Khandramai (Inactive) [ 06/Mar/21 ]

Hi Wayne Schneider,

now it works with VM options (before -jar):

-Dokapi_url=http://localhost:2222 -Dsecure_store_props=/Users/viachaslau_khandramai/Documents/test-ephemeral.properties

as well as with Program arguments (after -jar):

--Dokapi_url=http://localhost:2222 --Dsecure_store_props=/Users/viachaslau_khandramai/Documents/test-ephemeral.properties

or

--okapi_url=http://localhost:2222 --secure_store_props=/Users/viachaslau_khandramai/Documents/test-ephemeral.properties

The only one difference - two dashes in case of Program arguments. 

Thanks,

Slava

Comment by Wayne Schneider [ 09/Mar/21 ]

Thanks, Slava. That does not work the same as the other edge modules, however (which accept the single dash -D options as program arguments), so it still requires special configuration. But I've updated the build script to allow for more flexibility. In my testing on folio-snapshot-test, I get the following response to the request https://folio-snapshot-test.dev.folio.org:8000/asrService/asr/lookupNewAsrItems/12345?apikey=eyJzIjoidDJFZTU0a2s5bCIsInQiOiJkaWt1IiwidSI6ImRpa3UifQ==:

<asrItems/>

So it seems like at least the module is responding as expected. The Docker container log shows that the institutional user is logged in and able to complete requests.

Just a note – you may want to tune down the default logging from DEBUG to INFO.

Putting this issue in review. After the overnight reference environment builds, the module should be available in folio-testing and folio-snapshot.

Comment by Wayne Schneider [ 10/Mar/21 ]

Based on comments in EDGDEMATIC-9 Closed , it seems like this is working well enough for UAT. Closing this issue.

Generated at Thu Feb 08 23:24:58 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.