[FOLIO-1519] Automatic loading of sample and reference data Created: 19/Sep/18  Updated: 18/Aug/20  Resolved: 05/Apr/19

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Umbrella Priority: P2
Reporter: Jakub Skoczen Assignee: Adam Dickmeiss
Resolution: Done Votes: 0
Labels: ci, core, core-spike, platform-backlog, q1-2019, sprint47, sprint48, sprint49, sprint50, sprint51, sprint52, sprint53, sprint54
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks FOLIO-1577 Automated builds for FOLIO 'release' Closed
blocks UXPROD-1827 CI-integrated continuous deployment (... Closed
blocks FOLIO-2118 CI-integrated continuous deployment (... Closed
is blocked by FOLIO-1681 folio-ansible: Use tenant init to loa... Closed
is blocked by OKAPI-682 Tenant init parameters Closed
is blocked by OKAPI-688 tenant ref problem: own module not av... Closed
Relates
relates to FOLIO-1862 SPIKE: consider loadSample/loadRefere... Open
relates to CIRCSTORE-103 use loadSample to load sample data Closed
relates to CIRCSTORE-104 use loadReference to load reference data Closed
relates to MODINVSTOR-216 load reference data as part of the te... Closed
relates to MODINVSTOR-264 use loadSample to load sample data Closed
relates to MODTAG-14 use loadReference to load reference data Closed
relates to MODTAG-15 use loadSample to load sample data Closed
relates to MODUSERS-103 use loadSample/loadReference to load ... Closed
relates to MODORDSTOR-48 Rework how sample data is loaded Closed
relates to UXPROD-1815 support for upgrading schemas without... Closed
relates to UXPROD-2120 support for upgrading schemas without... Closed
relates to FOLIO-1548 SPIKE: a lighter-weight folio/testing... Closed
relates to FOLIO-1726 enable sample MARC record data loadin... Closed
relates to UXPROD-1827 CI-integrated continuous deployment (... Closed
relates to FOLIO-2118 CI-integrated continuous deployment (... Closed
relates to MODVEND-81 Load sample data via tenant API Closed
relates to OKAPI-655 Module scripting (e.g to support data... Closed
relates to RMB-296 tenant init for loading reference data Closed
relates to UXPROD-753 live db upgrades (db migration) Closed
relates to UXPROD-1508 Populate reference data as part of in... Closed
relates to UXPROD-2612 MARC Authority record - Load sample data Closed
Sprint:
Development Team: Core: Platform

 Description   

We have discussed the following extensions to the tenant and Okapi install calls:

  • a new version of the tenant init call that allows specifying the initialisation properties. This will allow us to control if and what kind of data is loaded during init (reference on/off, sample on/off, etc), By default only reference data will be loaded in the first version of the RMB implementation.
  • extension to the Okapi's install call to provide the global values for the initialisation properties. Those values would then be passed on to the individual modules during the init call.

In the first phase we will address the implementation of this new functionality in mod-inventory-storage.

Open questions:

  • should Okapi store the init props and resend them during module upgrade?
  • should we have an initial set of default init props that can be specified before any install call?

Reference data

The name of the parameter that controls reference data loading is loadReference with true|false (both in the tenantParams string query param and in the parameters kv array in the moduleInfo payload)

By default the value of loadReference is false

Sample data

The name of the parameter that controls sample data loading is loadSample with true|false (both in the tenantParams string query param and in the parameters kv array in the moduleInfo payload)

By default the value of loadSample is false



 Comments   
Comment by Jakub Skoczen [ 25/Sep/18 ]

Guys, Adam has begun work on OKAPI-655 Closed which is an attempt to provide capabilities to perform early initialisation of modules. Please provide feedback this week as the implementation will start asap.

Comment by Jakub Skoczen [ 11/Oct/18 ]

Adam Dickmeiss Wayne Schneider Niels Erik Nielsen Guys, do we need to get together on a call to try to push this forward? We need to come up with an approach suitable (especially) for loading inventory data. The way I see this we can expect the sample/reference data to be already processed to fit the modules schema (e.g I don't think we need to be able to load raw marc records for Inventory). Comments and thoughts?

Comment by Jakub Skoczen [ 30/Oct/18 ]

Adam Dickmeiss can you see if you can flesh out a propsal for this? Besides OKAPI-655 Closed we will need out-of-the-box support in RMB (I would propose bundling sample and reference data inside the jar and loading it from the classpath during start-up, but I am open for other suggestions).

Comment by Adam Dickmeiss [ 06/Nov/18 ]

After "local" discussions this morning, I think we are about to drop the scripting idea as outlined in OKAPI-655 Closed . Why?

1. modules deployed outside Okapi are problematic. We only specify a remote URL and that in no way can run a script. We could of course, haave something remote URI "script".. eg ag url to run a script remotely .. and then alternatively an Okapi local deployment way.. But that gives two ways and should module implements .

2. if a module already offers HTTP, why not use that anyway for loading data?

Using the existing tenant web service is better.. Why?

1. It works with local and remote deployed service.

2. A simple standard way of preparing a tenant to use a module. It also informs about versions (upgrade, downgrade etc..) as well as providing all necessary X-Okapi stuff to do the work.. Such as token etc..

3. Less "new" development .. which is always desirable.

If we decide on usingt the Tenant init service... we should consider if we need to extend it. SHould we allow it to be controlled if/when reference data should be included? Note that it is also possible to create a NEW module that depends on the BARE module with the only purpose of loading reference data (eg use the module ID to determine what to do)

Comment by Marc Johnson [ 06/Nov/18 ]

Adam Dickmeiss I like the idea of using the Tenant API for this purpose (given it has been decided that we are bundling the records with the module itself) and I think this is a good place to start.

I think it is valuable to be able to decide whether reference, and especially sample, records are loaded during this initialisation. The pre-loading of sample records in mod-configuration caused signifiant confusion for me when trying to determine why the tests produced certain results.

I imagine that many organisations (but maybe not all) may want the reference records in all of their environments. Do organisations want sample records for production? I think that many would not, or would need to delete them prior to loading their actual records.

I'm aware that initially, all modules / tenants may need to use the same policy, as the Tenant API request is made during module enabling/activation (Adam, I believe you raised this in the meeting last week).

I assume that these records will be loaded on the first initialisation of a module (and hence need to always be in line with that versions schema). What are we envisaging will happen on an upgrade (or is this part of the more general upgrade topic, as I don't believe we talked about how to handle record migration)?

(It might be a orthogonal topic, I think we may also need to consider the impact of the Tenant API doing more activities (and hence get slower) on the general activation process, and especially upgrades)

Comment by Wayne Schneider [ 04/Jan/19 ]

With the PR on mod-inventory-storage for MODINVSTOR-216 Closed , we have a sample implementation. See also FOLIO-1681 Closed .

Comment by Wayne Schneider [ 17/Jan/19 ]

For posterity, when working with the MODINVSTOR-216 Closed branch, I found:

  • If using Okapi to initialize the module for the tenant, the _tenant interface must be v1.2 or above
  • To pass the tenant parameters from the /_/proxy/tenants/<tenant>/install endpoint, use the tenantParameters query parameter: /_/proxy/tenants/<tenant>/install?tenantParameters=loadReference%3Dtrue

This is all documented in the Okapi guide. I also added documentation to mod-inventory-storage (since the support for the loadReference parameter is implemented there, not in RMB or elsewhere).

Comment by Adam Dickmeiss [ 18/Jan/19 ]

Thx. I think eventually we'll make some of the "private" methods in mod-inventory-storage part of an RMB util. But let's see what other modules have reference data (I see MODORDSTOR-48 Closed )

Comment by Jakub Skoczen [ 22/Jan/19 ]

Adam Dickmeiss We had a call today about loading "sample" data through this mechanism in the mod-source-record-storage and mod-order-storage. Wayne Schneider and I assume that the mechanism is the same, we just need new property called "loadSample" set in the CI and the rest belongs to the module itself. Correct?

Comment by Wayne Schneider [ 22/Jan/19 ]

Adam Dickmeiss would it make sense to extend the TenantModuleDescriptor schema to allow passing tenant parameters on that level? Right now, you can pass the parameters in query parameters in a POST to the /_/proxy/tenant/<id>/install endpoint. This would pass the parameters to all modules – what if a user only wanted to load reference/sample data for only some modules?

Comment by Craig McNally [ 22/Jan/19 ]

Wayne Schneider I guess I wasn't paying close enough attention. TBH that's the way I thought this was supposed to work, but now that I take another look I see that's not the case. I personally prefer this over the query argument approach...

e.g.

[ {
  "id" : "mod-users-14.2.1-SNAPSHOT.299",
  "action" : "enable"
}, {
  "id" : "permissions-module-4.0.4",
  "action" : "enable"
}, {
  "id" : "mod-login-3.1.1-SNAPSHOT.42",
  "action" : "enable"
}, {
  "id" : "mod-orders-3.0.0-SNAPSHOT.37",
  "action" : "enable",
  "loadSample": true,
}, {
  "id" : "mod-inventory-15.0.0",
  "action" : "enable",
  "loadReference": true
} ]
Comment by Jakub Skoczen [ 05/Apr/19 ]

Closing as all work has been completed.

Generated at Thu Feb 08 23:13:56 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.