[FOLIO-2060] Add "marccat" to snapshot and testing Created: 03/Jun/19  Updated: 03/Jun/20  Resolved: 07/Oct/19

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: David Crossley Assignee: David Crossley
Resolution: Done Votes: 0
Labels: marccat, platform-backlog
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: Text File SearchTest.txt     File logs-k8s-20190928.tar.gz    
Issue links:
Blocks
blocks UXPROD-1595 Connect SRS and MARCcat - Folijet wor... Closed
blocks UXPROD-2218 Connect SRS and MARCcat - AtCult work... Closed
blocks MODCAT-126 MARCcat Bib "suppress from discovery"... Closed
blocks MODSOURMAN-96 Update SRS 999 field to include MARCc... Closed
blocks MODSOURMAN-147 Update SRS MARC Bib when changes are ... Closed
blocks MODSOURMAN-148 Update MARCcat MARC Bib when changes ... Closed
blocks MODSOURMAN-154 SRS "suppress from display" flag shou... Closed
blocks MODSOURMAN-155 SRS MARC Bib "suppress from display" ... Closed
blocks UXPROD-1576 Connect MARCcat and SRS for MARC Auth... Closed
blocks UXPROD-1596 SRS-MARCcat integration for Holdings ... Closed
is blocked by MODCAT-111 Fix version number in ModuleDescripto... Closed
is blocked by MODCAT-118 The pathPattern in interface handlers... Closed
is blocked by MODCAT-119 ModuleDescriptor needs "configuration... Closed
is blocked by MODCAT-120 Improve explanation of the storage ar... Closed
is blocked by MODCAT-121 The mod-marccat needs to implement th... Closed
Defines
is defined by MODCAT-118 The pathPattern in interface handlers... Closed
Relates
relates to MODCAT-128 Use loadSample and loadReference to l... Closed
Sprint: CP: ready for planning, CP: sprint 73
Story Points: 13
Development Team: Core: Platform

 Description   

Add "ui-marccat" to platform-complete snapshot branch

https://github.com/folio-org/ui-marccat which depends on https://github.com/folio-org/mod-marccat



 Comments   
Comment by David Crossley [ 12/Jun/19 ]

It is too close to the Q2-2019 release to attempt to get these modules integrated (first in snapshot/testing and then in master) and there is additional information and fixes required. So will follow up after that.

Comment by Wayne Schneider [ 12/Jun/19 ]

From Mirko Fonzo:

1. MARCcat development team uses a database built by porting to Postgres a quite complex real production database (from data content perspective).
This allows to perform tests of many different and heterogeneous conditions. Database initialization has not yet introduced in MARCcat, but as you mentioned about other storage modules, it should be part of tenant initialization.

2. Yes, database connection information is stored in mod-configuration.
MARCcat needs 4 keys, below there is an example (at this moment password is not encrypted)

{"module": "MARCCAT", "configName": "datasource", "code": "url", "description": "Target JDBC URL", "default": true, "enabled": true, "value": "jdbc:postgresql://folio.frontside.atcult.it:5433/folio_marccat_test1" }
{"module": "MARCCAT", "configName": "datasource", "code": "port", "description": "Database listening port", "default": true, "enabled": true, "value": 5433 }
{"module": "MARCCAT", "configName": "datasource", "code": "user", "description": "Database username", "default": true, "enabled": true, "value": "marccat"}
{"module": "MARCCAT", "configName": "datasource", "code": "password", "description": "Database password", "default": true, "enabled": true, "value": "***"}

In mod-configuration there are also other configuration parameters used by MARCcat for different tasks. If needed we can provide the current list of settings.

3. Yes, each tenant requires its own database.
There are pros and cons having separate or a unique database per tenant. From our point of view there are more pros than cons having a dedicated database for each tenant. Please let me know if you see potential issues about our solution.

Comment by Wayne Schneider [ 12/Jun/19 ]

I think your solution is fine. It's just a question of understanding how to provision the module. If I understand correctly, to enable mod-marccat for a tenant in a FOLIO system, you must:

1. Register mod-marccat with Okapi (and optionally deploy it and register the URL with the Okapi discovery service) – this is the same as for any module
2. Provision a Postgres database for mod-marccat for the tenant, using some variation on the scripts in https://github.com/folio-org/mod-marccat/tree/master/database-setup
3. Register the configuration keys for the tenant with mod-configuration
4. Enable mod-marccat for the tenant (and optionally deploy at this time, if using Okapi orchestration)

Does this sound right?

Comment by Mirko Fonzo [ 13/Jun/19 ]

Yes, it sounds right.

Comment by Ann-Marie Breaux (Inactive) [ 01/Jul/19 ]

David Crossley it looks like all the blockers for this ticket to add MARCcat to snapshot and testing have been completed. Is there anything else holding it back? If so, could you or Tiziana Possemato add those as additional blockers? If not, then can this get unblocked and move forward?

Comment by David Crossley [ 02/Jul/19 ]

Added the ticket MODCAT-121 Closed to implement the Tenant API.

I gather that their development team is currently in the process of ensuring that mod-marccat operates with their local Okapi test instance to handle tenant activation and tenant separation.

Comment by Ann-Marie Breaux (Inactive) [ 02/Jul/19 ]

Thanks David Crossley So Tiziana Possemato looks like one more blocking story ( MODCAT-121 Closed ) for AtCult to take care of.

Comment by Wayne Schneider [ 23/Jul/19 ]

Given the current state of MODCAT-121 Closed , I think we have these options:

1. Include with defaults (hosted db) in the folio-testing reference environment, with no UI. Do not include in the testing-backend Vagrant box.

2. Include with defaults in both folio-testing and folio-snapshot. To include in folio-snapshot, we would either need to also include the UI in platform-complete or specifically call it out as an extra module (otherwise it won't be in the dependency graph). Again, do not include in Vagrant boxes.

Neither of these options require much work from the DevOps team, other than build configuration and testing. I think use of a hosted db makes it crazy to include in Q3.1 – anyone who stands up a system hits the same hosted DB? Clearly that doesn't work.

3. Attempt to adapt database and configuration scripts in CI (as an Ansible role), so that the database can be provisioned locally. This would allow us to include it in the Vagrant boxes. With appropriate updates to the folio-install instructions, we might also be able to consider including it in Q3.1. This option requires the most work from the DevOps team.

4. Wait until the "next version" of mod-marccat, when the module may behave in a manner more consistent with other storage modules.

Comment by Ann-Marie Breaux (Inactive) [ 31/Jul/19 ]

Wayne Schneider Now that MODCAT-121 Closed is closed, does that unblock this ticket?

Comment by Jakub Skoczen [ 01/Aug/19 ]

Added to the current sprint and asked David Crossley to give another look.

Comment by Ann-Marie Breaux (Inactive) [ 01/Aug/19 ]

Jakub Skoczen Thank you!!

Comment by David Crossley [ 05/Aug/19 ]

I did attempt a local install, but no success. See notes at MODCAT-121 Closed .

Comment by David Crossley [ 05/Aug/19 ]

Also i note that the "mod-marccat/sample-data" directory does have a data file, but it is empty.

Is the intention to add some basic sample-data to that repository?

Or if not, is there a way to obtain sample data for local testing.

Comment by Ann-Marie Breaux (Inactive) [ 05/Aug/19 ]

Hi David Crossley - good question.

Annalisa Di Sabato Tiziana Possemato has there been discussion of having sample records in MARCcat? If not, might it be possible to populate some in? Once we connect MARCcat and SRS, it'll probably be best to come up with a batch of records that can be loaded and populate into SRS, then into MARCcat and Inventory. Thanks, A-M

Comment by Annalisa Di Sabato [ 05/Aug/19 ]

Hi David Crossley and Ann-Marie Breaux,
at the moment the database is empty, but it will be possible to catalogue new records starting from 1. If you are able to start mod-marccat and indicate me in which FOLIO environment was deploied, I can create some records for you. This is the best and quick solution for all.

Let me know please,

Annalisa

Comment by David Crossley [ 06/Aug/19 ]

Note that the sample-data and reference-data are optional (see initial discussion about that in MODCAT-121 Closed ).

Glancing at the mod-marccat code it seems that it does not yet handle the loadSample and loadReference tenant parameters. So those empty directories might just be a placeholder for the future.

If the MARCcat project wants to have that, then i suggest that someone create a separate related Jira ticket, and that we focus this one on the initial installation without such data.

I was mainly wondering how we will know that this initial installation of mod-marccat is operational. The MARCcat developers probably have some other way to verify.

Comment by Tiziana Possemato [ 06/Aug/19 ]

Hi Ann-Marie Breaux, we added the https://folio-org.atlassian.net/browse/MODCAT-128 to trace the requirement to have a data sample in MARCcat, with references. Let us know if it is correct and if we need to link it to other issues. Thanks

Comment by Ann-Marie Breaux (Inactive) [ 10/Sep/19 ]

Tiziana Possemato Is there any hope of getting MARCcat into the 3.2/Daisy release? It will be pretty awful not to have any of MARCcat showing in Daisy, since there is a lot of manual testing and Inventory/SRS integration that needs to happen in Q4.

cc: Jakub Skoczen Mark Veksler Hkaplanian lew235 Cate Boerema

Comment by Ann-Marie Breaux (Inactive) [ 20/Sep/19 ]

Hi all

David Crossley Wayne Schneider Mirko Fonzo

It looks like there was progress on the last blocker: MODCAT-121 Closed . Is there hope that we can get MARCcat into the hosted ref envs by 25 Sept?

cc: Tiziana Possemato Jakub Skoczen Mark Veksler Hkaplanian lew235 Cate Boerema

Comment by Matteo Pascucci [ 23/Sep/19 ]

Hi, Ann-Marie Breaux we haven't been able to test the ui on our environments yet. Has an ui-marccat 1.8.5 installation been tested on folio host systems? if yes, what kind of problems were there? I'm following this guide https://github.com/folio-org/stripes-cli/blob/master/doc/user-guide.md#managing-ui-modules to test the operation of the ui and calls to okapi occur correctly. Am I missing something?

Comment by David Crossley [ 24/Sep/19 ]

We do not add it to the reference environments (snapshot and testing) until it can at least be installed locally.

So i did a basic test here today:
Used the vagrant VM "folio/snapshot-core".
Added mod-marccat as explained previously.
Followed that stripes-cli document to "enable" ui-marccat.
Now it is listed at 'localhost:3000/settings/about'

Comment by David Crossley [ 24/Sep/19 ]

So i will now proceed with this FOLIO-2060 Closed .

Comment by David Crossley [ 24/Sep/19 ]

The backend mod-marccat is now available in the "folio-snapshot" and "folio-testing"
https://dev.folio.org/guides/automation/#reference-environments

The next step is to add ui-marccat to platform-complete "snapshot" branch
(However there are troubles there today. See Slack #stripes channel.)

Comment by Ann-Marie Breaux (Inactive) [ 24/Sep/19 ]

Hi Matteo Pascucci I don't know if UI has ever been tested on the hosted environments. If there was no MOD until now, my guess is that UI has not been there previously either. CC to Tiziana Possemato to confirm.

Comment by David Crossley [ 24/Sep/19 ]

Please see my previous comments.

Comment by David Crossley [ 25/Sep/19 ]

A problem was encountered with mod-marccat today. So we needed to temporarily remove it from "snapshot" and "testing" environments. See MODCAT-121 Closed for the detail.

Comment by David Crossley [ 26/Sep/19 ]

carmentrazza Matteo Pascucci can you please make an actual release of mod-marcat. That will get it onto our new kubernetes cluster.

We will then make a PR for ui-marccat (being marked as "do not merge"). Please let us make that PR, which will utilise our new branch-preview system.

Comment by carmentrazza [ 27/Sep/19 ]

David Crossley We have prepared the release v2.0.0 for mod-marccat.
We have the same problem on Jenkins: the link "Build Now", in the left-hand panel, is not actived. I suppose that @Cult team isn't still authorized to do it. So, please, can you excute this command and then, when we'll have the authorization , we'll do it ourselves.
Thanks.

Comment by David Crossley [ 27/Sep/19 ]

Yep, i pressed that button some time ago. (We will re-investigate the permissions again later.)

So John Malconian it should be ready now for the next phase.

Comment by John Malconian [ 27/Sep/19 ]

I opened PR-425 in platform-core (https://github.com/folio-org/platform-core/pull/425) which is a test FOLIO build of the q3.2 release combined with the latest releases of ui-marccat (v1.8.5) and mod-marccat (2.0.0). I actually needed to run this build twice because I neglected to enable ui-marccat in the stripes config in the in the first build. There is a notable difference between the first and second FOLIO builds. (use standard diku tenant admin credentials)

The first build (marccat is not enabled) is here: http://platform-core-425-1.aws.indexdata.com
The second build (marccat is enabled) is here: http://platform-core-425-2.aws.indexdata.com

The second build with marccat enabled is unusable. Selecting any FOLIO app results in an error thrown by the stripes bundle (webpack).

Please refresh the page.
The following occurred resulting in the current page becoming unstable.
ERROR:state is not defined

    in t
    in main
    in eo
    in t
    in div
    in $n
    in n
    in WithModules(n)
    in t
    in withRouter(WithModules(n))
    in t
    in i
    in t
    in HotKeys
    in t
    in y
    in t
    in WithStripes(t)
    in t
    in InjectIntl(t)
    in t
    in IntlProvider
    in t
    in t
    in t
    in WithModules(t)
    in Connect(WithModules(t))
    in t

We've seen this error before in the past. See https://folio-org.atlassian.net/browse/STRIPES-633. In that previous issue we resolved the error by pinning react-hot-loader to version 4.8.8 in the platform's package.json. I have verified that this version is still pinned.

jenkins@eedfe49a4a5f:~/platform-core$ yarn list --pattern react-hot-loader
yarn list v1.16.0
└─ react-hot-loader@4.8.8
Done in 1.35s.

So I'm not sure what is the cause of this error this time around. I'm adding Zak Burke here in case he may have any additional input.

Comment by John Malconian [ 27/Sep/19 ]

One notable difference I see with ui-marccat 1.8.5 is that "@folio/stripes" is not defined as a peerDependency in its package.json

Comment by David Crossley [ 30/Sep/19 ]

It would be useful to compare with the package.json of other front-end modules (e.g. ui-notes, ui-checkin).

As noted elsewhere, you can use "stripes-cli" to verify ui-marccat against various backend systems (e.g. your local VM testing-backend with mod-marccat installed, or the branch-preview backend that John enabled).

Comment by David Crossley [ 30/Sep/19 ]

I did run the test set of curl queries again. The basic "search" query does work against a recent VM testing-backend with recent mod-marccat. However it fails against that branch-preview backend. I have obtained and attached the okapi and module logs (logs-k8s-20190928.tar.gz)

Comment by carmentrazza [ 30/Sep/19 ]

I have tested mod-marccat-2.1.0-SNAPSHOT.340” in my local VM testing-backend and executed the search curl (SearchTest.txt):
curl -D - -w '\n' \
-H "X-Okapi-Tenant: diku" \
"http://localhost:9130/marccat/search?lang=eng&q=NA+Manzoni&ml=170"
The response is OK:
HTTP/1.1 200 OK
I don't have the error reported in the log.
The error in the log depends by the configuration module on the get of the mod-marccat entries.

Comment by Matteo Pascucci [ 30/Sep/19 ]

John Malconian I compared the package.json with that of other modules and I fixed the dependencies that gave problems. I merged these changes in the master branch, can you test directly with this branch or is it better if I make a new release?

Comment by David Crossley [ 30/Sep/19 ]

Carmen, as i said earlier and in slack. It is the environment that John has provided above (see comment which gives the error. And see in the log file that i attached from that environment.

Yes as i said, the VM testing-backend does work for me too.

Comment by John Malconian [ 30/Sep/19 ]

Matteo Pascucci David Crossley I've have a new PR and preview build available. This PR replaces ui-marccat v1.8.5 with the current master branch of https://github.com/folio-org/ui-marccat

Github PR: https://github.com/folio-org/platform-core/pull/427
FOLIO Instance: http://platform-core-427-2.aws.indexdata.com

I'm not seeing the errors that were reported in the previous PR. If the marccat application behaves as expected, then you should proceed with a new ui-marccat release. If additional fixes to ui-marccat need to be made and are committed to the master branch, those changes can be previewed by running another build for this PR in Jenkins here:

https://jenkins-aws.indexdata.com/job/folio-org/job/platform-core/view/change-requests/job/PR-427/

Comment by David Crossley [ 01/Oct/19 ]

We determined a method to rebuild that PR-427 without there being a change yet to ui-marccat. I did the rebuild a few moments ago. This has picked up your new mod-marccat-2.1.5 release.

The FOLIO Instance is now: http://platform-core-427-5.aws.indexdata.com

And hooray, the curl Search query now does not fail on the backend.

It was strange that there was a difference between this system and the VM system. Carmen, do you know why that was so?

Comment by Ann-Marie Breaux (Inactive) [ 01/Oct/19 ]

OMG!! Hooray!!

A couple questions from a non-developer for David Crossley and Matteo Pascucci

1. How does MARCcat move from http://platform-core-427-5.aws.indexdata.com to folio-testing? Are there some checks or other manual work to complete first, or will it automatically get picked up?

2. Is the UI that's showing on http://platform-core-427-5.aws.indexdata.com the most recent UI, or is there a new release coming for ui-cat as well?

Comment by Matteo Pascucci [ 01/Oct/19 ]

Yes Ann-Marie Breaux, the UI you see on http://platform-core-427-5.aws.indexdata.com is the most recent UI for marccat

Comment by John Malconian [ 01/Oct/19 ]

Just to provide further clarification, the version of ui-marccat that you see in http://platform-core-427-5.aws.indexdata.com is from the master branch of ui-marccat. It is unreleased at this time.

Comment by Ann-Marie Breaux (Inactive) [ 01/Oct/19 ]

Matteo Pascucci John Malconian

Thank you for the clarifications on the UI portion - so what needs to happen to get MARCcat showing on folio-testing? More manual work? A release of ui-marccat? Or will folio-testing magically inherit it all from http://platform-core-427-5.aws.indexdata.com?

Comment by John Malconian [ 01/Oct/19 ]

I believe all that needs to happen is for David to enable the marccat modules in folio-testing/snapshot. In the meantime we'll keep this preview system available until that is complete.

Comment by Ann-Marie Breaux (Inactive) [ 01/Oct/19 ]

Exciting!!

Comment by Ann-Marie Breaux (Inactive) [ 01/Oct/19 ]

Tiziana Possemato We could have big news to share at MM SIG on Thursday, right??

Comment by Tiziana Possemato [ 01/Oct/19 ]

Finally, I think yes, we have!

Comment by David Crossley [ 02/Oct/19 ]

"Steady as she goes"

Since the last re-build (427-5) of this branch-preview environment, there were changes to the master snapshot of ui-marccat.
So i ran it again. Sorry to say that there are problems on the front-end.

The FOLIO Instance is now: http://platform-core-427-8.aws.indexdata.com

Log in and view the MARCcat interface: "ERROR:d.HEADERS is not a function. (In 'd.HEADERS(s)', 'd.HEADERS' is an instance of Object)"

Comment by Ann-Marie Breaux (Inactive) [ 02/Oct/19 ]

OK - sad face... Tiziana Possemato, if your UI folks need help with triage, it might be good to set up a call with Ryan Berger to look at the situation with them.

Comment by Tiziana Possemato [ 02/Oct/19 ]

Thanks Ann-Marie Breaux, I ask Matteo Pascucci to contact Ryan Berger as soon as possible. We hope to solve the problem soon.

Comment by David Crossley [ 02/Oct/19 ]

Remember that we can still use the previous installations, e.g. the "427-5" working one.

They are listed here:
Github PR: https://github.com/folio-org/platform-core/pull/427

Comment by Matteo Pascucci [ 02/Oct/19 ]

We made some changes and merged these changes into the master branch for both mod-marccat and ui-marccat.
For ui-marccat could you re-run the "branch-preview" PR 427?
Instead for mod-marccat we have to make a new release?
Thanks

Comment by David Crossley [ 02/Oct/19 ]

Yes, mod-marccat needs an actual "release" (Carmen is doing that now). The ui-marccat operates from the head of master branch.

Comment by David Crossley [ 02/Oct/19 ]

Instance available at http://platform-core-427-12.aws.indexdata.com

When the new mod-marcat is out, then we can run the branch-preview build again.

Comment by David Crossley [ 02/Oct/19 ]

Instance available at http://platform-core-427-17.aws.indexdata.com

On my tomorrow i intend to now add the mod-marccat to the daily "folio-testing" and "folio-snapshot" environments, and ui-marccat to platform-complete.

Comment by David Crossley [ 03/Oct/19 ]

The backend mod-marccat is now added to snapshot and testing.
Follow the links to each environment: https://dev.folio.org/guides/automation/#reference-environments
e.g. https://folio-snapshot.aws.indexdata.com/settings/about

Carmen, your verification script can now communicate with the "folio-snapshot-okapi" and "folio-testing-okapi" as explained there.

Matteo, your local development stripes-cli can use those for its "stripes serve --okapi ..."

My next step is adding the frontend ui-marccat to platform-complete.

Comment by David Crossley [ 03/Oct/19 ]

Welcome to MARCcat

The UI is in today's "folio-testing" instance. After the next daily run it will be in "folio-snapshot" too.

Comment by Matteo Pascucci [ 03/Oct/19 ]

Ok but I wanted to warn you that ui-marccat has an application problem that I'm trying to solve now and as soon as I finish release the changes on the master branch so if everything works as it should I do a release.

Comment by Ann-Marie Breaux (Inactive) [ 03/Oct/19 ]

Hi David Crossley Wait a minute - I thought Jakub Skoczen wanted to wait to promote it to folio-testing/snapshot until it was more stable. Did something change between yesterday and today?

cc: Tiziana Possemato and Charlotte Whitt

Comment by Tiziana Possemato [ 03/Oct/19 ]

Hi David Crossley, I totally agree with Ann-Marie Breaux: please, wait until the application will be stable and really working. I just ask you to wait to receive news from Matteo Pascucci (understanding that due to our different time zones your working day is quite at the end), just to anticipate the new release, in case we need.
Thank you for your great work and huge patience.

Comment by David Crossley [ 03/Oct/19 ]

Through the recent use of the special PR-preview environment, the modules were in a state that they can be added to the snapshot and testing, not to any release environment. This is just for normal daily development operations. Many modules are not in a fully working state.

Refer to the discussion above where we said what the next steps are. Also yesterday i indicated what was going to happen next.

I was not aware of any further holdups.

It will take some time to remove it from the daily infrastructure. I am out for the evening, so will follow on at our Team standup meeting later. (Ian Hardy can probably at least get the frontend out within a few hours.)

Comment by Ann-Marie Breaux (Inactive) [ 03/Oct/19 ]

Is Jakub Skoczen around? He was the one wanting us to be especially cautious. Plus if you look at SonarCloud, there's no automated testing yet, so neither the MOD or UI module meets the definition of done.

I'm really sorry about the confusion. I made a mistake thinking @jakub had discussed it with everyone yesterday. I apologize for that.

Comment by Ann-Marie Breaux (Inactive) [ 03/Oct/19 ]

OK - after a few conversations today, we're fine keeping it in folio-testing and snapshot. So David Crossley no need to reverse course. We're going to try to manage the community expectations while the usability issues get sorted out.

Thank you again for all of your help sorting this out over the past few weeks!

cc: Tiziana Possemato Jakub Skoczen

Comment by Matteo Pascucci [ 03/Oct/19 ]

I have just merged on the master branch a working version of ui-marccat after testing it also with mod-marccat inside the folio-testing VM. Thank you very much to all for your help and patience

Comment by David Crossley [ 07/Oct/19 ]

The folio-snapshot and folio-testing builds completed successfully with today's modules. So closing this ticket as done.

Generated at Thu Feb 08 23:17:55 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.