[FOLIO-3659] Create solution that will update MARC authorities when mapping rules have changed Created: 07/Dec/22  Updated: 01/Feb/23  Resolved: 13/Jan/23

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Pavlo Smahin Assignee: Shans Kaluhin
Resolution: Done Votes: 0
Labels: back-end, epam-spitfire
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File FOLIO-3659_ss_00(Record_after).png     PNG File FOLIO-3659_ss_00(Record_before).png     PNG File FOLIO-3659_ss_01(config).png     PNG File FOLIO-3659_ss_02(script).png     PNG File FOLIO-3659_ss_03(Data_import).png     PNG File image-2023-01-04-08-58-36-338.png    
Issue links:
Cloners
is cloned by FOLIO-3681 Extend solution that will update Inst... Closed
Defines
defines UXPROD-3911 MVP: Provide a way to update MARC aut... Closed
Relates
relates to UXPROD-3969 Improve solution that will refresh In... Open
relates to PERF-429 Standalone application used to update... Closed
Sprint: Spitfire Sprint 156, Spitfire Sprint 155
Story Points: 5
Development Team: Spitfire
Release: Orchid (R1 2023)
RCA Group: TBD

 Description   

Approach
Create a java app that has to do the update work:
1. Create data-import job profile for MARC-to-MARC matching by 999ff$s subfield and update authority record. Job/Match/Action/Mapping profiles will be hidden

  • Job Profile: Release Upgrade - Migrate MARC authority records 
  • Action Profile: Release Upgrade - Migrate MARC authority records 
  • Match Profile: Release Upgrade - Migrate MARC authority records 
  • Mapping Profile: Release Upgrade - Migrate MARC authority records **

2. Load authority records by pages of 50k (should be configurable) from SRS (GET /source-storage/records?recordType=MARC_AUTHORITY&state=ACTUAL&offset=<P>&limit=<N>
3. Prepare file or JSON payload (README)
4. Initialize data-import job
5. Wait until the data-import job finished (GET /change-manager/jobExecutions/<id> and check status of job)
6. Load the next page and repeat 3-6 until there are no authority records left.
7. Delete the job profile created in 1st step
8. Logging should exist to indicate N of batch that is in progress now.

Validate that a user can view Job status

  • job status is shown on the Data Import Logs UI
  • For each record update, user can view SRS/MOD-Inventory-Storage output

Documentation 

  • Instructions must be provided to Hosting providers/System administrators for using the standalone application 
    • Must consider that some libraries have already upgraded to Morning Glory weeks/months before this implementation. 
    • Include a note that this should be run off-hours
  • Release notes for Morning Glory and Nolana should be updated and include link to Instructions 

Testing - MORE details to discuss

  • Need a story for PTF
  • Need an environment(s) to test 
    • Upgrade Lotus > Morning Glory   
      • Work with FSE? 
    • Upgrade Morning Glory > Nolana
      • Work with FSE?   

 



 Comments   
Comment by Valery_Pilko [ 21/Dec/22 ]

Verified on Snapshot-2 environment:
1) The "MARC Authority" app has 110 records.
I picked one record to check if it will update after script execution:

2) I used the following config:

3) The script successfully executed:

4) 5 Jobs for updating of "MARC Authority" records is displayed in "Data import" app.:

5) The control record was updated (version and source):

Comment by Pavlo Smahin [ 22/Dec/22 ]

Investigate error
2022-12-22 15:24:43.733 [vert.x-worker-thread-4] [] [] [] [] WARN AdditionalFieldsUtil computeMarcRecord:: Error during the transformation to marc record
org.marc4j.util.JsonParser$Escape: Text was found preceding an array's opening bracket (this is usually caused by a missing comma or colon, or by using equals instead of a colon); Text="fields="; at Input Source: "MarcInput", Line: 1, Column: 9
at org.marc4j.util.JsonParser.parserError(JsonParser.java:973) ~[mod-source-record-storage-server-fat.jar:?]
at org.marc4j.util.JsonParser.parserError(JsonParser.java:968) ~[mod-source-record-storage-server-fat.jar:?]
at org.marc4j.util.JsonParser.parserError(JsonParser.java:964) ~[mod-source-record-storage-server-fat.jar:?]
at org.marc4j.util.JsonParser.next(JsonParser.java:704) ~[mod-source-record-storage-server-fat.jar:?]
at org.marc4j.MarcJsonReader.next(MarcJsonReader.java:208) ~[mod-source-record-storage-server-fat.jar:?]
at org.folio.services.util.AdditionalFieldsUtil.lambda$0(AdditionalFieldsUtil.java:80) ~[mod-source-record-storage-server-fat.jar:?]
at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$3(LocalLoadingCache.java:197) ~[mod-source-record-storage-server-fat.jar:?]
at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$2(LocalCache.java:165) ~[mod-source-record-storage-server-fat.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$13(BoundedLocalCache.java:2550) ~[mod-source-record-storage-server-fat.jar:?]
at java.util.concurrent.ConcurrentHashMap.compute(Unknown Source) ~[?:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2548) ~[mod-source-record-storage-server-fat.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2531) ~[mod-source-record-storage-server-fat.jar:?]
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:110) ~[mod-source-record-storage-server-fat.jar:?]
at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:58) ~[mod-source-record-storage-server-fat.jar:?]
at org.folio.services.util.AdditionalFieldsUtil.computeMarcRecord(AdditionalFieldsUtil.java:583) ~[mod-source-record-storage-server-fat.jar:?]
at org.folio.services.util.AdditionalFieldsUtil.getValueFromControlledField(AdditionalFieldsUtil.java:318) ~[mod-source-record-storage-server-fat.jar:?]
at org.folio.services.handlers.actions.AbstractUpdateModifyEventHandler.retrieveHrid(AbstractUpdateModifyEventHandler.java:167) ~[mod-source-record-storage-server-fat.jar:?]
at org.folio.services.handlers.actions.AbstractUpdateModifyEventHandler.handle(AbstractUpdateModifyEventHandler.java:80) ~[mod-source-record-storage-server-fat.jar:?]
at org.folio.processing.events.services.processor.EventProcessorImpl.process(EventProcessorImpl.java:36) ~[mod-source-record-storage-server-fat.jar:?]
at org.folio.processing.events.EventManager.handleEvent(EventManager.java:67) ~[mod-source-record-storage-server-fat.jar:?]
at org.folio.consumers.DataImportKafkaHandler.lambda$1(DataImportKafkaHandler.java:72) ~[mod-source-record-storage-server-fat.jar:?]
at java.util.Optional.map(Unknown Source) ~[?:?]
at org.folio.consumers.DataImportKafkaHandler.lambda$0(DataImportKafkaHandler.java:72) ~[mod-source-record-storage-server-fat.jar:?]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
at io.vertx.core.Future.lambda$toCompletionStage$3(Future.java:384) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:141) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.Promise.complete(Promise.java:66) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.Future.lambda$fromCompletionStage$4(Future.java:408) ~[mod-source-record-storage-server-fat.jar:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
at io.vertx.core.Future.lambda$toCompletionStage$3(Future.java:384) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:141) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.Promise.complete(Promise.java:66) ~[mod-source-record-storage-server-fat.jar:?]
at org.folio.dataimport.util.RestUtil.lambda$handleResponse$0(RestUtil.java:182) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.ext.web.client.impl.HttpContext.handleDispatchResponse(HttpContext.java:397) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.ext.web.client.impl.HttpContext.execute(HttpContext.java:384) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.ext.web.client.impl.HttpContext.next(HttpContext.java:362) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.ext.web.client.impl.HttpContext.fire(HttpContext.java:329) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.ext.web.client.impl.HttpContext.dispatchResponse(HttpContext.java:291) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.ext.web.client.impl.HttpContext.lambda$null$7(HttpContext.java:507) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.impl.WorkerContext.lambda$run$3(WorkerContext.java:106) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.impl.WorkerContext.lambda$null$1(WorkerContext.java:92) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[mod-source-record-storage-server-fat.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[mod-source-record-storage-server-fat.jar:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]

Comment by Pavlo Smahin [ 22/Dec/22 ]

And this one:
22-12-22 15:32:23.110 [vert.x-worker-thread-8] [] [] [] [] INFO KafkaEventPublisher Event with type: 'DI_ERROR' by jobExecutionId: 'f42f4627-04fc-432b-8a52-159329ab795a' and recordId: '6fe3f99e-7dc8-4850-b8ee-a3f689fb62e2' with chunkId: '0ab69390-bc15-44ac-88c9-5413e8921cd0' was sent to the topic 'folio-dev-spitfire.Default.diku.DI_ERROR'
2022-12-22 15:32:23.110 [vert.x-worker-thread-8] [] [] [] [] DEBUG KafkaConsumerWrapper businessHandlerCompletionHandler:: Consumer - id: 13 subscriptionPattern: SubscriptionDefinition(eventType=DI_SRS_MARC_AUTHORITY_RECORD_MATCHED, subscriptionPattern=folio-dev-spitfire\.Default\.\w

{1,}\.DI_SRS_MARC_AUTHORITY_RECORD_MATCHED) Committing offset: 285
2022-12-22 15:32:23.111 [vert.x-worker-thread-8] [] [] [] [] ERROR KafkaConsumerWrapper businessHandlerCompletionHandler:: Error while processing a record - id: 13 subscriptionPattern: SubscriptionDefinition(eventType=DI_SRS_MARC_AUTHORITY_RECORD_MATCHED, subscriptionPattern=folio-dev-spitfire\.Default\.\w{1,}

\.DI_SRS_MARC_AUTHORITY_RECORD_MATCHED) offset: 285
io.vertx.core.impl.NoStackTraceThrowable: handle:: Failed to process data import event payload from topic 'folio-dev-spitfire.Default.diku.DI_SRS_MARC_AUTHORITY_RECORD_MATCHED' by jobExecutionId: 'f42f4627-04fc-432b-8a52-159329ab795a' with recordId: '6fe3f99e-7dc8-4850-b8ee-a3f689fb62e2' and chunkId: '0ab69390-bc15-44ac-88c9-5413e8921cd0'
2022-12-22 15:32:23.111 [vert.x-worker-thread-8] [] [] [] [] WARN KafkaConsumerWrapper businessHandlerCompletionHandler:: Error handler has not been implemented for subscriptionPattern: SubscriptionDefinition(eventType=DI_SRS_MARC_AUTHORITY_RECORD_MATCHED, subscriptionPattern=folio-dev-spitfire\.Default\.\w

{1,}

\.DI_SRS_MARC_AUTHORITY_RECORD_MATCHED) failures
2022-12-22 15:32:23.111 [vert.x-worker-thread-8] [] [] [] [] DEBUG KafkaConsumerWrapper businessHandlerCompletionHandler:: Threshold is exceeded, preparing to resume, globalLoad: 1, currentLoad: 1, requestNo: -1
2022-12-22 15:32:23.302 [vert.x-worker-thread-13] [] [] [] [] WARN teModifyEventHandler handle:: Error while MARC record modifying
io.vertx.pgclient.PgException: ERROR: duplicate key value violates unique constraint "idx_records_matched_id_gen" (23505)
at io.vertx.pgclient.impl.codec.ErrorResponse.toException(ErrorResponse.java:31) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.pgclient.impl.codec.QueryCommandBaseCodec.handleErrorResponse(QueryCommandBaseCodec.java:57) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.pgclient.impl.codec.ExtendedQueryCommandCodec.handleErrorResponse(ExtendedQueryCommandCodec.java:90) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.pgclient.impl.codec.PgDecoder.decodeError(PgDecoder.java:246) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.pgclient.impl.codec.PgDecoder.decodeMessage(PgDecoder.java:132) ~[mod-source-record-storage-server-fat.jar:?]
at io.vertx.pgclient.impl.codec.PgDecoder.channelRead(PgDecoder.java:112) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[mod-source-record-storage-server-fat.jar:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[mod-source-record-storage-server-fat.jar:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]

Comment by Khalilah Gambrell [ 26/Dec/22 ]

Moving to In Progress based on Pavlo's notes. 

Comment by Khalilah Gambrell [ 13/Jan/23 ]

Successfully used on an environment.

Generated at Thu Feb 08 23:29:45 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.