Batch Importer (Bib/Acq) (UXPROD-47)

[MODSOURCE-89] Lookups in SRS for records by Instance UUID are timing out, part 1 Created: 20/Nov/19  Updated: 04/Feb/20  Resolved: 16/Dec/19

Status: Closed
Project: mod-source-record-storage
Components: None
Affects versions: None
Fix versions: 3.0.1
Parent: Batch Importer (Bib/Acq)

Type: Bug Priority: P2
Reporter: Theodor Tolstoy (One-Group.se) Assignee: Kateryna Senchenko
Resolution: Done Votes: 0
Labels: back-end, data-import, epam-folijet, q4-2019
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File image-2019-11-20-16-05-24-453.png     PNG File image-2019-11-20-16-05-49-935.png    
Issue links:
Blocks
blocks MODSOURCE-90 Release v3.0.1 Closed
Defines
defines UXPROD-2076 NFR: Data Import (Batch Importer for ... Closed
Relates
relates to MODSOURCE-92 Lookups in SRS for records by Instanc... Closed
relates to MODSOURCE-85 Timeout when making GET requests to /... Closed
relates to MODSOURCE-91 Timeout when making GET requests to /... Closed
relates to MODOAIPMH-82 ListRecords requests times out for Fi... Closed
Sprint: EPAM BatchLoader Sprint 31, EPAM BatchLoader Sprint 32
Story Points: 1
Development Team: Folijet
Epic Link: Batch Importer (Bib/Acq)

 Description   

When we had only 20 000 records in the Five colleges tenant, lookups by Instance UUID where fast, both through the UI, the API and also through the database. Now, with 7.6 million records, the requests are timing out.

I made test loads with 20 000 records using the exact same process as with the 7.6 million, so I am pretty certain that it is the number of records that is the cause for this.

Throught the ui:
Nothing happens

Through the API:
A request to baseUrl/source-storage/records/910890c8-3b58-4381-9ba8-725432ded07a?identifier=INSTANCE
gives me a HTTP 504 after 6minutes 40 seconds
Through the Database:

SELECT fs00001006_mod_source_record_storage.get_record_by_instance_id('910890c8-3b58-4381-9ba8-725432ded07a');

gives strange responses :

or



 Comments   
Comment by Theodor Tolstoy (One-Group.se) [ 20/Nov/19 ]

Ann-Marie BreauxCharlotte WhittCraig McNally are anyone of you able to bring this to Folijet's attention?

Comment by Ann-Marie Breaux (Inactive) [ 20/Nov/19 ]

Hi Theodor Tolstoy (One-Group.se) Yes, I'll pull it into Folijet's work. OK to change this FOLIO ticket into a MOD one of some kind?

BTW, Craig is on paternity leave for the next month. His wife had a little boy yesterday, Ethan!

Comment by Ann-Marie Breaux (Inactive) [ 20/Nov/19 ]

Hi Theodor Tolstoy (One-Group.se) One more thing - could you send info (or add to the description) on how to access the 5C tenant? Thank you!

Comment by Theodor Tolstoy (One-Group.se) [ 20/Nov/19 ]

Thank you Ann-Marie Breaux

I got a response from Ruslan in Slack saying that this issue is likely not an issue in the latest version. Question is though if that version is compatible with Daisy, or if there are other modules that needs to get updated.

I think it would be best to contact Anya for credentials, or patty.wanninger. Same issues are apparent in the UA tenant as well.

Congratulations to Craig McNally!

Comment by Anya [ 20/Nov/19 ]

Happy to provide them via email - who do I send it to?

Comment by Ann-Marie Breaux (Inactive) [ 20/Nov/19 ]

Hi Anya Please send credentials to me.

Theodor Tolstoy (One-Group.se) I have a feeling it will be better to wait for Edelweiss instead of retrofitting into Daisy, but Ruslan should be able to confirm tomorrow.

Comment by Anya [ 20/Nov/19 ]

Sent Thank you

Comment by Kateryna Senchenko [ 11/Dec/19 ]

Hello Theodor Tolstoy (One-Group.se),
Ruslan Lavrov and I checked retrieving of records by instanceId in newer version of SRS. It works as expected - request takes approximately 25s with 7.6 million records in the db.
Just to clarify a few points though:
1. Data that you shared with Ruslan contains "externalIdHolder" field instead of "externalIdsHolder"
2. The url to find a record by instance id is /source-storage/formattedRecords/{instanceId}?identifier=INSTANCE
Please let us know if you have any questions. Thank you.

Comment by Ann-Marie Breaux (Inactive) [ 12/Dec/19 ]

Hi Theodor Tolstoy (One-Group.se) Are we good to close this ticket? Any remaining outstanding questions with regards to it? Thank you!

cc: Kateryna Senchenko Ruslan Lavrov

Comment by Theodor Tolstoy (One-Group.se) [ 12/Dec/19 ]

I would need to test this in an environment with the right amount of records. Could Ruslan Lavrov let me have a look in his testing environment? Or could we perhaps get some help from the FSE Hosting team to set up an FOLIO instance like that? I can provide the FC Data.

Comment by Ann-Marie Breaux (Inactive) [ 12/Dec/19 ]

Theodor Tolstoy (One-Group.se) and Ruslan Lavrov Same as the other message. Hopefully you two can get together tomorrow? Maybe even today since it's still work time over there.

Comment by Ann-Marie Breaux (Inactive) [ 16/Dec/19 ]

Per conversation with Theodor Tolstoy (One-Group.se) and VBar We'll close this bug and release a MODSOURCE bug fix release with the improvements made so far, then create a Q1 2020 bug to continue to work on improving performance.

Taras Spashchenko Oleksii Kuzminov Kateryna Senchenko Igor Gorchakov Ruslan Lavrov

Generated at Thu Feb 08 23:20:07 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.