[MODEXPW-51] Records processing errors handling Created: 17/Nov/21  Updated: 20/Jan/22  Resolved: 20/Jan/22

Status: Closed
Project: mod-data-export-worker
Components: None
Affects versions: None
Fix versions: 1.3.0

Type: Story Priority: P3
Reporter: Illia Daliek Assignee: Viachaslau Khandramai (Inactive)
Resolution: Done Votes: 0
Labels: bulk-edit-pilot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File barcodes.csv     File c51ccc75-61b0-4f10-8dcd-11f3bac41bf2_barcodes.csv    
Issue links:
Blocks
blocks UIBULKED-13 Populating error accordion Closed
Defines
defines UXPROD-3318 User records - bulk edit - pilot impl... Closed
Gantt End to Start
has to be done after MODEXPW-45 Get matching records based on provide... Closed
has to be done after MODEXPW-46 Get matching records based on provide... Closed
Relates
relates to MODBULKED-2 [SPIKE]:File upload functionality Closed
Sprint: Firebird Sprint 130, Firebird Sprint 131
Story Points: 3
Development Team: Firebird

 Description   

Purpose/Overview:
It is required to handle the errors during the export process. The handled errors should be logged and saved. Therefore users will be able to see errors.

Requirements/Scope:

  1. If an error occurs, the identifier of the record and reason for the failure are provided.
  2. Encountered errors are logged – might not be in scope for the pilot program
  • record identifier (TODO: determine UUID or barcode)
  • reason for the error
  1. Invalid identifiers are logged (when processing the UUIDs file or record with id doesn't exist)
  • invalid identifier
  • reason for being invalid

Approach:
1) Database table for error logs added
2) Error handling provided for both export by UUIDs file and query jobs.
3) New endpoint for getting error logs implemented /data-export-worker/jobId/errors

Acceptance criteria:

  • Unsuccessful updates are logged and reported
  • List of errors is provided so that it can be displayed to the User
  • List of invalid identifiers is provided so that it can be displayed to the User

Out of scope: localization of the error messages is out of scope for the pilot project but should be considered in the later implementations



 Comments   
Comment by Magda Zacharska [ 19/Nov/21 ]

Illia Daliek Couple of things:

  • the story is about bulk edit not export
  • for the pilot program the errors should contain user barcode not the UUID as there is no way to search users by UUID in the UI

Also, I'm not sure what you mean by :

Invalid identifiers are logged (when processing the UUIDs file or record with id doesn't exist)

Comment by Illia Daliek [ 03/Dec/21 ]

Magda Zacharska, export means exporting users to a CSV file for editing.
"Invalid identifiers are logged" means that we will also log the errors.

Regarding barcodes, if the user wasn't found by UUID we cannot paste the barcode to the error log of a non-existent user because we cannot obtain this information in such a case.
If you'd like to save errors using user barcode then we can do it only if the user by provided UUID exists. In the case of CSV file with barcodes, we can save errors with the barcode.

Comment by Magda Zacharska [ 11/Jan/22 ]

Viachaslau Khandramai please provide examples of the requests and responses for the implemented functionality.

Comment by Viachaslau Khandramai (Inactive) [ 14/Jan/22 ]

Hi Magda Zacharska,

verified on https://folio-testing.dev.folio.org:
1) Create IDENTIFIERS job

POST /data-export-spring/jobs HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
x-okapi-url: http://localhost:9130
Content-Type: application/json
User-Agent: PostmanRuntime/7.28.4
Accept: */*
Postman-Token: 8c059cdc-4fd1-4843-84e7-ba16c0049903
Host: folio-testing-okapi.dev.folio.org:443
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 108
 
{"type":"BULK_EDIT_IDENTIFIERS","identifierType":"ID","entityType":"USER","exportTypeSpecificParameters":{}}
 
HTTP/1.1 201 Created
Date: Fri, 14 Jan 2022 09:09:43 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
vary: origin
 
{"id":"c51ccc75-61b0-4f10-8dcd-11f3bac41bf2","name":"000002","source":"diku_admin","isSystemSource":false,"type":"BULK_EDIT_IDENTIFIERS","exportTypeSpecificParameters":{},"status":"SCHEDULED","metadata":{"createdDate":1642151383085,"createdByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","createdByUsername":"diku_admin","updatedDate":1642151383085,"updatedByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","updatedByUsername":"diku_admin"},"identifierType":"ID","entityType":"USER"}

2) Upload file with barcodes barcodes.csv : c51ccc75-61b0-4f10-8dcd-11f3bac41bf2_barcodes.csv

POST /bulk-edit/c51ccc75-61b0-4f10-8dcd-11f3bac41bf2/upload HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryViCyJ5CEqiFuOFKZ
Accept: */*
User-Agent: PostmanRuntime/7.28.4
Postman-Token: da5aba9e-22eb-456e-9a5c-59a56f7299dd
Host: folio-testing-okapi.dev.folio.org:443
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 202
 
------WebKitFormBoundaryViCyJ5CEqiFuOFKZ
Content-Disposition: form-data; name="file"; filename="barcodes.csv"
<barcodes.csv>
------WebKitFormBoundaryViCyJ5CEqiFuOFKZ--
 
HTTP/1.1 200 OK
Date: Fri, 14 Jan 2022 09:09:48 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
vary: origin
 
3

3) Retrieve job and download file by link from Amazon:

GET /data-export-spring/jobs/c51ccc75-61b0-4f10-8dcd-11f3bac41bf2 HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
x-okapi-url: http://localhost:9130
Content-Type: text/plain
User-Agent: PostmanRuntime/7.28.4
Accept: */*
Postman-Token: b571ba01-2d91-43f0-b314-7dcf8470a5ff
Host: folio-testing-okapi.dev.folio.org:443
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 0
 
HTTP/1.1 200 OK
Date: Fri, 14 Jan 2022 09:09:51 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: close
vary: origin
 
{"id":"c51ccc75-61b0-4f10-8dcd-11f3bac41bf2","name":"000002","source":"diku_admin","isSystemSource":false,"type":"BULK_EDIT_IDENTIFIERS","exportTypeSpecificParameters":{},"status":"SUCCESSFUL","files":["https://folio-testing-export.s3.us-east-1.amazonaws.com/c51ccc75-61b0-4f10-8dcd-11f3bac41bf2_barcodes.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20220114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220114T090948Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=8fcb588e9848870659d132805cb07cd073034ccea8691c170e889bcac575780c"],"startTime":1642151388099,"endTime":1642151388562,"metadata":{"createdDate":1642151383085,"createdByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","createdByUsername":"diku_admin","updatedDate":1642151388857,"updatedByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","updatedByUsername":"diku_admin"},"identifierType":"ID","entityType":"USER"}

Break file, for example, break some ID ( c51ccc75-61b0-4f10-8dcd-11f3bac41bf2_barcodes.csv ):

4) Post UPLOAD job:

POST /data-export-spring/jobs HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
x-okapi-url: http://localhost:9130
Content-Type: application/json
User-Agent: PostmanRuntime/7.28.4
Accept: */*
Postman-Token: d982e2c2-2d9a-445c-94a6-8d43873fe15e
Host: folio-testing-okapi.dev.folio.org:443
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 108
 
{"type":"BULK_EDIT_UPDATE","identifierType":"BARCODE","entityType":"USER","exportTypeSpecificParameters":{}}
 
HTTP/1.1 201 Created
Date: Fri, 14 Jan 2022 09:11:31 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
vary: origin
 
{"id":"02ee2680-93fc-4357-847b-3823d08285b3","name":"000003","source":"diku_admin","isSystemSource":false,"type":"BULK_EDIT_UPDATE","exportTypeSpecificParameters":{},"status":"SCHEDULED","metadata":{"createdDate":1642151491799,"createdByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","createdByUsername":"diku_admin","updatedDate":1642151491799,"updatedByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","updatedByUsername":"diku_admin"},"identifierType":"BARCODE","entityType":"USER"}

5) Upload broken file:

POST /bulk-edit/02ee2680-93fc-4357-847b-3823d08285b3/upload HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryViCyJ5CEqiFuOFKZ
Accept: */*
Accept-Encoding: gzip, deflate, br
User-Agent: PostmanRuntime/7.28.4
Postman-Token: d7e99d93-825f-493c-a91b-29b0d27a0056
Host: folio-testing-okapi.dev.folio.org:443
Connection: keep-alive
Content-Length: 1039
 
------WebKitFormBoundaryViCyJ5CEqiFuOFKZ
Content-Disposition: form-data; name="file"; filename="c51ccc75-61b0-4f10-8dcd-11f3bac41bf2_barcodes.csv"
<c51ccc75-61b0-4f10-8dcd-11f3bac41bf2_barcodes.csv>
------WebKitFormBoundaryViCyJ5CEqiFuOFKZ--
 
HTTP/1.1 200 OK
Date: Fri, 14 Jan 2022 09:11:50 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
vary: origin
 
3

6) Start UPLOAD job:

POST /bulk-edit/02ee2680-93fc-4357-847b-3823d08285b3/start HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip, deflate, br
User-Agent: PostmanRuntime/7.28.4
Postman-Token: 574b80fb-4b2c-48af-a728-3c9d64d5c994
Host: folio-testing-okapi.dev.folio.org:443
Connection: keep-alive
Content-Length: 202
 
HTTP/1.1 200 OK
Date: Fri, 14 Jan 2022 09:11:53 GMT
Transfer-Encoding: chunked
Connection: keep-alive
vary: origin

7) Get UPDATE job and download csv file with errors:

GET /data-export-spring/jobs/02ee2680-93fc-4357-847b-3823d08285b3 HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
x-okapi-url: http://localhost:9130
Content-Type: text/plain
User-Agent: PostmanRuntime/7.28.4
Accept: */*
Postman-Token: ef8f54f6-2c19-43b0-9e43-8ca5d8bcba86
Host: folio-testing-okapi.dev.folio.org:443
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 0
 
HTTP/1.1 200 OK
Date: Fri, 14 Jan 2022 09:11:56 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: close
vary: origin
 
{"id":"02ee2680-93fc-4357-847b-3823d08285b3","name":"000003","source":"diku_admin","isSystemSource":false,"type":"BULK_EDIT_UPDATE","exportTypeSpecificParameters":{},"status":"SUCCESSFUL","files":["https://folio-testing-export.s3.us-east-1.amazonaws.com/2022-01-14-Errors-c51ccc75-61b0-4f10-8dcd-11f3bac41bf2_barcodes.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20220114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220114T091154Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=cf0994605af4c8e4ced44f1b3666da057f0383d496380228e9bccfe58b09a30f"],"startTime":1642151513780,"endTime":1642151513958,"metadata":{"createdDate":1642151491799,"createdByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","createdByUsername":"diku_admin","updatedDate":1642151514100,"updatedByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","updatedByUsername":"diku_admin"},"identifierType":"BARCODE","entityType":"USER","progress":{"total":3,"processed":3,"progress":100}}

Verify that file is available and it contains errors.

8) Retrieve errors preview:

GET /bulk-edit/02ee2680-93fc-4357-847b-3823d08285b3/errors?limit=10 HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
Content-Type: text/plain
User-Agent: PostmanRuntime/7.28.4
Accept: */*
Postman-Token: c71d310c-a93a-4d6f-bf4a-c75716555595
Host: folio-testing-okapi.dev.folio.org:443
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 0
 
HTTP/1.1 200 OK
Date: Fri, 14 Jan 2022 09:12:11 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: close
vary: origin
 
{"errors":[{"message":"789,[400 Bad Request] during [PUT] to [http://users/b4cee18d-f862-4ef1-95a5-879fdd61960] [UserClient#updateUser(User_String)]: [Invalid UUID format of id_ should be xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx where M is 1-5 and N is 8_ 9_ a_ b_ A or B and x is 0-9_ a-f or A-F.]","type":"BULK_EDIT_ERROR","code":null,"parameters":null}],"total_records":1}

Verify that preview is available.

Generated at Thu Feb 08 22:30:24 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.