[MODEXPW-51] Records processing errors handling Created: 17/Nov/21 Updated: 20/Jan/22 Resolved: 20/Jan/22 |
|
| Status: | Closed |
| Project: | mod-data-export-worker |
| Components: | None |
| Affects versions: | None |
| Fix versions: | 1.3.0 |
| Type: | Story | Priority: | P3 |
| Reporter: | Illia Daliek | Assignee: | Viachaslau Khandramai (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | bulk-edit-pilot | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||||||||||
| Issue links: |
|
||||||||||||||||||||||||||||||||||||
| Sprint: | Firebird Sprint 130, Firebird Sprint 131 | ||||||||||||||||||||||||||||||||||||
| Story Points: | 3 | ||||||||||||||||||||||||||||||||||||
| Development Team: | Firebird | ||||||||||||||||||||||||||||||||||||
| Description |
|
Purpose/Overview: Requirements/Scope:
Approach: Acceptance criteria:
Out of scope: localization of the error messages is out of scope for the pilot project but should be considered in the later implementations |
| Comments |
| Comment by Magda Zacharska [ 19/Nov/21 ] |
|
Illia Daliek Couple of things:
Also, I'm not sure what you mean by :
|
| Comment by Illia Daliek [ 03/Dec/21 ] |
|
Magda Zacharska, export means exporting users to a CSV file for editing. Regarding barcodes, if the user wasn't found by UUID we cannot paste the barcode to the error log of a non-existent user because we cannot obtain this information in such a case. |
| Comment by Magda Zacharska [ 11/Jan/22 ] |
|
Viachaslau Khandramai please provide examples of the requests and responses for the implemented functionality. |
| Comment by Viachaslau Khandramai (Inactive) [ 14/Jan/22 ] |
|
Hi Magda Zacharska, verified on https://folio-testing.dev.folio.org:
POST /data-export-spring/jobs HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
x-okapi-url: http://localhost:9130
Content-Type: application/json
User-Agent: PostmanRuntime/7.28.4
Accept: */*
Postman-Token: 8c059cdc-4fd1-4843-84e7-ba16c0049903
Host: folio-testing-okapi.dev.folio.org:443
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 108
{"type":"BULK_EDIT_IDENTIFIERS","identifierType":"ID","entityType":"USER","exportTypeSpecificParameters":{}}
HTTP/1.1 201 Created
Date: Fri, 14 Jan 2022 09:09:43 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
vary: origin
{"id":"c51ccc75-61b0-4f10-8dcd-11f3bac41bf2","name":"000002","source":"diku_admin","isSystemSource":false,"type":"BULK_EDIT_IDENTIFIERS","exportTypeSpecificParameters":{},"status":"SCHEDULED","metadata":{"createdDate":1642151383085,"createdByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","createdByUsername":"diku_admin","updatedDate":1642151383085,"updatedByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","updatedByUsername":"diku_admin"},"identifierType":"ID","entityType":"USER"}
2) Upload file with barcodes barcodes.csv
POST /bulk-edit/c51ccc75-61b0-4f10-8dcd-11f3bac41bf2/upload HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryViCyJ5CEqiFuOFKZ
Accept: */*
User-Agent: PostmanRuntime/7.28.4
Postman-Token: da5aba9e-22eb-456e-9a5c-59a56f7299dd
Host: folio-testing-okapi.dev.folio.org:443
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 202
------WebKitFormBoundaryViCyJ5CEqiFuOFKZ
Content-Disposition: form-data; name="file"; filename="barcodes.csv"
<barcodes.csv>
------WebKitFormBoundaryViCyJ5CEqiFuOFKZ--
HTTP/1.1 200 OK
Date: Fri, 14 Jan 2022 09:09:48 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
vary: origin
3
3) Retrieve job and download file by link from Amazon:
GET /data-export-spring/jobs/c51ccc75-61b0-4f10-8dcd-11f3bac41bf2 HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
x-okapi-url: http://localhost:9130
Content-Type: text/plain
User-Agent: PostmanRuntime/7.28.4
Accept: */*
Postman-Token: b571ba01-2d91-43f0-b314-7dcf8470a5ff
Host: folio-testing-okapi.dev.folio.org:443
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 0
HTTP/1.1 200 OK
Date: Fri, 14 Jan 2022 09:09:51 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: close
vary: origin
{"id":"c51ccc75-61b0-4f10-8dcd-11f3bac41bf2","name":"000002","source":"diku_admin","isSystemSource":false,"type":"BULK_EDIT_IDENTIFIERS","exportTypeSpecificParameters":{},"status":"SUCCESSFUL","files":["https://folio-testing-export.s3.us-east-1.amazonaws.com/c51ccc75-61b0-4f10-8dcd-11f3bac41bf2_barcodes.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20220114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220114T090948Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=8fcb588e9848870659d132805cb07cd073034ccea8691c170e889bcac575780c"],"startTime":1642151388099,"endTime":1642151388562,"metadata":{"createdDate":1642151383085,"createdByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","createdByUsername":"diku_admin","updatedDate":1642151388857,"updatedByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","updatedByUsername":"diku_admin"},"identifierType":"ID","entityType":"USER"}
Break file, for example, break some ID ( c51ccc75-61b0-4f10-8dcd-11f3bac41bf2_barcodes.csv 4) Post UPLOAD job:
POST /data-export-spring/jobs HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
x-okapi-url: http://localhost:9130
Content-Type: application/json
User-Agent: PostmanRuntime/7.28.4
Accept: */*
Postman-Token: d982e2c2-2d9a-445c-94a6-8d43873fe15e
Host: folio-testing-okapi.dev.folio.org:443
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 108
{"type":"BULK_EDIT_UPDATE","identifierType":"BARCODE","entityType":"USER","exportTypeSpecificParameters":{}}
HTTP/1.1 201 Created
Date: Fri, 14 Jan 2022 09:11:31 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
vary: origin
{"id":"02ee2680-93fc-4357-847b-3823d08285b3","name":"000003","source":"diku_admin","isSystemSource":false,"type":"BULK_EDIT_UPDATE","exportTypeSpecificParameters":{},"status":"SCHEDULED","metadata":{"createdDate":1642151491799,"createdByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","createdByUsername":"diku_admin","updatedDate":1642151491799,"updatedByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","updatedByUsername":"diku_admin"},"identifierType":"BARCODE","entityType":"USER"}
5) Upload broken file:
POST /bulk-edit/02ee2680-93fc-4357-847b-3823d08285b3/upload HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryViCyJ5CEqiFuOFKZ
Accept: */*
Accept-Encoding: gzip, deflate, br
User-Agent: PostmanRuntime/7.28.4
Postman-Token: d7e99d93-825f-493c-a91b-29b0d27a0056
Host: folio-testing-okapi.dev.folio.org:443
Connection: keep-alive
Content-Length: 1039
------WebKitFormBoundaryViCyJ5CEqiFuOFKZ
Content-Disposition: form-data; name="file"; filename="c51ccc75-61b0-4f10-8dcd-11f3bac41bf2_barcodes.csv"
<c51ccc75-61b0-4f10-8dcd-11f3bac41bf2_barcodes.csv>
------WebKitFormBoundaryViCyJ5CEqiFuOFKZ--
HTTP/1.1 200 OK
Date: Fri, 14 Jan 2022 09:11:50 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
vary: origin
3
6) Start UPLOAD job:
POST /bulk-edit/02ee2680-93fc-4357-847b-3823d08285b3/start HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip, deflate, br
User-Agent: PostmanRuntime/7.28.4
Postman-Token: 574b80fb-4b2c-48af-a728-3c9d64d5c994
Host: folio-testing-okapi.dev.folio.org:443
Connection: keep-alive
Content-Length: 202
HTTP/1.1 200 OK
Date: Fri, 14 Jan 2022 09:11:53 GMT
Transfer-Encoding: chunked
Connection: keep-alive
vary: origin
7) Get UPDATE job and download csv file with errors:
GET /data-export-spring/jobs/02ee2680-93fc-4357-847b-3823d08285b3 HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
x-okapi-url: http://localhost:9130
Content-Type: text/plain
User-Agent: PostmanRuntime/7.28.4
Accept: */*
Postman-Token: ef8f54f6-2c19-43b0-9e43-8ca5d8bcba86
Host: folio-testing-okapi.dev.folio.org:443
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 0
HTTP/1.1 200 OK
Date: Fri, 14 Jan 2022 09:11:56 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: close
vary: origin
{"id":"02ee2680-93fc-4357-847b-3823d08285b3","name":"000003","source":"diku_admin","isSystemSource":false,"type":"BULK_EDIT_UPDATE","exportTypeSpecificParameters":{},"status":"SUCCESSFUL","files":["https://folio-testing-export.s3.us-east-1.amazonaws.com/2022-01-14-Errors-c51ccc75-61b0-4f10-8dcd-11f3bac41bf2_barcodes.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20220114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220114T091154Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=cf0994605af4c8e4ced44f1b3666da057f0383d496380228e9bccfe58b09a30f"],"startTime":1642151513780,"endTime":1642151513958,"metadata":{"createdDate":1642151491799,"createdByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","createdByUsername":"diku_admin","updatedDate":1642151514100,"updatedByUserId":"a282bb36-6134-5f9e-a718-60a4163305ed","updatedByUsername":"diku_admin"},"identifierType":"BARCODE","entityType":"USER","progress":{"total":3,"processed":3,"progress":100}}
Verify that file is available and it contains errors. 8) Retrieve errors preview:
GET /bulk-edit/02ee2680-93fc-4357-847b-3823d08285b3/errors?limit=10 HTTP/1.1
x-okapi-token: ${TOKEN}
x-okapi-tenant: diku
Content-Type: text/plain
User-Agent: PostmanRuntime/7.28.4
Accept: */*
Postman-Token: c71d310c-a93a-4d6f-bf4a-c75716555595
Host: folio-testing-okapi.dev.folio.org:443
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 0
HTTP/1.1 200 OK
Date: Fri, 14 Jan 2022 09:12:11 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: close
vary: origin
{"errors":[{"message":"789,[400 Bad Request] during [PUT] to [http://users/b4cee18d-f862-4ef1-95a5-879fdd61960] [UserClient#updateUser(User_String)]: [Invalid UUID format of id_ should be xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx where M is 1-5 and N is 8_ 9_ a_ b_ A or B and x is 0-9_ a-f or A-F.]","type":"BULK_EDIT_ERROR","code":null,"parameters":null}],"total_records":1}
Verify that preview is available. |