[MODEXPW-47] Save errors in csv format Created: 24/Sep/21  Updated: 16/Dec/21  Resolved: 16/Dec/21

Status: Closed
Project: mod-data-export-worker
Components: None
Affects versions: None
Fix versions: None

Type: Story Priority: P3
Reporter: Magda Zacharska Assignee: Oleksandr Bozhko
Resolution: Done Votes: 0
Labels: bulk-edit-pilot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File 2021-12-09-Errors-user_barcodes.csv     File user_barcodes_with_one_corrent_one_not_exists_one_duplicated.csv    
Issue links:
Defines
defines UXPROD-3318 User records - bulk edit - pilot impl... Closed
Relates
relates to UIBULKED-19 Download errors (CSV) Closed
Sprint: Firebird Sprint 129
Story Points: 3
Development Team: Firebird

 Description   

Purpose/Overview:
If the errors occurred while retrieving records based on provided identifiers, those records will be displayed to the user and the user will need to be able to save them to their local machine.

Requirements/Scope:

  1. All encountered errors are saved in csv format
  2. Each row will specify affected identifier and the reason for error
  3. The file name follows the convention: <yyyy-mm-dd-Errors-<name of the file with identifiers>.csv

Approach:
Job for processing user-edited records should be enhanced with logic that will write invalid lines to a separate CSV file that by the end of the Job is available for the User

Acceptance criteria:

  • File is created and can be saved
  • File name follows the naming convention

Notes
Process efficiency



 Comments   
Comment by Illia Daliek [ 17/Nov/21 ]

Magda Zacharska, should we also store the CSV files with the errors in S3 for some period of time until it gets expired, or does the user has to get them only once?

Comment by Magda Zacharska [ 19/Nov/21 ]

Illia Daliek please do not make changes to the description of the jira after it was estimated by the team. Any required changes should be discussed and documented in the comments to the ticket.

Comment by Magda Zacharska [ 19/Nov/21 ]

For the pilot project we do not need to store the error files in S3

Comment by Oleksandr Bozhko [ 09/Dec/21 ]

Magda Zacharska - this story has been verified at snapshot environment.

1. Create a new job:

POST https://folio-snapshot-okapi.dev.folio.org:443/data-export-spring/jobs
X-Okapi-Tenant: diku
Content-Type: application/json
x-okapi-token: $TOKEN
{
"type": "BULK_EDIT_IDENTIFIERS",
"exportTypeSpecificParameters": {},
"entityType": "USER",
"identifierType": "BARCODE"
}
 
HTTP/1.1 201 Created
Content-Type: application/json
{
"id":"82e29654-921f-4d20b6dd617cf02a980d",
"name":"000008",
"source":"diku_admin",
"isSystemSource":false,
"type":"BULK_EDIT_IDENTIFIERS",
"exportTypeSpecificParameters":{},
"status":"SCHEDULED",
"metadata":
   {
    "createdDate":1639059432556,
    "createdByUserId":"c9eb3fe1-d774-5ef7-934f-be8a2315fb74",
    "createdByUsername":"diku_admin",
    "updatedDate":1639059432556,
    "updatedByUserId":"c9eb3fe1-d774-5ef7-934f-be8a2315fb74",
    "updatedByUsername":"diku_admin"
    },
"identifierType":"BARCODE",
"entityType":"USER"
}

2. Upload attached user_barcodes_with_one_corrent_one_not_exists_one_duplicated.csv file with user barcodes where first line contains a valid barcode, second line contains barcode that does not exist, and the last line contains the same barcode as in the first line:

POST https://folio-snapshot-okapi.dev.folio.org:443/bulk-edit/82e29654-921f-4d20-b6dd-617cf02a980d/upload
X-Okapi-Tenant: diku
Content-Type: multipart/form-data; 
x-okapi-token: $TOKEN
Accept: */*
Content-Disposition: form-data; name="file"; filename="user_barcodes.csv"
 
HTTP/1.1 200 OK

3. Get job by id:

GET https://folio-snapshot-okapi.dev.folio.org:443/data-export-spring/jobs/82e29654-921f-4d20-b6dd-617cf02a980d
X-Okapi-Tenant: diku
Content-Type: application/json
x-okapi-token: $TOKEN
Accept: */*
 
HTTP/1.1 200 OK
Content-Type: application/json
{
"id":"82e29654-921f-4d20-b6dd-617cf02a980d",
"name":"000008",
"source":"diku_admin",
"isSystemSource":false,
"type":"BULK_EDIT_IDENTIFIERS",
"exportTypeSpecificParameters":{},
"status":"SUCCESSFUL",
"files":["https://folio-snapshot-export.s3.us-east-1.amazonaws.com/user_barcodes.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211209%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211209T142031Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=bec05255f08ec384fd6fff1862cd04b6bc8a04fcb293f96db8a8562f4d6373d1",
"https://folio-snapshot-export.s3.us-east-1.amazonaws.com/2021-12-09-Errors-user_barcodes.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211209%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211209T142031Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=64114cb3394bc51538382263251f5134c127b3fa1202816d6cae6def4a5fe37b"],
"startTime":1639059630889,
"endTime":1639059630995,
"metadata":
    {
    "createdDate":1639059432556,
    "createdByUserId":"c9eb3fe1-d774-5ef7-934f-be8a2315fb74",
    "createdByUsername":"diku_admin",
    "updatedDate":1639059631114,
    "updatedByUserId":"c9eb3fe1-d774-5ef7-934f-be8a2315fb74",
    "updatedByUsername":"diku_admin"
    },
"identifierType":"BARCODE",
"entityType":"USER"
}

4. Use the second link in "files" and open 2021-12-09-Errors-user_barcodes.csv file:

GET https://folio-snapshot-export.s3.us-east-1.amazonaws.com/2021-12-09-Errors-user_barcodes.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211209%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211209T142031Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=64114cb3394bc51538382263251f5134c127b3fa1202816d6cae6def4a5fe37b

HTTP/1.1 200 OK

163901525593756911,No match found
1639015255937569116,Duplicate entry

Result is attached in 2021-12-09-Errors-user_barcodes.csv file.

Generated at Thu Feb 08 22:30:18 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.