[MODEXPS-48] Remove expired files Created: 17/Nov/21 Updated: 18/Nov/22 Resolved: 05/Jan/22 |
|
| Status: | Closed |
| Project: | mod-data-export-spring |
| Components: | None |
| Affects versions: | None |
| Fix versions: | 1.3.0 |
| Type: | Story | Priority: | P3 |
| Reporter: | Illia Daliek | Assignee: | Siarhei Charniak |
| Resolution: | Done | Votes: | 0 |
| Labels: | bulk-edit-pilot | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||
| Issue links: |
|
||||||||||||||||||||||||||||
| Sprint: | Firebird Sprint 130 | ||||||||||||||||||||||||||||
| Story Points: | 2 | ||||||||||||||||||||||||||||
| Development Team: | Firebird | ||||||||||||||||||||||||||||
| Description |
|
Purpose/Overview:
Approach: Acceptance criteria:
|
| Comments |
| Comment by Siarhei Charniak [ 24/Dec/21 ] |
|
Magda Zacharska 1. Prepare test data Check that job exists curl --location --request GET 'https://firebird-okapi.ci.folio.org:443/data-export-spring/jobs/68f88eda-e194-4f18-8480-52df4fa37c10' \ --header 'x-okapi-tenant: diku' \ --header "x-okapi-token: $TOKEN" \ --header 'Content-Type: application/json' { "id": "68f88eda-e194-4f18-8480-52df4fa37c10", "name": "000001", "source": "diku_admin", "isSystemSource": false, "type": "BULK_EDIT_QUERY", "exportTypeSpecificParameters": { "query": "(patronGroup==\"3684a786-6671-4268-8ed0-9db82ebca60b\") sortby personal.lastName" }, "status": "SUCCESSFUL", "files": [ "https://folio-firebird.s3.us-west-2.amazonaws.com/BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211223%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20211223T133026Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=2fdb37f5d95e7821154f907521e93fd9da6770e61b78e4f018751954ebca8162" ], "startTime": 1640266212058, "endTime": 1640266227658, "metadata": { "createdDate": 1640266205567, "createdByUserId": "1de56b34-2732-5a21-a9ed-a8bb1e301c85", "createdByUsername": "diku_admin", "updatedDate": 1638365427761, "updatedByUserId": "1de56b34-2732-5a21-a9ed-a8bb1e301c85", "updatedByUsername": "diku_admin" }, "entityType": "USER" } 2. Check result after next scheduled execution 14:11:48 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-spring] INFO ulkEditConfigService Found bulk-edit configuration: class ModelConfiguration { 24/12/2021 17:11:48 id: 9d9680a0-1cde-4d13-8372-9a817fe8b7c2 24/12/2021 17:11:48 module: BULKEDIT 24/12/2021 17:11:48 configName: general 24/12/2021 17:11:48 code: null 24/12/2021 17:11:48 description: null 24/12/2021 17:11:48 _default: true 24/12/2021 17:11:48 enabled: true 24/12/2021 17:11:48 value: {"jobExpirationPeriod":"14"} 24/12/2021 17:11:48 userId: null 24/12/2021 17:11:48 metadata: class Metadata { 24/12/2021 17:11:48 createdDate: Mon Dec 20 14:26:57 GMT 2021 24/12/2021 17:11:48 createdByUserId: null 24/12/2021 17:11:48 createdByUsername: null 24/12/2021 17:11:48 updatedDate: Mon Dec 20 14:26:57 GMT 2021 24/12/2021 17:11:48 updatedByUserId: null 24/12/2021 17:11:48 updatedByUsername: null 24/12/2021 17:11:48 } 24/12/2021 17:11:48 } 24/12/2021 17:11:48 14:11:48 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-spring] INFO JobServiceImpl Collecting old bulk-edit jobs with 'updatedDate' less than 2021-12-10T00:00:00.000+0000. 24/12/2021 17:11:48 14:11:48 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-spring] INFO JobServiceImpl Deleted old jobs [Job(id=68f88eda-e194-4f18-8480-52df4fa37c10, name=000001, description=null, source=diku_admin, isSystemSource=false, type=BULK_EDIT_QUERY, exportTypeSpecificParameters=class ExportTypeSpecificParameters { 24/12/2021 17:11:48 bursarFeeFines: null 24/12/2021 17:11:48 vendorEdiOrdersExportConfig: null 24/12/2021 17:11:48 query: (patronGroup=="3684a786-6671-4268-8ed0-9db82ebca60b") sortby personal.lastName 24/12/2021 17:11:48 }, status=SUCCESSFUL, files=[https://folio-firebird.s3.us-west-2.amazonaws.com/BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211223%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20211223T133026Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=2fdb37f5d95e7821154f907521e93fd9da6770e61b78e4f018751954ebca8162], startTime=2021-12-23 13:30:12.058, endTime=2021-12-23 13:30:27.658, createdDate=2021-12-23 13:30:05.567, createdByUserId=1de56b34-2732-5a21-a9ed-a8bb1e301c85, createdByUsername=diku_admin, updatedDate=2021-12-01 13:30:27.761, updatedByUserId=1de56b34-2732-5a21-a9ed-a8bb1e301c85, updatedByUsername=diku_admin, outputFormat=null, errorDetails=null, batchStatus=COMPLETED, exitStatus=exitCode=COMPLETED;exitDescription=, identifierType=null, entityType=USER, progress=null)]. 24/12/2021 17:11:49 14:11:49 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-spring] INFO KafkaService Sending JobCommand(type=DELETE, id=60fa2704-19c9-4f10-9ad3-35c76635cd2a, name=null, description=null, exportType=null, jobParameters={outputFilesInStorage=https://folio-firebird.s3.us-west-2.amazonaws.com/BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211223%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20211223T133026Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=2fdb37f5d95e7821154f907521e93fd9da6770e61b78e4f018751954ebca8162}, identifierType=null, entityType=null, progress=null). 24/12/2021 17:11:49 14:11:49 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-spring] INFO KafkaService Sent JobCommand(type=DELETE, id=60fa2704-19c9-4f10-9ad3-35c76635cd2a, name=null, description=null, exportType=null, jobParameters={outputFilesInStorage=https://folio-firebird.s3.us-west-2.amazonaws.com/BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211223%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20211223T133026Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=2fdb37f5d95e7821154f907521e93fd9da6770e61b78e4f018751954ebca8162}, identifierType=null, entityType=null, progress=null). 24/12/2021 17:11:49 14:11:49 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-spring] INFO ExportScheduler deleteOldJobs executed for jobId: at: 2021-12-24T14:11:48.573+0000 Check job execution at mod-data-export-worker module - file was deleted: 14:11:49 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-worker] INFO mandsReceiverService Received JobCommand(type=DELETE, id=60fa2704-19c9-4f10-9ad3-35c76635cd2a, name=null, description=null, exportType=null, jobParameters={outputFilesInStorage=https://folio-firebird.s3.us-west-2.amazonaws.com/BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211223%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20211223T133026Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=2fdb37f5d95e7821154f907521e93fd9da6770e61b78e4f018751954ebca8162}, identifierType=null, entityType=null, progress=null). 24/12/2021 17:11:49 14:11:49 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-worker] INFO mandsReceiverService Deleting old job files https://folio-firebird.s3.us-west-2.amazonaws.com/BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211223%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20211223T133026Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=2fdb37f5d95e7821154f907521e93fd9da6770e61b78e4f018751954ebca8162. 24/12/2021 17:11:49 14:11:49 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-worker] INFO ectStorageRepository Deleting objects [BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv]. Check that job was deleted curl --location --request GET 'https://firebird-okapi.ci.folio.org:443/data-export-spring/jobs/68f88eda-e194-4f18-8480-52df4fa37c10' \ --header 'x-okapi-tenant: diku' \ --header 'x-okapi-token: eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJkaWt1X2FkbWluIiwidXNlcl9pZCI6IjFkZTU2YjM0LTI3MzItNWEyMS1hOWVkLWE4YmIxZTMwMWM4NSIsImlhdCI6MTY0MDI2NjE4NCwidGVuYW50IjoiZGlrdSJ9.XTS55q1i5momIlrnj6l5N2KZCb_vhn7c_mzq9DaotUM' \ --header 'Content-Type: application/json' { "errors": [ { "message": "Job 68f88eda-e194-4f18-8480-52df4fa37c10 not found", "type": "NotFoundException", "code": "404 NOT_FOUND" } ], "total_records": 1 } |