[MODEXPS-48] Remove expired files Created: 17/Nov/21  Updated: 18/Nov/22  Resolved: 05/Jan/22

Status: Closed
Project: mod-data-export-spring
Components: None
Affects versions: None
Fix versions: 1.3.0

Type: Story Priority: P3
Reporter: Illia Daliek Assignee: Siarhei Charniak
Resolution: Done Votes: 0
Labels: bulk-edit-pilot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File pgadmin.png    
Issue links:
Defines
defines UXPROD-3318 User records - bulk edit - pilot impl... Closed
Gantt End to Start
has to be done after MODEXPW-45 Get matching records based on provide... Closed
Relates
relates to MODBULKED-2 [SPIKE]:File upload functionality Closed
relates to MODEXPW-326 Cannot re-run 1 day+ old exports Closed
Sprint: Firebird Sprint 130
Story Points: 2
Development Team: Firebird

 Description   

Purpose/Overview:
The files that stay more than the expiration time period should be deleted.
Requirements/Scope:

  1. Expired files should be auto cleaned
  2. The setting that defines the expiration time period should be configurable at the tenant level.

Approach:
1) Do integration with mod-configuration. Default values should be posted to mod-configuration during TenantApi.
2) Implement a scheduled job for file removal.

Acceptance criteria:

  • Expired files are removed
  • The expiration time period setting is configurable at the tenant level


 Comments   
Comment by Siarhei Charniak [ 24/Dec/21 ]

Magda Zacharska
Verified on firebird rancher environment

1. Prepare test data
Changed existing task updated date

Check that job exists

curl --location --request GET 'https://firebird-okapi.ci.folio.org:443/data-export-spring/jobs/68f88eda-e194-4f18-8480-52df4fa37c10' \
--header 'x-okapi-tenant: diku' \
--header "x-okapi-token: $TOKEN" \
--header 'Content-Type: application/json'
{
    "id": "68f88eda-e194-4f18-8480-52df4fa37c10",
    "name": "000001",
    "source": "diku_admin",
    "isSystemSource": false,
    "type": "BULK_EDIT_QUERY",
    "exportTypeSpecificParameters": {
        "query": "(patronGroup==\"3684a786-6671-4268-8ed0-9db82ebca60b\") sortby personal.lastName"
    },
    "status": "SUCCESSFUL",
    "files": [
        "https://folio-firebird.s3.us-west-2.amazonaws.com/BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211223%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20211223T133026Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=2fdb37f5d95e7821154f907521e93fd9da6770e61b78e4f018751954ebca8162"
    ],
    "startTime": 1640266212058,
    "endTime": 1640266227658,
    "metadata": {
        "createdDate": 1640266205567,
        "createdByUserId": "1de56b34-2732-5a21-a9ed-a8bb1e301c85",
        "createdByUsername": "diku_admin",
        "updatedDate": 1638365427761,
        "updatedByUserId": "1de56b34-2732-5a21-a9ed-a8bb1e301c85",
        "updatedByUsername": "diku_admin"
    },
    "entityType": "USER"
}

2. Check result after next scheduled execution
Check job execution at mod-data-export-spring module - expired job was found and DELETE message was sent to the mod-data-export-worker:

14:11:48 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-spring] INFO  ulkEditConfigService Found bulk-edit configuration: class ModelConfiguration {
24/12/2021 17:11:48     id: 9d9680a0-1cde-4d13-8372-9a817fe8b7c2
24/12/2021 17:11:48     module: BULKEDIT
24/12/2021 17:11:48     configName: general
24/12/2021 17:11:48     code: null
24/12/2021 17:11:48     description: null
24/12/2021 17:11:48     _default: true
24/12/2021 17:11:48     enabled: true
24/12/2021 17:11:48     value: {"jobExpirationPeriod":"14"}
24/12/2021 17:11:48     userId: null
24/12/2021 17:11:48     metadata: class Metadata {
24/12/2021 17:11:48         createdDate: Mon Dec 20 14:26:57 GMT 2021
24/12/2021 17:11:48         createdByUserId: null
24/12/2021 17:11:48         createdByUsername: null
24/12/2021 17:11:48         updatedDate: Mon Dec 20 14:26:57 GMT 2021
24/12/2021 17:11:48         updatedByUserId: null
24/12/2021 17:11:48         updatedByUsername: null
24/12/2021 17:11:48     }
24/12/2021 17:11:48 }
24/12/2021 17:11:48 14:11:48 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-spring] INFO  JobServiceImpl       Collecting old bulk-edit jobs with 'updatedDate' less than 2021-12-10T00:00:00.000+0000.
24/12/2021 17:11:48 14:11:48 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-spring] INFO  JobServiceImpl       Deleted old jobs [Job(id=68f88eda-e194-4f18-8480-52df4fa37c10, name=000001, description=null, source=diku_admin, isSystemSource=false, type=BULK_EDIT_QUERY, exportTypeSpecificParameters=class ExportTypeSpecificParameters {
24/12/2021 17:11:48     bursarFeeFines: null
24/12/2021 17:11:48     vendorEdiOrdersExportConfig: null
24/12/2021 17:11:48     query: (patronGroup=="3684a786-6671-4268-8ed0-9db82ebca60b") sortby personal.lastName
24/12/2021 17:11:48 }, status=SUCCESSFUL, files=[https://folio-firebird.s3.us-west-2.amazonaws.com/BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211223%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20211223T133026Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=2fdb37f5d95e7821154f907521e93fd9da6770e61b78e4f018751954ebca8162], startTime=2021-12-23 13:30:12.058, endTime=2021-12-23 13:30:27.658, createdDate=2021-12-23 13:30:05.567, createdByUserId=1de56b34-2732-5a21-a9ed-a8bb1e301c85, createdByUsername=diku_admin, updatedDate=2021-12-01 13:30:27.761, updatedByUserId=1de56b34-2732-5a21-a9ed-a8bb1e301c85, updatedByUsername=diku_admin, outputFormat=null, errorDetails=null, batchStatus=COMPLETED, exitStatus=exitCode=COMPLETED;exitDescription=, identifierType=null, entityType=USER, progress=null)].
24/12/2021 17:11:49 14:11:49 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-spring] INFO  KafkaService         Sending JobCommand(type=DELETE, id=60fa2704-19c9-4f10-9ad3-35c76635cd2a, name=null, description=null, exportType=null, jobParameters={outputFilesInStorage=https://folio-firebird.s3.us-west-2.amazonaws.com/BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211223%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20211223T133026Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=2fdb37f5d95e7821154f907521e93fd9da6770e61b78e4f018751954ebca8162}, identifierType=null, entityType=null, progress=null).
24/12/2021 17:11:49 14:11:49 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-spring] INFO  KafkaService         Sent JobCommand(type=DELETE, id=60fa2704-19c9-4f10-9ad3-35c76635cd2a, name=null, description=null, exportType=null, jobParameters={outputFilesInStorage=https://folio-firebird.s3.us-west-2.amazonaws.com/BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211223%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20211223T133026Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=2fdb37f5d95e7821154f907521e93fd9da6770e61b78e4f018751954ebca8162}, identifierType=null, entityType=null, progress=null).
24/12/2021 17:11:49 14:11:49 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-spring] INFO  ExportScheduler      deleteOldJobs executed for jobId:  at: 2021-12-24T14:11:48.573+0000

Check job execution at mod-data-export-worker module - file was deleted:

14:11:49 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-worker] INFO  mandsReceiverService Received JobCommand(type=DELETE, id=60fa2704-19c9-4f10-9ad3-35c76635cd2a, name=null, description=null, exportType=null, jobParameters={outputFilesInStorage=https://folio-firebird.s3.us-west-2.amazonaws.com/BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211223%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20211223T133026Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=2fdb37f5d95e7821154f907521e93fd9da6770e61b78e4f018751954ebca8162}, identifierType=null, entityType=null, progress=null).
24/12/2021 17:11:49 14:11:49 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-worker] INFO  mandsReceiverService Deleting old job files https://folio-firebird.s3.us-west-2.amazonaws.com/BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA2VGNPE73JV77RBGE%2F20211223%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20211223T133026Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=2fdb37f5d95e7821154f907521e93fd9da6770e61b78e4f018751954ebca8162.
24/12/2021 17:11:49 14:11:49 [258621/proxy;587674/tenant] [diku] [] [mod-data-export-worker] INFO  ectStorageRepository Deleting objects [BULK_EDIT_QUERY_2021-12-23_13%3A30%3A09_68f88eda-e194-4f18-8480-52df4fa37c10.csv].

Check that job was deleted

curl --location --request GET 'https://firebird-okapi.ci.folio.org:443/data-export-spring/jobs/68f88eda-e194-4f18-8480-52df4fa37c10' \
--header 'x-okapi-tenant: diku' \
--header 'x-okapi-token: eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJkaWt1X2FkbWluIiwidXNlcl9pZCI6IjFkZTU2YjM0LTI3MzItNWEyMS1hOWVkLWE4YmIxZTMwMWM4NSIsImlhdCI6MTY0MDI2NjE4NCwidGVuYW50IjoiZGlrdSJ9.XTS55q1i5momIlrnj6l5N2KZCb_vhn7c_mzq9DaotUM' \
--header 'Content-Type: application/json'
{
    "errors": [
        {
            "message": "Job 68f88eda-e194-4f18-8480-52df4fa37c10 not found",
            "type": "NotFoundException",
            "code": "404 NOT_FOUND"
        }
    ],
    "total_records": 1
}
Generated at Thu Feb 08 22:30:23 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.