/
Scripting Data Export Through API Calls

Scripting Data Export Through API Calls

  • In progress

    • Overview

    This document describes how to automate a data export job using the Data Export APIs.
    The process uploads a CSV file containing record UUIDs, triggers an export job using a predefined job profile, polls for job completion, and finally downloads the exported file.

    The workflow described here was validated using Postman, but the same sequence of API calls can be implemented in any scripting language (e.g., Bash, Python, Node.js).

    Assumptions & Preconditions

    • You already have a CSV file containing UUIDs.

      • The CSV must contain one UUID per line.

      • Automating the creation of this CSV is out of scope for this document.

      • Example use case: querying records updated since a particular date and extracting UUIDs for EDS population.

    • You have already validated endpoints and queries using Postman.

    • You are authenticated against Okapi and sending required headers

    • You have permission to access the Data Export endpoints.

    Example Query for Collecting UUIDs

    Example query used to identify records prior to export:

    https://folio-testing-okapi.dev.folio.org/instance-bulk/ids?limit=2147483647&query=(keyword all "nod") sortby title

    High-Level Workflow

    1. Identify the Data Export job profile

    2. Register a file definition

    3. Upload the CSV file

    4. Trigger the export job

    5. Poll job execution status

    6. Download the exported file

    Step 1: Retrieve the Job Profile ID

    You must first identify the job profile to use for the export.

    API Request

    GET /data-export/job-profiles

    Example Response

    { "jobProfiles": [ { "id": "6f7f3cd7-9f24-42eb-ae91-91af1cd54d0a", "name": "Default job profile", "destination": "fileSystem", "description": "Default job profile", "userInfo": { "firstName": "System", "lastName": "Process", "userName": "system_process" }, "mappingProfileId": "25d81cbe-9686-11ea-bb37-0242ac130002", "metadata": { "createdDate": "2020-07-28T00:00:00.000+0000", "createdByUserId": "00000000-0000-0000-0000-000000000000", "createdByUsername": "system_process", "updatedDate": "2020-07-28T00:00:00.000+0000", "updatedByUserId": "00000000-0000-0000-0000-000000000000", "updatedByUsername": "system_process" } } ], "totalRecords": 1 }

     

    ✅ Outcome

    • Capture the value of id for the desired job profile.

    • This value will be used later as:

      jobProfileId

    Step 2: Create a File Definition

    Before uploading the CSV, the server must be informed that a file is coming.

    API Request

    POST /data-export/file-definitions

    Request Body

    { "fileName": "test.csv" }

     

    Example Response

    { "id": "d0d6b4dc-2207-42ec-ac6b-3c30300e4307", "fileName": "test.csv", "status": "NEW", "metadata": { "createdDate": "2020-10-07T13:17:02.263+0000", "createdByUserId": "873bbfa4-7f40-4bd0-aa27-027494a4b5e9", "updatedDate": "2020-10-07T13:17:02.263+0000", "updatedByUserId": "873bbfa4-7f40-4bd0-aa27-027494a4b5e9" } }

    ``

    ✅ Outcome

    • Capture the value of id

    • This value will be used later as:

      fileDefinitionId = d0d6b4dc-2207-42ec-ac6b-3c30300e4307

    Step 3: Upload the CSV File

    This step uploads the actual CSV file contents to the server.

    API Request

    POST /data-export/file-definitions/{fileDefinitionId}/upload

    Required Headers

    Content-Type: application/octet-stream

    Request Body

    • Use binary mode

    • Upload the actual CSV file (test.csv)

    Postman Note
    Select Body → binary → Select File

    Example Response

    { "id": "d0d6b4dc-2207-42ec-ac6b-3c30300e4307", "fileName": "test.csv", "jobExecutionId": "22091d4a-0c70-4a9e-8843-5dcf96c7cdf5", "sourcePath": "./storage/files/d0d6b4dc-2207-42ec-ac6b-3c30300e4307/test.csv", "status": "COMPLETED", "metadata": { "createdDate": "2020-10-07T13:17:02.263+0000", "createdByUserId": "873bbfa4-7f40-4bd0-aa27-027494a4b5e9", "updatedDate": "2020-10-07T13:17:02.263+0000", "updatedByUserId": "873bbfa4-7f40-4bd0-aa27-027494a4b5e9" } }

    ✅ Outcome

    • Capture the jobExecutionId

    • This will be used to track the export job

    Step 4: Trigger the Export Job

    Now that the file is uploaded, instruct the server to execute the export.

    API Request

    POST /data-export/export

    Headers

    Content-Type: application/json

    Request Body

    { "fileDefinitionId": "<fileDefinitionId>", "jobProfileId": "<jobProfileId>" }

    Expected Response

    204 No Content

    ✅ Outcome

    • Export job is initiated

    • Processing occurs asynchronously

    Step 5: Poll for Job Completion

    Export execution is asynchronous. Your script must poll until completion.

    API Request

    GET /data-export/job-executions?query=id=="<jobExecutionId>"

    Example Response

    { "id": "22091d4a-0c70-4a9e-8843-5dcf96c7cdf5", "hrId": "112", "exportedFiles": [ { "fileId": "b975b199-fa3c-4cc8-8257-4e996e64addb", "fileName": "test-112.mrc" } ], "jobProfileId": "6f7f3cd7-9f24-42eb-ae91-91af1cd54d0a", "jobProfileName": "Default job profile", "progress": { "exported": 25, "failed": 0, "total": "25" }, "completedDate": "2020-10-07T13:18:20.364+0000", "startedDate": "2020-10-07T13:18:19.852+0000", "runBy": { "firstName": "admin", "lastName": "admin" }, "status": "SUCCESS" }

    Polling Logic

    • Continue polling until:

      • status == SUCCESS or status == FAIL

    • Recommended:

      • Add a timeout to prevent infinite loops

      • Example: max polling duration of N minutes

    ✅ Outcome

    • Capture:

      • id to be used later as jobExecutionId

      • fileId

    Step 6: Download the Exported File

    Once the job completes successfully, download the export file.

    API Request

    GET /data-export/job-executions/{jobExecutionId}/download/{fileId}

    Response

    { "fileId": "b975b199-fa3c-4cc8-8257-4e996e64addb", "link": "https://<download-link>" }

    ✅ Final Step

    • Use the returned link to download the file