Batch Importer (Bib/Acq) (UXPROD-47)

[UXPROD-656] Data Import (Batch Importer for Bib Acq) Infrastructure and Planning Created: 25/May/18  Updated: 08/Jun/21  Resolved: 22/Jan/19

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: Q4 2018
Parent: Batch Importer (Bib/Acq)

Type: New Feature Priority: P3
Reporter: Ann-Marie Breaux (Inactive) Assignee: Ann-Marie Breaux (Inactive)
Resolution: Done Votes: 0
Labels: acq-dataimp, data-import, delimited_files, marcimport, split
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Continues
is continued by UXPROD-1656 Data Import (Batch Importer for Bib A... Closed
Defines
defines UXPROD-47 Batch Importer (Bib/Acq) Analysis Complete
Relates
relates to MODDATAIMP-48 SPIKE: Create BackEnd Documentation f... Closed
relates to MODDATAIMP-50 Add DELETE endpoint fo Upload Definit... Closed
relates to MODDATAIMP-51 SPIKE: test our upload implementation... Closed
relates to MODDATAIMP-54 Add "uploadedDate" to the file model Closed
relates to MODSOURCE-18 Prepare module for release Closed
relates to MODSOURCE-21 Fix security vulnerabilities reported... Closed
relates to MODSOURMAN-41 Add endpoints for updating JobExecuti... Closed
relates to UIDATIMP-70 Release v1.0.0 for Q4 2018 Closed
relates to UXPROD-1376 CRUD for File Extensions Closed
relates to UXPROD-1377 CRUD for Job Profiles Closed
relates to UXPROD-1378 CRUD for Match Profiles Closed
relates to UXPROD-1379 CRUD for Action Profiles Closed
relates to UXPROD-1381 CRUD for EDIFACT Invoice Field Mappin... Closed
relates to UXPROD-2074 Bring Action Profiles to life: prep work Closed
relates to UXPROD-2075 MARC Bib Field Mapping Profiles Detai... Closed
relates to UXPROD-2209 Bring Action Profiles to life: next work Closed
relates to MODDATAIMP-1 Initial design for Batch Loader flow Closed
relates to MODDATAIMP-2 Batch Loader: Create Folio app skeleton Closed
relates to MODDATAIMP-3 SPIKE: PoC - usage of org.marc4j library Closed
relates to MODDATAIMP-4 Spike: Figure out relationship of Fol... Closed
relates to MODDATAIMP-5 SPIKE: Batch Loader Overview Closed
relates to MODDATAIMP-6 UI: Search panel Closed
relates to MODDATAIMP-10 SPIKE: UI: Upload component Closed
relates to MODDATAIMP-11 UI: Drag&Drop Closed
relates to MODDATAIMP-12 UI: Data import snapshot result Closed
relates to MODDATAIMP-13 UI: Results page Closed
relates to MODDATAIMP-14 Create mod-data-import module Closed
relates to MODDATAIMP-16 SPIKE: Investigate possibility to cal... Closed
relates to MODDATAIMP-19 Create sample module using vertx-rx-java Closed
relates to MODDATAIMP-22 SPIKE: Design Backend functionality t... Closed
relates to MODDATAIMP-24 UI: Drag&Drop: nest into panel Closed
relates to MODDATAIMP-25 UI: Drag&Drop: Configuration Closed
relates to MODDATAIMP-26 UI: Drag&Drop: Markup Closed
relates to MODDATAIMP-27 Create activify diagrams for 4 usecas... Closed
relates to MODDATAIMP-28 Add CRUD API for file upload and a up... Closed
relates to MODDATAIMP-29 Add services to to store and get uplo... Closed
relates to MODDATAIMP-31 Use description fields in RAML JSON s... Closed
relates to MODDATAIMP-32 Refine FOLIO-Data-Import-HighLevel-Ar... Closed
relates to MODDATAIMP-33 Create service's skeleton for upload ... Closed
relates to MODDATAIMP-34 Add implementation to UploadDefintion... Closed
relates to MODDATAIMP-35 Add implementation to File Upload\Del... Closed
relates to MODDATAIMP-36 Automate new modules’ registration wi... Closed
relates to MODDATAIMP-37 Security vulnerability reported in ja... Closed
relates to MODDATAIMP-38 Create sample data to return JobExecu... Closed
relates to MODDATAIMP-39 Create service for file chunking Closed
relates to MODDATAIMP-41 Add implementation to create job exec... Closed
relates to MODDATAIMP-42 Create sample data for the JobExecuti... Closed
relates to MODDATAIMP-43 Create sample data for the Log entity Closed
relates to MODDATAIMP-44 Create common RAML schema storage git Closed
relates to MODDATAIMP-45 Implement MARC file reader for chunki... Closed
relates to MODDATAIMP-46 Change module configuration to use Lo... Closed
relates to MODDATAIMP-47 Make a description of current work wi... Closed
relates to MODDATAIMP-49 Prepare module for the release Closed
relates to MODDATAIMP-52 Create Profile entity Closed
relates to MODDATAIMP-53 Create release branches for modules o... Closed
relates to MODDATAIMP-55 422 Error during POST /change-manager... Closed
relates to MODDATAIMP-56 Delete file from storage on DELETE fi... Closed
relates to MODDATAIMP-57 Create shared library with data utils... Closed
relates to MODDATAIMP-58 Create shared library with data utils... Closed
relates to MODDATAIMP-64 Fix security vulnerabilities reported... Closed
relates to MODDATAIMP-66 SPIKE: Become familiar with Apache Ka... Closed
relates to MODDATAIMP-88 Data Import architecture review prepa... Closed
relates to MODDATAIMP-89 "Data Import" use cases analysis. Closed
relates to MODDATAIMP-100 Prepare "Data Import" first stage demo. Closed
relates to MODDATAIMP-101 Don't work ParallelFileChunkingProcessor Closed
relates to MODDATAIMP-102 Update documentation for file upload API Closed
relates to MODDATAIMP-103 JobProfileInfo don't updates for job ... Closed
relates to MODSOURCE-1 Create mod-source-record-storage module Closed
relates to MODSOURCE-2 Create REST endpoint to return a list... Closed
relates to MODSOURCE-4 Create REST endpoint to return a list... Closed
relates to MODSOURCE-6 Add CRUD API for Snapshot at mod-sour... Closed
relates to MODSOURCE-7 Add CRUD API for Record entity at mod... Closed
relates to MODSOURCE-8 Add GET method for Result entity at m... Closed
relates to MODSOURCE-13 Add metadata to Result entity Closed
relates to MODSOURCE-14 Change totalRecords from recordsColle... Closed
relates to MODSOURCE-15 Spike: Investigate population of meta... Closed
relates to MODSOURCE-19 Create distributable HTTP Client for ... Closed
relates to MODSOURCE-23 Create endpoint Closed
relates to MODSOURCE-24 Create stub snapshot if module deploy... Closed
relates to MODSOURCE-25 Implement endpoint Closed
relates to MODSOURCE-26 Provide examples on how to use the en... Closed
relates to MODSOURMAN-1 Create Source records Manager module Closed
relates to MODSOURMAN-3 Create technical design for Change Ma... Closed
relates to MODSOURMAN-4 Create a RAML definition and endpoint... Closed
relates to MODSOURMAN-5 Make a design for creation and regist... Closed
relates to MODSOURMAN-6 Make a design for receiving and proce... Closed
relates to MODSOURMAN-7 MetadataRepository: Implement API for... Closed
relates to MODSOURMAN-8 Extend JobExecution entity with a new... Closed
relates to MODSOURMAN-9 Create ChangeManager component, expos... Closed
relates to MODSOURMAN-11 Create ChangeEngine component for par... Closed
relates to MODSOURMAN-18 Create endpoint for ChangeManager to ... Closed
relates to MODSOURMAN-19 ChangeManager: API for receiving chun... Closed
relates to MODSOURMAN-23 Implement methods for MetadataRepository Closed
relates to MODSOURMAN-24 Prepopulate repository with sample Jo... Closed
relates to MODSOURMAN-25 Change module configuration to use Lo... Closed
relates to MODSOURMAN-26 Create SQL script to prepopulate JobE... Closed
relates to MODSOURMAN-27 Change module structure to the common... Closed
relates to MODSOURMAN-28 Create RAML description for the new R... Closed
relates to MODSOURMAN-29 REST API implementation Closed
relates to MODSOURMAN-30 "Log" should be a DTO not a model object Closed
relates to MODSOURMAN-31 Change data type from "String" to "Da... Closed
relates to MODSOURMAN-32 Add a new field "UI Status" to the jo... Closed
relates to MODSOURMAN-34 Prepare module for the release Closed
relates to MODSOURMAN-35 Apply backend sorting for the Log ent... Closed
relates to MODSOURMAN-36 Select only COMMITTED JobExecutions (... Closed
relates to MODSOURMAN-37 Make request param "landingPage" opti... Closed
relates to MODSOURMAN-38 Not enough permissions to call Snapsh... Closed
relates to MODSOURMAN-40 Request to GET Jobs is failed Closed
relates to MODSOURMAN-43 Add endpoint for setting JobProfileNa... Closed
relates to MODSOURMAN-44 Optimize code in source-record-manager Closed
relates to MODSOURMAN-45 Eliminate potential problems with Job... Closed
relates to MODSOURMAN-46 Validate status on JobExecution statu... Closed
relates to MODSOURMAN-47 Exclude DISCARDED JobExecutions from ... Closed
relates to MODSOURMAN-48 Override default limit value of retur... Closed
relates to MODSOURMAN-101 Data Import online API road map (with... Closed
relates to UIDATIMP-4 Create landing page for Data Import a... Closed
relates to UIDATIMP-5 Create data import settings page 1: p... Closed
relates to UIDATIMP-44 Spike: Investigate the way to impleme... Closed
relates to UIDATIMP-49 Spike: Investigate state management o... Closed
relates to UIDATIMP-52 Change contracts Closed
relates to UIDATIMP-74 Add document icon to upload file items Closed
relates to UIDATIMP-75 Setup BigTest environment and write t... Closed
relates to UIDATIMP-78 Mock backend for testing Closed
relates to UIDATIMP-81 Translation problem on Data Import la... Closed
relates to UIDATIMP-82 Change jobExecutions status to much u... Closed
relates to UXPROD-130 Parse MARC21 format data records Closed
relates to UXPROD-1380 CRUD for MARC Field Mapping Profiles,... Closed
relates to UIDATIMP-71 Hide popover when user clicks on the ... Closed
relates to UXPROD-1382 CRUD for Delimited Data Mapping Profiles Draft
relates to UXPROD-1405 Data Import (Batch Importer for Bib A... Draft
relates to UIDATIMP-37 Create UI for File Upload, Part 3: Ac... Closed
relates to UIDATIMP-63 Creating UI for File Upload. When fil... Closed
relates to UIDATIMP-65 SPIKE: Create Frontend Documentation ... Closed
relates to UIDATIMP-67 Creating UI for File Upload. User nav... Closed
relates to UIDATIMP-72 Creating UI for File Upload. Deleting... Closed
relates to UIDATIMP-1 Batch Loader: Create Stripes app Closed
relates to UIDATIMP-2 Batch loader initial stripes app stru... Closed
relates to UIDATIMP-3 Setup Jenkins CI loop for ui-data-imp... Closed
relates to UIDATIMP-19 Data Import landing page Scenario 1: ... Closed
relates to UIDATIMP-20 Data Import landing page Scenario 2: ... Closed
relates to UIDATIMP-21 Data Import landing page Scenario 3: ... Closed
relates to UIDATIMP-22 Data Import landing page Scenario 4: ... Closed
relates to UIDATIMP-23 Data Import landing page Scenario 5: ... Closed
relates to UIDATIMP-24 Data Import landing page Scenario 6: ... Closed
relates to UIDATIMP-25 Data Import landing page Scenario 7: ... Closed
relates to UIDATIMP-26 Data Import landing page Scenario 8: ... Closed
relates to UIDATIMP-27 Data Import landing page Scenario 9: ... Closed
relates to UIDATIMP-28 Data Import landing page Scenario 10:... Closed
relates to UIDATIMP-29 Data Import landing page Scenario 11:... Closed
relates to UIDATIMP-31 Create a reusable stripes component f... Closed
relates to UIDATIMP-32 Gather UI related questions Closed
relates to UIDATIMP-33 Create UI for File Upload, Part 1: In... Closed
relates to UIDATIMP-35 Create UI for File Upload, Part 2: St... Closed
relates to UIDATIMP-38 Create UI for File Upload, Part 4: St... Closed
relates to UIDATIMP-39 Create UI for File Upload, Part 5: Ac... Closed
relates to UIDATIMP-40 Create End-of-list shared component Closed
relates to UIDATIMP-43 Data Import landing page Scenario 12:... Closed
relates to UIDATIMP-46 Create UI for File Upload, Part 4: St... Closed
relates to UXPROD-1220 Ability to assign Tags to Data Import... Closed
relates to UIDATIMP-50 Does not show proper app label in app... Closed
Epic Link: Batch Importer (Bib/Acq)
Analysis Estimate: Large < 10 days
Analysis Estimator: Niels Erik Nielsen
Front End Estimate: XL < 15 days
Front End Estimator: Niels Erik Nielsen
Front-End Confidence factor: Low
Back End Estimate: XXL < 30 days
Back End Estimator: Niels Erik Nielsen
Estimation Notes and Assumptions: As I understand the issue description there might be a UI where the user can configure rules for routing as well as parsing of incoming data.

I have no immediate basis for gauging the level of complexity in the bullet points, but it appears to me that it's potentially very complex.

I'm not sure exactly what profiles are.

There's no immediate indication as to how much analysis has already been done and how much is left.

For these reasons estimates are set relatively high and confidence factor to low.
Development Team: Folijet
Rank: Chalmers (Impl Aut 2019): R2
Rank: Chicago (MVP Sum 2020): R1
Rank: Cornell (Full Sum 2021): R1
Rank: 5Colleges (Full Jul 2021): R1
Rank: GBV (MVP Sum 2020): R1
Rank: Lehigh (MVP Summer 2020): R1
Rank: TAMU (MVP Jan 2021): R1
Rank: U of AL (MVP Oct 2020): R1

 Description   

Help the user set rules and profiles for dealing with incoming batch loads:

  • what type of incoming file
  • from what source
  • what FOLIO apps/record types should be affected
  • how to parse/map incoming data
  • create profiles related to the parsing & matching rules, including defaults
  • be able to mix and match pieces of profiles (or subprofiles) to create overall profiles - with varying mapping rules, matching rules, defaults, etc
  • matching rules and hierarchies, and what happens (create new, overlay, merge, delete or mark for deletion)
  • package up the rules (profile) and data to deliver to the appropriate FOLIO loader apps, to CRUD the proper kinds of records

Key elements:

  • Apps
    • ui-data-import (only landing page in this epic; other pages will come in subsequent epics)
    • mod-data-import, including file upload component
    • mod-source-record-storage
    • mod-source-record-manager
    • File upload component
  • UI:
    • App landing page, with preview and running jobs, most recent 25 imports, file upload area
    • Outline of settings pages for file extensions, job profiles, match profiles, action profiles, data mapping profiles
    • File upload component

See https://docs.google.com/drawings/d/1IN2t0T838HtHytNctdugQN0nL8_KSJ-PzysQxSyZ1Hc/edit for a schematic



 Comments   
Comment by Ann-Marie Breaux (Inactive) [ 03/Dec/18 ]

Broke a number of elements into separate features
UXPROD-1376 Closed file extension settings
UXPROD-1377 Closed CRUD for job profiles
UXPROD-1378 Closed CRUD for mapping profiles
UXPROD-1379 Closed CRUD for action profiles
UXPROD-1380 Closed CRUD for MARC mapping profiles
UXPROD-1381 Closed CRUD for EDIFACT mapping profiles
UXPROD-1382 Draft CRUD for delimited mapping profiles

Generated at Fri Feb 09 00:09:25 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.