Page Comparison

...

1. The current implementation resides in the mod-data-export-worker module. This functionality must be moved into a separate module to create a sustainable basis for further feature development.
2. Expected volumes of data to be processed:
   1. there is a confluence page that contains the volumes of data that are processed by the different institutions. Volumes vary greatly by institution. Bulk Edit - edited data size
      
   2. For the Nolana Orchid release, the Bulk Operation application must support the processing of up to 50,000 records by one operation.
3. Expected execution time:
   1. It is clear that the processing of big datasets could take some significant time, up to several hours. For the Nolana Orchid release, there is no strict requirement for the execution time. This discussion will continue after we get real results from the production environment for the Nolana Orchid release. 

Functional requirements

...

It is expected that back-end storage modules will implement optimistic locking (optimistic concurrency controls) for update operations. So the Bulk Operations application does not handle special cases when the records change between their selection, the modifications are chosen, and the edits are being applied. In the cases when target storage modules do not provide optimistic locking capabilities, there is a possibility for lost changes. Any additional effort related to adding optimistic locking to the back-end storage modules is out of scope for the Bulk Operation application. 

The same behavior is expected from special APIs that should be used for update operations (e.g. changing the due date).  It is expected from the API provider (a back-end module) to correctly handle "outdated" requests.

...

New

Retrieving records

Saving records locally

Data Modification

Review Changes

Apply Changes

Suspended

Completed

Completed with errors

Cancelled

Scheduled 

Failed

...

General Flow Diagram part 1:

Image RemovedImage Added

General Flow Diagram part 2:

Image RemovedImage Added

This ER diagram represents the data model for the new module.

Image RemovedImage Added

• bulk_operation - this is the main table that represents any Bulk Operation.

• buld_operation_rule - this table contains the rules that are/will be applied to a particular Bulk Operation in the case of the In-App approach for data modification.

• buld_operation_rule_details - this table contains the rules details

• bulk_operation_data_processing - this table represents the state of the data processing operation when records retrieved from the Export Manager are processed according to the rules defined for a particular Bulk Operation. In the case of errors, this operation could be resumed.

• bulk_operation_execution - this table represents an attempt to apply the changes to the target storage module. The changes are applied in chunks.

• bulk_operation_execution_chunk - this table represents the chunk of the modified records that are sent to the target storage module to be stored.

• bulk_operation_execution_content - this table holds the information for every particular modified record regarding if it has been successfully updated or not.

  
  

This component diagram represents the basic high-level structure for the new back-end module.

Image RemovedImage Added

• Bulk Operation Controller - this component represents all end-points for the Bulk Operation beck-end module.
• Bulk Operation Service - it contains the implementation of all business logic for the Bulk Operation application.
• Error Handler Service - this component is responsible for error handling and storing errors in the database for further audit/analysis
• Thread Pool Executor - the component used by the Bulk Operation Service to run asynchronous tasks 
• Kafka consumer - this component receives messages from the Data Export Manager it sends on completing the data export job. Once receives a message, the Kafka consumer passes control to the Bulk Operation Service.
• Data Export Client - it encapsulates the details of the HTTP interaction with the Data Export Manager.
• Remote File System Client - it encapsulates the details of interaction with S3-like storage that is available to all instances of the module.
• Bulk Operation Repository - it is a set of JPA repositories used to deal with the database.
• Mod Client - it is a set of clients that encapsulate the details of HTTP interaction with storage modules like mod-users, mod-inventory-storage, etc.
• Mod Client<T> - it is a generalized interface used by the Bulk Operation Service to retrieve actual data from different storage modules.
• Mod Client Adapter - this is a set of adapters used to cast Mod Clients to Mod Client<T> for Bulk Operation Service.
• Mod Client Adapter Factory - it is a component used to produce Mod Client Adapters by entity type.
• Data Processor<T> - this is a generalized interface used by the Bulk Operation Service to process a file with records using the rules specified by a user.
• Data Processor - this is a set of components that implement the logic for file processing for different entities.  
• Data Processor Factory - this is a component used to produce Data Processors by entity type.
• Process Callback - this interface is used by the Data Processors to update the progress of processing. Bulk Operation Service implements this interface.
• Update Processor<T> - this is a generalized interface used by the Bulk Operation Service to update records in storage modules.
• Update Processor - it is a set of components that implement the logic for updating records in storage modules using a file as a source. Each Update Processor uses a corresponding Mod Client to perform HTTP calls to an appropriate backend module.
• Update Processor Factory - this is a component used by the Bulk Operation Service to produce Update Processors by entity type.

Unified table representation of data.

For the sake of simplicity, the back-end module uses the unified data structure to represent data for preview. It means that the UI application does not need to know the exact data type and/or data structure used to represent a particular type of entity. It simply receives the unified tabular data structure and renders the preview table for a user.

Below is the class diagram that represents the data structures used for that.

Image Added

It makes sense to consider the usage of Query tool data structure to represent the results and preview to users.

BuklEditController implementation details

BuklEditService implementation details