Overview

Most of the finance schemas are well out of date and need to be updated.  Recent clarifications on requirements have also introduced new complexities and challenges that need to be overcome.  Given the scope of these changes and how interrelated the various schemas and APIs will be, creating a wiki page to capture this information seemed to make sense.

Schemas

Schemas exist for most of the financial records, but will need to be updated as they're out of date.  The Acquisitions Interface Fields maybe be helpful to some degree, but isn't a direct 1-to-1 mapping with the schemas due to intricacies of the data model, e.g. internal/storage use are needed which aren't captured in that spreadsheet.

Acquisition Units

The complex relationships between the various record types makes it difficult to use an inheritance pattern like we've used in other apps.  For this reason, most finance records will have an acquisitionUnits field.  How these are applied is slightly different as well.  The fiscal year detail view for instance displays ledgers, groups and funds (budgets).  In order to keep things reasonably simple, if the user has the folio permission to view the fiscal year, and belongs to one of the acquisition units assigned to the fiscal year record, the ledger, group and fund (budget) data can also be viewed.  The same concept applies across the board - e.g. ledgers detail view shows groups and funds (budgets).  We will need to get creative with our APIs and database views to make this work.

Transactions

• Immutable
• Payments, credits, allocations, transfers, encumbrance (see below)
• Each of these will have their own API to allow for business logic rules, but may share a table in the storage layer.
• Since there isn't a directly link to a budget, Querying by toFundId, fromFundId, and fiscalYearId is needed to satisfy:
  • GET transactions by Budget
• Single transactions table, shared by multiple APIs

Schemas

transaction

Encumbrances

encumbrance

awaiting_payment

invoice_transaction_summary (deprecated in Quesnelia)

temporary_invoice_transactions (deprecated in Quesnelia)

order_transaction_summary (deprecated in Quesnelia)

temporary_order_transactions (deprecated in Quesnelia)

APIs

Business Logic Module

Storage Module

Encumbrance as a Transaction

• needs to be viewed as a transaction, but can't be immutable 
•  Option A: make encumbrance an superset of transaction, store in separate table, and use views/joins to satisfy search/filter needs.
  • Some of the field names are unintuitive - "amount" for instance is vague and potentially confusing given that we have other fields: "initalAmountEncumbered", "amountAwaitingPayment", and "amountExpended"
•  Option B: make encumbrance a special-case of transaction.  This involves adding a bunch of optional fields to the transaction schema which would only be used for the encumbrance type of transaction.  
  • Putting these extra fields in an "encumbrance" sub-object might make sense - would allow for some schema validation to be used instead of needing to do it all in code.
  • This would allow us to store all transaction types in a single table and avoid having to deal with views/joins and confusing field names.
  • Need to implement a PUT endpoint for transactions, to support this approach, but can restrict its use (in code) to only transactions of type "encumbrance".  I don't like putting even this much business logic in the storage layer, but it's fairly limited and allows for a cleaner design.
    • See below the API table about possibly not needing this endpoint.

Updating Allocated/Unavailable/Available and Other Running Totals

• Option A:  Have the business logic module perform calculations as money is moved
  • Harder to guarantee consistency/accuracy - is eventual consistency OK for finances?
  • Keeps business logic out of the storage module, consistent with approach taken in other acquisition apps
• Option B:  Perform summary/total calculations in the storage module and update all tables within a transaction.  Either they all succeed, or the entire operation fails
  • Much better control over consistency/accuracy
  • Bleeds business logic into the storage module
  • Other business logic remains in mod-finance - e.g. acquisitionUnits, what can be modified vs what can't, etc.
• Option C:  Calculate these numbers on-the-fly in the business logic module
  • Difficult/inefficient to do if these numbers are needed to be shown in search results (they are)
  • Likely means we'll need a more complicated data model with more views and schemas
  • Since we're calculating these numbers on the fly, consistency is less of a problem... the numbers only live in one place
• Option D: Hybrid of B and C
  • Calculate most of these totals in the storage module and update all tables in a transaction
  • For some special cases, we need to perform calculations on the fly
    • e.g. GET groups for a ledger/FY - Only sum the amounts from the funds in those groups which are part of the ledger for this FY...

Preventing Partially Paid Invoices (deprecated method)

When an invoice is paid, numerous transactions (payments, credits) will be generated.  Either all or none of these need to be successfully processed; we can't have partially paid invoices.  This presents a challenge of scale.  There could feasibly be hundreds of payments for an invoice.  Several options were weight and the most attractive approach involves collecting or aggregating payments/credits in the storage layer until all are received, then processing them all at once in a database transaction.  The other options are listed in the appendix.

This approach keeps the payment "simple" and consistent with the other transaction APIs.  The storage layer performs some aggregation in the database of payments/credits based on the sourceInvoiceId specified in each payment.  The flow looks something like this:

Introduce new tables: 

• invoice_transaction_summary - Contains an invoiceId and the total number of transactions (payments/credits) expected when "paying" this invoice.
• temp_invoice_payments - Temporary storage of payments until all payments related to the invoice are received.

When an invoice is paid

• mod-invoice calls POST /finance/invoice-transaction-summary 
  • mod-finance calls mod-finance-storage to create the entry if one doesn't already exist
• mod-invoice generates payments and calls POST /finance/payments
  • mod-finance checks calls POST /finance-storage/payments

When an payment is received, 

• Persist the payment record in a temp_invoice_payments table
• Count the number of records in the temp_invoice_payments table.  If this is the last payment, (the number of entries in the temp table == numTransactions)
  • In a database transaction:
    • Apply the payments/credits from the temp_invoice_payments table and perform the summary # updates (e.g. update budget, ledger, encumbrances, etc. - unavailable/available/encumbered/expended/etc.)
      • If everything works, cleanup the temp_invoice_payments table and return 200 OK
      • If something fails, no actual records are updated.  The payment API call can fail and can be retried (POST payment/credit are idempotent)

When an "approved" invoice is removed:

• mod-invoice makes a call to mod-finance to ensure that the invoice_transaction_summary has been cleaned up.  These records are small so even if on is orphaned (which shouldn't happen often if ever) it's not a huge deal. 
• DELETE /finance-storage/invoice-transaction-summary/<invoiceId>

If a payment request fails

• mod-finance can safely retry it.
• After N failed tries an error can be returned to mod-invoice and/or the client who can later try to pay the invoice again once the (network?) problem is resolved.

"All or Nothing" Operations (deprecated method)

There are several situations in acquisitions where a bunch of transactions need to happen all at once.  Often the situation is such that either all of these transactions need to succeed, or none of them should.  We first crossed this in the context of preventing partially paid invoices (see above).  Fortunately, these cases are similar enough to one another that the design work and lessons learned during that work can be borrowed/reused.  There scenarios can be split into two categories:

Transactions applying to an invoice

• When approving an invoice, money needs to move from encumbered to awaitingPayment in both the encumbrances and the corresponding budgets.  It's also possible that encumbrances need to be released at this point, e.g. we encumbered $100 but the invoice is for $90, so we "release" $10 back to the budget's "available" pool. 
  • For tracking purposes we create pending payment for every fund distribution  
    • amount= fundDistribution amount
    • awaitingPayment.encumbranceId = fundDistribution.encumbrance
    • NOTE:  This means that the crossed-out statement below "The number of transactions should be the same for both of these events" is again true.  In order to avoid complicated data migraiton scripts, we're adding a new field to invoice-transaction-summary for indicating the number of new pending payment created upon invoice approval.  See the schema table for details.
    • NOTE:  This also simplifies the calculations that need to occur when applyling payments/credits as there should always be an associated pending payment.
  • In order to accomodate this we need to add awaitngPayment sub-object to transaction schema. 
• When paying an invoice, payments and credits are made.  See the earlier section "Preventing Partially Paid Invoices" for details.

Here I think we can re-use the "invoice_transaction_summary" and temporary transaction table almost as-is for both of these cases.  We might want to rename "invoice_transaction_summary" to a more generic "invoice_transaction_summary".  What this means is that a mod-invoice would create the invoice summary record when the invoice is approved instead of when the invoice is paid.  The number of transactions should be the same for both of these events The number of encumbrances might differ from the number of payments/credits, hence separate fields for the two counts.  It's important to note that the invoice shouldn't be changing after it's approved.  This is what allows us to calculate the number of payments/credits upon invoice approval.  When determining if all of the payments/credits have been received, we'll need to also look at the transaction type to essentially ignore the "encumbrance" transactions in the temporary table.  The temporary table can be cleaned up, but there's not guarantee on the timing of that, so we can't assume that only payments/credits will be present at the time of an invoice being paid.

Transactions applying to an Order

• When opening an order, encumbrances are created.  We need to ensure that all encumbrances for an order are created before opening an order.
• When closing an order, encumbrances need to be released.  That is if we encumbered money that we didn't end up spending, it needs to be "released" or moved back into the budget's "available" pool.

We looked at handling this in a similar fashion to creation of records in inventory; i.e. when opening the order we try to create all of the encumbrances, and if any of them fail, we return an appropriate error response and leave the order in "Pending" state.  However, it was determined that was not a viable approach since it opens the door to having encumbrances in the system for "pending" orders.  

The idea for handling this is similar to what we do above for invoices... 

mod-finance-storage

• Create an order_transaction_summaries table and API to sit in front of it
• Create a temporary table for holding encumbrances (transactions)
• Add a link to the order (POL?) in the encumbrance sub-object of transaction.  This is akin to sourceInvoiceId/sourceInvoiceLineId and is needed to accumulate all of the encumbrances associated with an order before applying changes.
• Once all encumbrances for an order are received, apply the changes in a DB transaction

These new tables will be used for both opening an order (create encumbrances) as well as closing an order (releasing encumbrances).  The encumbrance status field will help differentiate the transactions for each of these events in the temporary table.

Business Logic API changes

The current design includes a generic PUT /finance/encumbrances/<id> endpoint that would be sufficient for this approach.  However, this generic PUT endpoint doesn't really allow the caller to specify exactly what they're trying to do, making it difficult to properly validate the input.  By incorporating specific endpoints for the various actions, we clarify the context.  By having these endpoints take input that isn't a full blown transaction, we can save ourselves a lot of validation.

• POST /finance/awaiting-payment - takes a new schema w/ encumbranceId, amount.  Retrieves the specified encumbrance from storage, applies the change and sends the updated record back to storage.
• POST /finance/release-encumbrance/<id> - no request payload.  Retrieves the specified encumbrance, applies the changes and sends the updated record back to storage.

NOTE:  These endpoints are slightly different from the rest of the finance API in that they're command endpoints, and aren't very RESTful in their design.  That said, there is some precedence for this in several other places in acquisitions, e.g. mod-orders receiving/check-in endpoints

Batch all-or-nothing operations (new method)

See "Quesnelia changes to the transaction API". There are no transaction summaries or temporary transactions with the new endpoint.

Calculations

Some budget read-only fields (such as unavailable, overExpended or overEncumbered) are recalculated on the fly based on the stored fields. They do not need to be updated when transactions are applied.

For instance:

• totalFunding = allocated + netTransfers
• unavailable = encumbered + awaitingPayment + expended
• available = totalFunding - unavailable
• overExpended = max(0, expended + awaitingPayment - max(0, totalFunding))

Allocations

Update Budget identified by the transactions fiscal year (fiscalYearId) and source fund (fromFundId)

• If "fromFundId" is present,
  • allocated decreases by transaction amount
  • available decreases by transaction amount
  • recalculate overEncumbered
  • recalculate overExpended

Update Budget identified by the transaction's fiscal year (fiscalYearId) and the destination fund (toFundId)

• allocated increases by the transaction amount
• available increases by the transaction amount
• recalculate overEncumbered
• recalculate overExpended

Update LedgerFY identified by the transaction's fiscal year (fiscalYearId) and the source fund (fromFundId)

• If "fromFundId" is present
  • allocated decreases by transaction amount
  • available decreases by transaction amount

Update LedgerFY identified by the transaction's fiscal year (fiscalYearId) and the source fund (toFundId)

• allocated increases by the transaction amount
• available increases by the transaction amount

Transfers

Update Budget identified by the transactions fiscal year (fiscalYearId) and source fund (fromFundId)

• If "fromFundId" is present,
  • available decreases by the transaction amount
  • recalculate overEncumbered
  • recalculate overExpended

Update Budget identified by the transactions fiscal year (fiscalYearId) and source fund (toFundId)

• available increases by the transaction amount
• recalculate overEncumbered
• recalculate overExpended

Update LedgerFY identified by the transactions fiscal year (fiscalYearId) and source fund (fromFundId)

• If "fromFundId" is present,
  • available decreases by the transaction amount

Update LedgerFY identified by the transactions fiscal year (fiscalYearId) and source fund (toFundId)

• available increases by the transaction amount

Payments

Update the encumbrance identified by the transactions (paymentEncumbranceId)

• expended increases by transaction amount
• amount (effectiveEncumbrance) is recalculated

Delete related pending payment transaction

Update Budget identified by the transaction's fiscal year (fiscalYearId) and the source fund (fromFundId)

• Always
  • awaitingPayment decreases by transaction amount
  • expenditures increases by transaction amount 
  • recalculate overExpended

Credits

Update the encumbrance identified by the transactions (paymentEncumbranceId)

• expended decreases by transaction amount
• amount (effectiveEncumbrance) is recalculated

Delete related pending payment transaction

Update Budget identified by the transaction's fiscal year (fiscalYearId) and the destination fund (toFundId)

• Always
  • awaitingPayment increases by transaction amount
  • expenditures decreases by transaction amount
  • recalculate overExpended

Encumbrances

Upon Creation

• Update Budget identified by the transaction fiscal year (fiscalYearId) and the source fund (fromFundId)
  • encumbered increases by the transaction amount
  • available decreases by the transaction amount
  • unavailable increases by the transaction amount
  • overEncumbered is recalculated
• Update LedgerFY identified by the transaction fiscal year (fiscalYearId) and the source fund (fromFundId)
  • available decreases by the transaction amount
  • unavailable increases by the transaction amount

Upon Update

• ALWAYS DO THIS PART:

  • Update Budget identified by the transaction fiscal year (fiscalYearId) and the source fund (fromFundId)
    • TBD

  IN ADDITION TO ABOVE, DO THIS PART IF ENCUMBRANCE IS RELEASED:

  • If encumbrance.status = Released 
    • Update the encumbrance
      • transaction.amount becomes 0 (save the original value for updating the budget)
    • Update the budget identified by the transaction fiscal year (fiscalYearId) and the source fund (fromFundId)
      • encumbered decreases by the amount being released
      • available increases by the amount being released
      • unavailable decreases by the amount being released (min 0)
    • Update LedgerFY identified by the transactions fiscal year (fiscalYearId) and source fund (fromFundId)
      • available increases by the transaction amount (min 0)
      • unavailable decreases by the transaction amount

Pending payments

Upon Creation

• if awaitingPayment.encumbranceId specified
  • ALWAYS DO THIS PART:
    • Update Budget identified by the transaction fiscal year (fiscalYearId) and the source fund (fromFundId)
      • encumbered decreases by the transaction.amount
      • awaitingPayment increases by the same amount
    • update related encumbrance:
      • transaction(encumbrance).amount is updated to (transaction(encumbrance.)amount - transaction(pending payment).amount)
    • if pendingPayment.amout > encumbrance.amount
      • Update the budget identified by the transaction(encumbrance) fiscal year (fiscalYearId) and the source fund (fromFundId)
        • available decreases by the  pendingPayment.amout - encumbrance.amount
        • unavailable increases by the pendingPayment.amout - encumbrance.amount  (min 0)
        • recalculate overExpended?
  • IN ADDITION TO ABOVE, DO THIS PART IF ENCUMBRANCE IS RELEASED:
    • If encumbrance.status = Released 
      • Update the related encumbrance
        • transaction.amount becomes 0 (save the original value for updating the budget)
        • Update the budget identified by the transaction(encumbrance) fiscal year (fiscalYearId) and the source fund (fromFundId)
          • encumbered decreases by the amount being released
          • available increases by the amount being released
          • unavailable decreases by the amount being released (min 0)
        • Update LedgerFY identified by the transactions fiscal year (fiscalYearId) and source fund (fromFundId)
          • available increases by the transaction amount (min 0)
          • unavailable decreases by the transaction amount
• if awaitingPayment.encumbranceId is empty
  • Update budget identified by the transaction fiscal year (fiscalYearId) and the source fund (fromFundId)
    • available decreases by transaction amount (min 0)
    • unavailable increases by the transaction amount
    • awaitingPayment increases by the transaction.amount

  • Update LedgerFY identified by the transactions fiscal year (fiscalYearId) and destination fund (toFundId)

    • available increases by the transaction amount 
    • unavailable decreases by the transaction amount (min 0)

Example payment:

Budget:

allocated: 100

unavailable: 0

available: 100

encumbrance:

amount: 50

initialEncumbered: 50

Budget:

allocated: 100

unavailable: 50

available: 50

encumbered: 50

Pending payment:

amount: 51

encumbrance:

amount: 0

awaitingPayment: 51

initialEncumbered: 50

Budget:

allocated: 100

unavailable: 51 

available: 49

encumbered: 0

awaitngPayment: 51

Payment:

amount: 51

encumbrance:

amount: 0

awaitingPayment: 0

initialEncumbered: 50

expended: 51

Budget:

allocated: 100

unavailable: 51 

available: 49

encumbered: 0

awaitngPayment: 0

expended: 51

overExpended: rcalculate

Example credit:

Budget:

allocated: 100

unavailable: 0

available: 100

encumbrance:

amount: 50

initialEncumbered: 50

Budget:

allocated: 100

unavailable: 50

available: 50

encumbered: 50

Pending payment:

amount: -10

encumbrance:

amount: 50 + 10 = 60

awaitingPayment:  0

initialEncumbered: 50

Budget:

allocated: 100

unavailable: 40 

available: 60

encumbered: 60

awaitngPayment: 0

Credit:

amount: 10

encumbrance:

amount: 50

awaitingPayment: 0

initialEncumbered: 50

expended: -10

Budget:

allocated: 100

unavailable: 40 

available: 60

encumbered: 50

awaitngPayment: 0

expended: -10

overExpended: rcalculate

release:

encumbrance:

amount: 0 

awaitingPayment: 0

initialEncumbered: 50

expended: -10

status: released

Budget:

allocated: 100

unavailable: max(40 -50, 0) = 0 -10

available: 110

encumbered: 0

awaitngPayment: 0

expended: -10

overExpended: rcalculate

Without encumbrance:

Budget:

allocated: 100

unavailable: 0

available: 100

Pending payment:

amount: 50

Budget:

allocated: 100

unavailable: 50 

available: 50

encumbered: 0

awaitngPayment: 50

expended: 0

Payment:

amount: 50

Budget:

allocated: 100

unavailable: 50 

available: 50

encumbered: 0

awaitngPayment: 0

expended: 50

overExpended: rcalculate

Transaction Restrictions

It's often necessary to check if a transaction should be allowed to happen based on the current budget values.  This section explains these restrictions and how they're applied.

Payments

There are several factors that come into play here.  The flowchart below shows how we determine if a payment can be accepted or should be rejected.  

The "Remaining Allowable Exp." in the flowchart is calculated as follows:

[remaining amount we can expend] = (totalFunding * allowableExpenditure) - unavailable + relatedEncumbranceForPendingPayment

[remaining amount] = (allocated + netTransfers) * allowableExpenditure - (awaitingPayment + expended)

When restricted expenditures are enabled, the remaining amount must remain positive or equal to 0.

The "restrictExpenditures" flag actually lives in the ledger, so we'll need to find traverse:  budget → fund → ledger.restrictExpenditures  

Encumbrance

Creating encumbrances requires a similar set of checks as described above for payments.

The "Remaining Allowable Enc." in the flowchar is calculated as follows:

[remaining amount we can encumber] = (totalFunding * allowableEncumbered) - unavailable

[remaining amount] = (allocated + netTransfers) * allowableEncumbered - (encumbered + awaitingPayment + expended)

When restricted encumbrances are enabled, the remaining amount must remain positive or equal to 0.

The "restrictEncumbrances" flag actually lives in the ledger, so we'll need to find traverse:  budget → fund → ledger.restrictEncumbrances  

Tags

Depending on the type of transaction, tags are in some cases inherited from another record.

• Payments / Credits
  • Inherit the tags that were associated with the invoice line
• Encumbrances
  • Inherit the tags that were associated with the order line if one is associated with the encumbrance
  • DB: Yes. Otherwise, Inherit the tags that were associated with the invoice line.Ann-Marie Breaux (Deactivated) or Dennis Bridges to clarify this
• Allocations / Transfers
  • Don't inherit tags from anywhere - the user should be allowed to assign tags at the point of creating the allocation or transfer

FundTypes

• Simple controlled vocabulary

Schemas

fund_type

APIs

Business Logic Module

Storage Module

Funds

• Record with info that doesn't change from FY to FY (name, code, acq. units, etc.)
• Foreign Keys;
  
  • funds.ledgerId = ledgers.id

Schemas

fund

compositeFund

This schema is essentially a fund w/ one additional field - groupIds.  The business logic module's fund APIs will use this schema, and will manage the fund and group/fund relationships independently via calls to the storage APIs.  This simplifies things on the UI end.