FYRO Error Troubleshooting Knowledge Sharing

#1 Rollover poLines by chunks failed (Encumbrance should be released before deletion in mod-orders logs) - Nolana

Error description

Error is connected to order rollover process and incorrect status of encumbrances for closed orders - they should be released or even do not exist in new FY

Investigation

From logs it's clear that transactions for closed orders should be removed during order rollover, but as they have unreleased status they can not be removed. The reason why closed orders have unreleased encumbrances in new FY - incorrect encumbrance.orderStatus, they have Open instead of Closed.

Steps taken to fix the issue

Run fix_encumbrances script, after that rollover shouldn't fail with "Encumbrance should be released before deletion" error

IMPORTANT

1. If you are on testing env and can restore state before the rollover, fix_encumbrances should be run for current FY before the rollover
2. If you did a rollover and it failed, fix_encumbrances should be run for rolled FY (next one), after that order rollover process should be started manually via API (POST /orders/rollover)

#2 duplicate key value violates unique constraint "transaction_pkey"

Error description

duplicate key value violates unique constraint "transaction_pkey"

This error is thrown on DB level when FYRO script tries to create transaction with already existing UUID.

Investigation

During investigation we found that old order lines (previous issues with incorrect links between order line and encumbrances) migrated encumbrances from previous FY to current FY as a result in one FY there were base encumbrance (previous FY) and rolled over encumbrance (current FY). As we use encumbrance UUID to create new encumbrance UUID having base in the same FY produced an error to create already existing UUID.

Results

After fixing encumbrances (moving them back to previous FY or removing them) allowed to run FY without errors

Steps taken to fix the issue

Next SQL helps to identify duplicated encumbrances that need to be fixed: moved back to previous FY or removed

SELECT
  tr.fromfundid,
  jsonb_extract_path_text(tr.jsonb, 'encumbrance', 'sourcePoLineId') as sourcePoLineId,
  jsonb_extract_path_text(tr.jsonb, 'fiscalYearId') as trFiscalYearId,
  tr.expenseclassid as expenseClass,
  count(*)
FROM {tenant}_mod_finance_storage.transaction as tr
LEFT JOIN {tenant}_mod_finance_storage.fund as fund
ON fund.id = tr.fromfundid
WHERE
  tr.jsonb->'encumbrance'->>'sourcePoLineId' IS NOT NULL
  AND tr.jsonb->'encumbrance'->>'orderStatus' <> 'Closed'
  AND tr.fiscalyearid = {fiscalyearid}
GROUP BY tr.fromfundid, trFiscalYearId, sourcePoLineId, expenseClass
HAVING count(*) > 1;

> fiscalyearid is id (UUID) of FY library rollovers from
> tenant is tenant id

After duplicate encumbrances are fixed, fix_encumbrances should be executed to update links, recalculate budgets etc

1. run SQL to find duplicates
2. SQL returns sourcePoLineId so you need to review transactions connected to this order line and its fund distribution (funds, expense classes, etc) to decide remove or move back to previous year:
   To decide what action apply (remove or move back) additional information is required:
   
   0. Find order line id from transaction (run when order line is unknown)
   
   

   select jsonb->'encumbrance'->>'sourcePoLineId' as sourcePoLineId
   from {tenant}_mod_finance_storage.transaction
   where id in ({transactionIds});
   
   > transactionIds - list of transaction UUIDs to analize
   > tenant is tenant id

   
   1. Find transactions for order line in FY rollover from
   
   

   SELECT id, jsonb, creation_date, created_by, fiscalyearid, fromfundid, sourcefiscalyearid, tofundid, expenseclassid
   FROM {tenant}_mod_finance_storage.transaction
   WHERE jsonb->'encumbrance'->>'sourcePoLineId' in ({...sourcePoLineIds})
     AND fiscalyearid = '{fiscalyearid}';
   
   > sourcePoLineIds - list of order line UUIDs to analize
   > fiscalyearid is id (UUID) of FY library rollovers from
   > tenant is tenant id

   
   2. Find POL fund distributions
   
   

   SELECT id, jsonb->>'fundDistribution' as fundDistribution
   FROM {tenant}_mod_orders_storage.po_line
   WHERE id in ({...sourcePoLineIds});
   
   > tenant is tenant id
   > sourcePoLineIds - list of order line UUIDs to analize

   
   
   
   1.  move back - I rely only on this query. If this query returns already existing UUID, it means that this transaction is from past
      > SELECT public.uuid_generate_v5(public.uuid_nil(), concat('BER1', {transactionId}))
      IMPORTANT: only 2.a is enough to fix "duplicate key constraint error", 2.b is used to have right values and get rid of data inconsistency
   2. when you see that there are still duplicates, you can decide what to remove based on order line fund distribution
      
      
      
3. run fix_encumbrances script in Dry run mode to find duplicates (2 and 3 steps report them)
4. repeat #2 for found duplicated lines
5. run script in normal mode

#3 Your FOLIO system is configured to limit the number of PO Lines on each order - Nolana

Error description

Error is connected to the count of requests made by mod-orders to mod-orders-storage during order rollover process - Some order line update requests are failed.

Investigation

It was found that error can occur in Nolana, in Orchid another approach to send requests is used. To fix it special node.js script should be executed to parse logs of failed order lines requests

Steps taken to fix the issue

1. Collect logs from mod-orders module in range when order rollover was running and by messages that contains Exception calling PUT /orders-storage/po-lines

   
   

   Logs should be exported in CSV or TXT formats.
2. Follow instructions to run post_order_rollover script
3. Optional: Once script is finished and there are no other errors reported during Rollover, Rollover progress status can be changed to Success in DB.