SPIKE: Closing declared lost loan (Lost and paid status) for set cost

while not ideal, I think hosting providers could come up with some sort of log monitoring to detect errors in pub sub modules (or any errors in general) and send an alert to interested parties.

Assuming that an error produces a log entry.

Some logs are next to impossible to use for visually diagnosing issues, so we (EBSCO) are likely getting to this stage anyway unless we revisit logging in FOLIO (which I am an advocate for, since, even with log monitoring tools, multi-tenant logs are still a problem).

I think this leads us into the bigger topic of operational observability, which I agree is an important topic that FOLIO likely needs to invest in at some point.

We currently have a similar situation with Okapi timer tasks where they can fail silently and remain undetected. For example, the session time out timer task was failing with a server side error for the first couple of weeks in the Chalmers deployment of Edelweiss due to a property type mismatch in the JSON config.

Indeed, the Okapi timer interface is used to implement scheduled tasks (some older tasks use RAML Module Builder's internal scheduler). I agree, this other kind of asynchronous process also has challenges with making folks aware of failures and compensating for them.

I just happened to notice a persistent 5xx error in a cloudwatch chart while I was looking into another error. The system appeared to work fine and no one seemed to notice that their circ sessions were not timing out automatically.

This is a key challenge, it is difficult for folks to notice that something didn't happen. Testing for this can be challenging (I discussed it briefly on the pull request to resolve this issue, as the defaulting approach is still hard to observe)

, while not ideal, I think hosting providers could come up with some sort of log monitoring to detect errors in pub sub modules (or any errors in general) and send an alert to interested parties. Some logs are next to impossible to use for visually diagnosing issues, so we (EBSCO) are likely getting to this stage anyway unless we revisit logging in FOLIO (which I am an advocate for, since, even with log monitoring tools, multi-tenant logs are still a problem).

We currently have a similar situation with Okapi timer tasks where they can fail silently and remain undetected. For example, the session time out timer task was failing with a server side error for the first couple of weeks in the Chalmers deployment of Edelweiss due to a property type mismatch in the JSON config. I just happened to notice a persistent 5xx error in a cloudwatch chart while I was looking into another error. The system appeared to work fine and no one seemed to notice that their circ sessions were not timing out automatically.

This came up in another meeting and let us know that this is a known issue with PubSub and that people are already looking into how to solve it.

That is good to know, I was not aware that it was being actively progressed at the moment.

Please could you point me in the direction of this work.

Given that, I don't think lack of failure handling should prevent us from using PubSub for this feature. Are there any other reasons we shouldn't just use PubSub? If not, I think we can move forward.

As long as we are happy with the process being asynchronous, I am not aware of any reason not to proceed with pub-sub.

BTW, will be calling a meeting on the topic of failure handling in PubSub (with a few folks such as you, , and ).

Oh, I wasn't aware of this either. Either I missed both of these, or a lot has changed since I raised the topic yesterday in the Tech Leads meeting.

Thanks . This came up in another meeting and let us know that this is a known issue with PubSub and that people are already looking into how to solve it. Given that, I don't think lack of failure handling should prevent us from using PubSub for this feature. Are there any other reasons we shouldn't just use PubSub? If not, I think we can move forward.

BTW, will be calling a meeting on the topic of failure handling in PubSub (with a few folks such as you, Craig, Vince and Taras).

+

what is your preference?

Short Answer

I think the pub-sub based asynchronous process is best aligned with how a real world process could work and with the technical direction. I think FOLIO will need to address the handling of failures (in general) at some point.

Long Answer

Longer answer: I think an asynchronous process where the circulation reacts to a fine being closed is the closest alignment to how this process might work in reality (e.g. a accounts staff member closes the fine, and later, after being informed by accounts, a circulation staff member closes the loan, as two separate tasks).

presented two asynchronous options. I think the pub-sub option more closely aligns with both how the process could work in the real world and with the technical direction FOLIO is taking.

It sounds like the primary challenge with PubSub is handling of the (rare) case when something goes wrong and there is an error closing the loan.

Yes, the two most important aspects of this decision are:

• how are (hopefully rare) errors handled

• loans will not be closed as soon as the fee in closed (it might be very soon after)

As I understand it, this is a general issue with PubSub and other asynchronous approaches and the lack of solution hasn't prevented other teams from moving forward with such approaches.

Yes, FOLIO has already adopted some asynchronous processes with scheduling (request expiration, loan anonymization) and with pub-sub (data import, and I think automated patron blocks will use this).

The lack of agreed solutions to handling failure hasn't stopped these moving forward. I raised this with the Technical Leads yesterday and will be writing a document describing the challenges for further discussion.

Given that, I am not inclined to let it hold us back either. I think we can probably go into this knowing we are incurring tech debt that will need to be resolved later (not just for this feature, but for the project as a whole).

Agreed. I only wanted to make sure folks were aware of these challenges whilst making the decision. These aspects will need addressing at some point. I don't know who needs to sponsor investing in that (maybe the Technical Council?).