[CIRC-1131] Some or all circ notices are missing from scheduled notices, ditto some on-demand notices Created: 21/Apr/21  Updated: 17/Feb/22  Resolved: 02/Jun/21

Status: Closed
Project: mod-circulation
Components: None
Affects versions: None
Fix versions: None

Type: Bug Priority: P2
Reporter: Anne L. Highsmith Assignee: julie.bickle
Resolution: Cannot Reproduce Votes: 0
Labels: Support, patron_notice
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified
Environment:

TAMU's test environment running Honeysuckle (seen on both Goldenrod and Honeysuckle).


Attachments: Text File circ_rules.txt     File missing_notices.tar.gz     PNG File missing_notices_owners.png    
Issue links:
Defines
defines CIRC-1157 [SPIKE] Find approach to capturing an... Closed
Sprint:
Development Team: Vega
Affected Institution:
TAMU

 Description   

Overview:
Because we are not in production, we send all circ notices to the same email address, folio_user@library.tamu.edu, which is set up as separate mailbox. Late in the period when we were up on Goldenrod, we noticed we weren't getting any scheduled notices (overdues & courtesies). We waited until we got Honeysuckle going until we did any investigation. Once Honeysuckle was up, we started receiving some notices daily, but not all of the ones we were supposed to received, based on a comparison of tamu_mod_email.email_statistics (which lists the notices delivered to smtp) and tamu_mod_audit.circulation_logs (which lists the notices the system thinks it delivered). 

On several different days I cross-checked the email stats vs the circ log and found that scheduled notices delivered were only 40%-60% of what the circ log said should have been delivered. I tried to find a pattern in the failures as to service point or template, but there was none. Each service point saw some notices delivered and some fail, ditto template. We have also worked with our computer center to see if there was some kind of smtp error in that the notices were being created but not delivered. They didn't see any such errors; as far as we can tell, the html containing the notice just isn't being created at all.

Now we are seeing the same phenomenon we saw in Goldenrod, i.e. there are been NO notices created for our tamu instance since 4/13. Yet the circ log says we should have received the following notices for the past several days.

4/14  56
4/15  61
4/16  96
4/17  84
4/18  72
4/19  38
4/20  99
4/21  97

Finally, we aren't receiving all of our on-demand notices either. When I create a fee/fine against a user record or take an action against a fee/fine, sometimes it creates the appropriate notice and sometimes not. 

Expected Results:

Number and type of notices scheduled to be delivered or generated by manual processes are crafted and delivered to mailbox.
Actual Results:

Every day, some or all of scheduled notices are not crafted. Manual processes do not always reliable deliver notices either.
Additional Information:
URL:
Interested parties:



 Comments   
Comment by Darcy Branchini [ 21/Apr/21 ]

Hi Anne L. Highsmith, based on your comments above, I wanted to start with a few observations. Service points and notices aren't associated with each other so I wouldn't expect to see a pattern by service point. Perhaps you meant location since service points are associated with location and circulation rules refer to locations? Notices are generated based on three parts - a matching circulation rule (based on patron group, loan type, material type and/or location), the notice policy and the notice template. It might be helpful to see your circulation rules, notice policies and notice templates. Perhaps you can share those? Also all manual fee/fine charges and all fee/fine actions are setup differently the other types of notices. The templates are created in the same place - Settings > Circulation > Patron notice templates with a Manual fee/fine charge or Fee/fine action (in Honeysuckle); however, they are associated with fees/fines under Users > Fee/fine > Manual charges. If a notice isn't setup there, then the user shouldn't be given the option to send a notice, but also the user does need to be sure they check the box, "send patron notice." It would be helpful to see the fee/fine owner screen too to see how you have those setup. If there are problems with both types of notices, then I believe it will be two different reasons since they're handled so differently.

Comment by Anne L. Highsmith [ 22/Apr/21 ]

Here are the notice policies, templates, and circ rules. Will provide screenshot of the manual charges in a bit.

Comment by Anne L. Highsmith [ 22/Apr/21 ]

screen shot of the main library's (owner) manual charges.

Comment by Oleksandr Vidinieiev [ 27/Apr/21 ]

Hi Anne L. Highsmith, in order to investigate this issue I need the access to TAMU environment. Can you provide the credentials (FOLIO username and password) that I can use? Having access to Okapi and/or mod-circulation logs would be nice too.

Comment by Anne L. Highsmith [ 27/Apr/21 ]

Hello, Oleksandr Vidinieiev. Our folio instance is behind the campus firewall, so we would have to have some kind of mediated video session on slack/teams/zoom where we show you stuff as you request it. This would probably turn out to be more useful anyway, as both our SysAdmin, jroot and I would participate to answer questions.

Here are some times in the upcoming week that are available:
Thursday, 4/29 9-10 AM central time, 11 AM-1 PM central time,2:30-4:30 PM central time
Friday, 4/30 10 AM-4 PM central time

If neither of those fits your schedule, please let me know and I will check on times for next week.

Comment by Kelly Drake [ 28/Apr/21 ]

Anne L. Highsmith - I'm asking on behalf on the Support SIG.  Have you attempted to reproduce this behavior - maybe just for the on demand notices - on any of the community environments?  We would want to know if this is a folio error, or specific to the TAMU implementation.

Comment by Darcy Branchini [ 28/Apr/21 ]

Anne L. Highsmith, as you can see from Kelly Drake's comment above, this ticket is being rerouted through the Support SIG. Kelly also asked in the implementators Slack channel to see if anyone else is seeing behavior as you describe in the issue. Also, I see three different tickets here - manual fee/fine notices (since the configuration is so different for those), on-demand "regular" notices and scheduled "regular" notices.

I checked your configuration for scheduled notices and I didn't see anything that would be a problem. The first thing that I check for is multiple loans #loans /loans on a template for reminder and overdue loan that are set to process overnight or that it's not there for reminder and overdue loan notices that are not set to process overnight. On the policy, these are now labeled, "Send overnight with multiple loans/items by patron. Useful for long-term loans" and "Send throughout the day without multiple loans/items. Useful for short-term loans." Unfortunately, on the backend and in the JSON, you'll see these labeled as "realTime" set to true or false. That looks find to me.

I also wondered about the on-demand or item status notices, such as item recalled or requested item is available notices. The realTime attribute should be ignored on these (used to be an option). You could try realTime set to true on these since it used to be required. I don't think it matters, but it's worth a try.

As far as the fee/fine notices, I don't see any problems. Do you have default charge and action notices setup or just specific notices for each type of charge? I also noticed that you have migrated manual charges and newly created (duplicate) charges. I don't see any problems with that unless for some reason, people are selecting the wrong charge when testing this, but then it shouldn't show up in the circulation log. I also wondered if migrations from Goldenrod to Honeysuckle might have caused any problems and/or corrupt data.

Those are my few thoughts on this matter. I believe the Support SIG will pick it up from here... to do additional triage.

Comment by Anya [ 03/May/21 ]

support: is this able to be reproduced in Snapshot or in a different environment?

Another thought: could it be possible that fetch limits are causing this? for example UICIRC-568 Closed

 

If we can reproduce it in one of the ref environment - this needs to be a P1 -  (for awareness Anton Emelianov)

Comment by Anne L. Highsmith [ 03/May/21 ]

For the scheduled notices, at least, this is beginning to look like a local issue, i.e. a mismatch between preferred contact type and the presence/absence of an email address in the user record. I am continuing to test that part of it daily by comparing the "missing" notices with the list of users who were supposed to receive notices that day but lack emails in the user record.

As for on-demand notices, I will test those also as I have time.

Comment by jroot [ 03/May/21 ]

To expand on Anne's comment: This does not explain why notices stop sending altogether. After restarting the back-end modules as well as Okapi - they started to send again.

Comment by Anne L. Highsmith [ 04/May/21 ]

This morning I tested creation of a series of on-demand notices, such as lost item processing/replacement fees, and automatic notices produced as the result of an event, such as a recall or checkin of recalled item. All situations produced a notice EXCEPT discharge of an overdue item to which an automatic overdue fine was applied. According to the user record, the fee was created. But when I checked the item barcode in the circ log, it didn't show that a notice was sent, although it showed that the item had been billed. Looking at the text version of our circ rules, the relevant line would be line 47: 

t bluray cassette cd dvd vhs: l day-7-renew-0-recall-n r no-request n notices-basic o day-1-max-35 i lost-item-fee-policy

Given the information I've provided previously, i.e. policies, rules, and templates, can you see any reason why the system wouldn't have produced a notice? And by the way, loans have loan policy, lost policy, and overdue policy stored in them – where is the notice policy that relates to a given loan stored in the database?

thanks.

Comment by Debra Howell [ 17/May/21 ]

Anne L. Highsmith and jroot 

From SUPPORT SIG:  is this able to be reproduced in Snapshot or in a different environment?

 

If we can reproduce it in one of the ref environment - this needs to be a P1 -  (for awareness Anton Emelianov)

Comment by Darcy Branchini [ 01/Jun/21 ]

Anne L. Highsmith and jroot have you been able to reproduce this issue on another environment? Should we talk about this issue so I can get a clear idea of it's status?

Comment by Anne L. Highsmith [ 01/Jun/21 ]

Darcy Branchini I think we have resolved this issue locally. Let me spend a little time on it today and I'll either close it or update it. (jroot FYI)

Comment by Erin Nettifee [ 01/Jun/21 ]

Anne L. Highsmith if you have solved it, can you share the root cause / solution here?

Comment by Anne L. Highsmith [ 01/Jun/21 ]

Darcy Branchini (julie.bickle) (jroot) You can close this ticket; I believe it has been resolved locally. If we have any problems in future we'll open a new one.

Comment by Anne L. Highsmith [ 01/Jun/21 ]

Erin Nettifee It was a mismatch between setting all users to preferred contact type = 'email' and having 14K users in the legacy system who didn't have email addresses. When we did the migration, we defaulted the email to 'folio_user@tamu.edu' for all users who already HAD emails (because we didn't want to actually deliver them from a test system), but not for those who had no email in the legacy system, with the result that the system tried to send emails to users who had none listed in FOLIO. The circulation log thought it had sent those emails, but the *_mod_email.email_statistics table showed that they hadn't actually been sent. 

The 14k users who have no email in the legacy system are either ILS partners or community users or university users whose records are so old that we have no email available. We're making a systematic effort now to add email addresses to ILS partner records. Beyond that, we will continue to default to 'folio_user@tamu.edu' for user records that have no emails, which will send those to a specially-defined email inbox. The service points will have to review those emails periodically to process the ones that have no target address.

Since, as I understand, there is currently no way to print an email with a snail mail address for mailing, it would be nice if the system blocked the attempt to send an email to a user whose email address is null. But maybe it's better just to update the system so that it can print a notice in such a case.

Comment by Erin Nettifee [ 01/Jun/21 ]

Oh wow - yah, that makes sense to me. I know we'd discuss the need for reporting on users without valid email addresses (which in my mind would include no email) but I can easily see how this would happen and how confusing it would have been to untangle. Kudos on you all for figuring it out! Maybe we should start a considerations for migrations - users page that could capture this info.

Comment by Erin Nettifee [ 01/Jun/21 ]

OK, I started a super skinny page but it at least captures the experience here hopefully: https://folio-org.atlassian.net/wiki/display/FOLIOtips/Considerations+for+Data+Migration+-+Users

The bounced email report jira is here: https://folio-org.atlassian.net/browse/UXPROD-932
There's also an audit history of notices feature here: https://folio-org.atlassian.net/browse/UXPROD-1627
And there's a jira for validating the email on https://folio-org.atlassian.net/browse/UIU-1291

Darcy Branchini and julie.bickle - do we know for a fact that if the patron preferred notice type right now is anything other than email, that nothing will happen with the notice? Should we capture a bug to reflect what happened here with the system looking like emails were delivered but they actually weren't?

Comment by Darcy Branchini [ 02/Jun/21 ]

I'll write up a new bug to capture this behavior. Thanks for the update Anne L. Highsmith! I really appreciate the details here.

Comment by Darcy Branchini [ 02/Jun/21 ]

See CIRC-1157 Closed for new task filed.

Generated at Fri Feb 09 00:28:55 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.