Improve handling of COUNTER API Exceptions

Improve handling of COUNTER API Exceptions

Description

The eUsage app should handle individual CoP error codes during harvesting. At the moment, the program only recognizes exceptions, but does not differentiate between them in terms of content. For this reason, the program must be expanded to respond to the various cases of exceptions.

Use case

Statistics show that CoP errors occur in the form of peaks. This means that as soon as errors occurs, it multiplies. Example: more than 1000 reports with CoP error codes during peak periods. This can happen, for example, if credentials changes or servers or reports are unavailable. This number of errors can only be handled by the user with great effort. These errors must be automatically intercepted and processed accordingly.

Workaround

  1. Manual error analysis by users.

  2. Deleting reports with errors by users.

  3. Restarting harvesting.

Harvesting could be started manually and errors could be handled manually, but depending on the number of UDPs this could be a lot of work.

Concept

The following recommendation assumes that the exceptions are used as specified in the Code of Practice (CoP). If a usage data provider uses exceptions in a wrong way (which should be reported to COUNTER), a manual handling of the exceptions might be required.

Handling of exception codes

See also: COUNTER Code of Practice Release 5.1, Appendix D: Handling Errors and Exceptions

EPIC https://folio-org.atlassian.net/browse/UIEUS-429

Exception Code

UXPROD

Status

Exception Message

Comment

Implementation plan

Retry?

Retry after

Retry count

Exception Code

UXPROD

Status

Exception Message

Comment

Implementation plan

Retry?

Retry after

Retry count

0

 

closed (Won’t Do)

{Info or Debug Message}

Exception with codes below 1000 are only for custom information, debugging, warning messages. They must not be used instead of the standard exceptions, so it should be possible to safely ignore them.

No individual handling. The report ends as a report with an error.

no

 

 

1-999

 

closed (Won’t Do)

{Warning Message}

No individual handling. The report ends as a report with an error.

no

 

 

1030

UXPROD-5440

closed (Won’t Do)

Insufficient Information to Process Request

Should not occur with FOLIO, unless the usage data host wronly uses this exception instead of the more specific ones below.

No individual handling. The report ends as a report with an error.

no

 

no

2011

UXPROD-5444

closed (Won’t Do)

Global Reports Not Supported

 

Global reports may be handled in more detail with eUsage at a later date. There is currently no use case.

Cannot be handled like CoP 3000

Consultation with PO from the FOLIO App Open Access:

‘A load-compatible variant should be available in the long term.’

3020

UXPROD-5448

closed (Won’t Do)

Invalid Date Arguments

Should not occur with FOLIO, unless wrongly used by the usage data host.

No individual handling. The report ends as a report with an error.

no

 

no

3040

UXPROD-5452

needs to be discussed (SME)

Partial Data Returned

These should be permanent conditions, unless wrongly used by the usage data host.

 

 

 

 

3050

UXPROD-5453

closed (Won’t Do)

Parameter Not Recognized in this Context

Should not occur with FOLIO unless a usage data host is not COUNTER compliant or not handling the request correctly.

Exception 3063 should not occur with FOLIO unless component support would be added (very complex, only supported by a few usage data host).

No individual handling. The report ends as a report with an error.

no

 

no

3060

UXPROD-5454

closed (Won’t Do)

Invalid ReportFilter Value

3061

UXPROD-5455

closed (Won’t Do)

Incongruous ReportFilter Value

3062

UXPROD-5456

closed (Won’t Do)

Invalid ReportAttribute Value

3063

UXPROD-5457

closed (Won’t Do)

Components Not Supported

 

 

 

 

3070

UXPROD-5458

closed (Won’t Do)

Required ReportFilter Missing

No individual handling. The report ends as a report with an error.

no

 

no

3071

Counter 4?

closed (Won’t Do)

Required ReportAttribute Missing

These exceptions should never occur, they were removed from the Code of Practice in Release 5.0.2.

The harvester for Counter 4 will be removed.

3080

Counter 4?

closed (Won’t Do)

Limit Requested Greater than Maximum Server Limit

The most error handling can be grouped into categories in eUsage. Most exceptions can be handled with these groups. The remaining cases will be added later.

Exception handling for CoP Error codes 1XXX

User story

https://folio-org.atlassian.net/browse/UXPROD-5746

Exception Code

UXPROD

Status

Exception Message

Comment

Implementation plan

Retry?

Retry after

Retry count

1000

UXPROD-5438

needs to be discussed (SME)

Service Not Available

These should be a temporary conditions for the UDP (not report specific). If the retries would be counted, more failed attempts should be allowed than for 3031.

Solution proposed by the developer:

Optional additional individual settings for the harvester per UDP. These optional settings would be created in the edit mode of the UDP record when configuring the harvester. This solution allows you to control behavior according to UDP.

yes

10 min

no?

1010

UXPROD-5081

needs to be discussed (SME)

Service Busy

yes

10 min

no?

1011

UXPROD-4967

needs to be discussed (SME)

Report Queued for Processing

Not an error, try again after a short time.

yes

10 min

no

1020

UXPROD-5439

needs to be discussed (SME)

Client has made too many requests

Same as 1000/1010. Could be report specific, but sometimes is not.

The handling of the exception will be changed. The harvester should stop and make no further attempts. Instead, an error message is displayed in the Detail View. This can also be filtered to search for it.

no

After correcting the configuration

no

Exception handling for CoP Error codes 2XXX

Please note: 2011 is not included.

User story

https://folio-org.atlassian.net/browse/UXPROD-5754

  • A new feature is being developed that will enable credentials to be checked initially when creating a UDP.

  • The harvester is adjusted so that it responds to CoP error codes.

Exception Code

UXPROD

Status

Exception Message

Comment

Implementation plan

Retry?

Retry after

Retry count

2000

UXPROD-5442

discussed - in preparation for Refinement

Requestor Not Authorized to Access Service

UDP configuration needs to be fixed. Automatically retry after fix?

Exception 2011 could only occur if Customer_ID 0000000000000000 would be used to request global reports.

 

 

  1. If the exception occurs, stop harvesting. Otherwise, attempts would be made to load reports with incorrect data, which could cause a large number of errors. Users should have the option to correct their credentials.

  2. A solution must be found to restart the harvester after correcting the credentials.

  • Option 1: Restart the harvester manually.

  • Option 2: “Auto-Retry-function” after the user changed the credentials or a feature to check the credentials within the form. SME: Try it with a report in the background

Exception 2011

Requires special handling.

 

Developer:

There is a separation between UDP and Harvester in the backend. A solution is being sought that will allow the harvester to read the errors. Only then can the harvester respond accordingly.

One suggestion would be to only allow the initial harvesting to start after prior verification of credentials. The credentials must therefore be checked. A solution is being sought during ongoing operations. A possible solution for ongoing harvesting would be that when something is reported, the harvester stops and the initial check must be repeated. The trigger for it must be checked from the technical standpoint.

SME recommends checking the credentials by means of a retrieval, i.e., an attempt should be made to download a report during the initial check.

no

After correcting the configuration

no

2010

UXPROD-5443

discussed - in preparation for Refinement

Requestor is Not Authorized to Access Usage for Institution

no

After correcting the configuration

no

2020

UXPROD-5445

discussed - in preparation for Refinement

APIKey Invalid

no

After correcting the configuration

no

Exception handling for CoP Error codes 3XXX

User story

https://folio-org.atlassian.net/browse/UXPROD-5711

Exception Code

UXPROD

Status

Exception Message

Comment

Implementation plan

Retry?

Retry after

Retry count

3000

UXPROD-5446

discussed - in preparation for Refinement

Report Not Supported

SME: R5 only, in R5.1 the HTTP status code 404 is used instead.

Possible redesign of report type selection.

no

After correcting the configuration

no

3010

UXPROD-5447

discussed - in preparation for Refinement

Report Version Not Supported

no

After correcting the configuration

no

Exception handling for CoP Error codes 303X (UI only)

In these cases, only the icon used is changed. Therefore, nothing is changed in the harvester and only the UI controls what is displayed.

User stories

Exception Code

UXPROD

Status

Exception Message

Comment

Implementation plan

Retry?

Retry after

Retry count

3030

UXPROD-5449

closed (Done)

No Usage Available for Requested Dates

Not an error, unless wrongly used by the usage data host. Maybe this exception could be shown with a different symbol or color (lighter or more yellowish green?), so that this case can easily be identified.

New Icon in the overview of the Counter Report statistics. Lighter green and a 0 as the symbol. Also called 0-reports. The reason for the symbol with a 0.

yes

User configuration in the settings

yes

3031

UXPROD-5450

discussed - in preparation for Refinement

Usage Not Ready for Requested Dates

Reports should be available within four weeks after the end of the month, but often are available much ealier, depending on the usage data host. How many retries should be permitted and how ofter a retry should be attempted depends on when the first attempt is started.

Only new Icon: calendar (yellow)

yes

3 days?

User configuration in the settings

yes

3032

UXPROD-5451

discussed - in preparation for Refinement

Usage No Longer Available for Requested Dates

These should be permanent conditions, unless wrongly used by the usage data host.

Only new Icon: calendar (red)

yes

User configuration in the settings

yes

Harvester (Backend)

The backend of the harvester needs to be redesigned. The current design cannot meet the new requirements. See https://folio-org.atlassian.net/browse/UXPROD-5578