Okapi Log Formatting

Owned by Martin Tran
Last updated: May 28, 2022 by Nassib Nassar

2022-05-27 - Giraffe is now maintained at https://github.com/library-data-platform/giraffe and has been updated to support the new Okapi log format.

Because Giraffe was created to use an outdated version of Okapi's log, and Giraffe hasn't been updated to consume the most recent version, the current Okapi log has to be formatted to match that of the old format. 

Old format:

2020-08-19 14:39:45 INFO  ProxyContext         793863/inventory REQ 10.23.10.106:27716 fs09000000 GET /inventory/items  mod-authtoken-2.5.1 mod-inventory-16.0.1
2020-08-19 14:39:45 INFO  ProxyContext         793863/inventory RES 202 5848us mod-authtoken-2.5.1 http://pvt.lb.gcp1.folio-eis.us-east-1:8051/mod-authtoken/inventory/items?query=barcode==H6369373
2020-08-19 14:39:45 INFO  ProxyContext         554600/item-storage REQ 10.23.10.106:27728 fs09000000 GET /item-storage/items  mod-authtoken-2.5.1 mod-inventory-storage-19.3.1
2020-08-19 14:39:45 INFO  ProxyContext         554600/item-storage RES 202 5531us mod-authtoken-2.5.1 http://pvt.lb.gcp1.folio-eis.us-east-1:8051/mod-authtoken/item-storage/items?query=barcode%3D%3DH6369373&limit=10&offset=0
2020-08-19 14:39:45 INFO  ProxyContext         793863/inventory;574902/holdings-storage REQ 10.23.10.224:60282 fs09000000 GET /holdings-storage/holdings  mod-authtoken-2.5.1 mod-inventory-storage-19.3.1
2020-08-19 14:39:45 INFO  ProxyContext         554600/item-storage RES 200 36014us mod-inventory-storage-19.3.1 http://pvt.lb.gcp1.folio-eis.us-east-1:8051/mod-inventory-storage/item-storage/items?query=barcode%3D%3DH6369373&limit=10&offset=0
2020-08-19 14:39:45 INFO  ProxyContext         793863/inventory;574902/holdings-storage RES 202 6736us mod-authtoken-2.5.1 http://pvt.lb.gcp1.folio-eis.us-east-1:8051/mod-authtoken/holdings-storage/holdings?query=id%3D%3D%282300cee6-9f44-432f-8741-2f5ded381f3f%29&limit=1&offset=0


New Format:

2022-02-16T19:30:33.271-05:00	00:30:33 [301110/source-storage] [fs09000000] [9eb67301-6f6e-468f-9b1a-6134dc39a684] [] INFO ProxyContext 301110/source-storage REQ 10.23.35.20:26378 fs09000000 POST /source-storage/snapshots mod-authtoken-2.10.0-SNAPSHOT.214 mod-source-record-storage-5.3.0-SNAPSHOT.127
2022-02-16T19:30:33.284-05:00	00:30:33 [301110/source-storage] [fs09000000] [9eb67301-6f6e-468f-9b1a-6134dc39a684] [] INFO ProxyContext 301110/source-storage RES 202 6852us mod-authtoken-2.10.0-SNAPSHOT.214 http://mod-authtoken-b.lcp1.folio-eis.us-east-1:8051/mod-authtoken/source-storage/snapshots?
2022-02-16T19:30:33.305-05:00	00:30:33 [301110/source-storage] [fs09000000] [9eb67301-6f6e-468f-9b1a-6134dc39a684] [] INFO ProxyContext 301110/source-storage RES 201 20856us mod-source-record-storage-5.3.0-SNAPSHOT.127 http://mod-source-record-storage-b.lcp1.folio-eis.us-east-1:8051/mod-source-record-storage/source-storage/snapshots?
2022-02-16T19:30:33.336-05:00	00:30:33 [534313/data-import-profiles] [fs09000000] [9eb67301-6f6e-468f-9b1a-6134dc39a684] [] INFO ProxyContext 534313/data-import-profiles REQ 10.23.35.20:26378 fs09000000 GET /data-import-profiles/jobProfiles/91f9b8d6-d80e-4727-9783-73fb53e3c786 mod-authtoken-2.10.0-SNAPSHOT.214 mod-data-import-converter-storage-1.13.0-SNAPSHOT.69


Steps to transform the new format into the old format simply by Find and Replace:

  1.  Change the timestamp with the following regular expression:
    1. Find "2022-02-16T19:30:[0-9][0-9].[0-9][0-9][0-9]-05:00" and replace with "2022-02-16"  (... content must be adapted to the actual log. Here the content is applicable to the log above).
  2. Remove all the contents inside the brackets. The following regular expression looks for everything that's between the first open bracket and last closing bracket, including the open and closing brackets in between.
    1. Find \[(.*?)\] and replace with "" (empty strings)
  3. Tabs must be replaced by spaces.  Between the date and the timestamp there is a tab
    1. Find "\t" and replace with " " (1 space)

Final Log:

2022-02-16 00:30:33     INFO ProxyContext 638594/source-storage REQ 10.23.35.125:3662 fs09000000 PUT /source-storage/snapshots/36214572-bf2c-4b29-9c5c-7e5201d5f0b0 mod-authtoken-2.10.0-SNAPSHOT.214 mod-source-record-storage-5.3.0-SNAPSHOT.127
2022-02-16 00:30:33     INFO ProxyContext 638594/source-storage RES 202 6721us mod-authtoken-2.10.0-SNAPSHOT.214 http://mod-authtoken-b.lcp1.folio-eis.us-east-1:8051/mod-authtoken/source-storage/snapshots/36214572-bf2c-4b29-9c5c-7e5201d5f0b0?
2022-02-16 00:30:33     INFO ProxyContext 638594/source-storage RES 200 15980us mod-source-record-storage-5.3.0-SNAPSHOT.127 http://mod-source-record-storage-b.lcp1.folio-eis.us-east-1:8051/mod-source-record-storage/source-storage/snapshots/36214572-bf2c-4b29-9c5c-7e5201d5f0b0?
2022-02-16 00:30:33     INFO ProxyContext 057800/data-import-profiles REQ 10.23.35.125:3712 fs09000000 GET /data-import-profiles/jobProfileSnapshots/3150ae1f-3381-4668-93ab-46377d10d705 mod-authtoken-2.10.0-SNAPSHOT.214 mod-data-import-converter-storage-1.13.0-SNAPSHOT.69
2022-02-16 00:30:33     INFO ProxyContext 057800/data-import-profiles RES 202 6754us mod-authtoken-2.10.0-SNAPSHOT.214 http://mod-authtoken-b.lcp1.folio-eis.us-east-1:8051/mod-authtoken/data-import-profiles/jobProfileSnapshots/3150ae1f-3381-4668-93ab-46377d10d705