[FOLIO-2102] Timestamp metadata missing from bootstrap data Created: 14/Jun/19  Updated: 03/Jun/20  Resolved: 06/Sep/19

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Bug Priority: P2
Reporter: Cate Boerema (Inactive) Assignee: Julian Ladisch
Resolution: Done Votes: 0
Labels: platform-backlog
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks MODINV-152 Missing and Incomplete Data in Hosted... Closed
blocks UIIN-597 Item record. Record metadata is missi... Closed
blocks UIIN-598 Holdings record. Record metadata is m... Closed
blocks UIIN-599 Instance record. Record metadata is m... Closed
blocks UIU-1069 Record created date populate Invalid ... Closed
is blocked by RMB-459 Populate metadata for reference data/... Closed
Relates
relates to STCOM-538 Display "Unknown" when record metadat... Closed
relates to UIIN-626 Record Metadata Component Should Disp... Closed
relates to STSMACOM-230 ViewMetaData component barfs if creat... Closed
relates to MODUSERBL-75 Update raml-util submodule making met... Closed
relates to MODINV-142 Issue with metadata in inventory item... Closed
Sprint: CP: sprint 70, CP: sprint 69, CP: sprint 71
Story Points: 2
Development Team: Core: Platform

 Description   

Description

The automatically loaded (bootsrapped) data should include timestamps (created at / updated at).

Context

We have suddenly found that records metadata seems to be missing from our bootstrap data (Users, Instances, Holdings, Items - maybe others not yet discovered). This causes the record metadata component to disappear or to display "invalid" where data should be.

  • Several bugs were filed when this was first discovered ( UIIN-599 Closed , UIIN-598 Closed , UIIN-597 Closed , UIU-1069 Closed ) and those will be blocked on this one
  • We have also filed STCOM-538 Closed so that, if data is missing in the future, the component will handle it more gracefully
  • Ideally we would also make the data required in the schema but, per comments in UIIN-597 Closed , the cost of making that change is quite high - probably not a priority right now.

After RMB has been updated as explained in RMB-459 Closed the backend modules need to use the new RMB version.

How to test: When this issue is done, please test UIIN-599 Closed , UIIN-598 Closed , UIIN-597 Closed and UIU-1069 Closed



 Comments   
Comment by Jakub Skoczen [ 14/Jun/19 ]

Marc Johnson Cate Boerema is it all metadata including timestamps (createdDate..) or only related metadata?

Comment by Marc Johnson [ 14/Jun/19 ]

Jakub Skoczen I think Zak Burke investigated briefly and mentioned that all of the change metadata was missing.

Theodor Tolstoy (One-Group.se) confirmed that reason for this in the bug fest environment is due to the data being loaded directly.

We think the same has been observed in the regular environments.

Comment by Cate Boerema (Inactive) [ 01/Jul/19 ]

Jakub Skoczen this seems to have resolved itself. I am now seeing record metadata in folio-snapshot. I'll close this and all the various linked bugs.

It would be interesting to know what happened here, though. It caused a ton of bug writing and discussion that would have better been avoided. Do you have thoughts on that? Maybe John Malconian knows what happened?

Comment by Cate Boerema (Inactive) [ 01/Jul/19 ]

Actually, this is still a problem in folio-testing but not in folio-snapshot. Can you guys please fix it so we can close all these bugs?

Comment by Cate Boerema (Inactive) [ 01/Jul/19 ]

Sorry guys - it looks like this is STILL a problem for both folio-testing and folio-snapshot. I mistakenly thought it was fixed for snapshot because the records I checked had been updated by a manual tester (that will set the last updated data). But if you find a record that hasn't been touched, the data is still missing in both environments.

Comment by Jakub Skoczen [ 05/Jul/19 ]

Zak Burke Marc Johnson Adam Dickmeiss Guys, I don't get this issue. I understand the problem with the bugfest env was due to the fact the data was loaded directly into the DB and hence had to metadata. But for reference envs data loading is performed by the modules themselves so we should see "timestamps" but no users in the metadata.

Comment by Marc Johnson [ 05/Jul/19 ]

the problem with the bugfest env was due to the fact the data was loaded directly into the DB and hence had to metadata.

Agreed, this is a red herring.

for reference envs data loading is performed by the modules themselves so we should see "timestamps" but no users in the metadata.

It seems the modules behave differently (see example records below).

For users is no metadata property at all.

For instances, sample records loaded at activation have a metadata object with explicitly null values for each property. For records loaded via the MODS ingest, the metadata is as expected.

I don't think we know why this occurs.

Could it be that some versions of the RAML Module Builder abort setting any of the properties if no user header is present? (I vaguely recall some conversations about this a few months ago).

And that this behaviour varies by different versions of the RAML Module Builder?

Users

{
  "username" : "theodora",
  "id" : "e59bc21a-884e-42a2-8792-163efc3662e7",
  "barcode" : "144826156017341",
  "active" : true,
  "type" : "patron",
  "patronGroup" : "3684a786-6671-4268-8ed0-9db82ebca60b",
  "proxyFor" : [ ],
  "personal" : {
    "lastName" : "Dickinson",
    "firstName" : "Francisca",
    "middleName" : "Dock",
    "email" : "rollin@jacobson-emmerich-and-schaefer.kg",
    "phone" : "(071)378-8133 x66872",
    "mobilePhone" : "373-226-0411",
    "dateOfBirth" : "1945-12-20T00:00:00.000+0000",
    "addresses" : [ {
      "countryId" : "US",
      "addressLine1" : "76540 Jaden Meadows #999",
      "city" : "Toledo",
      "region" : "MN",
      "postalCode" : "49708",
      "addressTypeId" : "93d3d88d-499b-45d0-9bc7-ac73c3a19880",
      "primaryAddress" : true
    } ],
    "preferredContactTypeId" : "003"
  },
  "enrollmentDate" : "2016-08-13T00:00:00.000+0000",
  "expirationDate" : "2020-03-18T00:00:00.000+0000",
  "createdDate" : "2019-07-05T03:32:54.297+0000",
  "updatedDate" : "2019-07-05T03:32:54.297+0000"
}

Instances

Sample Records

{
  "@context" : "http://folio-snapshot-okapi.aws.indexdata.com/inventory/instances/context",
  "id" : "69640328-788e-43fc-9c3c-af39e243f3b7",
  "hrid" : "in00000020",
  "source" : "MARC",
  "title" : "ABA Journal",
  "parentInstances" : [ ],
  "childInstances" : [ ],
  "alternativeTitles" : [ ],
  "editions" : [ ],
  "series" : [ ],
  "identifiers" : [ {
    "identifierTypeId" : "913300b2-03ed-469a-8179-c1092c991227",
    "value" : "0747-0088"
  }, {
    "identifierTypeId" : "c858e4f2-2b6b-4385-842b-60732ee14abb",
    "value" : "84641839"
  } ],
  "contributors" : [ ],
  "subjects" : [ ],
  "classifications" : [ ],
  "publication" : [ {
    "publisher" : "American Bar Association",
    "place" : "Chicago, Ill.",
    "dateOfPublication" : "1915-1983",
    "role" : null
  } ],
  "publicationFrequency" : [ ],
  "publicationRange" : [ ],
  "electronicAccess" : [ ],
  "instanceTypeId" : "6312d172-f0cf-40f6-b27d-9fa8feaf332f",
  "instanceFormatIds" : [ ],
  "physicalDescriptions" : [ ],
  "languages" : [ ],
  "notes" : [ ],
  "statisticalCodeIds" : [ ],
  "metadata" : {
    "createdDate" : null,
    "createdByUserId" : null,
    "updatedDate" : null,
    "updatedByUserId" : null
  },
  "links" : {
    "self" : "http://folio-snapshot-okapi.aws.indexdata.com/inventory/instances/69640328-788e-43fc-9c3c-af39e243f3b7"
  }
}

MODS Ingest

{
  "@context" : "http://folio-snapshot-okapi.aws.indexdata.com/inventory/instances/context",
  "id" : "8d56cae7-3b5f-44c5-9a62-53b7d21fb06c",
  "hrid" : "in00000120",
  "source" : "Local: MODS",
  "title" : "1963 Birmingham church bombing",
  "parentInstances" : [ ],
  "childInstances" : [ ],
  "alternativeTitles" : [ ],
  "editions" : [ ],
  "series" : [ ],
  "identifiers" : [ {
    "identifierTypeId" : "8261054f-be78-422d-bd51-4ed9f33c3422",
    "value" : "15440252"
  }, {
    "identifierTypeId" : "8261054f-be78-422d-bd51-4ed9f33c3422",
    "value" : "9780756540920 (library binding)"
  }, {
    "identifierTypeId" : "8261054f-be78-422d-bd51-4ed9f33c3422",
    "value" : "2008038921"
  } ],
  "contributors" : [ {
    "contributorNameTypeId" : "2b94c631-fca9-4892-a730-03ee529ffe2a",
    "name" : "Klobuchar, Lisa.",
    "contributorTypeId" : "",
    "contributorTypeText" : "",
    "primary" : null
  } ],
  "subjects" : [ ],
  "classifications" : [ ],
  "publication" : [ ],
  "publicationFrequency" : [ ],
  "publicationRange" : [ ],
  "electronicAccess" : [ ],
  "instanceTypeId" : "6312d172-f0cf-40f6-b27d-9fa8feaf332f",
  "instanceFormatIds" : [ ],
  "physicalDescriptions" : [ ],
  "languages" : [ ],
  "notes" : [ ],
  "statisticalCodeIds" : [ ],
  "metadata" : {
    "createdDate" : "2019-07-05T03:35:57.709+0000",
    "createdByUserId" : "a5df7bf3-c1c0-5937-aa2a-1b2f301eba49",
    "updatedDate" : "2019-07-05T03:35:57.709+0000",
    "updatedByUserId" : "a5df7bf3-c1c0-5937-aa2a-1b2f301eba49"
  },
  "links" : {
    "self" : "http://folio-snapshot-okapi.aws.indexdata.com/inventory/instances/8d56cae7-3b5f-44c5-9a62-53b7d21fb06c"
  }
}
Comment by Zak Burke [ 05/Jul/19 ]

I think STCOM-538 Closed is done save for unit tests. I've been pulled in other directions and haven't been able to finish it.

Would completing that ticket be another path to closing these related tickets? I pushed the branch to GitHub if somebody else wants to pick it up. I think all that's left is unit tests.

Comment by Marc Johnson [ 05/Jul/19 ]

Zak Burke

Would completing that ticket be another path to closing these related tickets?

I think completing STCOM-538 Closed makes diagnosis of missing change metadata easier to diagnose.

I believe the expectation is that it should be present, and so these issues won't be resolved until the correct aspects of change metadata are present.

I shall leave it to Cate Boerema to specify what properties have to be present for this to be acceptable. Above, Jakub Skoczen suggested that only the date time properties were needed to resolve this. Is that the case?

Comment by Zak Burke [ 05/Jul/19 ]

STCOM-538 Closed won't help with diagnosing missing data in inventory. In those components, metadata-display is consistently wrapped in code like "if (metadata exists) then show_metadata(...)" so all those if-clauses will need to be removed to allow the shared component to be in charge of what is displayed.

Comment by Cate Boerema (Inactive) [ 05/Jul/19 ]

I shall leave it to Cate Boerema to specify what properties have to be present for this to be acceptable. Above, Jakub Skoczen suggested that only the date time properties were needed to resolve this. Is that the case?

This is what Jakub and I agreed to. Since the data is auto-generated in our test environments, we didn't really know what to use as the source. This is why we suggested modifying STCOM-538 Closed so the component would display "Automated process" when the source was null. I'm not sure if Zak Burke saw that change yet. I didn't realize the ticket was already in progress. Might have been too late to add requirement changes.

STCOM-538 Closed won't help with diagnosing missing data in inventory. In those components, metadata-display is consistently wrapped in code like "if (metadata exists) then show_metadata(...)" so all those if-clauses will need to be removed to allow the shared component to be in charge of what is displayed.

UIIN issue filed for this: UIIN-626 Closed

Thanks Marc Johnson and Zak Burke

Comment by Oleksii Popov [ 10/Jul/19 ]

As a result of investigation, we should come with a solution and an estimation.

Comment by Cate Boerema (Inactive) [ 29/Jul/19 ]

Jakub Skoczen is this planned for sprint 69?

Comment by Julian Ladisch [ 13/Aug/19 ]

See RMB-459 Closed why metadata is not populated and how to fix it.
I propose to

  • Remove "spike" from the title.
  • Set this issue to blocked.
Comment by Julian Ladisch [ 27/Aug/19 ]

Fixed in https://folio-testing.aws.indexdata.com/ and https://folio-snapshot.aws.indexdata.com/
Not yet propagated to https://folio-snapshot-stable.aws.indexdata.com/

Comment by Ann-Marie Breaux (Inactive) [ 05/Sep/19 ]

Julian Ladisch and Cate Boerema this showed up in the manual testing queue, but I don't think there's anything for us to test. Everything here seems to have been closed. Do you want to close this ticket? Or let us know if there's something we should test manually. Thank you!

Comment by Cate Boerema (Inactive) [ 06/Sep/19 ]

Oh - yay! This is finally fixed. Thank you Julian Ladisch. So great to see all the dependent bugs closed, as well.

Generated at Thu Feb 08 23:18:13 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.