/
mod-source-record-storage performance for standard requests on big database.

mod-source-record-storage performance for standard requests on big database.

Nov 10, 2020

 

1. For measuring srs(mod-source-record-storage) performance was created and deployed testing vagrant-box locally. It has specific resources: vb.cpus = 3, vb.memory = "20000". SRS has maximum heap zise parameter: -Xmx256m

2. The database for srs(diku_mod_source_record_storage schema) was filled via 100_000 records. So, tables "records_lb", "raw_records_lb", "marc_records_lb" contains 100_000 records.

 

Record example from "records_lb"-table: 

id

snapshot_id

matched_id

generation

record_type

instance_id

state

leader_record_status

order

suppress_discovery

created_by_user_id

created_date

updated_by_user_id

updated_date

instance_hrid

id

snapshot_id

matched_id

generation

record_type

instance_id

state

leader_record_status

order

suppress_discovery

created_by_user_id

created_date

updated_by_user_id

updated_date

instance_hrid

cd02a7a1-0c9a-4d2c-8bf3-b82a6e9dfa7c

63ae895a-0315-4f80-9238-58ede7f58f66

2b55fe13-c4f4-40f1-b6b5-cce5c706635b

0

MARC

c5c95970-df21-4801-aae3-84a1ed3bc450

ACTUAL

c

<null>

false

<null>

2020-11-09 14:13:53.491183 +00:00

<null>

2020-11-09 14:13:53.491183 +00:00

in00000000004

 

Record example from "raw_records_lb"-table: 

id

content

id

content

2b55fe13-c4f4-40f1-b6b5-cce5c706635b

{"leader":"01897cas a2200493 a 4500","fields":[{"001":"in00000000002"},{"005":"20200717090254.0"},{"006":"m o d "},{"007":"cr cnu||||||||"},{"008":"060504c20069999txufr pso 0 a0eng c"},{"010":{"subfields":[{"a":" 2006214613"}],"ind1":" ","ind2":" "}},{"035":{"subfields":[{"a":"(OCoLC)ocm68188263"}],"ind1":" ","ind2":" "}},{"040":{"subfields":[{"a":"NSD"},{"b":"eng"},{"c":"NSD"},{"d":"WAU"},{"d":"DLC"},{"d":"HUL"},{"d":"OCLCQ"},{"d":"OCLCF"},{"d":"OCL"},{"d":"AU@"}],"ind1":" ","ind2":" "}},{"012":{"subfields":[{"l":"1"}],"ind1":" ","ind2":" "}},{"019":{"subfields":[{"a":"1058285745"}],"ind1":" ","ind2":" "}},{"022":{"subfields":[{"a":"1931-7603"},{"l":"1931-7603"},{"2":"1"}],"ind1":"0","ind2":" "}},{"035":{"subfields":[{"a":"(OCoLC)68188263"},{"z":"(OCoLC)1058285745"}],"ind1":" ","ind2":" "}},{"042":{"subfields":[{"a":"pcc"},{"a":"nsdp"}],"ind1":" ","ind2":" "}},{"050":{"subfields":[{"a":"ISSN RECORD"}],"ind1":"1","ind2":"0"}},{"050":{"subfields":[{"a":"QL640"}],"ind1":"1","ind2":"4"}},{"082":{"subfields":[{"a":"598.1"},{"2":"14"}],"ind1":"1","ind2":"0"}},{"049":{"subfields":[{"a":"ILGA"}],"ind1":" ","ind2":" "}},{"130":{"subfields":[{"a":"Herpetological conservation and biology (Online)"}],"ind1":"0","ind2":" "}},{"210":{"subfields":[{"a":"Herpetol. conserv. biol."},{"b":"(Online)"}],"ind1":"0","ind2":" "}},{"222":{"subfields":[{"a":"Herpetological conservation and biology"},{"b":"(Online)"}],"ind1":" ","ind2":"0"}},{"245":{"subfields":[{"a":"Herpetological conservation and biology."}],"ind1":"1","ind2":"0"}},{"246":{"subfields":[{"a":"HCB"}],"ind1":"1","ind2":"3"}},{"260":{"subfields":[{"a":"[Texarkana, Tex.] :"},{"b":"[publisher not identified],"},{"c":"[2006]"}],"ind1":" ","ind2":" "}},{"310":{"subfields":[{"a":"Semiannual"}],"ind1":" ","ind2":" "}},{"336":{"subfields":[{"a":"text"},{"b":"txt"},{"2":"rdacontent"}],"ind1":" ","ind2":" "}},{"337":{"subfields":[{"a":"computer"},{"b":"c"},{"2":"rdamedia"}],"ind1":" ","ind2":" "}},{"338":{"subfields":[{"a":"online resource"},{"b":"cr"},{"2":"rdacarrier"}],"ind1":" ","ind2":" "}},{"362":{"subfields":[{"a":"Began with: Vol. 1, issue 1 (Sept. 2006)."}],"ind1":"1","ind2":" "}},{"500":{"subfields":[{"a":"Published in partnership with: Partners in Amphibian & Reptile Conservation (PARC), World Congress of Herpetology, Las Vegas Springs Preserve."}],"ind1":" ","ind2":" "}},{"650":{"subfields":[{"a":"Herpetology"},{"x":"Conservation"},{"v":"Periodicals."}],"ind1":" ","ind2":"0"}},{"650":{"subfields":[{"a":"Amphibians"},{"x":"Conservation"},{"v":"Periodicals."}],"ind1":" ","ind2":"0"}},{"650":{"subfields":[{"a":"Reptiles"},{"x":"Conservation"},{"v":"Periodicals."}],"ind1":" ","ind2":"0"}},{"655":{"subfields":[{"a":"Electronic journals."}],"ind1":" ","ind2":"0"}},{"710":{"subfields":[{"a":"Partners in Amphibian and Reptile Conservation."}],"ind1":"2","ind2":" "}},{"711":{"subfields":[{"a":"World Congress of Herpetology."}],"ind1":"2","ind2":" "}},{"776":{"subfields":[{"t":"Herpetological conservation and biology (Print)"},{"x":"2151-0733"},{"w":"(DLC) 2009202029"},{"w":"(OCoLC)427887140"}],"ind1":"1","ind2":" "}},{"841":{"subfields":[{"a":"v.1- (1992-)"}],"ind1":" ","ind2":" "}},{"856":{"subfields":[{"u":"http://www.herpconbio.org"},{"z":"Available to Lehigh users"}],"ind1":"4","ind2":"0"}},{"999":{"subfields":[{"s":"b5688c27-a352-4596-8396-802f27f035ea"},{"i":"93c42489-0ef0-4c1e-8c97-9ec8dbbd110d"}],"ind1":"f","ind2":"f"}}]}

 

Record example from "marc_records_lb"-table: 

id

content

id

content

2b55fe13-c4f4-40f1-b6b5-cce5c706635b

"{\"leader\":\"01897cas a2200493 a 4500\",\"fields\":[{\"001\":\"in00000000002\"},{\"005\":\"20200717090254.0\"},{\"006\":\"m o d \"},{\"007\":\"cr cnu||||||||\"},{\"008\":\"060504c20069999txufr pso 0 a0eng c\"},{\"010\":{\"subfields\":[{\"a\":\" 2006214613\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"035\":{\"subfields\":[{\"a\":\"(OCoLC)ocm68188263\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"040\":{\"subfields\":[{\"a\":\"NSD\"},{\"b\":\"eng\"},{\"c\":\"NSD\"},{\"d\":\"WAU\"},{\"d\":\"DLC\"},{\"d\":\"HUL\"},{\"d\":\"OCLCQ\"},{\"d\":\"OCLCF\"},{\"d\":\"OCL\"},{\"d\":\"AU@\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"012\":{\"subfields\":[{\"l\":\"1\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"019\":{\"subfields\":[{\"a\":\"1058285745\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"022\":{\"subfields\":[{\"a\":\"1931-7603\"},{\"l\":\"1931-7603\"},{\"2\":\"1\"}],\"ind1\":\"0\",\"ind2\":\" \"}},{\"035\":{\"subfields\":[{\"a\":\"(OCoLC)68188263\"},{\"z\":\"(OCoLC)1058285745\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"042\":{\"subfields\":[{\"a\":\"pcc\"},{\"a\":\"nsdp\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"050\":{\"subfields\":[{\"a\":\"ISSN RECORD\"}],\"ind1\":\"1\",\"ind2\":\"0\"}},{\"050\":{\"subfields\":[{\"a\":\"QL640\"}],\"ind1\":\"1\",\"ind2\":\"4\"}},{\"082\":{\"subfields\":[{\"a\":\"598.1\"},{\"2\":\"14\"}],\"ind1\":\"1\",\"ind2\":\"0\"}},{\"049\":{\"subfields\":[{\"a\":\"ILGA\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"130\":{\"subfields\":[{\"a\":\"Herpetological conservation and biology (Online)\"}],\"ind1\":\"0\",\"ind2\":\" \"}},{\"210\":{\"subfields\":[{\"a\":\"Herpetol. conserv. biol.\"},{\"b\":\"(Online)\"}],\"ind1\":\"0\",\"ind2\":\" \"}},{\"222\":{\"subfields\":[{\"a\":\"Herpetological conservation and biology\"},{\"b\":\"(Online)\"}],\"ind1\":\" \",\"ind2\":\"0\"}},{\"245\":{\"subfields\":[{\"a\":\"Herpetological conservation and biology.\"}],\"ind1\":\"1\",\"ind2\":\"0\"}},{\"246\":{\"subfields\":[{\"a\":\"HCB\"}],\"ind1\":\"1\",\"ind2\":\"3\"}},{\"260\":{\"subfields\":[{\"a\":\"[Texarkana, Tex.] :\"},{\"b\":\"[publisher not identified],\"},{\"c\":\"[2006]\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"310\":{\"subfields\":[{\"a\":\"Semiannual\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"336\":{\"subfields\":[{\"a\":\"text\"},{\"b\":\"txt\"},{\"2\":\"rdacontent\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"337\":{\"subfields\":[{\"a\":\"computer\"},{\"b\":\"c\"},{\"2\":\"rdamedia\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"338\":{\"subfields\":[{\"a\":\"online resource\"},{\"b\":\"cr\"},{\"2\":\"rdacarrier\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"362\":{\"subfields\":[{\"a\":\"Began with: Vol. 1, issue 1 (Sept. 2006).\"}],\"ind1\":\"1\",\"ind2\":\" \"}},{\"500\":{\"subfields\":[{\"a\":\"Published in partnership with: Partners in Amphibian & Reptile Conservation (PARC), World Congress of Herpetology, Las Vegas Springs Preserve.\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"650\":{\"subfields\":[{\"a\":\"Herpetology\"},{\"x\":\"Conservation\"},{\"v\":\"Periodicals.\"}],\"ind1\":\" \",\"ind2\":\"0\"}},{\"650\":{\"subfields\":[{\"a\":\"Amphibians\"},{\"x\":\"Conservation\"},{\"v\":\"Periodicals.\"}],\"ind1\":\" \",\"ind2\":\"0\"}},{\"650\":{\"subfields\":[{\"a\":\"Reptiles\"},{\"x\":\"Conservation\"},{\"v\":\"Periodicals.\"}],\"ind1\":\" \",\"ind2\":\"0\"}},{\"655\":{\"subfields\":[{\"a\":\"Electronic journals.\"}],\"ind1\":\" \",\"ind2\":\"0\"}},{\"710\":{\"subfields\":[{\"a\":\"Partners in Amphibian and Reptile Conservation.\"}],\"ind1\":\"2\",\"ind2\":\" \"}},{\"711\":{\"subfields\":[{\"a\":\"World Congress of Herpetology.\"}],\"ind1\":\"2\",\"ind2\":\" \"}},{\"776\":{\"subfields\":[{\"t\":\"Herpetological conservation and biology (Print)\"},{\"x\":\"2151-0733\"},{\"w\":\"(DLC) 2009202029\"},{\"w\":\"(OCoLC)427887140\"}],\"ind1\":\"1\",\"ind2\":\" \"}},{\"841\":{\"subfields\":[{\"a\":\"v.1- (1992-)\"}],\"ind1\":\" \",\"ind2\":\" \"}},{\"856\":{\"subfields\":[{\"u\":\"http://www.herpconbio.org\"},{\"z\":\"Available to Lehigh users\"}],\"ind1\":\"4\",\"ind2\":\"0\"}},{\"999\":{\"subfields\":[{\"s\":\"b5688c27-a352-4596-8396-802f27f035ea\"},{\"i\":\"93c42489-0ef0-4c1e-8c97-9ec8dbbd110d\"}],\"ind1\":\"f\",\"ind2\":\"f\"}}]}"

So, there are 100_000 the same records (with other UUIDs) for each of this tables.

 

3. Let`s measure response time for standard GET-endpoints in srs for 100_000 records. There will be 10 requests without timeout between them, and will be calculated average time for specific request. Moreover, first request always slower then other ones.

 

- /source-storage/records/{id} :

         

 Average time=50ms:

 

 

- /source-storage/records:

Parameters: Limit = 1000

 

Average time=4818ms(4.8seconds):

 

 

- /source-storage/records/{id}/formatted:

 

Average time=73ms:

 

 

- /source-storage/source-records/{id}:

 

Average time=61ms:

 

 

- /source-storage/source-records:

Parameters: Limit = 1000

 

 

Average time=1266ms (1.27seconds):

 

After this measuring, DB was filled with 500_000 records (with the same contents) for checking the difference. Resources are the same.

 

 

  • /source-storage/records/{id}: Average = 171ms(VS 50ms for 100_000 records)

  • /source-storage/records: Average = The same as for 100_000 records

  • /source-storage/records/{id}/formatted: Average = The same as for 100_000 records

  • /source-storage/source-records: Average = 1513ms(VS 1266ms for 100_000 records)

  • /source-storage/source-records/{id}: Average = The same as for 100_000 records