<!-- 
RSS generated by JIRA (1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d) at Thu Feb 08 23:07:45 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary add field=key&field=summary to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>FOLIO Jira</title>
    <link>https://folio-org.atlassian.net</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>1001.0.0-SNAPSHOT</version>
        <build-number>100246</build-number>
        <build-date>07-02-2024</build-date>
    </build-info>

<item>
            <title>[FOLIO-704] Investigate a central compound object index</title>
                <link>https://folio-org.atlassian.net/browse/FOLIO-704</link>
                <project id="10290" key="FOLIO">FOLIO</project>
                    <description>&lt;p&gt;As the front-end is currently desirous of a way to search for users based on related objects (e.g., get a list of all the users with permissions X, Y and Z), it makes sense to evaluate the benefits of maintaining an index of composite objects to facilitate this.&lt;/p&gt;

&lt;p&gt;For example, to execute the previously mentioned request, we have to query the permissions module first to get a list of all permission-association objects that contain permissions X, Y and Z. Then, from this list of objects, we have to compile a list of usernames that these objects possess. Then we have to query the user module to get a list of all user records that match our list of usernames. Then, depending on the return format requested by the client, we may have to join the user objects together with the permissions objects to return the compound result.&lt;/p&gt;

&lt;p&gt;The logic gets more complicated when we add in features like sorting the result set based on a field in a module outside of the users module, or doing pagination. We also run into the issue of degraded performance for the &quot;manual searching&quot; approach when we increase the number of records that need to be searched.&lt;/p&gt;

&lt;p&gt;A possible solution would be to use something that&apos;s happy indexing huge amounts of data (e.g. solr) and inserting composite records into this index.&lt;/p&gt;

&lt;p&gt;For the schema, we could adopt a dot notation to indicate the object and subfield. For example, to search by userid, you&apos;d use field &quot;user.id&quot;.  For a permission name, something like permissionsUser.permission_name.&lt;/p&gt;

&lt;p&gt;The good news is that this makes record retrieval stupidly easy. It&apos;s just a straightforward query, and translating between CQL and Solr queries is not hard. We&apos;d be free to introduce as much complex logic as we wanted, as well as choose fields to sort on and suchlike.&lt;/p&gt;

&lt;p&gt;The downside, of course, is building and maintaining the index. How can this be done in a reliable fashion with the least amount of burden on the maintainers of the individual modules that contribute to the composite record?&lt;/p&gt;

&lt;p&gt;For example, we could maintain a message queue and allow modules to push messages to it whenever a record is created, updated or deleted. These messages would then be consumed in order and used to update the composite record index.&lt;/p&gt;

&lt;p&gt;It is worth noting that Solr supports partial document updating...which seems particular relevant here, since no one module would be sending enough information to update an entire document. &lt;a href=&quot;https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Another option would be to try to implement this with no changes to the storage modules, and have a way to externally monitor for changes and then write them back to the indexes. One way to do this might be some kind of low level database trigger, though I worry that this might really violate the KISS principle. Another possibility could be a filter-level Okapi module that would listen for request types to various modules (e.g. PUT, POST, DELETE) and then create some kind of message into a queue for some process to query the module for changes and write these back to the index.&lt;/p&gt;

&lt;p&gt;Since this seems to potentially overlap several different issues, I&apos;d like to determine fairly quickly whether or not this is a road worth going down, or if we want to try to implement the &quot;manual searching&quot; solution for the short term, at least.&lt;/p&gt;</description>
                <environment></environment>
        <key id="79883">FOLIO-704</key>
            <summary>Investigate a central compound object index</summary>
                <type id="10003" iconUrl="https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10318?size=medium">Task</type>
                                            <priority id="10002" iconUrl="https://dev.folio.org/assets/jira-priority/jira-p3.svg">P3</priority>
                        <status id="10000" iconUrl="https://folio-org.atlassian.net/images/icons/statuses/generic.png" description="(Migrated on 4 Feb 2024 13:41 UTC)">Draft</status>
                    <statusCategory id="2" key="new" colorName="blue-gray"/>
                                    <resolution id="-1">Unresolved</resolution>
                                                        <assignee accountid="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c">shale99</assignee>
                                                                <reporter accountid="5c38e8d616ac1e4f7cbc660a">Kurt Nordstrom</reporter>
                                    <labels>
                            <label>core</label>
                            <label>for-next-sprint</label>
                            <label>sprint17</label>
                            <label>sprint18</label>
                            <label>sprint19</label>
                    </labels>
                <created>Thu, 29 Jun 2017 20:58:25 +0000</created>
                <updated>Mon, 14 Feb 2022 10:22:32 +0000</updated>
                                                                                <due></due>
                            <votes>0</votes>
                                    <watches>5</watches>
                                                    <timespent seconds="3600">1 hour</timespent>
                                <comments>
                                                            <comment id="190015" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Fri, 30 Jun 2017 11:11:33 +0000"  >&lt;p&gt;i think this is a good summary. I am 100% in favor of the solution using a messaging queue to push partial records into solr. i do not think this should be on the okapi level and should be kept at the storage / application layer. &lt;/p&gt;

&lt;p&gt;i see this as the kind of missing link of a distributed architecture that must combine data as part of the querying process. i dont see another way to do this efficiently. the partial record update may need some tweaking but still allows us to keep the modules independent while allowing biz modules to query across multiple modules efficiently.&lt;/p&gt;

</comment>
                                                            <comment id="190016" author="5c38e8d616ac1e4f7cbc660a" created="Fri, 30 Jun 2017 13:19:24 +0000"  >&lt;p&gt;I was hoping to avoid adding any complexity to the existing storage modules, but it is possible that this is an unrealistic goal. Of course, if there was a standard way to do this &quot;update queue push&quot; operation, it would simplify things.&lt;/p&gt;

&lt;p&gt;Would we see both the Solr&lt;img class=&quot;emoticon&quot; src=&quot;/images/icons/emoticons/help_16.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; index and the message queue as being provisioned by the same module?&lt;/p&gt;</comment>
                                                            <comment id="190017" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Fri, 30 Jun 2017 14:04:09 +0000"  >&lt;p&gt;i think we can template whats needed by a storage module so its simple enough to integrate. i think there are a bunch of details to cover here , but i think the first stage is to get to a decision on this and if we are going with this type of solution&lt;/p&gt;</comment>
                                                            <comment id="190018" author="5c38e8d616ac1e4f7cbc660a" created="Fri, 30 Jun 2017 14:25:30 +0000"  >&lt;p&gt;Agreed.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3Ab8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; class=&quot;user-hover&quot; rel=&quot;557058:b8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; data-account-id=&quot;557058:b8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; accountid=&quot;557058:b8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; rel=&quot;noreferrer&quot;&gt;Jakub Skoczen&lt;/a&gt; Do you have thoughts on adding this component to the system? Would it be worth prototyping?&lt;/p&gt;</comment>
                                                            <comment id="190019" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Mon, 3 Jul 2017 18:53:17 +0000"  >&lt;p&gt;two issues i thought about &lt;/p&gt;

&lt;p&gt;1. what fields do we store in the index? remember the key to the index is being small, this is key. do we store any fields in the index at all? or just index the fields for searching - then return an id of the matched composite record to the biz logic module - and have the biz logic module request records from the storage modules for display (based on what needs to be displayed). &amp;lt;- i lean towards this with maybe a handful of must have fields stored in the index itself&lt;/p&gt;

&lt;p&gt;2. lets take the user composite record - composed of - a user record + perms + creds + groups&lt;br/&gt;
The user record is indexed when a user is created - there is an id&lt;br/&gt;
perms are then created and associated with that user - the perms record has no reference to a user id - they are connected via the username -  hence we need to map the id to the username so that when we index the perms part of the composite record in the index we know that based on the username which user we need to update &amp;lt;- so we either map id to unique fields that are used by other modules for indexing parts of the composite or solr might have an easier way to do this&lt;/p&gt;</comment>
                                                            <comment id="190020" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Wed, 5 Jul 2017 11:28:35 +0000"  >&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;63940_thumb&quot; href=&quot;/rest/api/3/attachment/content/63940&quot; title=&quot;screenshot-1.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;63940&quot; file-preview-title=&quot;screenshot-1.png&quot;&gt;&lt;jira-attachment-thumbnail url=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/63940?default=false&quot; jira-url=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/63940&quot; filename=&quot;screenshot-1.png&quot;&gt;&lt;img src=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/63940&quot; data-attachment-name=&quot;screenshot-1.png&quot; data-attachment-type=&quot;thumbnail&quot; data-media-services-id=&quot;40780753-949e-4f69-850a-0dfad4a70c52&quot; data-media-services-type=&quot;file&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/jira-attachment-thumbnail&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;uploaded a screen shot &lt;/p&gt;

&lt;p&gt;i think if we start down this path we can start with a trigger in postgres which will call pg_notify - this is a wrapper for the Notify / Listen mechanism in postgres which allows external processes to receives notifications from postgres on things like data updates / deletes / inserts.&lt;/p&gt;

&lt;p&gt;there will be a need for an async driver that supports this (there are), and then all changes are streamed async to the client (can be solr / or something light above solr running in the same web server - dont know yet)&lt;/p&gt;

&lt;p&gt;the notify mechanism has channels so that each tenant can have his own (mod-users.nyu channel for example)&lt;/p&gt;

&lt;p&gt;solr has near rel time searching so that within a few seconds indexed data is searchable. this means that this data has not yet been persisted to disk so that these entries may need to be replayed on index crash&lt;/p&gt;

&lt;p&gt;open issues:&lt;br/&gt;
1. max payload 8kb - should be more then enough for metadata - will be problematic if we need to index full texts or something in the future&lt;br/&gt;
2. error handling - havent looked into this enough - what happens if indexing fails to records that have been streamed by the trigger to the client&lt;br/&gt;
3. how does this work with multiple clients&lt;/p&gt;</comment>
                                                            <comment id="190021" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Thu, 6 Jul 2017 13:28:57 +0000"  >&lt;p&gt;phase 1: simple implementation - more error prone:&lt;/p&gt;

&lt;p&gt;I have a simple java app prototype which listens on changes to a table in postgres (Notify / Listen Postgres mechanism) - there is an async jdbc driver option but that doesnt work on windows &lt;img class=&quot;emoticon&quot; src=&quot;/images/icons/emoticons/sad.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; so currently using jdbc driver. and then indexes the records to solr. &lt;/p&gt;

&lt;p&gt;so basically &lt;br/&gt;
1. storage module persists data (as today)&lt;br/&gt;
2. new trigger is called which notifies / queues data &lt;tt&gt;perform pg_notify(&apos;moduleId_tenantId&apos;, NEW.jsonb::text);&lt;/tt&gt; a delete will send out &lt;tt&gt;OLD&lt;/tt&gt;&lt;br/&gt;
   a. this is per channel - so that we set up a channel per tenant per module &apos;moduleid_tenantid&apos;&lt;br/&gt;
3. listener app gets data from postgres&apos;s message queue (jsonb) - this can probably be part of the solr instance potentially&lt;br/&gt;
4. app has a mapping (configured) to add a partial record (for example perms) to an existing record (for example user) in the index via username. app sends json to solr for indexing (need more clarity here)&lt;br/&gt;
6. commit / soft commit options&lt;/p&gt;

&lt;p&gt;this will be better served using a replayable / highly avail / persistent message queue - where we can protect against crashes between soft and hard commits / etc...&lt;/p&gt;

&lt;p&gt;the one BIG problem with this:&lt;br/&gt;
1. postgres will not queue messages when no clients are listening - this is bad. meaning if there are network issues / crash of the client apps getting the data - we lose data. if we have a replicated kafka / amq / etc... queue then this may make the use case less likely - but still an issue&lt;/p&gt;

&lt;p&gt;other options:&lt;br/&gt;
1. trigger on data updates into a new table in postgres - can contain only ids - and then indexer pulls records and deletes once processing (indexing) comlete&lt;br/&gt;
2. has a status table in postgres with date of last index - select from data table newer records then this date&lt;/p&gt;</comment>
                                                            <comment id="190022" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Sun, 9 Jul 2017 15:56:45 +0000"  >&lt;p&gt;some additional info about the notify / listen mechanism in postgres.&lt;br/&gt;
if multiple records are inserted with the same content in a transaction - the notify will only issue a notify on the last record. this de-duplication seems to happen with postgres comparing the content of the data to send for the notification&lt;br/&gt;
a simple test of:&lt;/p&gt;

&lt;p&gt;inserting &lt;b&gt;10,000&lt;/b&gt; new records with the same json content - with the notification data containing the json itself - creates a single notification (&lt;b&gt;250&lt;/b&gt; milliseconds db time)&lt;/p&gt;

&lt;p&gt;inserting &lt;b&gt;10,000&lt;/b&gt; new records with the same json content - with the notification data containing the json itself with a dynamic id - &lt;br/&gt;
creates 10,000 notification (&lt;b&gt;750&lt;/b&gt; milliseconds db time) &lt;/p&gt;

&lt;p&gt;the performance spikes exponentially at &lt;b&gt;100,000&lt;/b&gt; inserts as it seems postgres has an O(n2) of duplication checks over the inserted records - or something else that causes a major slowdown....&lt;/p&gt;

&lt;p&gt;inserting &lt;b&gt;100,000&lt;/b&gt; new records with the same json content - with the notification sending the json itself with a dynamic id - &lt;br/&gt;
creates 100,000 notification (&lt;b&gt;2 minutes&lt;/b&gt; db time)&lt;br/&gt;
if only the json content is sent (which is the same, without the dynamic id, this returns in under a second)&lt;/p&gt;
</comment>
                                                            <comment id="190023" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Mon, 10 Jul 2017 07:01:45 +0000"  >&lt;p&gt;ok, so of course SOLR has an answer to what we need - should have started there!&lt;br/&gt;
READ ONLY THIS..... &lt;img class=&quot;emoticon&quot; src=&quot;/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;in SOLR there is a dataImportHandler&lt;br/&gt;
this allows connecting either to database tables directly or to web services and mapping results to records in SOLR. SOLR uses a last update date internally to keep track of the last records it requested from the database.&lt;/p&gt;

&lt;p&gt;an example how to user this for the mod-users dev:&lt;br/&gt;
1. select * from users where update_date &amp;gt; value_solr_saved from the last select and join this via groupId on the groups table &lt;br/&gt;
2. map the json fields returned from the join into fields in the composite document fields in solr&lt;br/&gt;
3. for each record returned extract the username and select permissions for the username from the permissions table and map those to fields in the composite user document fields in solr&lt;/p&gt;

&lt;p&gt;1. yes! this can also be done via http and not only via direct database access&lt;br/&gt;
2. i think! there is a scheduler to run this automatically, but there is for sure an http api to force updates. you can pass a query param to the api to indicate if you want a delta update (via the last update date) or a full re-index -&amp;gt; no update date passed in&lt;/p&gt;

&lt;p&gt;my current open issue:&lt;br/&gt;
1. how to have changes in permissions - which is embedded in the user composite record in the index update the master composite record. triggering an update when the user is updated is not enough &lt;br/&gt;
2. performance - we would not want to query permissions via usernames one at a time - need an alternative&lt;/p&gt;

&lt;p&gt;getting fancy:&lt;br/&gt;
combine postgres&apos;s NOTIFY / LISTEN mechanism so that we trigger updates immediately into solr when they occur in the db. and in case of a solr crash, after crash recovery issue an api call to solr to update what it missed via the last update date while it was down&lt;/p&gt;
</comment>
                                                            <comment id="190024" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Mon, 10 Jul 2017 08:07:32 +0000"  >&lt;p&gt;Another issue is handling deletes.&lt;br/&gt;
deletes remove records from the table which means querying the table wont return those records so we wont know to delete them from the index - so we either need to mark deleted records as deleted and not physically delete them immediately or have a trigger pushing all inserts / updates / deletes into a queue table (with a column indicating the operation). then solr would need to issue a query similar to:&lt;/p&gt;


&lt;div class=&quot;code panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;codeContent panelContent&quot;&gt;
&lt;pre class=&quot;code-java&quot;&gt;
DELETE FROM queue
WHERE userId = (
  SELECT userId 
  FROM queue
  ORDER BY date
  FOR UPDATE SKIP LOCKED &amp;lt;------not nessarily needed
  LIMIT 1
)
RETURNING *;
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;


&lt;p&gt;this will pull the record from the queue table (without locking other records so that this can happen concurrently if needed) and will delete on commit - need to see if this is supported in the data import handler (sending a query and then a commit - or if a customer handler is needed)&lt;/p&gt;</comment>
                                                            <comment id="190025" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Sun, 16 Jul 2017 11:00:44 +0000"  >&lt;p&gt;SUMMARY:&lt;/p&gt;

&lt;p&gt;I think we should take a step back from the central index for the following reasons:&lt;/p&gt;

&lt;p&gt;if we have a requirement that a tenant&apos;s data should be in a single DB (by a single DB i mean , this can be a db cluster of multiple postgres servers which holds data for lets say 100 tenants)&lt;/p&gt;

&lt;p&gt;then:&lt;br/&gt;
since we are using the schema per tenant concept we can actually join across schemas by just including the schema as a namespace (prefix) &lt;br/&gt;
we still want to abstract the master data tables within the schemas (modules) from this joining / querying (searching) functionality - so each module can actually&lt;/p&gt;

&lt;p&gt;create a &lt;b&gt;read only view&lt;/b&gt; of tables a module would like to expose. this &lt;b&gt;read only view&lt;/b&gt; can serve as an &lt;em&gt;interface&lt;/em&gt; to the data of the module - note that a view in this case is actually just a prepared statement (a query) - so that when there are upgrades the module can decide whether the query should be updated. If we take mod-users for example, this view can be a join query between the users and groups tables to create the read only data interface exposed by the module.&lt;br/&gt;
We can then have a dedicated search module (preferable, since this can be a storage module with access to the db), or the biz modules themselves (will need to send the query to a storage module so not as clean) getting an indication of the views exposed by the storage modules to the data the biz module needs and then a cross view join can be executed to retrieve / sort / etc... the results&lt;/p&gt;

&lt;p&gt;what is the upside:&lt;br/&gt;
1. no data replication&lt;br/&gt;
2. no sync issues between persisted data and indexed data&lt;br/&gt;
3. less modules to manage&lt;/p&gt;

</comment>
                                                            <comment id="190026" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Mon, 17 Jul 2017 09:44:55 +0000"  >&lt;p&gt;a bit more details (comments welcomed as i may be just going off on a tangent)&lt;/p&gt;

&lt;p&gt;a storage module exposes the interface for its entities (aka a read only view it creates which does not have to have the same content as the actual data tables)&lt;br/&gt;
for example:&lt;br/&gt;
&lt;b&gt;Permissions:&lt;/b&gt;&lt;/p&gt;
&lt;div class=&quot;code panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;codeContent panelContent&quot;&gt;
&lt;pre class=&quot;code-java&quot;&gt;
{
    &lt;span class=&quot;code-quote&quot;&gt;&quot;entities&quot;&lt;/span&gt;: [
        {
        &lt;span class=&quot;code-quote&quot;&gt;&quot;entity&quot;&lt;/span&gt; : &lt;span class=&quot;code-quote&quot;&gt;&quot;perms&quot;&lt;/span&gt; , 
        &lt;span class=&quot;code-quote&quot;&gt;&quot;view&quot;&lt;/span&gt; : &lt;span class=&quot;code-quote&quot;&gt;&quot;myuniversity_mymodule.perms&quot;&lt;/span&gt;,
        &lt;span class=&quot;code-quote&quot;&gt;&quot;references&quot;&lt;/span&gt;: [
            {&lt;span class=&quot;code-quote&quot;&gt;&quot;entity&quot;&lt;/span&gt; : &lt;span class=&quot;code-quote&quot;&gt;&quot;users&quot;&lt;/span&gt;, &lt;span class=&quot;code-quote&quot;&gt;&quot;key&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;userName&quot;&lt;/span&gt;, &lt;span class=&quot;code-quote&quot;&gt;&quot;foriegnKey&quot;&lt;/span&gt; :&lt;span class=&quot;code-quote&quot;&gt;&quot;userName&quot;&lt;/span&gt;}
        ]},
    ]
}
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;b&gt;Users:&lt;/b&gt;&lt;/p&gt;
&lt;div class=&quot;code panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;codeContent panelContent&quot;&gt;
&lt;pre class=&quot;code-java&quot;&gt;
{
    &lt;span class=&quot;code-quote&quot;&gt;&quot;entities&quot;&lt;/span&gt;: [
        {
        &lt;span class=&quot;code-quote&quot;&gt;&quot;entity&quot;&lt;/span&gt; : &lt;span class=&quot;code-quote&quot;&gt;&quot;users&quot;&lt;/span&gt; , 
        &lt;span class=&quot;code-quote&quot;&gt;&quot;view&quot;&lt;/span&gt; : &lt;span class=&quot;code-quote&quot;&gt;&quot;myuniversity_mymodule.users&quot;&lt;/span&gt;,
        &lt;span class=&quot;code-quote&quot;&gt;&quot;references&quot;&lt;/span&gt;: [
            {&lt;span class=&quot;code-quote&quot;&gt;&quot;entity&quot;&lt;/span&gt; : &lt;span class=&quot;code-quote&quot;&gt;&quot;groups&quot;&lt;/span&gt;, &lt;span class=&quot;code-quote&quot;&gt;&quot;key&quot;&lt;/span&gt; :&lt;span class=&quot;code-quote&quot;&gt;&quot;groupId&quot;&lt;/span&gt;}
        ]},
        {&lt;span class=&quot;code-quote&quot;&gt;&quot;entity&quot;&lt;/span&gt; : &lt;span class=&quot;code-quote&quot;&gt;&quot;groups&quot;&lt;/span&gt; , &lt;span class=&quot;code-quote&quot;&gt;&quot;view&quot;&lt;/span&gt; : &lt;span class=&quot;code-quote&quot;&gt;&quot;myuniversity_mymodule.groups&quot;&lt;/span&gt;}
    ]
}
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;for the simplicity of the example, lets say mod-users-bl got a query param &lt;tt&gt;include=perms&lt;/tt&gt; - so that it wants a composite user objects with added permissions.&lt;/p&gt;

&lt;p&gt;1. mod-users-bl makes a request to mod-search (this can even be mod-users &lt;span class=&quot;error&quot;&gt;&amp;#91;as it is the core module&amp;#93;&lt;/span&gt; that supplies the core entity without the need for another module)&lt;br/&gt;
The request made contains the request for users + perms + cql&lt;br/&gt;
2. mod-users calls the perms api to get the data interface (above) - this api can be handled by rmb and can look for a json file created by the developer for the particular module&apos;s version&lt;br/&gt;
3. mod-users then joins its interface with the perms interface using the reference keys (the keys are based on the view) and appends the cql translated to sql clause&lt;br/&gt;
4. mod-users returns the correct composite object / objects&lt;/p&gt;

&lt;p&gt;thoughts???&lt;/p&gt;</comment>
                                                            <comment id="190027" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Mon, 17 Jul 2017 12:09:12 +0000"  >&lt;p&gt;last zig zag i hope (it&apos;s a tough subject)&lt;/p&gt;

&lt;p&gt;The above will work well for retrieving composite records and sorting, etc... &lt;br/&gt;
It will however be limited in the search functionality it will provide.&lt;br/&gt;
For example, we can find solutions for case insensitive searching as well as stop words removal (as these can be handled within the query). it starts getting a bit more complicated with things like stemming support and the likes. there is stuff we can do in the query to some extent - but with limitations - for example, if someone queries &lt;b&gt;all the kings men&lt;/b&gt; - we can run a stemmer on a per language basis and fire off 4 queries (need to understand the performance implications) with things like:&lt;br/&gt;
all the king man&lt;br/&gt;
all the king men&lt;br/&gt;
all the kings man&lt;br/&gt;
all the kings men (the original query)&lt;br/&gt;
but ranking will not be something we can really do in order to order the results correctly.&lt;/p&gt;

&lt;p&gt;To summarize:&lt;/p&gt;

&lt;p&gt;the simplest solution which would allow case insensitive search and stop words removal search (this would be good enough for matching partial fields / full fields) with sorting , etc... can be done via the db search solution suggested.&lt;/p&gt;

&lt;p&gt;if we need more advanced search features and ranking, then we need an alternative.&lt;br/&gt;
either solr / elastic - or something like postgres full text (i do not know enough about it to give an opinion)&lt;/p&gt;
</comment>
                                                            <comment id="190028" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Mon, 24 Jul 2017 10:00:28 +0000"  >&lt;p&gt;a few more thoughts (hopefully the last until we discuss)&lt;/p&gt;

&lt;p&gt;1. if we do not go with a dedicated index (solr, es) we would probably need a dedicated mod-search &lt;br/&gt;
    a. the reason for this , i think, is because storage modules will not be composite object aware - so they wont have a concept of returning data that belongs to other modules&lt;br/&gt;
    b. and the biz logic module does not have db access&lt;br/&gt;
    c. so there needs to be a module that can return composite objects and does have db access&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;63942_thumb&quot; href=&quot;/rest/api/3/attachment/content/63942&quot; title=&quot;screenshot-3.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;63942&quot; file-preview-title=&quot;screenshot-3.png&quot;&gt;&lt;jira-attachment-thumbnail url=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/63942?default=false&quot; jira-url=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/63942&quot; filename=&quot;screenshot-3.png&quot;&gt;&lt;img src=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/63942&quot; data-attachment-name=&quot;screenshot-3.png&quot; data-attachment-type=&quot;thumbnail&quot; data-media-services-id=&quot;52ee038e-269c-4192-b259-d52fbbc548fd&quot; data-media-services-type=&quot;file&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/jira-attachment-thumbnail&gt;&lt;/a&gt;&lt;/span&gt;  &lt;/p&gt;

&lt;p&gt;biz modules would query by indicating the composite object and a cql&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10000">
                    <name>Blocks</name>
                                            <outwardlinks description="blocks">
                                        <issuelink>
            <issuekey id="35349">MODUSERBL-7</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="79877">FOLIO-677</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10003">
                    <name>Relates</name>
                                                                <inwardlinks description="relates to">
                                        <issuelink>
            <issuekey id="79814">FOLIO-611</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="35336">MODUSERBL-3</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="80221">FOLIO-705</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="63940" name="screenshot-1.png" size="39705" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Wed, 5 Jul 2017 11:21:59 +0000"/>
                            <attachment id="63941" name="screenshot-2.png" size="27262" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Sun, 9 Jul 2017 11:27:35 +0000"/>
                            <attachment id="63942" name="screenshot-3.png" size="29992" author="712020:32bb56ac-50e7-4787-b4af-ed3089d9401c" created="Mon, 24 Jul 2017 09:59:36 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10000" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummarycf">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10057" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Development Team</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10144"><![CDATA[Core: Platform]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10019" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|hzxqcn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10020" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="773">CP: Roadmap backlog</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10024" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>[CHART] Date of First Response</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 30 Jun 2017 11:11:33 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                </customfields>
    </item>
</channel>
</rss>