Upgrade R2-2021 to R3-2021 (Kiwi)

Owned by Ingolf Kuss
Last updated: May 11, 2022
12 min read

Installing "platform-complete" on a Single Server (Single Virtual Machine). This assumes you have already installed Juniper (GA or Hotfix#3, but it doesn't really matter which hotfix).

OS: Ubuntu 20.04.2 LTS

Deployment with Okapi
Database contents of Juniper Install will be migrated to Kiwi automatically

Hardware Requirements

 8 CPUs
 40 GB RAM
recommended: 350 GB SSD HD

First do Ubuntu Updates & Upgrades
  

sudo apt-get update
sudo apt-get upgrade
sudo reboot

This may need to be done twice, until you get the message "0 updates can be applied immediately" upon login.
Check if all Services have been restarted after reboot: Okapi, postgres, docker; the docker containers  (do: docker ps --all | more ).

Is the Stripes container running ? If not, restart it.

Kiwi Release Notes (only for reference; I describe here the actions that I have done to successfully upgrade) :  https://folio-org.atlassian.net/wiki/display/REL/Kiwi+%28R3+2021%29+Release+Notes

Actions that I took (based on the Release Notes) before the upgrade: 

I.) Before the Upgrade:

i.) Change all duplicate item barcodes

Item barcode is unique now. Duplicate item barcodes fail the upgrade.
Before upgrade: Change all duplicate item barcodes. Find them with this SQL:
psql folio
SET search_path TO diku_mod_inventory_storage;
SELECT lower(jsonb->>'barcode')
FROM item
GROUP BY 1
HAVING count(*) > 1;
 lower
-------
(0 rows)
Use Inventory Item Barcode search to edit the duplicate barcode.

ii.) Change all holdings sources to FOLIO

ii.)  Holdings created by a MARC Bib, not a MARC Holdings, showed the source = MARC. The behavior was changed to show source = FOLIO for such holdings. DB tables might contain holdings records with incorrect source value.
see here MODSOURMAN-627 - Script for retrieving holding by specific conditions. :
# Select Holdings where sourceName are not FOLIO and MARC

SET search_path TO diku_mod_inventory_storage;
SELECT *
FROM holdings_record
WHERE holdings_record.jsonb ->> 'sourceId' = (
  SELECT id::text
  FROM holdings_records_source
  WHERE holdings_records_source.jsonb ->> 'name' != 'FOLIO' AND
    holdings_records_source.jsonb ->> 'name' != 'MARC');
(0 rows)

(END)

If the holdings source is anything other than FOLIO or MARC (e.g. -), then change to FOLIO. 

iii.) Install Elasticsearch

If you have not already done so in your Juniper Install (it was optional there) install Elasticsearch now in your running Juniper instance. Follow this guide to install a 3-node Elasticsearch cluster on a Single Server: Installation of Elasticsearch. This also install mod-search and the frontend modules  folio_inventory-es and folio_search in Juniper.

iv.) Install a minIO-Server

If you want to use Data Export, you either have to use Amazon S3 or a minIO server.
So, for anyone who plans to use MinIO server instead of Amazon S3:
External storage for generated MARC records should be configured to MinIO server by changing ENV variable AWS_URL.
Installation of a MinIO server is not being covered in this documentation (yet). Refer to:
https://docs.min.io/minio/baremetal/installation/deployment-and-management.html#minio-installation
https://docs.min.io/minio/baremetal/installation/deploy-minio-standalone.html#deploy-minio-standalone-container

v.) .... there might be more preparatory steps one should take, but I haven't done any of them (yet ! Maybe I will need to, later) If you are unsure what other steps you might need to take, study the Release Notes.

II.) BEGIN of Main Processing: Upgrade Juniper => Kiwi (R3-2021)

This documentation assumes that you have Juniper Hotfix#3 running. Upgrade procedures for other Hotfixes or the GA Release might vary slightly. In particular, if this documentation refers to Juniper Release module versions, check if you have exactly that version running and if not, use the version that you had deployed.

II.i) Upgrade the Okapi Version / Restart Okapi

Install and configure Okapi

This needs to be done first, otherwise Okapi can not pull the new modules.

The Okapi Release Version can be read in platform-complete/install.json, so this needs to be downloaded first:

Clone the repository , change into that directory: 

git clone https://github.com/folio-org/platform-complete
cd platform-complete
git fetch

There is a new Branch R3-2021-hotfix-2. We will deploy this version.
Check out this Branch.
Stash local changes. This should only pertain to stripes.config.js .
Discard any changes which you might have made in Juniper on install.json etc.:

 git restore install.json
 git restore okapi-install.json
 git restore stripes-install.json
 git restore package.json

 git stash save
 git checkout master
 git pull
 git checkout R3-2021-hotfix-2
 git stash pop

Read  the R3 Okapi version from install.json: okapi-4.11.1

Fetch Okapi as a Debian package from repository.folio.org . Import the FOLIO signing key, add the FOLIO apt repository, install okapi (of this release):

wget --quiet -O - https://repository.folio.org/packages/debian/folio-apt-archive-key.asc | sudo apt-key add -
sudo add-apt-repository "deb https://repository.folio.org/packages/ubuntu focal/"
sudo apt-get update
sudo apt-get -y --allow-change-held-packages install okapi=4.11.1-1

Start Okapi in cluster mode

I install Okapi in cluster mode, because this is the appropriate way to install it in a production environment (although it is being done on a single server here, this procedure could also be applied to a multi-server environment). Strictly speaking, on a single server, it is not necessary to deploy Okapi in cluster mode. So you might want to stay with the default role "dev", instead.

Change the port range in okapi.conf . Compared to Juniper, this needs to be done now, because Elasticsearch will occupy ports 9200 and 9300 (or is already occupying them) :

vim /etc/folio/okapi/okapi.conf
# then change the following lines to:
      - role="cluster"
      - cluster_config="-hazelcast-config-file /etc/folio/okapi/hazelcast.xml"
      - cluster_port="9001"
      - port_start="9301"
      - port_end="9520"
      - host="10.9.2.85"  # change to your host's IP address
      - nodename="10.9.2.85"

You can find out the node name that Okapi uses like this: curl -X GET http://localhost:9130/_/discovery/nodes . You have to use that value if you deploy Okapi in  cluster mode. Not your host name and not "localhost".

Edit interface and members in hazelcast.xml (if you deploy Okapi in cluster mode). If you run Okapi on a single server, this is your local IP address. 

vim /etc/folio/okapi/hazelcast.xml
...
<tcp-ip enabled="true">
                <interface>10.9.2.85</interface>
                <member-list>
                    <member>10.9.2.85</member>
                </member-list>
</tcp-ip>

Send new Environment Variables for Hazelcast to Okapi (in case you haven't been using Hazelcast so far):

curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"OKAPI_CLUSTERHOST\",\"value\":\"10.9.2.85\"}" http://localhost:9130/_/env
curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"HAZELCAST_IP\",\"value\":\"10.9.2.85\"}" http://localhost:9130/_/env
curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"HAZELCAST_PORT\",\"value\":\"5701\"}" http://localhost:9130/_/env
curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"HAZELCAST_FILE\",\"value\":\"/etc/folio/okapi/hazelcast.xml\"}" http://localhost:9130/_/env

Restart Okapi:

sudo systemctl daemon-reload
sudo systemctl restart okapi.service

Follow /var/log/folio/okapi/okapi.log . You should read something like this:

INFO DeploymentManager shutdown
...

Now Okapi will re-start your modules. Follow the okapi.log. It will run for 2 or 3 minutes. Check if all modules are running:

docker ps --all | grep "mod-" | wc
  62

Retrieve the list of modules which are now being enabled for your tenant (just for your information):

curl -w '\n' -XGET http://localhost:9130/_/proxy/tenants/diku/modules
...
}, {
  "id" : "okapi-4.11.1"
} ]

You should see 9 Edge modules (Starting from Juniper HF#3; if you have started from Juniper-GA, you will see only 8 Edge modules), 52 Frontend modules (folio_*), 62 Backend modules (mod-*) (These are the modules of Juniper, platform-complete) + the Kiwi-Version of Okapi (4.11.1).

II.ii) Pull module descriptors from central registry

curl -w '\n' -D - -X POST -H "Content-type: application/json" -d '{ "urls": [ "https://folio-registry.dev.folio.org" ] }' http://localhost:9130/_/proxy/pull/modules

Okapi log should show something like

2022-03-17T12:51:44,073 INFO  ProxyContext         283828/proxy REQ 127.0.0.1:51424 supertenant POST /_/proxy/pull/modules  okapi-4.11.1
2022-03-17T12:51:46,222 INFO  PullManager          Remote registry at https://folio-registry.dev.folio.org is version 4.11.1
2022-03-17T12:51:46,222 INFO  PullManager          pull smart
...
2022-03-17T12:53:14,148 INFO  PullManager          pull: 3466 MDs to insert
2022-03-17T12:53:17,170 INFO  ProxyContext         283828/proxy RES 200 93096323us okapi-4.11.1 /_/proxy/pull/modules


II.iii) Deploy a compatible FOLIO backend

1.) Post data source information to the Okapi environment for use by deployed modules

Send new environment variables for Elasticsearch and mod-search to the Okapi environment:

  curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"ELASTICSEARCH_HOST\",\"value\":\"10.9.2.85\"}" http://localhost:9130/_/env;
  curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"ELASTICSEARCH_URL\",\"value\":\"http://10.9.2.85:9200\"}" http://localhost:9130/_/env;
  curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"ELASTICSEARCH_USERNAME\",\"value\":\"elastic\"}" http://localhost:9130/_/env;
  curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"ELASTICSEARCH_PASSWORD\",\"value\":\"s3cret\"}" http://localhost:9130/_/env;
  curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"INITIAL_LANGUAGES\",\"value\":\"eng, ger, swe\"}" http://localhost:9130/_/env;

Change 10.9.2.85 to your local IP address. Choose a  safe password ELASTICSEARCH_PASSWORD.  You can choose up to five INITIAL_LANGUAGES. To find out the language codes, view here folio-org/mod-search (github.com).

The Okapi environment should now look something like this:

  curl -X GET http://localhost:9130/_/env
[ {
  "name" : "DB_DATABASE",
  "value" : "folio"
}, {
  "name" : "DB_HOST",
  "value" : "10.9.2.62"
}, {
  "name" : "DB_PASSWORD",
  "value" : "folio123"
}, {
  "name" : "DB_PORT",
  "value" : "5432"
}, {
  "name" : "DB_USERNAME",
  "value" : "folio"
}, {
  "name" : "ELASTICSEARCH_HOST",
  "value" : "10.9.2.85"
}, {
  "name" : "ELASTICSEARCH_PASSWORD",
  "value" : "s3cret"
}, {
  "name" : "ELASTICSEARCH_URL",
  "value" : "http://10.9.2.85:9200"
}, {
  "name" : "ELASTICSEARCH_USERNAME",
  "value" : "elastic"
}, {
  "name" : "HAZELCAST_FILE",
  "value" : "/etc/folio/okapi/hazelcast.xml"
}, {
  "name" : "HAZELCAST_IP",
  "value" : "10.9.2.85"
}, {
  "name" : "HAZELCAST_PORT",
  "value" : "5701"
}, {
  "name" : "INITIAL_LANGUAGES",
  "value" : "ger, eng, fre, spa"
}, {
  "name" : "KAFKA_HOST",
  "value" : "10.9.2.85"
}, {
  "name" : "KAFKA_PORT",
  "value" : "9092"
}, {
  "name" : "OKAPI_CLUSTERHOST",
  "value" : "10.9.2.85"
}, {
  "name" : "OKAPI_URL",
  "value" : "http://10.9.2.85:9130"
}, {
  "name" : "SYSTEM_USER_PASSWORD",
  "value" : "pub-sub"
} ]

If you choose a different SYSTEM_USER_PASSWORD than the default (which you should do on a production system), then you will have to change the password of the user "pub-sub" in the system to the same value. Change the password of the user pub-sub via Settings - Passwords menu. Do this before you re-deploy pubsub. Thus, do this now, as the re-deployment of pub-sub is the next step.

2.) Deploy the backend modules

Sidestep: Look, how many containers are running already now:

sudo docker ps | grep -v "^CONTAINER" | wc -l

 68 containers are running:
 - 62 Backend Modules of R2-2021
 - Stripes with nginx
 - 3 Nodes Elasticsearch
 - Kafka & Zookeeper

2.1. Upgrade and enable mod-pubsub

 curl -w '\n' -D - -X POST -H "Content-type: application/json" -d '[ { "id" : "mod-pubsub-2.4.3", "action" : "enable" } ]' http://localhost:9130/_/proxy/tenants/diku/install?simulate=true
  curl -w '\n' -D - -X POST -H "Content-type: application/json" -d '[ { "id" : "mod-pubsub-2.4.3", "action" : "enable" } ]' http://localhost:9130/_/proxy/tenants/diku/install?deploy=true\&preRelease=false\&tenantParameters=loadReference%3Dtrue
HTTP/1.1 200 OK

The old module instance, mod-pubsub-2.3.3, has been automatically undeployed by this operation.

2.2 Deploy all the other backend modules

Remove mod-pubsub-2.4.3 from the list ~/platform-complete/okapi-install.json because it has already been deployed.

Deploy all backend modules with this single script deploy-all-backend-modules.sh (you will also need this script deploy-backend-module.sh) :

This will download all the necessary container images for the backend modules from Docker Hub and deploy them as containers to the local system :

./deploy-all-backend-modules.sh ~/platform-complete/okapi-install.json 10.9.2.85

This script will run for approx. 15 minutes. It will spin up one Docker container for each backend module using Okapi's /discovery/modules endpoint.

You can follow the progress on the terminal screen and/or in /var/lib/folio/okapi/okapi.log .

Now the R3 (Kiwi) backend modules have been deployed, but have not yet been enabled for your tenant.

In addition, the R2 (Juniper) containers are still running on your system. The latter are enabled. This means that, right now, the system is still in the state "R2-2021 (Juniper)", except for Okapi (but the Okapi is downward compatible to R2) and mod-pubsub.

There are 61 backend modules of R2 (all, except for mod-pubsub) and 65 backend modules of R3 running as containers on your system. Check this by typing

docker ps --all | grep "mod-" | wc
    126

Side Remark: 3 of the backend modules have been deployed twice in the same version, because for those backend modules, the R2 version is the same as the R3 version. These modules are:

  mod-service-interaction:1.0.0
  mod-graphql:1.9.0
  mod-z3950-2.4.0  .

II.iv) Enable the modules for your tenant

 Enable the R3 modules for your tenant

Remove mod-pubsub-2.4.3 and okapi from ~/platform-complete/install.json because they have already been enabled.

Enable frontend and backend modules in a single post. But first, do a simulate run. Don't deploy the modules because they have already been deployed (use the parameter deploy=false). Load reference data of the modules, but no sample data. If you don't load reference data for the new modules, you might not be able to utilize the modules properly.

curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @/usr/folio/platform-complete/install.json http://localhost:9130/_/proxy/tenants/diku/install?simulate=true\&preRelease=false
curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @/usr/folio/platform-complete/install.json http://localhost:9130/_/proxy/tenants/diku/install?deploy=false\&preRelease=false\&tenantParameters=loadReference%3Dtrue
...
HTTP/1.1 100 Continue

HTTP/1.1 200 OK

Side Remark: folio_inventory-es-6.4.0, a frontend module POC for using mod-inventory with Elasticsearch, has been kept and even updated. But in Kiwi, you won`t need it, because there, mod-inventory and folio_inventory work with Elasticsearch, anyway. Therefore, you might want to disable folio_inventory-es-6.4.0.

Just for your information: Look, how many backend modules have now been deployed on your server:

 curl -w '\n' -D - http://localhost:9130/_/discovery/modules | grep srvcId | wc
  128

The number 128 includes 5  Edge modules, also. Another important information: Get a list of modules (frontend + backend) that have now been enabled for your tenant:

curl -w '\n' -XGET http://localhost:9130/_/proxy/tenants/diku/modules | grep id | wc
  132

This number is the sum of the following:

 - 56 Frontend modules (including folio_inventory-es)
 - 10 Edge modules
 - 65 Backend modules (R3-2021)
 - 1 Okapi module (4.11.1)

These are all R3 (Kiwi) modules.

II.v) Build an R3-2021 FOLIO Stripes platform

Install Stripes and nginx in a Docker container. Use the docker file of platform-complete. Change OKAPI_URL to your server name or SAN - usually the name for which you have an SSL certificate. Change the TENANT_ID to your tenant ID:

cd ~/platform-complete
vim docker/Dockerfile
    ARG OKAPI_URL=https://demo.folio.hbz-nrw.de/okapi
    ARG TENANT_ID=diku

Configure webserver to serve Stripes webpack: Edit docker/nginx.conf. Place your local server name – this is not necessarily equal to the server name for which you have an SSL certificate ! –  in server_name and use your IP address in proxy_pass :

vim docker/nginx.conf
server {
  listen 80;
  server_name folio-hbz2.hbz-nrw.de;
  charset utf-8;
  access_log  /var/log/nginx/host.access.log  combined;

  # front-end requests:
  # Serve index.html for any request not found
  location / {
    # Set path
    root        /usr/share/nginx/html;
    index       index.html index.htm;
    include mime.types;
    types {
      text/plain lock;
    }
    try_files $uri /index.html;
  }

  # back-end requests:
  location /okapi {
    rewrite ^/okapi/(.*) /$1 break;
    proxy_pass http://10.9.2.85:9130/;
  }
}

Change the okapi URL and your tenant also in stripes.config.js :

 vim stripes.config.js
      okapi: { 'url':'https://demo.folio.hbz-nrw.de/okapi', 'tenant':'diku' },

Finally, build the docker container which will contain Stripes and nginx :

sudo su
docker build -f docker/Dockerfile --build-arg OKAPI_URL=https://demo.folio.hbz-nrw.de/okapi --build-arg TENANT_ID=diku -t stripes .
Sending build context to Docker daemon  1.138GB
Step 1/19 : FROM node:15-alpine as stripes_build
...
Step 19/19 : ENTRYPOINT ["/usr/bin/entrypoint.sh"]
 ---> Running in 11b702c198a2
Removing intermediate container 11b702c198a2
 ---> 97f46658fa94
Successfully built 97f46658fa94
Successfully tagged stripes:latest

This will run for approximately 15 minutes. 

Stop the old Stripes container: docker stop <container id of your old stripes container which is still running>

Completely free your port 80. Look if something is still running there: e.g., do

netstat -taupn | grep 80

If there should be something still running on port 80, kill these processes.

Start the stripes container:

nohup docker run -d -p 80:80 stripes

Log in to your frontend: E.g., go to https://demo.folio.hbz-nrw.de/ in your browser. 

  - Can you see the R3 modules in Settings - Installation details ?

  - Do you see the right okapi version, 4.11.1-1 ? 

  - Does everything look good ?

If so, remove the old stripes container: docker rm <container id of your old stripes container> .


II.vi) Cleanup

Clean up. Undeploy all unused containers.

In general, all R2 modules have to be removed now. But care has to be taken about which modules might be removed:

  • Sometimes, R2 versions are the same as R3 versions. In this case, you have to find out which instance-ID of the module needs to be removed (if the module has been deployed twice). You can find this out via the port number (lower port numbers have been used for the R2 modules, higher numbers for the R3 modules, which have been installed later).
  • If you have more than one tenant on your server, some tenants may still need R2 modules ! Check for your supertenant; did you enable anything else than okapi for it ?

Create a list of containers which are in your okapi discovery

 curl -w '\n' -XGET http://localhost:9130/_/discovery/modules | jq '.[] | .srvcId + "/" + .instId' > dockerps.sh
 sort dockerps.sh > dockerps.todelete.sh

This should still contain the module versions of the old release.

   => backend modules (mod-*) of the old release (61 modules) + backend module of the new release (62 = 65 - 3 which are the same in the old release) + 5 Edge modules =128 modules.

Of those 128 modules, 58 have now to be undeployed:
   Undeploy all R2 modules (which are still running), except for mod-service-interaction:1.0.0, mod-graphql:1.9.0, mod-z3950-2.4.0.

If you have deployed mod-service-interaction:1.0.0, mod-graphql:1.9.0, mod-z3950-2.4.0 twice, you will need to undeploy 61 modules.

Compare the list with list of R3 modules

Compare  dockerps.todelete.sh with the list ~/platform-complete/okapi-install.sh (the R3 backend modules). First, sort this list:

sort okapi-install.json > okapi-install.json.sorted

Now, throw out all modules which are in okapi-install.json.sorted out of the list dockerps.todelete.sh . 
This should leave you with a list of 58 modules which are to be deleted now. If you decided to delete one of the instances of those modules which have been deployed twice (be careful about which one to delete !!!) , you should now have a list of 61 modules.
Edit  dockerps.todelete.sh once again to make each line look like this on each line (e.g.) :

curl -w '\n' -D - -XDELETE http://localhost:9130/_/discovery/modules/mod-agreements-4.1.1/<instId>

Finally, remove all unused modules

Add a line #!/bin/bash at the top of your delete script, make the script executable and then call your delete script:

 ./dockerps.todelete.sh

This will be acknowledged by an "HTTP/1.1 204 No Content" for each module.


Now, finally once again get a list of the deployed backend modules:

curl -w '\n' -D - http://localhost:9130/_/discovery/modules | grep srvcId | wc
 70

This should only contain the module versions of the new release now : 65 backend modules of R3-2021 + 5 Edge modules.
Compare this with the number of your running docker containers:

 docker ps --all | grep "mod-" | wc
    65
 docker ps --all | wc
    77


The following containers are running on your system, but do not contain backend modules:

  • 5 containers with Edge modules
  • Stripes
  • 3x Elasticsearch
  • Kafka
  • Zookeper

In sum, these are 11 containers without backend modules (if one doesn't count Edge modules as backend modules).

Also subtract the header line (of "docker ps"), et voilà on arrive à 77 - 12 = 65 containers with backend modules (the figure of the first wc).

C'EST FINI !