Upgrade R1-2021 to R2-2021 "Juniper"
- Ingolf Kuss
Upgrade R1-2021 => R2-2021 "Juniper"
Based on "platform-complete"
with automatic migration of the database contents
Deployment with Okapi, "Single Server" on BS Ubuntu 20.04.2 LTS
Increase server hardware
Required for an upgrade R1 → R2 :
8 CPUs
40 GB RAM
recommended: 350 GB SSD HD
Install Ubuntu Updates
sudo apt-get update
sudo apt-get upgrade
sudo reboot
The following has to restart after a system reboot:
- postgres
systemctl start postgresql
- docker
systemctl start docker.service
- okapi
systemctl start okapi.service
- nginx
systemctl start nginx
Upgrade of Okapi Version / Restart Okapi
You have to do that, otherwise you can't pull the new modules.
Installation and configuration of Okapi.
The Okapi version is in the platform-complete/install.json file, so a current version of this file must be downloaded first:
# Clone the platform-complete repository if you haven't already done so
# git clone https://github.com/folio-org/platform-complete
Change into that directory:
cd ~/platform-complete
git fetch
# new branch R2-2021, new Tag R2-2021-GA
# Check out that branch.
# Save your local changes. This should only pertain to stripes.config.js .
# Discard changes to install.json etc:
git restore install.json
git restore okapi-install.json
git restore stripes-install.json
git restore package.json
git stash save
git checkout master
git pull
git checkout R2-2021-GA
git stash pop
# Read R2 Okapi Version from install.json: okapi-4.8.2
Get Okapi as Debian-Paket from repository.folio.org
Importiert the FOLIO signing key, add the FOLIO apt repository and install Okapi (of this version).
First save okapi.conf :
sudo su
cp -p /etc/folio/okapi/okapi.conf /usr/folio/upgrade/
wget --quiet -O - https://repository.folio.org/packages/debian/folio-apt-archive-key.asc | sudo apt-key add -
OK
sudo add-apt-repository "deb https://repository.folio.org/packages/ubuntu focal/"
sudo apt-get update
sudo apt-get -y --allow-change-held-packages install okapi=4.8.2-1 # R2-2021 Okapi Version
# play back okapi.conf :
sudo cp -p ~/upgrade/okapi.conf /etc/folio/okapi/okapi.conf
Port range of Okapi may need to be enlarged (9131-9350); Edit okapi.conf. Look how many ports are alreday in use (sudo docker ps --all | wc)
At least 60 ports must still be free !
sudo apt-mark hold okapi
okapi set on hold.
New start Okapi
sudo systemctl daemon-reload
sudo systemctl restart okapi.service
In /var/log/folio/okapi/okapi.log there should be:
INFO DeploymentManager fast shutdown
# and directly after
INFO InternalModule InternalModule starting okapiversion=4.3.3
...
INFO MainVerticle Deploy completed succesfully
# Get the list of modules for this tenant (merely informative):
curl -w '\n' -XGET http://localhost:9130/_/proxy/tenants/diku/modules
...
}, {
"id" : "okapi-4.8.2"
} ]
# 8x edge, 50x folio_ (Frontend), 60x mod- (Backend) = the R1-Modules + the R2-Okapi
Pull Module descriptors from the central repository
vim ~/folio-install/runbooks/single-server/scripts/okapi-pull.json
{
"urls": [
"https://folio-registry.dev.folio.org"
]
}
cd ~
curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @folio-install/runbooks/single-server/scripts/okapi-pull.json http://localhost:9130/_/proxy/pull/modules
2021-08-25T13:11:06,833 INFO ProxyContext 619855/proxy REQ 127.0.0.1:45316 supertenant POST /_/proxy/pull/modules okapi-4.8.2
2021-08-25T13:11:17,625 INFO PullManager Remote registry at https://folio-registry.dev.folio.org is version 4.7.1
2021-08-25T13:11:17,625 INFO PullManager pull smart
2021-08-25T13:11:31,578 INFO PullManager pull: 2457 MDs to insert
2021-08-25T13:11:33,823 INFO ProxyContext 619855/proxy RES 200 26989705us okapi-4.8.2 /_/proxy/pull/modules
Deploy a compatible FOLIO backend and enable for the tenant.
Post Environment variables to be used by te deployed modules in the Okapi environment:
curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"DB_HOST\",\"value\":\"10.9.2.62\"}" http://localhost:9130/_/env;
curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"DB_PORT\",\"value\":\"5432\"}" http://localhost:9130/_/env;
curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"DB_DATABASE\",\"value\":\"folio\"}" http://localhost:9130/_/env;
curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"DB_USERNAME\",\"value\":\"folio\"}" http://localhost:9130/_/env;
curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"DB_PASSWORD\",\"value\":\"folio123\"}" http://localhost:9130/_/env;
curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"KAFKA_HOST\",\"value\":\"10.9.2.85\"}" http://localhost:9130/_/env;
curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"KAFKA_PORT\",\"value\":\"9092\"}" http://localhost:9130/_/env;
curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d "{\"name\":\"OKAPI_URL\",\"value\":\"http://10.9.2.85:9130\"}" http://localhost:9130/_/env;
# Merely informativ: Look, how many docker containers are running now:
sudo docker ps | grep -v "^CONTAINER" | wc -l
# there are running approx. 60 containers of the R1 release.
First change environment variables of mod_pubsub ändern, because of Kommunikation with Kafka
Set KAFKA_HOST and OKAPI_URL to the real IP address:
cd ~/upgrade
# The R2 version of pubsub is mod-pubsub-2.3.3.
# The R2 2021 versions of the modules are in the file ~/platform-complete/okapi-install.json
wget https://folio-registry.dev.folio.org/_/proxy/modules/mod-pubsub-2.3.3 -O pubsub-module-descriptor.json
vim pubsub-module-descriptor.json
# Adjust in the launch descriptor:
}, {
"name" : "KAFKA_HOST",
"value" : "10.9.2.85"
}, {
"name" : "OKAPI_URL",
"value" : "http://10.9.2.85:9130"
}, {
"name" : "SYSTEM_USER_PASSWORD",
"value" : "***secret***"
} ],
curl -X DELETE -D - -w '\n' http://localhost:9130/_/proxy/modules/mod-pubsub-2.3.3
curl -i -w '\n' -X POST -H 'Content-type: application/json' -d @pubsub-module-descriptor.json http://localhost:9130/_/proxy/modules
Do the same (without OKAPI_URL) for these Modules
Look up R2 versions of the modules in the okapi-install.json :
mod-data-import
mod-data-export-spring
mod-data-export-worker
mod-ebsconet
mod-inventory
mod-inventory-storage
mod-invoice
mod-quick-marc
mod-remote-storage
# mod-search ==> don't install, otherwise elasticsearch has to be installed also
mod-source-record-manager
mod-source-record-storage
First upgrade and enable mod-pubsub
curl -w '\n' -D - -X POST -H "Content-type: application/json" -d '[ { "id" : "mod-pubsub-2.3.3", "action" : "enable" } ]' http://localhost:9130/_/proxy/tenants/diku/install?simulate=true
curl -w '\n' -D - -X POST -H "Content-type: application/json" -d '[ { "id" : "mod-pubsub-2.3.3", "action" : "enable" } ]' http://localhost:9130/_/proxy/tenants/diku/install?deploy=true\&preRelease=false\&tenantParameters=loadReference%3Dtrue
Finally deploy all backend modules and enable:
- Don't install elasticsearch (et.al. !)
cd ~/platform-complete
Remove mod-search and folio_inventory-es from install.json. Also remove mod-kb-ebsco-java, folio_search, edge-caiasoft, edge-dematic, mod-gobi, mod-ebsconet and mod-codex-ekb, if you don't need them
Remove mod-search from okapi-install.json. Also remove mod-kb-ebsco-java, mod-gobi, mod-ebsconet, mod-codex-ekb.
Remove folio_inventory-es from stripes-install.json. Also remove folio_search, edge-caiasoft, edge-dematic.
Remove @folio/inventory-es from package.json. Remove @folio/search.
# *****************************************************************************************
# Deploy the modules one by one. For each module a docker container will be started.
# *****************************************************************************************
# **********************************************
# Deploy everthing that has not yet been deployed
# **********************************************
For a single module it works like this:
cat > circulation-deployment-descriptor.json <<END
{
"srvcId": "mod-circulation-22.0.4",
"nodeId": "10.9.2.85"
}
END
curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @circulation-deployment-descriptor.json http://localhost:9130/_/discovery/modules
Go through the list okapi-install.json module by module and deploy !
=> Now the R2 containers have been deployed, but not yet enabled.
In addittion the R1 containers are still running on the system. I.e. at the moment the system is still in the state "R1-2021", except for Okapi, which is already on R2-2021 (but backward compatible).
Enable the R2 backend modules for the tenant
# Only enable, don't deploy because the container ist running already. You need to post the complete list "okapi-install.json", otherwise the module dependencies will not be resolved:
curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @/usr/folio/platform-complete/okapi-install.json http://localhost:9130/_/proxy/tenants/diku/install?simulate=true\&preRelease=false
curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @/usr/folio/platform-complete/okapi-install.json http://localhost:9130/_/proxy/tenants/diku/install?deploy=false\&preRelease=false\&tenantParameters=loadReference%3Dtrue
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
vary: origin
Content-Type: application/json
X-Okapi-Trace: POST okapi-4.8.2 /_/proxy/tenants/diku/install.. : 200 3670882us
List the deployed modules (only Backend):
curl -w '\n' -D - http://localhost:9130/_/discovery/modules | grep srvcId | wc
List the enabled modules (Frontend + Backend)
curl -w '\n' -XGET http://localhost:9130/_/proxy/tenants/diku/modules | grep id | wc
Post the list of frontend modules to enable them for the tenant
Enable them but no deploy; they are no containers:
curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @/usr/folio/platform-complete/stripes-install.json http://localhost:9130/_/proxy/tenants/diku/install?simulate=true\&preRelease=false
curl -w '\n' -D - -X POST -H "Content-type: application/json" -d @/usr/folio/platform-complete/stripes-install.json http://localhost:9130/_/proxy/tenants/diku/install?preRelease=false
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
vary: origin
Content-Type: application/json
X-Okapi-Trace: POST okapi-4.8.2 /_/proxy/tenants/diku/install.. : 200 8014004us
Installation Frontend: Build an R2 version of the FOLIO Stripes platform
Installation of Stripes and nginx in a Docker Container
# Quelle: https://github.com/folio-org/platform-complete
Adjust Docker File and nginx Configuration
cd ~/platform-complete
vim docker/Dockerfile
ARG OKAPI_URL=http://10.9.2.85:9130 # change the IP address of the server
ARG TENANT_ID=diku # change tenant
vim docker/nginx.conf
server {
listen 80;
server_name folio-hbz2.hbz-nrw.de;
charset utf-8;
access_log /var/log/nginx/host.access.log combined;
# front-end requests:
# Serve index.html for any request not found
location / {
# Set path
root /usr/share/nginx/html;
index index.html index.htm;
include mime.types;
types {
text/plain lock;
}
try_files $uri /index.html;
}
# back-end requests:
location /okapi {
rewrite ^/okapi/(.*) /$1 break;
proxy_pass http://10.9.2.85:9130/;
}
}
vim stripes.config.js
Adjust Tenant and Okapi-URL :
okapi: { 'url':'http://folio-hbz2.hbz-nrw.de/okapi', 'tenant':'diku' },
The URL must be reachable from the outside, thus from a browser. The endpoint "/okapi" will be redirected in nginx.conf to the internal port 9130.
Remove this line (don't install the search app) :
'@folio/search' : {},
Adjust the logo and the favicon icon in stripes.config.js as you like.
Build the docker container
sudo su
docker build -f docker/Dockerfile --build-arg OKAPI_URL=http://10.9.2.85:9130 --build-arg TENANT_ID=diku -t stripes .
Sending build context to Docker daemon 1.138GB
Step 1/19 : FROM node:15-alpine as stripes_build
...
Step 19/19 : ENTRYPOINT ["/usr/bin/entrypoint.sh"]
---> Running in a47dce4e3b3e
Removing intermediate container a47dce4e3b3e
---> 48a532266f21
Successfully built 48a532266f21
Successfully tagged stripes:latest
# This will run approx. 15 minutes.
Start the docker container
Redirect Port 80 from the outer world to Port 80 of the container. For SSL, also port 443 needs to be redirected (which is not part of this documentation):
nohup docker run -d -p 80:80 stripes
Stop nginx on the server:
sudo service nginx stop
# Log in to the docker container
docker exec -it <id> sh
check whether the config file has been copied correctly:
vi /etc/nginx/conf.d/default.conf
# follow the log of the web server inside the container:
tail -f /var/log/nginx/host.access.log
Clean up.
This step is important, because otherwise the server might easily go down in performance. Especially during your next upgrade this will be of critical importance !!
Undeploy unused containers:
cd ~/upgrade
# Which containers are enabled ?
curl -w '\n' -XGET http://localhost:9130/_/proxy/tenants/supertenant/modules
curl -w '\n' -XGET http://localhost:9130/_/proxy/tenants/diku/modules
# Which containers are in FOLIO discovery and are running ?
curl -w '\n' -XGET http://localhost:9130/_/discovery/modules | jq '.[] | .srvcId + "/" + .instId' > dockerps.sh
# This should contain the module versions of the old release
# => Backend-Modules (mod-*) of the old release + backend modules of the new release (This should be approx. 120 modules).
# Delete unused, i.e. not enabled, containers:
z.B.: curl -w '\n' -D - -XDELETE http://localhost:9130/_/discovery/modules/mod-orders-12.0.1
you will receive an "HTTP/1.1 204 No Content" for each post.
# List the modules which are being deployed now:
curl -w '\n' -D - http://localhost:9130/_/discovery/modules | grep srvcId | wc
60
# This should only contain the module versions of the new release
# Compare with the number of running docker containers:
docker ps --all | wc
63 788 15088
# subtract Kafka, Zookeeper and the header line=> 60 containers with backend modules (The Sums are matching).
DONE
Log in as "diku_admin:admin" (or with your sys admin login) on http://folio-hbz2.hbz-nrw.de (replace with your server name).
Check if the old data are still there ! E.g, look in Catalogue or User Managament, if your inventory is still there and if your users are still there.