Data-import testing approach on Rancher

Merging order:

data-import-processing-core
1. raml-storage
mod-source-record-storage
1. - raml-storage
2. data-import-processing-core
  1. raml-storage
mod-source-record-manager
1. data-import-processing-core
  1. raml-storage
2. mod-source-record-storage-client
mod-inventory
1. data-import-processing-core
2. mod-source-record-storage-client
mod-data-import
1. mod-source-record-manager-client
2. data-import-processing-core
mod-data-import-converter-storage
mod-qucik-marc

data-import-processing-core

This module is a library, so it does not have any dependency to other modules involved into the data-import process, so the only steps needed are:

build project

build data-import-processing-core from the branch that contains your changes

mod-source-record-storage

Module has following dependency

data-import-processing-core

To make mod-source-record-storage testable on Rancher env, do next steps:

add a data-import-processing-core lib to the project

build data-import-processing-core from the branch that contains your changes
create a folder (for instance with name 'lib') in mod-source-record-storage-server
copy data-import-processing-core-fat.jar to 'lib' folder

update mod-source-record-storage-server pom.xml file

add a local library as a system dependency

<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE-->
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>data-import-processing-core</artifactId>
	<scope>system</scope>
	<version>version_of_lib</version>
	<systemPath>path_to_lib</systemPath>
</dependency>

update maven-shade-plugin with property

<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar</Class-Path>

update .dockerignore file

#*
!Dockerfile
!docker*
!mod-source-record-storage-server/target/*.jar

add a rancher pipeline file

.rancher-pipeline.yml

stages:
  - name: Build
	steps:
	- runScriptConfig:
		image: postgres:12.2
		shellScript: |-
			apt update
			apt install -y maven git
			docker-entrypoint.sh "postgres" &
			mvn package -DskipTests -Ddocker.skip -Ddocker.host.address=localhost -Ddocker.container.db.ip=127.0.0.1 -Ddb.port=5432 -Djava.util.logging.config.file=vertx-default-jul-logging.properties
		env:
			POSTGRES_DB: postgres
			POSTGRES_USER: postgres
			POSTGRES_PASSWORD: postgres
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
	  dockerfilePath: ./Dockerfile
	  buildContext: .
	  tag: docker.dev.folio.org/mod-source-record-storage:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  pushRemote: true
	  registry: docker.dev.folio.org
  - name: Deploy
 	steps:
	- applyAppConfig:
	  catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-source-record-storage
	  version: 0.1.32
	  answers:
		image.repository: docker.dev.folio.org/mod-source-record-storage
		image.tag: <team_space_name>-${CICD_EXECUTION_SEQUENCE}
		targetNamespace: <team_space_name>
	  name: mod-source-record-storage
timeout: 60
notification: {}

update Dockerfile

additional step

# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-storage-server/lib/* ${LIB_DIR}/

Dockerfile

FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-source-record-storage-server-fat.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY mod-source-record-storage-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-storage-server/lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 8081

mod-source-record-manager

Module has following dependencies

data-import-processing-core
mod-data-import-converter-storage-client
mod-source-record-storage-client

To make mod-source-record-manager testable on Rancher env, do next steps:

add libs to the project

add data-import-processing-core lib
- build data-import-processing-core from the branch that contains your changes
- create a folder (for instance with name 'lib') in mod-source-record-manager-server
- copy data-import-processing-core-fat.jar to 'lib' folder

add mod-data-import-converter-storage-client lib
- build mod-data-import-converter-storage-client from the branch that contains your changes
- copy mod-data-import-converter-storage-client-<version>.jar to 'lib' folder

perform similar steps for mod-source-record-storage-client if needed

update mod-source-record-manager-server pom.xml file

add a data-import-processing-core library as a system dependency

<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency> <groupId>org.folio</groupId> 
	<artifactId>data-import-processing-core</artifactId> 
	<scope>system</scope> 
	<version>version_of_lib</version> 
	<systemPath>path_to_lib</systemPath> 
</dependency>

add a mod-data-import-converter-storage-client library as a system dependency

<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>mod-data-import-converter-storage-client</artifactId>
	<scope>system</scope>
	<version>version_of_lib</version>
	<systemPath>path_to_lib</systemPath>
</dependency>

add a mod-source-record-storage-client library as a system dependency(if needed)

update maven-shade-plugin with property

<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-data-import-converter-storage-client-1.11.0-SNAPSHOT.jar</Class-Path>

update .dockerignore file

#*
!Dockerfile
!docker*
!mod-source-record-manager-server/target/*.jar

add a rancher pipeline file

.rancher-pipeline.yml

stages:
  - name: Build
 	steps:
	- runScriptConfig:
	  image: maven:3-adoptopenjdk-11
	  shellScript: mvn package -DskipTests -Djava.util.logging.config.file=vertx-default-jul-logging.properties
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
	  dockerfilePath: ./Dockerfile
	  buildContext: .
	  tag: docker.dev.folio.org/mod-source-record-manager:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  pushRemote: true
	  registry: docker.dev.folio.org
  - name: Deploy
	steps:
	- applyAppConfig:
	  catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-source-record-manager
	  version: 0.1.32
	  answers:
	  image.repository: docker.dev.folio.org/mod-source-record-manager
	  image.tag: spitfire-${CICD_EXECUTION_SEQUENCE}
	  targetNamespace: <team_space_name>
	  name: mod-source-record-manager
timeout: 60
notification: {}

update Dockerfile

additional step

# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-manager-server/lib/* ${LIB_DIR}/

Dockerfile

FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-source-record-storage-server-fat.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY mod-source-record-manager-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-manager-server/lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 8081

for simplifying the steps described above there is a branch with name 'rancher' which you may use.

mod-inventory

There are two dependencies exists for mod-inventory

data-import-processing-core
mod-source-record-storage-client

To make mod-source-record-manager testable on Rancher env, do next steps:

add libs to the project

add data-import-processing-core lib
- build data-import-processing-core from the branch that contains your changes
- create a folder (for instance with name 'lib') in mod-source-record-manager-server
- copy data-import-processing-core-fat.jar to 'lib' folder

add mod-source-record-storage-client lib
- build mod-source-record-storage-client from the branch that contains your changes
- copy mod-source-record-storage-client-<version>.jar to 'lib' folder

update mod-source-record-manager-server pom.xml file

add a data-import-processing-core library as a system dependency

<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency> <groupId>org.folio</groupId> 
	<artifactId>data-import-processing-core</artifactId> 
	<scope>system</scope> 
	<version>version_of_lib</version> 
	<systemPath>path_to_lib</systemPath> 
</dependency>

add a mod-source-record-storage-client library as a system dependency

<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>mod-source-record-storage-client</artifactId>
	<version>version_of_lib</version>
	<scope>system</scope>
	<systemPath>path_to_lib</systemPath>
</dependency>

update maven-shade-plugin with property

<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-source-record-storage-client-5.1.0.0-SNAPSHOT.jar</Class-Path>

add a rancher pipeline file

.rancher-pipeline.yml

stages:
  - name: Clone submodules
	steps:
	- runScriptConfig:
		image: alpine/git
		shellScript: git submodule update --init --recursive
  - name: Build
	steps:
	- runScriptConfig:
		image: 'maven:3-openjdk-11'
		shellScript: mvn package -DskipTests
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
		dockerfilePath: ./Dockerfile
		buildContext: .
		tag: docker.dev.folio.org/mod-inventory:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
		pushRemote: true
		registry: docker.dev.folio.org
  - name: Deploy
	steps:
	- applyAppConfig:
	catalogTemplate: 'p-<team_prefix>:<team_space_name>-helmcharts-mod-inventory'
	version: 0.1.32
	answers:
	  image.repository: docker.dev.folio.org/mod-inventory
	  image.tag: '<team_space_name>-${CICD_EXECUTION_SEQUENCE}'
 	  postJob.enabled: true
	targetNamespace: <team_space_name>
	name: mod-inventory
timeout: 60
notification: {}

update Dockerfile

additional step

# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/

FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-inventory.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 9403

mod-data-import

Module has following dependency

data-import-processing-core
mod-source-record-manager-client

To make mod-source-record-storage testable on Rancher env, do next steps:

add libs to the project

add data-import-processing-core lib
- build data-import-processing-core from the branch that contains your changes
- create a folder (for instance with name 'lib') in mod-source-record-manager-server
- copy data-import-processing-core-fat.jar to 'lib' folder

add mod-source-record-manager-client lib
- build mod-source-record-manager-client from the branch that contains your changes
- copy mod-source-record-manager-client-<version>.jar to 'lib' folder

update pom.xml file

add a data-import-processing-core library as a system dependency

<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency> <groupId>org.folio</groupId> 
	<artifactId>data-import-processing-core</artifactId> 
	<scope>system</scope> 
	<version>version_of_lib</version> 
	<systemPath>path_to_lib</systemPath> 
</dependency>

add a mod-source-record-manager-client library as a system dependency

<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>mod-source-record-manager-client</artifactId>
	<scope>system</scope>
	<version>version_of_lib</version>
	<systemPath>path_to_lib</systemPath>
</dependency>

update maven-shade-plugin with property