Data-import testing approach on Rancher

Owned by Natalia Zaitseva
Last updated: Oct 20, 2021 by Oleksandr_Dekin

Merging order:

  1. data-import-processing-core
    1. raml-storage
  2. mod-source-record-storage
    1. - raml-storage
    2. data-import-processing-core
      1. raml-storage 
  3. mod-source-record-manager 
    1. data-import-processing-core
      1. raml-storage
    2. mod-source-record-storage-client
  4. mod-inventory
    1. data-import-processing-core 
    2. mod-source-record-storage-client
  5. mod-data-import
    1. mod-source-record-manager-client
    2. data-import-processing-core
  6. mod-data-import-converter-storage
  7. mod-qucik-marc

data-import-processing-core

This module is a library, so it does not have any dependency to other modules involved into the data-import process, so the only steps needed are:

 build project
  • build data-import-processing-core from the branch that contains your changes

mod-source-record-storage

Module has following dependency 

  • data-import-processing-core

To make mod-source-record-storage testable on Rancher env, do next steps:

 add a data-import-processing-core lib to the project
  • build data-import-processing-core from the branch that contains your changes
  • create a folder (for instance with name 'lib') in mod-source-record-storage-server 
  • copy data-import-processing-core-fat.jar  to 'lib' folder

 update mod-source-record-storage-server pom.xml file
  • add a local library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE-->
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>data-import-processing-core</artifactId>
	<scope>system</scope>
	<version>version_of_lib</version>
	<systemPath>path_to_lib</systemPath>
</dependency>
  • update maven-shade-plugin with property
<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar</Class-Path>

 update .dockerignore file
#*
!Dockerfile
!docker*
!mod-source-record-storage-server/target/*.jar
 add a rancher pipeline file
.rancher-pipeline.yml
stages:
  - name: Build
	steps:
	- runScriptConfig:
		image: postgres:12.2
		shellScript: |-
			apt update
			apt install -y maven git
			docker-entrypoint.sh "postgres" &
			mvn package -DskipTests -Ddocker.skip -Ddocker.host.address=localhost -Ddocker.container.db.ip=127.0.0.1 -Ddb.port=5432 -Djava.util.logging.config.file=vertx-default-jul-logging.properties
		env:
			POSTGRES_DB: postgres
			POSTGRES_USER: postgres
			POSTGRES_PASSWORD: postgres
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
	  dockerfilePath: ./Dockerfile
	  buildContext: .
	  tag: docker.dev.folio.org/mod-source-record-storage:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  pushRemote: true
	  registry: docker.dev.folio.org
  - name: Deploy
 	steps:
	- applyAppConfig:
	  catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-source-record-storage
	  version: 0.1.32
	  answers:
		image.repository: docker.dev.folio.org/mod-source-record-storage
		image.tag: <team_space_name>-${CICD_EXECUTION_SEQUENCE}
		targetNamespace: <team_space_name>
	  name: mod-source-record-storage
timeout: 60
notification: {}

 update Dockerfile

additional step 


# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-storage-server/lib/* ${LIB_DIR}/

Dockerfile
FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-source-record-storage-server-fat.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY mod-source-record-storage-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-storage-server/lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 8081

mod-source-record-manager

Module has following dependencies 

  • data-import-processing-core
  • mod-data-import-converter-storage-client

  • mod-source-record-storage-client

To make mod-source-record-manager testable on Rancher env, do next steps:

 add libs to the project
  • add  data-import-processing-core lib
    • build data-import-processing-core from the branch that contains your changes
    • create a folder (for instance with name 'lib') in mod-source-record-manager-server 
    • copy data-import-processing-core-fat.jar  to 'lib' folder


  • add mod-data-import-converter-storage-client lib
    • build mod-data-import-converter-storage-client from the branch that contains your changes
    • copy mod-data-import-converter-storage-client-<version>.jar to 'lib' folder


  • perform similar steps for mod-source-record-storage-client if needed

 update mod-source-record-manager-server pom.xml file
  • add a data-import-processing-core library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency> <groupId>org.folio</groupId> 
	<artifactId>data-import-processing-core</artifactId> 
	<scope>system</scope> 
	<version>version_of_lib</version> 
	<systemPath>path_to_lib</systemPath> 
</dependency>
  • add a mod-data-import-converter-storage-client library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>mod-data-import-converter-storage-client</artifactId>
	<scope>system</scope>
	<version>version_of_lib</version>
	<systemPath>path_to_lib</systemPath>
</dependency>
  • add a mod-source-record-storage-client library as a system dependency(if needed)


  • update maven-shade-plugin with property
<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-data-import-converter-storage-client-1.11.0-SNAPSHOT.jar</Class-Path>
 update .dockerignore file
#*
!Dockerfile
!docker*
!mod-source-record-manager-server/target/*.jar
 add a rancher pipeline file
.rancher-pipeline.yml
stages:
  - name: Build
 	steps:
	- runScriptConfig:
	  image: maven:3-adoptopenjdk-11
	  shellScript: mvn package -DskipTests -Djava.util.logging.config.file=vertx-default-jul-logging.properties
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
	  dockerfilePath: ./Dockerfile
	  buildContext: .
	  tag: docker.dev.folio.org/mod-source-record-manager:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  pushRemote: true
	  registry: docker.dev.folio.org
  - name: Deploy
	steps:
	- applyAppConfig:
	  catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-source-record-manager
	  version: 0.1.32
	  answers:
	  image.repository: docker.dev.folio.org/mod-source-record-manager
	  image.tag: spitfire-${CICD_EXECUTION_SEQUENCE}
	  targetNamespace: <team_space_name>
	  name: mod-source-record-manager
timeout: 60
notification: {}

 update Dockerfile

additional step 


# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-manager-server/lib/* ${LIB_DIR}/

Dockerfile
FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-source-record-storage-server-fat.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY mod-source-record-manager-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-manager-server/lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 8081

for simplifying the steps described above there is a branch with name 'rancher' which you may use.


mod-inventory

There are two dependencies exists for mod-inventory

  • data-import-processing-core 
  • mod-source-record-storage-client

To make mod-source-record-manager testable on Rancher env, do next steps:

 add libs to the project
  • add  data-import-processing-core lib
    • build data-import-processing-core from the branch that contains your changes
    • create a folder (for instance with name 'lib') in mod-source-record-manager-server 
    • copy data-import-processing-core-fat.jar  to 'lib' folder


  • add mod-source-record-storage-client lib
    • build mod-source-record-storage-client from the branch that contains your changes
    • copy mod-source-record-storage-client-<version>.jar to 'lib' folder


 update mod-source-record-manager-server pom.xml file
  • add a data-import-processing-core library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency> <groupId>org.folio</groupId> 
	<artifactId>data-import-processing-core</artifactId> 
	<scope>system</scope> 
	<version>version_of_lib</version> 
	<systemPath>path_to_lib</systemPath> 
</dependency>
  • add a mod-source-record-storage-client library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>mod-source-record-storage-client</artifactId>
	<version>version_of_lib</version>
	<scope>system</scope>
	<systemPath>path_to_lib</systemPath>
</dependency>


  • update maven-shade-plugin with property
<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-source-record-storage-client-5.1.0.0-SNAPSHOT.jar</Class-Path>
 add a rancher pipeline file 
.rancher-pipeline.yml
stages:
  - name: Clone submodules
	steps:
	- runScriptConfig:
		image: alpine/git
		shellScript: git submodule update --init --recursive
  - name: Build
	steps:
	- runScriptConfig:
		image: 'maven:3-openjdk-11'
		shellScript: mvn package -DskipTests
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
		dockerfilePath: ./Dockerfile
		buildContext: .
		tag: docker.dev.folio.org/mod-inventory:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
		pushRemote: true
		registry: docker.dev.folio.org
  - name: Deploy
	steps:
	- applyAppConfig:
	catalogTemplate: 'p-<team_prefix>:<team_space_name>-helmcharts-mod-inventory'
	version: 0.1.32
	answers:
	  image.repository: docker.dev.folio.org/mod-inventory
	  image.tag: '<team_space_name>-${CICD_EXECUTION_SEQUENCE}'
 	  postJob.enabled: true
	targetNamespace: <team_space_name>
	name: mod-inventory
timeout: 60
notification: {}
 update Dockerfile

additional step 


# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/



FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-inventory.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 9403


mod-data-import

Module has following dependency 

  • data-import-processing-core

  • mod-source-record-manager-client

To make mod-source-record-storage testable on Rancher env, do next steps:

 add libs to the project
  • add  data-import-processing-core lib
    • build data-import-processing-core from the branch that contains your changes
    • create a folder (for instance with name 'lib') in mod-source-record-manager-server 
    • copy data-import-processing-core-fat.jar  to 'lib' folder


  • add mod-source-record-manager-client lib
    • build mod-source-record-manager-client from the branch that contains your changes
    • copy mod-source-record-manager-client-<version>.jar to 'lib' folder


 update pom.xml file
  • add a data-import-processing-core library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency> <groupId>org.folio</groupId> 
	<artifactId>data-import-processing-core</artifactId> 
	<scope>system</scope> 
	<version>version_of_lib</version> 
	<systemPath>path_to_lib</systemPath> 
</dependency>
  • add a mod-source-record-manager-client library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>mod-source-record-manager-client</artifactId>
	<scope>system</scope>
	<version>version_of_lib</version>
	<systemPath>path_to_lib</systemPath>
</dependency>


  • update maven-shade-plugin with property
<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-source-record-manager-client-3.1.0-SNAPSHOT.jar</Class-Path>
 add a rancher pipeline file 
.rancher-pipeline.yml
stages:
  - name: Build
	steps:
	- runScriptConfig:
	  image: maven:3.6.3-openjdk-11
      shellScript: mvn package -DskipTests -Djava.util.logging.config.file=vertx-default-jul-logging.properties
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
	  dockerfilePath: ./Dockerfile
	  buildContext: .
	  tag: docker.dev.folio.org/mod-data-<team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  pushRemote: true
	  registry: docker.dev.folio.org
  - name: Deploy
	steps:
	- applyAppConfig:
	  catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-data-import
	  version: 0.1.32
	  answers:
	  image.repository: docker.dev.folio.org/mod-data-import
	  image.tag: <team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  targetNamespace: <team_space_name>
	  name: mod-data-import
timeout: 60
notification: {}
 update Dockerfile

additional step 


# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/



FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-data-import-fat.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container 
ENV LIB_DIR ${VERTICLE_HOME}/lib 
RUN mkdir -p ${LIB_DIR} 
COPY lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 8081

mod-data-import-converter-storage

This module does not have any dependency to other modules involved into the data-import process, so the only steps needed are:

 build project
  • build mod-data-import-converter-storage from the branch that contains your changes
 add a rancher pipeline file 
.rancher-pipeline.yml
stages:
  - name: Build
	steps:
	- runScriptConfig:
	image: maven:3.6.3-openjdk-11
	shellScript: mvn package -DskipTests -Djava.util.logging.config.file=vertx-default-jul-logging.properties
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
	  dockerfilePath: ./Dockerfile
	  buildContext: .
	  tag: docker.dev.folio.org/mod-data-import-converter-storage:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  pushRemote: true
	  registry: docker.dev.folio.org
  - name: Deploy
	steps:
	- applyAppConfig:
	  catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-data-import-converter-storage
	  version: 0.1.32
	  answers:
	  image.repository: docker.dev.folio.org/mod-data-import-converter-storage
	  image.tag: <team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  targetNamespace: <team_space_name>
	  name: mod-data-import-converter-storage
timeout: 60
notification: {}
mod-quick-marc

First of all, you should generate jar from SRM client and add lib into folder lib for quick marc (for example: mod-quick-marc → lib → mod-source-record-manager-client-3.2.0-SNAPSHOT.jar)

Change docker file:

 Dockerfile
# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles
# - should be a single jar file
ARG JAR_FILE=./target/mod-quick-marc.jar

////////////////////

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/

Example of Dockerfile:

 Dockerfile
FROM folioci/alpine-jre-openjdk11:latest

# Copy your fat jar to the container
ENV APP_FILE mod-quick-marc-fat.jar
# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles
# - should be a single jar file
ARG JAR_FILE=./target/mod-quick-marc.jar
# - copy
COPY ${JAR_FILE} ${JAVA_APP_DIR}/${APP_FILE}

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 8081

Change pom.xml by adding plugin maven-shade-plugin and add properties 

 pom.xml

<properties>
  <start-class>org.folio.qm.ModQuickMarcApplication</start-class>
///////////////////////////

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-shade-plugin</artifactId>
      <version>3.2.4</version>
      <executions>
        <execution>
          <phase>package</phase>
          <goals>
            <goal>shade</goal>
          </goals>
          <configuration>
            <transformers>
              <transformer
                implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                <resource>META-INF/spring.handlers</resource>
              </transformer>
              <transformer
                implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
                <resource>META-INF/spring.factories</resource>
              </transformer>
              <transformer
                implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                <resource>META-INF/spring.schemas</resource>
              </transformer>
              <transformer
                implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
              <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                <manifestEntries>
                  <Main-Class>${start-class}</Main-Class>
                  <Multi-Release>true</Multi-Release>
                  <Class-Path>lib/mod-source-record-manager-client-3.2.0-SNAPSHOT.jar</Class-Path>
                </manifestEntries>
              </transformer>
            </transformers>
            <artifactSet />
            <outputFile>${project.build.directory}/${project.artifactId}.jar</outputFile>
          </configuration>
        </execution>
      </executions>
    </plugin>

Also ew should change SRM client in the pom.xml by adding systemPath and scope (system)

 pom.xml

<dependency>
  <groupId>org.folio</groupId>
  <artifactId>mod-source-record-manager-client</artifactId>
  <scope>system</scope>
  <systemPath>${basedir}/lib/mod-source-record-manager-client-3.2.0-SNAPSHOT.jar</systemPath>
  <version>${mod-srm-client.version}</version>
  <exclusions>
    <exclusion>
      <groupId>org.folio</groupId>
      <artifactId>domain-models-runtime</artifactId>
    </exclusion>
    <exclusion>
      <groupId>org.folio</groupId>
      <artifactId>domain-models-api-interfaces</artifactId>
    </exclusion>
  </exclusions>
</dependency>