Data-import testing approach on Rancher

Merging order:

  1. data-import-processing-core
    1. raml-storage
  2. mod-source-record-storage
    1. - raml-storage
    2. data-import-processing-core
      1. raml-storage 
  3. mod-source-record-manager 
    1. data-import-processing-core
      1. raml-storage
    2. mod-source-record-storage-client
  4. mod-inventory
    1. data-import-processing-core 
    2. mod-source-record-storage-client
  5. mod-data-import
    1. mod-source-record-manager-client
    2. data-import-processing-core
  6. mod-data-import-converter-storage
  7. mod-qucik-marc

data-import-processing-core

This module is a library, so it does not have any dependency to other modules involved into the data-import process, so the only steps needed are:

 build project
  • build data-import-processing-core from the branch that contains your changes

mod-source-record-storage

Module has following dependency 

  • data-import-processing-core

To make mod-source-record-storage testable on Rancher env, do next steps:

 add a data-import-processing-core lib to the project
  • build data-import-processing-core from the branch that contains your changes
  • create a folder (for instance with name 'lib') in mod-source-record-storage-server 
  • copy data-import-processing-core-fat.jar  to 'lib' folder

 update mod-source-record-storage-server pom.xml file
  • add a local library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE-->
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>data-import-processing-core</artifactId>
	<scope>system</scope>
	<version>version_of_lib</version>
	<systemPath>path_to_lib</systemPath>
</dependency>
  • update maven-shade-plugin with property
<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar</Class-Path>

 update .dockerignore file
#*
!Dockerfile
!docker*
!mod-source-record-storage-server/target/*.jar
 add a rancher pipeline file
.rancher-pipeline.yml
stages:
  - name: Build
	steps:
	- runScriptConfig:
		image: postgres:12.2
		shellScript: |-
			apt update
			apt install -y maven git
			docker-entrypoint.sh "postgres" &
			mvn package -DskipTests -Ddocker.skip -Ddocker.host.address=localhost -Ddocker.container.db.ip=127.0.0.1 -Ddb.port=5432 -Djava.util.logging.config.file=vertx-default-jul-logging.properties
		env:
			POSTGRES_DB: postgres
			POSTGRES_USER: postgres
			POSTGRES_PASSWORD: postgres
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
	  dockerfilePath: ./Dockerfile
	  buildContext: .
	  tag: docker.dev.folio.org/mod-source-record-storage:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  pushRemote: true
	  registry: docker.dev.folio.org
  - name: Deploy
 	steps:
	- applyAppConfig:
	  catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-source-record-storage
	  version: 0.1.32
	  answers:
		image.repository: docker.dev.folio.org/mod-source-record-storage
		image.tag: <team_space_name>-${CICD_EXECUTION_SEQUENCE}
		targetNamespace: <team_space_name>
	  name: mod-source-record-storage
timeout: 60
notification: {}


 update Dockerfile

additional step 


# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-storage-server/lib/* ${LIB_DIR}/

Dockerfile
FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-source-record-storage-server-fat.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY mod-source-record-storage-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-storage-server/lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 8081



mod-source-record-manager

Module has following dependencies 

  • data-import-processing-core
  • mod-data-import-converter-storage-client
  • mod-source-record-storage-client

To make mod-source-record-manager testable on Rancher env, do next steps:

 add libs to the project
  • add  data-import-processing-core lib
    • build data-import-processing-core from the branch that contains your changes
    • create a folder (for instance with name 'lib') in mod-source-record-manager-server 
    • copy data-import-processing-core-fat.jar  to 'lib' folder


  • add mod-data-import-converter-storage-client lib
    • build mod-data-import-converter-storage-client from the branch that contains your changes
    • copy mod-data-import-converter-storage-client-<version>.jar to 'lib' folder


  • perform similar steps for mod-source-record-storage-client if needed

 update mod-source-record-manager-server pom.xml file
  • add a data-import-processing-core library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency> <groupId>org.folio</groupId> 
	<artifactId>data-import-processing-core</artifactId> 
	<scope>system</scope> 
	<version>version_of_lib</version> 
	<systemPath>path_to_lib</systemPath> 
</dependency>
  • add a mod-data-import-converter-storage-client library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>mod-data-import-converter-storage-client</artifactId>
	<scope>system</scope>
	<version>version_of_lib</version>
	<systemPath>path_to_lib</systemPath>
</dependency>
  • add a mod-source-record-storage-client library as a system dependency(if needed)


  • update maven-shade-plugin with property
<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-data-import-converter-storage-client-1.11.0-SNAPSHOT.jar</Class-Path>
 update .dockerignore file
#*
!Dockerfile
!docker*
!mod-source-record-manager-server/target/*.jar
 add a rancher pipeline file
.rancher-pipeline.yml
stages:
  - name: Build
 	steps:
	- runScriptConfig:
	  image: maven:3-adoptopenjdk-11
	  shellScript: mvn package -DskipTests -Djava.util.logging.config.file=vertx-default-jul-logging.properties
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
	  dockerfilePath: ./Dockerfile
	  buildContext: .
	  tag: docker.dev.folio.org/mod-source-record-manager:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  pushRemote: true
	  registry: docker.dev.folio.org
  - name: Deploy
	steps:
	- applyAppConfig:
	  catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-source-record-manager
	  version: 0.1.32
	  answers:
	  image.repository: docker.dev.folio.org/mod-source-record-manager
	  image.tag: spitfire-${CICD_EXECUTION_SEQUENCE}
	  targetNamespace: <team_space_name>
	  name: mod-source-record-manager
timeout: 60
notification: {}



 update Dockerfile

additional step 


# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-manager-server/lib/* ${LIB_DIR}/

Dockerfile
FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-source-record-storage-server-fat.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY mod-source-record-manager-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-manager-server/lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 8081

for simplifying the steps described above there is a branch with name 'rancher' which you may use.



mod-inventory

There are two dependencies exists for mod-inventory

  • data-import-processing-core 
  • mod-source-record-storage-client

To make mod-source-record-manager testable on Rancher env, do next steps:

 add libs to the project
  • add  data-import-processing-core lib
    • build data-import-processing-core from the branch that contains your changes
    • create a folder (for instance with name 'lib') in mod-source-record-manager-server 
    • copy data-import-processing-core-fat.jar  to 'lib' folder


  • add mod-source-record-storage-client lib
    • build mod-source-record-storage-client from the branch that contains your changes
    • copy mod-source-record-storage-client-<version>.jar to 'lib' folder


 update mod-source-record-manager-server pom.xml file
  • add a data-import-processing-core library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency> <groupId>org.folio</groupId> 
	<artifactId>data-import-processing-core</artifactId> 
	<scope>system</scope> 
	<version>version_of_lib</version> 
	<systemPath>path_to_lib</systemPath> 
</dependency>
  • add a mod-source-record-storage-client library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>mod-source-record-storage-client</artifactId>
	<version>version_of_lib</version>
	<scope>system</scope>
	<systemPath>path_to_lib</systemPath>
</dependency>


  • update maven-shade-plugin with property
<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-source-record-storage-client-5.1.0.0-SNAPSHOT.jar</Class-Path>
 add a rancher pipeline file 
.rancher-pipeline.yml
stages:
  - name: Clone submodules
	steps:
	- runScriptConfig:
		image: alpine/git
		shellScript: git submodule update --init --recursive
  - name: Build
	steps:
	- runScriptConfig:
		image: 'maven:3-openjdk-11'
		shellScript: mvn package -DskipTests
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
		dockerfilePath: ./Dockerfile
		buildContext: .
		tag: docker.dev.folio.org/mod-inventory:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
		pushRemote: true
		registry: docker.dev.folio.org
  - name: Deploy
	steps:
	- applyAppConfig:
	catalogTemplate: 'p-<team_prefix>:<team_space_name>-helmcharts-mod-inventory'
	version: 0.1.32
	answers:
	  image.repository: docker.dev.folio.org/mod-inventory
	  image.tag: '<team_space_name>-${CICD_EXECUTION_SEQUENCE}'
 	  postJob.enabled: true
	targetNamespace: <team_space_name>
	name: mod-inventory
timeout: 60
notification: {}
 update Dockerfile

additional step 


# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/



FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-inventory.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 9403



mod-data-import

Module has following dependency 

  • data-import-processing-core
  • mod-source-record-manager-client

To make mod-source-record-storage testable on Rancher env, do next steps:

 add libs to the project
  • add  data-import-processing-core lib
    • build data-import-processing-core from the branch that contains your changes
    • create a folder (for instance with name 'lib') in mod-source-record-manager-server 
    • copy data-import-processing-core-fat.jar  to 'lib' folder


  • add mod-source-record-manager-client lib
    • build mod-source-record-manager-client from the branch that contains your changes
    • copy mod-source-record-manager-client-<version>.jar to 'lib' folder


 update pom.xml file
  • add a data-import-processing-core library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency> <groupId>org.folio</groupId> 
	<artifactId>data-import-processing-core</artifactId> 
	<scope>system</scope> 
	<version>version_of_lib</version> 
	<systemPath>path_to_lib</systemPath> 
</dependency>
  • add a mod-source-record-manager-client library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>mod-source-record-manager-client</artifactId>
	<scope>system</scope>
	<version>version_of_lib</version>
	<systemPath>path_to_lib</systemPath>
</dependency>


  • update maven-shade-plugin with property