Skip to end of banner
Go to start of banner

Data-import testing approach on Rancher

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 17 Next »

Merging order:

  1. data-import-processing-core
    1. raml-storage
  2. mod-source-record-storage
    1. - raml-storage
    2. data-import-processing-core
      1. raml-storage 
  3. mod-source-record-manager 
    1. data-import-processing-core
      1. raml-storage
    2. mod-source-record-storage-client
  4. mod-inventory
    1. data-import-processing-core 
    2. mod-source-record-storage-client
  5. mod-data-import
    1. mod-source-record-manager-client
    2. data-import-processing-core
  6. mod-data-import-converter-storage

data-import-processing-core

This module is a library, so it does not have any dependency to other modules involved into the data-import process, so the only steps needed are:

 build project
  • build data-import-processing-core from the branch that contains your changes

mod-source-record-storage

Module has following dependency 

  • data-import-processing-core

To make mod-source-record-storage testable on Rancher env, do next steps:

 add a data-import-processing-core lib to the project
  • build data-import-processing-core from the branch that contains your changes
  • create a folder (for instance with name 'lib') in mod-source-record-storage-server 
  • copy data-import-processing-core-fat.jar  to 'lib' folder

 update mod-source-record-storage-server pom.xml file
  • add a local library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE-->
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>data-import-processing-core</artifactId>
	<scope>system</scope>
	<version>version_of_lib</version>
	<systemPath>path_to_lib</systemPath>
</dependency>
  • update maven-shade-plugin with property
<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar</Class-Path>

 update .dockerignore file
#*
!Dockerfile
!docker*
!mod-source-record-storage-server/target/*.jar
 add a rancher pipeline file
.rancher-pipeline.yml
stages:
  - name: Build
	steps:
	- runScriptConfig:
		image: postgres:12.2
		shellScript: |-
			apt update
			apt install -y maven git
			docker-entrypoint.sh "postgres" &
			mvn package -DskipTests -Ddocker.skip -Ddocker.host.address=localhost -Ddocker.container.db.ip=127.0.0.1 -Ddb.port=5432 -Djava.util.logging.config.file=vertx-default-jul-logging.properties
		env:
			POSTGRES_DB: postgres
			POSTGRES_USER: postgres
			POSTGRES_PASSWORD: postgres
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
	  dockerfilePath: ./Dockerfile
	  buildContext: .
	  tag: docker.dev.folio.org/mod-source-record-storage:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  pushRemote: true
	  registry: docker.dev.folio.org
  - name: Deploy
 	steps:
	- applyAppConfig:
	  catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-source-record-storage
	  version: 0.1.32
	  answers:
		image.repository: docker.dev.folio.org/mod-source-record-storage
		image.tag: <team_space_name>-${CICD_EXECUTION_SEQUENCE}
		targetNamespace: <team_space_name>
	  name: mod-source-record-storage
timeout: 60
notification: {}

 update Dockerfile

additional step 


# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-storage-server/lib/* ${LIB_DIR}/

Dockerfile
FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-source-record-storage-server-fat.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY mod-source-record-storage-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-storage-server/lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 8081

mod-source-record-manager

Module has following dependencies 

  • data-import-processing-core
  • mod-data-import-converter-storage-client

  • mod-source-record-storage-client

To make mod-source-record-manager testable on Rancher env, do next steps:

 add libs to the project
  • add  data-import-processing-core lib
    • build data-import-processing-core from the branch that contains your changes
    • create a folder (for instance with name 'lib') in mod-source-record-manager-server 
    • copy data-import-processing-core-fat.jar  to 'lib' folder


  • add mod-data-import-converter-storage-client lib
    • build mod-data-import-converter-storage-client from the branch that contains your changes
    • copy mod-data-import-converter-storage-client-<version>.jar to 'lib' folder


  • perform similar steps for mod-source-record-storage-client if needed

 update mod-source-record-manager-server pom.xml file
  • add a data-import-processing-core library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency> <groupId>org.folio</groupId> 
	<artifactId>data-import-processing-core</artifactId> 
	<scope>system</scope> 
	<version>version_of_lib</version> 
	<systemPath>path_to_lib</systemPath> 
</dependency>
  • add a mod-data-import-converter-storage-client library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>mod-data-import-converter-storage-client</artifactId>
	<scope>system</scope>
	<version>version_of_lib</version>
	<systemPath>path_to_lib</systemPath>
</dependency>
  • add a mod-source-record-storage-client library as a system dependency(if needed)


  • update maven-shade-plugin with property
<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-data-import-converter-storage-client-1.11.0-SNAPSHOT.jar</Class-Path>
 update .dockerignore file
#*
!Dockerfile
!docker*
!mod-source-record-manager-server/target/*.jar
 add a rancher pipeline file
.rancher-pipeline.yml
stages:
  - name: Build
 	steps:
	- runScriptConfig:
	  image: maven:3-adoptopenjdk-11
	  shellScript: mvn package -DskipTests -Djava.util.logging.config.file=vertx-default-jul-logging.properties
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
	  dockerfilePath: ./Dockerfile
	  buildContext: .
	  tag: docker.dev.folio.org/mod-source-record-manager:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  pushRemote: true
	  registry: docker.dev.folio.org
  - name: Deploy
	steps:
	- applyAppConfig:
	  catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-source-record-manager
	  version: 0.1.32
	  answers:
	  image.repository: docker.dev.folio.org/mod-source-record-manager
	  image.tag: spitfire-${CICD_EXECUTION_SEQUENCE}
	  targetNamespace: <team_space_name>
	  name: mod-source-record-manager
timeout: 60
notification: {}

 update Dockerfile

additional step 


# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-storage-server/lib/* ${LIB_DIR}/

Dockerfile
FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-source-record-storage-server-fat.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY mod-source-record-manager-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-manager-server/lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 8081

for simplifying the steps described above there is a branch with name 'rancher' which you may use.


mod-inventory

There are two dependencies exists for mod-inventory

  • data-import-processing-core 
  • mod-source-record-storage-client

To make mod-source-record-manager testable on Rancher env, do next steps:

 add libs to the project
  • add  data-import-processing-core lib
    • build data-import-processing-core from the branch that contains your changes
    • create a folder (for instance with name 'lib') in mod-source-record-manager-server 
    • copy data-import-processing-core-fat.jar  to 'lib' folder


  • add mod-source-record-storage-client lib
    • build mod-source-record-storage-client from the branch that contains your changes
    • copy mod-source-record-storage-client-<version>.jar to 'lib' folder


 update mod-source-record-manager-server pom.xml file
  • add a data-import-processing-core library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency> <groupId>org.folio</groupId> 
	<artifactId>data-import-processing-core</artifactId> 
	<scope>system</scope> 
	<version>version_of_lib</version> 
	<systemPath>path_to_lib</systemPath> 
</dependency>
  • add a mod-source-record-storage-client library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>mod-source-record-storage-client</artifactId>
	<version>version_of_lib</version>
	<scope>system</scope>
	<systemPath>path_to_lib</systemPath>
</dependency>


  • update maven-shade-plugin with property
<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-source-record-storage-client-5.1.0.0-SNAPSHOT.jar</Class-Path>
 add a rancher pipeline file 
.rancher-pipeline.yml
stages:
  - name: Clone submodules
	steps:
	- runScriptConfig:
		image: alpine/git
		shellScript: git submodule update --init --recursive
  - name: Build
	steps:
	- runScriptConfig:
		image: 'maven:3-openjdk-11'
		shellScript: mvn package -DskipTests
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
		dockerfilePath: ./Dockerfile
		buildContext: .
		tag: docker.dev.folio.org/mod-inventory:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
		pushRemote: true
		registry: docker.dev.folio.org
  - name: Deploy
	steps:
	- applyAppConfig:
	catalogTemplate: 'p-<team_prefix>:<team_space_name>-helmcharts-mod-inventory'
	version: 0.1.32
	answers:
	  image.repository: docker.dev.folio.org/mod-inventory
	  image.tag: '<team_space_name>-${CICD_EXECUTION_SEQUENCE}'
 	  postJob.enabled: true
	targetNamespace: <team_space_name>
	name: mod-inventory
timeout: 60
notification: {}
 update Dockerfile

additional step 


# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/



FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-inventory.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 9403


mod-data-import

Module has following dependency 

  • data-import-processing-core

  • mod-source-record-manager-client

To make mod-source-record-storage testable on Rancher env, do next steps:

 add libs to the project
  • add  data-import-processing-core lib
    • build data-import-processing-core from the branch that contains your changes
    • create a folder (for instance with name 'lib') in mod-source-record-manager-server 
    • copy data-import-processing-core-fat.jar  to 'lib' folder


  • add mod-source-record-manager-client lib
    • build mod-source-record-manager-client from the branch that contains your changes
    • copy mod-source-record-manager-client-<version>.jar to 'lib' folder


 update pom.xml file
  • add a data-import-processing-core library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency> <groupId>org.folio</groupId> 
	<artifactId>data-import-processing-core</artifactId> 
	<scope>system</scope> 
	<version>version_of_lib</version> 
	<systemPath>path_to_lib</systemPath> 
</dependency>
  • add a mod-source-record-manager-client library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> 
<dependency>
	<groupId>org.folio</groupId>
	<artifactId>mod-source-record-manager-client</artifactId>
	<scope>system</scope>
	<version>version_of_lib</version>
	<systemPath>path_to_lib</systemPath>
</dependency>


  • update maven-shade-plugin with property
<Class-Path>path_to_dependency_jar_file</Class-Path>

Example

<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-source-record-manager-client-3.1.0-SNAPSHOT.jar</Class-Path>
 add a rancher pipeline file 
.rancher-pipeline.yml
stages:
  - name: Build
	steps:
	- runScriptConfig:
	  image: maven:3.6.3-openjdk-11
      shellScript: mvn package -DskipTests -Djava.util.logging.config.file=vertx-default-jul-logging.properties
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
	  dockerfilePath: ./Dockerfile
	  buildContext: .
	  tag: docker.dev.folio.org/mod-data-<team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  pushRemote: true
	  registry: docker.dev.folio.org
  - name: Deploy
	steps:
	- applyAppConfig:
	  catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-data-import
	  version: 0.1.32
	  answers:
	  image.repository: docker.dev.folio.org/mod-data-import
	  image.tag: <team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  targetNamespace: <team_space_name>
	  name: mod-data-import
timeout: 60
notification: {}
 update Dockerfile

additional step 


# Copy local lib to the container

ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/



FROM folioci/alpine-jre-openjdk11:latest

ENV VERTICLE_FILE mod-data-import-fat.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Copy local lib to the container 
ENV LIB_DIR ${VERTICLE_HOME}/lib 
RUN mkdir -p ${LIB_DIR} 
COPY lib/* ${LIB_DIR}/

# Expose this port locally in the container.
EXPOSE 8081

mod-data-import-converter-storage

This module does not have any dependency to other modules involved into the data-import process, so the only steps needed are:

 build project
  • build mod-data-import-converter-storage from the branch that contains your changes
 add a rancher pipeline file 
.rancher-pipeline.yml
stages:
  - name: Build
	steps:
	- runScriptConfig:
	image: maven:3.6.3-openjdk-11
	shellScript: mvn package -DskipTests -Djava.util.logging.config.file=vertx-default-jul-logging.properties
  - name: Build Docker with DIND
	steps:
	- publishImageConfig:
	  dockerfilePath: ./Dockerfile
	  buildContext: .
	  tag: docker.dev.folio.org/mod-data-import-converter-storage:<team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  pushRemote: true
	  registry: docker.dev.folio.org
  - name: Deploy
	steps:
	- applyAppConfig:
	  catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-data-import-converter-storage
	  version: 0.1.32
	  answers:
	  image.repository: docker.dev.folio.org/mod-data-import-converter-storage
	  image.tag: <team_space_name>-${CICD_EXECUTION_SEQUENCE}
	  targetNamespace: <team_space_name>
	  name: mod-data-import-converter-storage
timeout: 60
notification: {}
  • No labels