Data-import testing approach on Rancher
Natalia Zaitseva
Oleksandr_Dekin
Merging order:
- data-import-processing-core
- raml-storage
- mod-source-record-storage
- - raml-storage
- data-import-processing-core
- raml-storage
- mod-source-record-manager
- data-import-processing-core
- raml-storage
- mod-source-record-storage-client
- data-import-processing-core
- mod-inventory
- data-import-processing-core
- mod-source-record-storage-client
- mod-data-import
- mod-source-record-manager-client
- data-import-processing-core
- mod-data-import-converter-storage
- mod-qucik-marc
data-import-processing-core
This module is a library, so it does not have any dependency to other modules involved into the data-import process, so the only steps needed are:
- build
data-import-processing-core
from the branch that contains your changes
mod-source-record-storage
Module has following dependency
data-import-processing-core
To make mod-source-record-storage
testable on Rancher env, do next steps:
- build
data-import-processing-core
from the branch that contains your changes - create a folder (for instance with name
'lib'
) inmod-source-record-storage-server
- copy
data-import-processing-core-fat.jar
to'lib'
folder
- add a local library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>data-import-processing-core</artifactId> <scope>system</scope> <version>version_of_lib</version> <systemPath>path_to_lib</systemPath> </dependency>
- update
maven-shade-plugin
with property
<Class-Path>path_to_dependency_jar_file</Class-Path>
Example
<Class-Path>lib/data-import-processing-core-fat.jar</Class-Path>
#* !Dockerfile !docker* !mod-source-record-storage-server/target/*.jar
stages: - name: Build steps: - runScriptConfig: image: postgres:12.2 shellScript: |- apt update apt install -y maven git docker-entrypoint.sh "postgres" & mvn package -DskipTests -Ddocker.skip -Ddocker.host.address=localhost -Ddocker.container.db.ip=127.0.0.1 -Ddb.port=5432 -Djava.util.logging.config.file=vertx-default-jul-logging.properties env: POSTGRES_DB: postgres POSTGRES_USER: postgres POSTGRES_PASSWORD: postgres - name: Build Docker with DIND steps: - publishImageConfig: dockerfilePath: ./Dockerfile buildContext: . tag: docker.dev.folio.org/mod-source-record-storage:<team_space_name>-${CICD_EXECUTION_SEQUENCE} pushRemote: true registry: docker.dev.folio.org - name: Deploy steps: - applyAppConfig: catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-source-record-storage version: 0.1.32 answers: image.repository: docker.dev.folio.org/mod-source-record-storage image.tag: <team_space_name>-${CICD_EXECUTION_SEQUENCE} targetNamespace: <team_space_name> name: mod-source-record-storage timeout: 60 notification: {}
additional step
# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-storage-server/lib/* ${LIB_DIR}/
FROM folioci/alpine-jre-openjdk11:latest ENV VERTICLE_FILE mod-source-record-storage-server-fat.jar # Set the location of the verticles ENV VERTICLE_HOME /usr/verticles # Copy your fat jar to the container COPY mod-source-record-storage-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE} # Copy local lib to the container ENV LIB_DIR ${VERTICLE_HOME}/lib RUN mkdir -p ${LIB_DIR} COPY mod-source-record-storage-server/lib/* ${LIB_DIR}/ # Expose this port locally in the container. EXPOSE 8081
mod-source-record-manager
Module has following dependencies
data-import-processing-core
mod-data-import-converter-storage-client
mod-source-record-storage-client
To make mod-source-record-manager
testable on Rancher env, do next steps:
- add
data-import-processing-core
lib- build
data-import-processing-core
from the branch that contains your changes - create a folder (for instance with name
'lib'
) inmod-source-record-manager-server
- copy
data-import-processing-core-fat.jar
to'lib'
folder
- build
- add
mod-data-import-converter-storage-client
lib- build
mod-data-import-converter-storage-client
from the branch that contains your changes - copy
mod-data-import-converter-storage-client-<version>
.jar
to'lib'
folder
- build
- perform similar steps for
mod-source-record-storage-client
if needed
- add a
data-import-processing-core
library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>data-import-processing-core</artifactId> <scope>system</scope> <version>version_of_lib</version> <systemPath>path_to_lib</systemPath> </dependency>
- add a
mod-data-import-converter-storage-client
library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>mod-data-import-converter-storage-client</artifactId> <scope>system</scope> <version>version_of_lib</version> <systemPath>path_to_lib</systemPath> </dependency>
- add a
mod-source-record-storage-client
library as a system dependency(if needed)
- update
maven-shade-plugin
with property
<Class-Path>path_to_dependency_jar_file</Class-Path>
Example
<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-data-import-converter-storage-client-1.11.0-SNAPSHOT.jar</Class-Path>
#* !Dockerfile !docker* !mod-source-record-manager-server/target/*.jar
stages: - name: Build steps: - runScriptConfig: image: maven:3-adoptopenjdk-11 shellScript: mvn package -DskipTests -Djava.util.logging.config.file=vertx-default-jul-logging.properties - name: Build Docker with DIND steps: - publishImageConfig: dockerfilePath: ./Dockerfile buildContext: . tag: docker.dev.folio.org/mod-source-record-manager:<team_space_name>-${CICD_EXECUTION_SEQUENCE} pushRemote: true registry: docker.dev.folio.org - name: Deploy steps: - applyAppConfig: catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-source-record-manager version: 0.1.32 answers: image.repository: docker.dev.folio.org/mod-source-record-manager image.tag: spitfire-${CICD_EXECUTION_SEQUENCE} targetNamespace: <team_space_name> name: mod-source-record-manager timeout: 60 notification: {}
additional step
# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-manager-server/lib/* ${LIB_DIR}/
FROM folioci/alpine-jre-openjdk11:latest ENV VERTICLE_FILE mod-source-record-storage-server-fat.jar # Set the location of the verticles ENV VERTICLE_HOME /usr/verticles # Copy your fat jar to the container COPY mod-source-record-manager-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE} # Copy local lib to the container ENV LIB_DIR ${VERTICLE_HOME}/lib RUN mkdir -p ${LIB_DIR} COPY mod-source-record-manager-server/lib/* ${LIB_DIR}/ # Expose this port locally in the container. EXPOSE 8081
for simplifying the steps described above there is a branch with name 'rancher' which you may use.
mod-inventory
There are two dependencies exists for mod-inventory
- data-import-processing-core
- mod-source-record-storage-client
To make mod-source-record-manager
testable on Rancher env, do next steps:
- add
data-import-processing-core
lib- build
data-import-processing-core
from the branch that contains your changes - create a folder (for instance with name
'lib'
) inmod-source-record-manager-server
- copy
data-import-processing-core-fat.jar
to'lib'
folder
- build
- add
mod-source-record-storage-client
lib- build
mod-source-record-storage-client
from the branch that contains your changes - copy
mod-source-record-storage-client-<version>
.jar
to'lib'
folder
- build
- add a
data-import-processing-core
library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>data-import-processing-core</artifactId> <scope>system</scope> <version>version_of_lib</version> <systemPath>path_to_lib</systemPath> </dependency>
- add a
mod-source-record-storage-client
library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>mod-source-record-storage-client</artifactId> <version>version_of_lib</version> <scope>system</scope> <systemPath>path_to_lib</systemPath> </dependency>
- update
maven-shade-plugin
with property
<Class-Path>path_to_dependency_jar_file</Class-Path>
Example
<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-source-record-storage-client-5.1.0.0-SNAPSHOT.jar</Class-Path>
stages: - name: Clone submodules steps: - runScriptConfig: image: alpine/git shellScript: git submodule update --init --recursive - name: Build steps: - runScriptConfig: image: 'maven:3-openjdk-11' shellScript: mvn package -DskipTests - name: Build Docker with DIND steps: - publishImageConfig: dockerfilePath: ./Dockerfile buildContext: . tag: docker.dev.folio.org/mod-inventory:<team_space_name>-${CICD_EXECUTION_SEQUENCE} pushRemote: true registry: docker.dev.folio.org - name: Deploy steps: - applyAppConfig: catalogTemplate: 'p-<team_prefix>:<team_space_name>-helmcharts-mod-inventory' version: 0.1.32 answers: image.repository: docker.dev.folio.org/mod-inventory image.tag: '<team_space_name>-${CICD_EXECUTION_SEQUENCE}' postJob.enabled: true targetNamespace: <team_space_name> name: mod-inventory timeout: 60 notification: {}
additional step
# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/
FROM folioci/alpine-jre-openjdk11:latest ENV VERTICLE_FILE mod-inventory.jar # Set the location of the verticles ENV VERTICLE_HOME /usr/verticles # Copy your fat jar to the container COPY target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE} # Copy local lib to the container ENV LIB_DIR ${VERTICLE_HOME}/lib RUN mkdir -p ${LIB_DIR} COPY lib/* ${LIB_DIR}/ # Expose this port locally in the container. EXPOSE 9403
mod-data-import
Module has following dependency
data-import-processing-core
mod-source-record-manager-client
To make mod-source-record-storage
testable on Rancher env, do next steps:
- add
data-import-processing-core
lib- build
data-import-processing-core
from the branch that contains your changes - create a folder (for instance with name
'lib'
) inmod-source-record-manager-server
- copy
data-import-processing-core-fat.jar
to'lib'
folder
- build
- add
mod-source-record-manager-client
lib- build
mod-source-record-manager-client
from the branch that contains your changes - copy
mod-source-record-manager-client
-<version>
.jar
to'lib'
folder
- build
- add a
data-import-processing-core
library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>data-import-processing-core</artifactId> <scope>system</scope> <version>version_of_lib</version> <systemPath>path_to_lib</systemPath> </dependency>
- add a
mod-source-record-manager-client
library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>mod-source-record-manager-client</artifactId> <scope>system</scope> <version>version_of_lib</version> <systemPath>path_to_lib</systemPath> </dependency>
- update
maven-shade-plugin
with property
<Class-Path>path_to_dependency_jar_file</Class-Path>
Example
<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-source-record-manager-client-3.1.0-SNAPSHOT.jar</Class-Path>
stages: - name: Build steps: - runScriptConfig: image: maven:3.6.3-openjdk-11 shellScript: mvn package -DskipTests -Djava.util.logging.config.file=vertx-default-jul-logging.properties - name: Build Docker with DIND steps: - publishImageConfig: dockerfilePath: ./Dockerfile buildContext: . tag: docker.dev.folio.org/mod-data-<team_space_name>-${CICD_EXECUTION_SEQUENCE} pushRemote: true registry: docker.dev.folio.org - name: Deploy steps: - applyAppConfig: catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-data-import version: 0.1.32 answers: image.repository: docker.dev.folio.org/mod-data-import image.tag: <team_space_name>-${CICD_EXECUTION_SEQUENCE} targetNamespace: <team_space_name> name: mod-data-import timeout: 60 notification: {}
additional step
# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/
FROM folioci/alpine-jre-openjdk11:latest ENV VERTICLE_FILE mod-data-import-fat.jar # Set the location of the verticles ENV VERTICLE_HOME /usr/verticles # Copy your fat jar to the container COPY target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE} # Copy local lib to the container ENV LIB_DIR ${VERTICLE_HOME}/lib RUN mkdir -p ${LIB_DIR} COPY lib/* ${LIB_DIR}/ # Expose this port locally in the container. EXPOSE 8081
mod-data-import-converter-storage
This module does not have any dependency to other modules involved into the data-import process, so the only steps needed are:
- build
mod-data-import-converter-storage
from the branch that contains your changes
stages: - name: Build steps: - runScriptConfig: image: maven:3.6.3-openjdk-11 shellScript: mvn package -DskipTests -Djava.util.logging.config.file=vertx-default-jul-logging.properties - name: Build Docker with DIND steps: - publishImageConfig: dockerfilePath: ./Dockerfile buildContext: . tag: docker.dev.folio.org/mod-data-import-converter-storage:<team_space_name>-${CICD_EXECUTION_SEQUENCE} pushRemote: true registry: docker.dev.folio.org - name: Deploy steps: - applyAppConfig: catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-data-import-converter-storage version: 0.1.32 answers: image.repository: docker.dev.folio.org/mod-data-import-converter-storage image.tag: <team_space_name>-${CICD_EXECUTION_SEQUENCE} targetNamespace: <team_space_name> name: mod-data-import-converter-storage timeout: 60 notification: {}
mod-quick-marc
First of all, you should generate jar from SRM client and add lib into folder lib for quick marc (for example: mod-quick-marc → lib → mod-source-record-manager-client-3.2.0-SNAPSHOT.jar)
Change docker file:
# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles
# - should be a single jar file
ARG JAR_FILE=./target/mod-quick-marc.jar
////////////////////
# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/
Example of Dockerfile:
FROM folioci/alpine-jre-openjdk11:latest
# Copy your fat jar to the container
ENV APP_FILE mod-quick-marc-fat.jar
# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles
# - should be a single jar file
ARG JAR_FILE=./target/mod-quick-marc.jar
# - copy
COPY ${JAR_FILE} ${JAVA_APP_DIR}/${APP_FILE}
# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/
# Expose this port locally in the container.
EXPOSE 8081
Change pom.xml by adding plugin maven-shade-plugin and add properties
<properties>
<start-class>org.folio.qm.ModQuickMarcApplication</start-class>
///////////////////////////
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.handlers</resource>
</transformer>
<transformer
implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
<resource>META-INF/spring.factories</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.schemas</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>${start-class}</Main-Class>
<Multi-Release>true</Multi-Release>
<Class-Path>lib/mod-source-record-manager-client-3.2.0-SNAPSHOT.jar</Class-Path>
</manifestEntries>
</transformer>
</transformers>
<artifactSet />
<outputFile>${project.build.directory}/${project.artifactId}.jar</outputFile>
</configuration>
</execution>
</executions>
</plugin>
Also ew should change SRM client in the pom.xml by adding systemPath and scope (system)
<dependency>
<groupId>org.folio</groupId>
<artifactId>mod-source-record-manager-client</artifactId>
<scope>system</scope>
<systemPath>${basedir}/lib/mod-source-record-manager-client-3.2.0-SNAPSHOT.jar</systemPath>
<version>${mod-srm-client.version}</version>
<exclusions>
<exclusion>
<groupId>org.folio</groupId>
<artifactId>domain-models-runtime</artifactId>
</exclusion>
<exclusion>
<groupId>org.folio</groupId>
<artifactId>domain-models-api-interfaces</artifactId>
</exclusion>
</exclusions>
</dependency>