Data-import testing approach on Rancher
- Natalia Zaitseva
- Oleksandr_Dekin
Merging order:
- data-import-processing-core
- raml-storage
- mod-source-record-storage
- - raml-storage
- data-import-processing-core
- raml-storage
- mod-source-record-manager
- data-import-processing-core
- raml-storage
- mod-source-record-storage-client
- data-import-processing-core
- mod-inventory
- data-import-processing-core
- mod-source-record-storage-client
- mod-data-import
- mod-source-record-manager-client
- data-import-processing-core
- mod-data-import-converter-storage
- mod-qucik-marc
data-import-processing-core
This module is a library, so it does not have any dependency to other modules involved into the data-import process, so the only steps needed are:
- build
data-import-processing-core
from the branch that contains your changes
mod-source-record-storage
Module has following dependency
data-import-processing-core
To make mod-source-record-storage
testable on Rancher env, do next steps:
- build
data-import-processing-core
from the branch that contains your changes - create a folder (for instance with name
'lib'
) inmod-source-record-storage-server
- copy
data-import-processing-core-fat.jar
to'lib'
folder
- add a local library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>data-import-processing-core</artifactId> <scope>system</scope> <version>version_of_lib</version> <systemPath>path_to_lib</systemPath> </dependency>
- update
maven-shade-plugin
with property
<Class-Path>path_to_dependency_jar_file</Class-Path>
Example
<Class-Path>lib/data-import-processing-core-fat.jar</Class-Path>
#* !Dockerfile !docker* !mod-source-record-storage-server/target/*.jar
stages: - name: Build steps: - runScriptConfig: image: postgres:12.2 shellScript: |- apt update apt install -y maven git docker-entrypoint.sh "postgres" & mvn package -DskipTests -Ddocker.skip -Ddocker.host.address=localhost -Ddocker.container.db.ip=127.0.0.1 -Ddb.port=5432 -Djava.util.logging.config.file=vertx-default-jul-logging.properties env: POSTGRES_DB: postgres POSTGRES_USER: postgres POSTGRES_PASSWORD: postgres - name: Build Docker with DIND steps: - publishImageConfig: dockerfilePath: ./Dockerfile buildContext: . tag: docker.dev.folio.org/mod-source-record-storage:<team_space_name>-${CICD_EXECUTION_SEQUENCE} pushRemote: true registry: docker.dev.folio.org - name: Deploy steps: - applyAppConfig: catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-source-record-storage version: 0.1.32 answers: image.repository: docker.dev.folio.org/mod-source-record-storage image.tag: <team_space_name>-${CICD_EXECUTION_SEQUENCE} targetNamespace: <team_space_name> name: mod-source-record-storage timeout: 60 notification: {}
additional step
# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-storage-server/lib/* ${LIB_DIR}/
FROM folioci/alpine-jre-openjdk11:latest ENV VERTICLE_FILE mod-source-record-storage-server-fat.jar # Set the location of the verticles ENV VERTICLE_HOME /usr/verticles # Copy your fat jar to the container COPY mod-source-record-storage-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE} # Copy local lib to the container ENV LIB_DIR ${VERTICLE_HOME}/lib RUN mkdir -p ${LIB_DIR} COPY mod-source-record-storage-server/lib/* ${LIB_DIR}/ # Expose this port locally in the container. EXPOSE 8081
mod-source-record-manager
Module has following dependencies
data-import-processing-core
mod-data-import-converter-storage-client
mod-source-record-storage-client
To make mod-source-record-manager
testable on Rancher env, do next steps:
- add
data-import-processing-core
lib- build
data-import-processing-core
from the branch that contains your changes - create a folder (for instance with name
'lib'
) inmod-source-record-manager-server
- copy
data-import-processing-core-fat.jar
to'lib'
folder
- build
- add
mod-data-import-converter-storage-client
lib- build
mod-data-import-converter-storage-client
from the branch that contains your changes - copy
mod-data-import-converter-storage-client-<version>
.jar
to'lib'
folder
- build
- perform similar steps for
mod-source-record-storage-client
if needed
- add a
data-import-processing-core
library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>data-import-processing-core</artifactId> <scope>system</scope> <version>version_of_lib</version> <systemPath>path_to_lib</systemPath> </dependency>
- add a
mod-data-import-converter-storage-client
library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>mod-data-import-converter-storage-client</artifactId> <scope>system</scope> <version>version_of_lib</version> <systemPath>path_to_lib</systemPath> </dependency>
- add a
mod-source-record-storage-client
library as a system dependency(if needed)
- update
maven-shade-plugin
with property
<Class-Path>path_to_dependency_jar_file</Class-Path>
Example
<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-data-import-converter-storage-client-1.11.0-SNAPSHOT.jar</Class-Path>
#* !Dockerfile !docker* !mod-source-record-manager-server/target/*.jar
stages: - name: Build steps: - runScriptConfig: image: maven:3-adoptopenjdk-11 shellScript: mvn package -DskipTests -Djava.util.logging.config.file=vertx-default-jul-logging.properties - name: Build Docker with DIND steps: - publishImageConfig: dockerfilePath: ./Dockerfile buildContext: . tag: docker.dev.folio.org/mod-source-record-manager:<team_space_name>-${CICD_EXECUTION_SEQUENCE} pushRemote: true registry: docker.dev.folio.org - name: Deploy steps: - applyAppConfig: catalogTemplate: p-<team_prefix>:<team_space_name>-helmcharts-mod-source-record-manager version: 0.1.32 answers: image.repository: docker.dev.folio.org/mod-source-record-manager image.tag: spitfire-${CICD_EXECUTION_SEQUENCE} targetNamespace: <team_space_name> name: mod-source-record-manager timeout: 60 notification: {}
additional step
# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY mod-source-record-manager-server/lib/* ${LIB_DIR}/
FROM folioci/alpine-jre-openjdk11:latest ENV VERTICLE_FILE mod-source-record-storage-server-fat.jar # Set the location of the verticles ENV VERTICLE_HOME /usr/verticles # Copy your fat jar to the container COPY mod-source-record-manager-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE} # Copy local lib to the container ENV LIB_DIR ${VERTICLE_HOME}/lib RUN mkdir -p ${LIB_DIR} COPY mod-source-record-manager-server/lib/* ${LIB_DIR}/ # Expose this port locally in the container. EXPOSE 8081
for simplifying the steps described above there is a branch with name 'rancher' which you may use.
mod-inventory
There are two dependencies exists for mod-inventory
- data-import-processing-core
- mod-source-record-storage-client
To make mod-source-record-manager
testable on Rancher env, do next steps:
- add
data-import-processing-core
lib- build
data-import-processing-core
from the branch that contains your changes - create a folder (for instance with name
'lib'
) inmod-source-record-manager-server
- copy
data-import-processing-core-fat.jar
to'lib'
folder
- build
- add
mod-source-record-storage-client
lib- build
mod-source-record-storage-client
from the branch that contains your changes - copy
mod-source-record-storage-client-<version>
.jar
to'lib'
folder
- build
- add a
data-import-processing-core
library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>data-import-processing-core</artifactId> <scope>system</scope> <version>version_of_lib</version> <systemPath>path_to_lib</systemPath> </dependency>
- add a
mod-source-record-storage-client
library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>mod-source-record-storage-client</artifactId> <version>version_of_lib</version> <scope>system</scope> <systemPath>path_to_lib</systemPath> </dependency>
- update
maven-shade-plugin
with property
<Class-Path>path_to_dependency_jar_file</Class-Path>
Example
<Class-Path>lib/data-import-processing-core-fat.jar lib/mod-source-record-storage-client-5.1.0.0-SNAPSHOT.jar</Class-Path>
stages: - name: Clone submodules steps: - runScriptConfig: image: alpine/git shellScript: git submodule update --init --recursive - name: Build steps: - runScriptConfig: image: 'maven:3-openjdk-11' shellScript: mvn package -DskipTests - name: Build Docker with DIND steps: - publishImageConfig: dockerfilePath: ./Dockerfile buildContext: . tag: docker.dev.folio.org/mod-inventory:<team_space_name>-${CICD_EXECUTION_SEQUENCE} pushRemote: true registry: docker.dev.folio.org - name: Deploy steps: - applyAppConfig: catalogTemplate: 'p-<team_prefix>:<team_space_name>-helmcharts-mod-inventory' version: 0.1.32 answers: image.repository: docker.dev.folio.org/mod-inventory image.tag: '<team_space_name>-${CICD_EXECUTION_SEQUENCE}' postJob.enabled: true targetNamespace: <team_space_name> name: mod-inventory timeout: 60 notification: {}
additional step
# Copy local lib to the container
ENV LIB_DIR ${VERTICLE_HOME}/lib
RUN mkdir -p ${LIB_DIR}
COPY lib/* ${LIB_DIR}/
FROM folioci/alpine-jre-openjdk11:latest ENV VERTICLE_FILE mod-inventory.jar # Set the location of the verticles ENV VERTICLE_HOME /usr/verticles # Copy your fat jar to the container COPY target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE} # Copy local lib to the container ENV LIB_DIR ${VERTICLE_HOME}/lib RUN mkdir -p ${LIB_DIR} COPY lib/* ${LIB_DIR}/ # Expose this port locally in the container. EXPOSE 9403
mod-data-import
Module has following dependency
data-import-processing-core
mod-source-record-manager-client
To make mod-source-record-storage
testable on Rancher env, do next steps:
- add
data-import-processing-core
lib- build
data-import-processing-core
from the branch that contains your changes - create a folder (for instance with name
'lib'
) inmod-source-record-manager-server
- copy
data-import-processing-core-fat.jar
to'lib'
folder
- build
- add
mod-source-record-manager-client
lib- build
mod-source-record-manager-client
from the branch that contains your changes - copy
mod-source-record-manager-client
-<version>
.jar
to'lib'
folder
- build
- add a
data-import-processing-core
library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>data-import-processing-core</artifactId> <scope>system</scope> <version>version_of_lib</version> <systemPath>path_to_lib</systemPath> </dependency>
- add a
mod-source-record-manager-client
library as a system dependency
<!-- TODO: THE VERSION OF LIBRARY MUST BE CHANGED BACK ONCE TESTING ON RANCHER IS COMPLETE--> <dependency> <groupId>org.folio</groupId> <artifactId>mod-source-record-manager-client</artifactId> <scope>system</scope> <version>version_of_lib</version> <systemPath>path_to_lib</systemPath> </dependency>
- update
maven-shade-plugin
with property