/
Thunderjet - Onboarding plan

Thunderjet - Onboarding plan

(this is a draft)

 

This documentation provides some help for backend developers to get started. It is most likely incomplete, so please don't hesitate to ask and add information here as needed.

See also: End user documentation, Overview for a new developer and Newcomer's first steps

FOLIO Design & Architecture

Introduction to FOLIO Acquisitions

Some demo videos are useful to understand FOLIO acquisition apps (these are using old versions):

See also: Thunderjet - Guidelines for backend code organization, unit tests and Karate tests

Running in IDEA

Installation

  1. Install Java JDK 21

  2. Install IDEA as IDE : https://www.jetbrains.com/idea/

  3. Clone from git modules and add all of them in one IDEA project (import as Maven)

    • ThunderJet back-end modules : mod-orders, mod-orders-storage, mod-invoice, mod-invoice-storage, mod-finance, mod-finance-storage, mod-organizations, mod-organizations-storage, mod-gobi, mod-mosaic, mod-consortia-keycloak,edge-orders, edge-common, mod-ebsconet, acq-models

    • Modules with integration tests: folio-integration-tests

  4. Use eureka-cli to run a local environment, see requirements there; use the combined config

IDEA project / modules organization and plugins

In IDEA, a single project can use the acquisition git repositories as modules. It can use a project directory containing all the git repositories.

A module is added with the Maven configuration. After compile, the directory target/generated-sources should be set as a generated source folder for the module.

The following plugins can be useful: uppercut (to run karate tests and for syntax highlighting), YAML.

Running a module locally

Create a run/debug configuration, with (example for mod-orders):

  • JVM: Java 21

  • Classpath: mod-orders

  • VM options: -Dhttp.port=36002 (or whatever free port you wish to use)

  • Main class: org.folio.rest.RestLauncher

  • Program arguments: org.folio.rest.RestVerticle

  • Working directory: path/to/mod-orders

  • Environment variables: KAFKA_HOST=localhost;KAFKA_PORT=9092 (this should match the eureka-cli config; it is possible to use an env file to specify many variables; PostgreSQL env variables like DB_HOST, DB_PORT, DB_DATABASE, DB_USERNAME and DB_PASSWORD might also be needed when running modules using direct database access such as -storage modules)

To intercept the eureka-cli module with the one running in IDEA, see Intercept a module in the eureka-cli documentation.

For instance, to intercept mod-orders:

eureka-cli interceptModule -p combined -n mod-orders -gm 36002 -s 37002

(the -gm port needs to match the one used to run the module; the -s one is for the sidecar module)

To stop the intercept:

eureka-cli interceptModule -p combined -n mod-orders -r

Debugging

A "Remote JVM Debug" configuration can be used to debug a module in IDEA, without having to redeploy it (check the port with docker).

Modules can be executed in debug mode in IDEA, they just need to be intercepted (see below). Unit tests can also be debugged in the same way.

change log level for mod-invoice:
curl -X PUT 'http://localhost:30017/admin/loglevel?level=DEBUG'

(port might change, check with docker ps)

Creating and running unit tests

Unit tests are executed with Maven. The files are in src/test.

Tests are using Mockito, RestAssured, JUnit and Hamcrest.

This step might not be needed anymore when running eureka-cli locally: Remove src/main/resources/postgres-conf.json + mvn clean compile  before running unit tests (if the file is present); put it back afterwards and recompile (it's needed to run storage mods)

Creating and running integration tests

Most of the time, a local environment with eureka-cli (karate.env: dev) should be used for running tests.

Integration tests are using Karate.

Create karate tests in folio-integration-tests (this should be started before submitting a PR, to check results).

Make sure karate tests can run in parallel.

Tests can be executed either with an ApiTest class in acquisitions/src/test/java/org/folio, or by running the karate test (such as orders.feature) directly with an IDEA plugin such as the karate plugin (not free) or uppercut (free). It is possible to run tests in parallel in both cases.

To run a JUnit class, use the following VM options: -ea -Dkarate.env=dev -Dtest.mode=no-shared-pool

Because the list of tests is generated dynamically, this runs all the tests for the module. After this, a single test can be selected to run with right-click on the list of tests in the results.

When using uppercut to run a feature file or scenario:

  • In the IDEA settings, Tools - Karate: use Default environment: dev and Default parallelism: 4 (number of threads).

  • To run them the first time: right-click on the file, More Run/Debug, Modify Run Configuration, and in the dialog, set: Parallelism: 4, Environment: dev (to run locally with eureka-cli) and change "Working Directory" to the acquisitions directory so it saves reports in the target directory there (as with junit). Then just "Run". The configuration should be saved so the test can just be started with "Run" the next time.

See Karate Coding Standards & Best Practices.

Troubleshooting

Error when running tests:
setup-users.feature:105 - status code was: 422, expected: 201, response time: 18, url: http://localhost:9130/perms/users, response: {"errors":[{"code":"-1","message":"Unable to update derived fields: Attempting to add non-existent permissions invoice.all to permission user with id 4334831c-02d7-4936-849c-2f44d5491300","type":"1",

-> just deploy mod-invoice or whatever permissions are required for

 

Permission problems, for instance
2021-05-19T14:00:10,684 ERROR [vert.x-eventloop-thread-1] RestClient Exception calling GET /organizations-storage/organizations/c6dace5d-4574-411e-8ba1-036102fcdc9b - org.folio.rest.core.models.RequestContext@4286b5d9
java.util.concurrent.CompletionException: org.folio.invoices.rest.exceptions.HttpException: Access requires permission: organizations-storage.organizations.item.get

-> Check descriptor (clean+compile), redeploy

 

maven-resources-test:mod-finance-storage: java.lang.OutOfMemoryError: Java heap space

-> Increase Settings - Build, Execution etc - Compiler - Shared build process heap size and reload all projects

PR workflow

API and module versions

When modifying the API in ModuleDescriptor-template, update the API version number in the related path in it, for instance:
      "id": "finance.transactions",
      "version": "4.2",
Make a field required -> major API version (for storage + business)
Change a pattern (validation) -> no API version change.
Migration → no module version change
Major API version change -> major module version change.
Never modify pom.xml or NEWS.md unless this is a release PR.

Basic git operations

(assuming the JIRA ticket worked on is "MODORDSTOR-212")

Create the new branch locally:
git checkout -b MODORDSTOR-212
(even if there are changes: they will be moved to the new branch)
(if that doesn't work, try "git switch -c <new-branch>")

Commit changes:
git add .     (to undo, use git reset)
git commit -m "[MODORDSTOR-212] ..."  (to undo, git reset HEAD~ )

Push to the repo the first time (this is reminded when using simply "git push")
git push --set-upstream origin MODORDSTOR-212

after:
git push

To go back to the master branch:
git checkout master

Updating submodules

This is useful when modifying the acq-models schemas: in a particular module, the acq-models submodule has to be updated.

cd ramls/acq-models (for instance)
git fetch
git checkout master
git pull
(compile at this point will revert changes in ramls/acq-models - a commit needs to be done first)
cd ../..
git add ramls/acq-models
git commit -m ""

Finish the PR creation on github

After PR is created

  • Include a link to the Jira ticket in the description, with the ticket title.

  • Describe the changes so other developers understand the purpose and method. This is important when looking at old code, to figure out the developer intent and how code was modified.

  • Include links to related PRs.

  • Fix code smells etc.

  • If the build fails with continuous-integration/jenkins/pr-merge, try to click login and rerun buttons on top in jenkins.

  • Attach a screenshot with Karate tests to show it does not introduce a regression (no new test needed at this point).

Integration test PR

  • Create a new PR for the karate tests in folio-integration-tests (creating a new test or updating one), using the same branch name based on the JIRA ticket.

  • In the test PR, link to the module PR. Also link to the test PR in the module PR.

  • Possibly test user interface and add a comment to the JIRA ticket about how to test.

Code review

  • Add folio-org/acquisitions team to reviewers on github for both PRs

  • Change JIRA ticket to "Code review".

  • After a PR is approved by 2 people, the initial poster has to merge it.

  • Merge with master before if needed (the master branch is protected, so github won’t let you merge if the PR is not up to date anyway)

  • Check boxes in the PR.

  • Squash&Merge all related PRs at the same time

  • After the pull requests are merged in github:
    Change the "Fix Version" (version can be found in pom.xml, just remove "-SNAPSHOT") and change JIRA ticket to "Review".

Review

  • Summarize the changes in a comment in the Jira - this should be understandable for non-developers

  • Set the JIRA ticket "Assignee" to "Unassigned".

  • Based on your steps other guy from team will recheck it again and close if everything fine.