The Performance Task Force needs dedicated resources to host carrier-io and FOLIO. Carrier-io is a suite of tools: Galloper, Loki, Traefik, InfluxDB, Grafana, Redis, Vault, pgHero, Jenkins that are CPU and memory intensive. These tools need to be run on an EC2 instance that is fast and has a lot of disk space. Carrier-io test services need a big volume disk space to store test execution data, especially profiled data. Eventually, the data would be backed up and archived on an S3 bucket so that one day if there's a need to bring up data for comparison we simply restore the backed up data.
PTF will start out with one carrier-io instance, but if there are many tests to run and if we don't want to wait to run tests, PTF will need a second carrier-io machine to do its job efficiently.
PTF also needs a dedicated FOLIO environment to run tests against. The software will need to be hosted on a production-grade compatible hardware in order for the test results to have any value. There is a likely a need to have a second FOLIO environment in which all modules are loaded with the Uber profiler so that they can profile the modules. When the tests are not running, we could stop the second FOLIO environment machines to save cost.
...
Motivation
While working on the Data Import test using AWS MSK, Kafka messages we getting blocked in Kafka topic and we had to clear out messages from all DI topics. Messages can get stuck in a topic for many reasons such as consumers not having enough resources to consume all topics, the consumer goes down or AWS MSK Broker becomes unstable. Depending on the number of tenants per FOLIO instance, there can be thousands of topics per FOLIO instance. Instead of manually clearing out all messages from each of these topics, we decided to automate it to make this process faster.
To clear out messages from Kafka topic, there are 3 steps:
- Get the list of topics to clear out messages
- Download the latest Kafka release and extract it - https://kafka.apache.org/quickstart
- Navigate to the bin directory kafka_2.12-2.7.0/bin/ - we will be using kafka-topics.sh and kafka-configs.sh scripts out of the box
- Get list of topics and write to file topics.txt
./kafka-topics.sh --describe --zookeeper <zookeeper-plaintext-connection-url> --topic "imtc.Default.fs00001020.*" | grep Configs | awk '{printf "%s\n", $2}' > topics.txt
- Alter topic by adding custom config by setting retention time to 1 second or less
- Revert topic by deleting the custom config