/
Eureka environments deployment known issues - AI generated KB for RAG purpose.

Eureka environments deployment known issues - AI generated KB for RAG purpose.

Eureka Environments Deployment: Known Issues & Resolutions

1. Deployment Errors: Missing Dependencies & Module Enablement Failures

  • Issue: Errors like RequestValidationException: Missing dependencies found... during tenant enablement; failures when enabling modules out of dependency order or using missing images.

  • Resolution:

    • Validate dependencies using the FOLIO API Dependencies Tool.

    • Deploy modules in this order: Kong β†’ Keycloak β†’ mgr-* modules β†’ tenant modules.

    • Wait for all pods/modules to be fully ready before beginning tenant enablement and entitlement.

    • If issues persist, inspect logs and confirm module versions in the platform-complete repository.

(Source: Kubernetes Example Deployment, RANCHER-1817, RANCHER-1860, Updated: 2025-07-30)

2. Resource Limitations & Infrastructure Problems

  • Issue: InternalServerErrorException error 500: Connection refused or doPostTenant/entitlement failures related to CPU/RAM exhaustion, resulting in throttling or pod eviction.

  • Resolution:

    • Allocate sufficient CPU/RAM quotas for each node.

    • Monitor pod resources in Rancher/Grafana, and assign module-level resource limits to avoid rebalancing.

    • For persistent problems, increase node size or cluster resources.

(Source: Kubernetes Example Deployment, Build/Deploy UI for Eureka environment, Updated: 2025-07-30)

3. Entitlement and Kafka Sidecar Errors

  • Issue: Errors like The module is not entitled on tenant ..., often caused by sidecar-Kafka misalignment or lost connections.

  • Resolution:

    • Monitor kafka-ui for Kafka and sidecar health.

    • Restart modules if sidecars lose Kafka connection; retry entitlement when fully healthy.

    • Ensure correct deployment and redeployment sequence for modules and mgr-tenant-entitlement.

(Source: Kubernetes Example Deployment, Updated: 2025-07-30)

4. Kong Upstream Timeout/504 Errors

  • Issue: Long-running requests via Kong result in 504: Upstream server is timing out.

  • Resolution:

    • Increase Kong/Nginx timeout values using environment variables.

    • For more details, see Kong documentation.

(Source: Kubernetes Example Deployment, Updated: 2025-07-30)

5. Issues with Module Enablement/Updating

  • Issue: New module versions fail to enable due to blocked PRs or incomplete descriptor changes.

  • Resolution:

    • Confirm descriptor PRs are merged and available.

    • Wait for upstream snapshot builds; only enable modules after all dependencies are deployed.

(Source: RANCHER-1868, RANCHER-1880, Updated: 2024-11-05)

6. UI and Build Pipeline Issues

  • Issue: UI build failures, inconsistent UI flow after changes, or snapshot deployment errors.

  • Resolution:

    • Use buildAndDeployUIEureka pipeline with the correct parameters.

    • Investigate Jenkins pipeline failures, fix parameters, and manually rebuild/reset if necessary.

(Source: Build/Deploy UI for Eureka environment, Updated: 2025-03-03; RANCHER-2181, Updated: 2025-03-12)

7. Environment Inaccessibility or Partial Failure

  • Issue: Env/Database/UI inaccessible after Jenkins job or pod restart.

  • Resolution:

    • After failed create/start env job, delete the old namespace fully using deleteNamespace before re-creation.

    • For persistent access issues, use recreateTeamNamespace and monitor pods in Rancher UI.

(Source: How to recreate Eureka Rancher environment, RANCHER-1990, Updated: 2025-03-06)

8. Login Issues

  • Issue: Users cannot log in, even when environment and modules appear enabled.

  • Resolution:

    • Run loginIssueFix pipeline. Wait 5 minutes after it completes before retrying login.

(Source: Fix Eureka login issue, Updated: 2025-02-17)

9. Capabilities/Capability Sets Missing After Entitlement

  • Issue: Unable to assign capabilities immediately after entitlement due to Kafka processing lag.

  • Resolution:

    • Wait for mod-roles-keycloak to consume the relevant Kafka queue, then proceed with assignments.

(Source: Kubernetes Example Deployment, Updated: 2025-07-30)

10. Sprint Testing/Module Update Issues

  • Issue: Environments do not reflect the correct release version or do not include expected fixes/features for testing.

  • Resolution:

    • Use deployModulesFromJson pipeline before/after environment provisioning to update module versions as needed.

(Source: Sprint testing folio-testing-sprint, Updated: 2024-12-13)

11. 🚨 CRITICAL: Existing Namespace Conflict During Environment Creation

  • Issue:

    • createNamespaceFromBranch pipeline fails with existing environment conflicts

    • Error occurs even if previous deployment attempts failed

    • Namespace remains in inconsistent state preventing new deployments

    • Build console shows resource conflicts or allocation issues

  • Symptoms:

    • Jenkins job fails during initial namespace setup phase

    • Console logs indicate existing resources blocking creation

    • Environment appears partially deployed or in error state

    • Subsequent creation attempts continue to fail

  • ⚠️ MANDATORY Resolution Workflow:

    1. Check Current Namespace Status: Navigate to Projects(Namespaces)

    2. Delete Existing Environment (REQUIRED): Use deleteNamespace β€” ⏱️ Wait for complete deletion before proceeding

    3. Verify Clean State: Confirm namespace no longer appears in Rancher UI

    4. Retry Environment Creation: Run createNamespaceFromBranch with original parameters

  • ⚑ Quick Alternative: Use recreateTeamNamespace which automatically handles delete+create sequence

(Source: Jenkins Console Analysis build #6280, Slack thread C08FXR6L6G5/1760449472.893459, Updated: 2025-10-07)

12. Jenkins Module Deployment Stuck at Helm Deploy State

  • Symptoms:

    • Jenkins deployModuleFromFeatureBranchEureka jobs hang at "helm deploy" step for 30+ minutes

    • Job shows "running" status but no actual progress in deployment steps

    • Module pods may show unfinished deployments or problematic states in Rancher

  • Root Causes:

    • Resource pressure or pod evictions in namespace

    • Jenkins agent/session issues during long waits

    • Heavy startup/migration or OOM on modules

    • Telepresence traffic-agents interfering with module deployment

    • Problematic modules like mod-service-interaction causing cascading issues

  • Resolution Steps:

    • Option 1: Re-run Job (Most Common Fix)

    • Option 2: Address Resource Issues β€” clean evicted pods, adjust CONFIG_TYPE

    • Option 3: Remove Telepresence Agents

    • Option 4: Environment Recreation (Last Resort) using recreateTeamNamespace pipeline

(Source: Slack thread C017RFAGBK2/1759250204.744059, RANCHER-1970, RANCHER-1996, RANCHER-1805, Updated: 2025-10-01)

13. mod-search Feature Branch Deployment Failures with SIMPLIFIED Workaround πŸ”

  • Issue: deployModuleFromFeatureBranchEureka fails for mod-search feature branches during Helm deployment phase.

  • βœ… PROVEN Resolution: Re-run the Failed Jenkins Job with SIMPLIFIED=true parameter β€” bypasses complex application descriptor flows.

  • When to Use SIMPLIFIED:

    • βœ… Feature branch deployments that fail with standard approach

    • βœ… DB schema changes or migrations in the module

    • ❌ Major capability changes that affect other modules

(Source: Slack thread C017RFAGBK2/1760536397.064279, Jenkins build #3877, Updated: 2025-12-16)

14. πŸ” SSO Keycloak Configuration: Alternative User Matching Methods

  • Issue: SSO configuration fails when IDP doesn't provide external_system_id attribute.

  • Solutions:

    • Option 1: Username-Based Matching β€” Use "Detect existing broker user" + uidβ†’username mapping

    • Option 2: Email-Based Matching β€” Map IDP EmailAddress attribute to Keycloak email field

    • Option 3: Custom Attribute Matching via Keycloak extension

(Source: Slack thread C07SL94PAPR/1762209775.358279, Stanford IDP success case, Updated: 2025-01-20)

15. Best Practices & Troubleshooting

  • Always monitor Rancher pod status post-deployment, acting quickly to address any evictions or failures.

  • For unresolved or critical issues, create a Rancher Jira and inform the Kitfox team in #folio-rancher-support Slack.

  • For routine problems (module update, entitlement, login, UI), use recommended Jenkins pipelines.

16. mod-serials-management schema reset workaround

  • Workaround:

    1. In Rancher, scale the mod-serials-management deployment to 0 replicas.

    2. Using pgAdmin, delete (CASCADE) the mod_serials_management__system database schema.

    3. Scale mod-serials-management back up to 1 replica.

17. mod-service-interaction 502 Bad Gateway during deployModuleFromFeatureBranchEureka (schema reset workaround)

  • Symptoms: Jenkins deployModuleFromFeatureBranchEureka fails with HTTP 502: "An invalid response was received from the upstream server".

  • Workaround:

    1. In Rancher, scale the mod-service-interaction deployment to 0 replicas.

    2. In pgAdmin, drop (CASCADE) the schema: mod_service_interaction__system.

    3. Scale mod-service-interaction back up to 1 replica.

    4. Wait until the pod is Running/Ready.

    5. Re-run the failed Jenkins job.

18. Deploying Locally Built Container Images in Eureka Environments

  • Issue: Need to deploy a locally built module container image to Rancher environment.

  • Solutions:

    • Option 1: Direct Container Image Swap β€” push to public registry, edit deployment in Rancher UI (works same as OKAPI environments)

    • Option 2: Use Jenkins Build Pipeline with Custom Branch β€” push changes to feature branch, use buildAndPushModule + deployModuleFromFeatureBranchEureka

19. ⏳ deleteNamespace Job Running Unusually Long Due to Large Number of AWS SSM Parameter Store Entries

  • Issue: The deleteNamespace Jenkins job takes significantly longer than the expected ~5–10 minutes (delete phase) β€” in some cases running for 1+ hours.

  • Root Cause: The number of AWS SSM Parameter Store entries associated with the namespace directly determines the deletion job duration. The more Karate (API automation) test runs executed against the environment, the more SSM entries are created. Environments with heavy Karate test usage can accumulate 1,000+ SSM parameter entries, causing the cleanup phase of deleteNamespace to run proportionally longer.

  • Symptoms:

    • deleteNamespace job running for 1+ hours (vs. expected ~5–10 min delete phase)

    • Job console output shows: "Deleting AWS SSM Param store entries..."

    • Progress indicator appears red/in-progress but is still actively working

    • Counter shows 1,000+ entries already deleted and still counting

  • ⚠️ Important: This is NOT a stuck job. The job is actively deleting SSM entries and will complete. Do NOT abort the job prematurely.

  • How to Verify:

    1. Open the Jenkins job console output: deleteNamespace console

    2. Look for the line "Deleting AWS SSM Param store entries..." with a running count

    3. If the count is actively increasing, the job is working normally β€” just processing a large volume of entries

  • Prevention / Best Practice:

    • The volume of SSM entries scales with the number of Karate test executions run against the environment

    • Teams running frequent/large-scale Karate test suites should expect longer deletion times when recreating environments

    • Allow sufficient time (potentially 1–2 hours) for deleteNamespace to complete on heavily used environments before starting createNamespaceFromBranch

  • Validated Case:

    • Environment: Vega (folio-edev cluster)

    • Observation: Job ran 1 hour 18 minutes vs. Spitfire team's ~8 minutes for same job

    • Cause: Large number of Karate test runs against the Vega environment resulted in 1,000+ SSM entries

    • Resolution: Waited for job completion β€” no manual intervention needed βœ…

(Source: Slack thread C017RFAGBK2/1777560895.503979, Updated: 2025-05-28)