PoC for hosted development environment for FOLIO teams (FOLIO-2601)

[FOLIO-2609] Enable EFK log aggregation stack on FOLIO cluster Created: 20/May/20  Updated: 22/Jul/20  Resolved: 22/Jul/20

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None
Parent: PoC for hosted development environment for FOLIO teams

Type: Task Priority: P2
Reporter: Jakub Skoczen Assignee: John Malconian
Resolution: Done Votes: 0
Labels: dev-environment
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Sprint: DevOps: sprint 92, DevOps: sprint 90
Development Team: FOLIO DevOps
Epic Link: PoC for hosted development environment for FOLIO teams

 Description   

Implement EFK log stack on FOLIO cluster. The log stack should be able to ship logs from any container to an Elasticsearch instance that can be accessible via a Kibana dashboard. Github OAUTH should be enabled to allow access to the dashboard from any member of github.com/folio-org. The entire stack should be run on the K8s cluster.



 Comments   
Comment by John Malconian [ 16/Jul/20 ]

I've created an EFK log aggregation stack at: https://logs.ci.folio.org. Authentication is via OAUTH2 and restricted to github.com/folio-org members. Navigate to 'Discovery' from the home page to get to the main logging dashboard.

Comment by Jakub Skoczen [ 17/Jul/20 ]

Back in progress to implement index retention.

Comment by John Malconian [ 22/Jul/20 ]

Increased disk capacity of Elasticsearch nodes in order to hold at least a few days worth of logs and configured the following ES Lifecycle Policy for filebeat (logs) indexes.

PUT _ilm/policy/filebeat
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_size": "50gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "actions": {
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "3d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

This may need further refinement, but the policy should rotate log indexes daily (or if the index reaches 50GB first). Rotated indexes will be deleted after three days. This should be ok for a dev environment. My best guess at this point is that we will generate ~50GB daily.

Generated at Thu Feb 08 23:21:56 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.