之前在Lab環境搞了一套ELK去收log, 但因為資源問題, 硬碟空間沒這麼大, 很容易被塞滿.

一開始是用一連串curl的方式去查詢跟刪除:

先利用curl的指令打server的ip + /_cat/indices?v&h=i"

# search
`curl -u elastic:changeme "http://<elasticsearch_host_ip>:<port>/_cat/indices?v&h=i"` > query.txt

接著用cat + grep 去過濾想刪除的資訊

# filter
cat query.txt | grep *.<date_or_some_keyeord> > filter.txt

最後寫個簡單的迴圈shell script去帶入filter.txt

#!/bin/bash
while read line
do
        curl -XDELETE -u <user>:<pass> "http://<ip>:<port>/$line"
done < filter.txt

後來直接寫了一個簡單的Python + Shell去處理整件事

#!/usr/bin/env python3

"""
Author : GordonWei
Date : 01/04/21
Comment : Get Expire Data (3 Days Ago)And Delete It Automation.
"""

import requests, subprocess
from requests.auth import HTTPBasicAuth
from datetime import datetime, timedelta, date

# Get Today Time
today = date.today() - timedelta(days=3)
today = today.strftime('%Y.%m.%d')

# Get Kibana Index Pattern.
getIndex = requests.get("http://kibanaURL:port/_cat/indices?v&h=i", auth = HTTPBasicAuth('elastic', 'ChangeMe'))
writeFile = open('./query.txt', 'w')
writeFile.writelines(getIndex.text)
writeFile.close()

# Filter Keyword 
subprocess.call('cat query.txt | grep ' + today + ' > filter.txt', shell = True)

# Read Filter File And Request With Delete Functoin To Clean Index Pattern
readFile = open('./filter.txt', 'r')
readLines = readFile.readlines()
for line in readLines:
  line = line.replace('\n','')
  requests.delete('http://kibanaURL:port/'+ line, auth = HTTPBasicAuth('elastic', 'ChangeMe'))

這份Python在弱弱的github上也有放一份, 有需要的客倌可以自行取用喔~

github 傳送門