[Python] 使用Python自動刪除Kibana Data
之前在Lab環境搞了一套ELK
去收log, 但因為資源問題, 硬碟空間沒這麼大, 很容易被塞滿.
一開始是用一連串curl
的方式去查詢跟刪除:
先利用curl
的指令打server的ip + /_cat/indices?v&h=i"
# search
`curl -u elastic:changeme "http://<elasticsearch_host_ip>:<port>/_cat/indices?v&h=i"` > query.txt
接著用cat
+ grep
去過濾想刪除的資訊
# filter
cat query.txt | grep *.<date_or_some_keyeord> > filter.txt
最後寫個簡單的迴圈shell script
去帶入filter.txt
#!/bin/bash
while read line
do
curl -XDELETE -u <user>:<pass> "http://<ip>:<port>/$line"
done < filter.txt
後來直接寫了一個簡單的Python + Shell
去處理整件事
#!/usr/bin/env python3
"""
Author : GordonWei
Date : 01/04/21
Comment : Get Expire Data (3 Days Ago)And Delete It Automation.
"""
import requests, subprocess
from requests.auth import HTTPBasicAuth
from datetime import datetime, timedelta, date
# Get Today Time
today = date.today() - timedelta(days=3)
today = today.strftime('%Y.%m.%d')
# Get Kibana Index Pattern.
getIndex = requests.get("http://kibanaURL:port/_cat/indices?v&h=i", auth = HTTPBasicAuth('elastic', 'ChangeMe'))
writeFile = open('./query.txt', 'w')
writeFile.writelines(getIndex.text)
writeFile.close()
# Filter Keyword
subprocess.call('cat query.txt | grep ' + today + ' > filter.txt', shell = True)
# Read Filter File And Request With Delete Functoin To Clean Index Pattern
readFile = open('./filter.txt', 'r')
readLines = readFile.readlines()
for line in readLines:
line = line.replace('\n','')
requests.delete('http://kibanaURL:port/'+ line, auth = HTTPBasicAuth('elastic', 'ChangeMe'))
這份Python在弱弱的github上也有放一份, 有需要的客倌可以自行取用喔~