https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/StartSplunk
yum -y install splunk-8.2.3-cd0848707637-linux-2.6-x86_64.rpm
splunk start
...
accept licence
dataset
import pandas as pd
import random
import datetime
row = []
start = datetime.datetime(2021,1,1)
for i in range(10000):
row.append({
"date": str(start + datetime.timedelta(hours=i)),
"letter": random.choice("abcdefghijklmno"),
"age": random.randint(10, 80),
"fruit": random.choice(["apple", "blueberry", "banana", "grape"]),
"v1": random.randint(10, 80),
"v2": random.randint(10, 800),
})
pd.DataFrame(row).to_csv("random-splunk-data.csv", index=False)
query
source="random-splunk-data (2).csv" host="splunk-lab" index="testdata2" sourcetype="csv" fruit="apple" | eval tsdate=strptime(date, "%Y-%m-%d %H:%M:%S") | where tsdate >= 1635724800.0 and tsdate <= 1638316800.0
https://docs.splunk.com/Documentation/Splunk/8.2.3/AdvancedDev/CustomAlertConvertScripted?ref=hk
alert -> scripts (bash, python)
is deprecated 
webhook 
https://docs.splunk.com/Documentation/Splunk/8.2.3/Alert/Webhooks?ref=hk
{
"result": {
"sourcetype" : "mongod",
"count" : "8"
},
"sid" : "scheduler_admin_search_W2_at_14232356_132",
"results_link" : "http://web.example.local:8000/app/search/@go?sid=scheduler_admin_search_W2_at_14232356_132",
"search_name" : null,
"owner" : "admin",
"app" : "search"
}
webhook payload log
{'app': 'search',
'owner': 'admin',
'result': {'_bkt': 'testdata2~1~CFA091AE-A09D-4BD7-908F-28F1A6424D90',
'_cd': '1:20565',
'_eventtype_color': '',
'_indextime': '1637205169',
'_raw': '22,2021-11-01 03:00:00,apple,d,75,359',
'_serial': '104',
'_si': ['splunk-lab', 'testdata2'],
'_sourcetype': 'csv',
'_time': '1635746400',
'age': '22',
'date': '2021-11-01 03:00:00',
'date_hour': '3',
'date_mday': '1',
'date_minute': '0',
'date_month': 'november',
'date_second': '0',
'date_wday': 'monday',
'date_year': '2021',
'date_zone': 'local',
'eventtype': '',
'fruit': 'apple',
'host': 'splunk-lab',
'index': 'testdata2',
'letter': 'd',
'linecount': '1',
'punct': ',--_::,,,,',
'source': 'random-splunk-data (2).csv',
'sourcetype': 'csv',
'splunk_server': 'splunk-lab',
'splunk_server_group': '',
'timeendpos': '22',
'timestartpos': '3',
'tsdate': '1635746400.000000',
'v1': '75',
'v2': '359'},
'results_link': 'http://splunk-lab:8000/app/search/search?q=%7Cloadjob%20scheduler__admin__search__RMD5354c629e1e04adb1_at_1637207280_30%20%7C%20head%20105%20%7C%20tail%201&earliest=0&latest=now',
'search_name': 'test alert',
'sid': 'scheduler__admin__search__RMD5354c629e1e04adb1_at_1637207280_30'}
{'app': 'search',
'owner': 'admin',
'result': {'_bkt': 'testdata2~1~CFA091AE-A09D-4BD7-908F-28F1A6424D90',
'_cd': '1:20549',
'_eventtype_color': '',
'_indextime': '1637205169',
'_raw': '76,2021-10-31 23:00:00,apple,a,55,769',
'_serial': '105',
'_si': ['splunk-lab', 'testdata2'],
'_sourcetype': 'csv',
'_time': '1635732000',
'age': '76',
'date': '2021-10-31 23:00:00',
'date_hour': '23',
'date_mday': '31',
'date_minute': '0',
'date_month': 'october',
'date_second': '0',
'date_wday': 'sunday',
'date_year': '2021',
'date_zone': 'local',
'eventtype': '',
'fruit': 'apple',
'host': 'splunk-lab',
'index': 'testdata2',
'letter': 'a',
'linecount': '1',
'punct': ',--_::,,,,',
'source': 'random-splunk-data (2).csv',
'sourcetype': 'csv',
'splunk_server': 'splunk-lab',
'splunk_server_group': '',
'timeendpos': '22',
'timestartpos': '3',
'tsdate': '1635732000.000000',
'v1': '55',
'v2': '769'},
'results_link': 'http://splunk-lab:8000/app/search/search?q=%7Cloadjob%20scheduler__admin__search__RMD5354c629e1e04adb1_at_1637207280_30%20%7C%20head%20106%20%7C%20tail%201&earliest=0&latest=now',
'search_name': 'test alert',
'sid': 'scheduler__admin__search__RMD5354c629e1e04adb1_at_1637207280_30'}
Retrieving data from Splunk Dashboard Panels via API
https://avleonov.com/2019/02/07/retrieving-data-from-splunk-dashboard-panels-via-api/
List all alerts configured
https://community.splunk.com/t5/Alerting/How-can-i-query-to-get-all-alerts-which-are-configured/m-p/288845
all apps
|rest/servicesNS/-/-/saved/searches | search alert.track=1 | fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule
search app only
|rest/servicesNS/-/search/saved/searches | search alert.track=1 | fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule
Count Group by category status
index="iulb-test"
| eventstats count as viptotal by vipname
| eventstats count(eval(state=="active")) as active by vipname
| eval perc=round(active*100/viptotal,2)
| sort perc desc | stats values(viptotal), values(perc) by vipname
index="iulb-test"
| eventstats count as viptotal by vipname
| eventstats count(eval(state=="active")) as active by vipname
| eval perc=round(active*100/viptotal,2)
| stats values(perc) as perc by vipname
| sort 0 - perc
Caveats
nested json embrace in single quotes
https://community.splunk.com/t5/Splunk-Search/How-to-use-JSON-subfields-with-the-eval-command/m-p/222635