Netwitness Integrations 101
· 33 min read
Vassal @ 特贸 Cyber
Collection of tips on how to create NetWitness integrations, mainly using 3rd-party APIs not currently (at the time of publishing, circa 2022) supported by the official release.
Introduction
This post compiles knowledge from the old times, when Netwitness still dominating the SIEM/Threat Intel arena and flowrished prior to the massive cloud-scale era.
I did some work on the platform and I liked the architectural choices and its operations. I did some customizations as needed, and also built some integrations, mainly parsing for unsupported appliances from diverse vendors.
Here I try my best to distill this information I've found useful and important at that time. Chances are nothing of it still works as-is, so do your due dilligence if you are touching the platform.
Log Integration in RSA NetWitness
Sometimes, log integrations is a pain in the arse. Sometimes, you need to use any silvertape to glue the missing pieces together during your analysis due to the lack of a built-in native interface in the platform for easily integration.
The non-existent example of a simple REST plugin to pull data from APIs across different vendors if they are not readly supported baffles the mind of the unsuspecting.
This guide is intendend not only to provide some already integrations previously made by me, but to guide you in the path of log parser construction, by point out the fastest routes possible to get it done in a timely manner - because sometimes Netwitness Community is a very deep jungle with resources scattered here and there.
How to Log Collect in NW
Netwitness provides distinct ways of log collection that could be leveraged depending on your type of implementation.
As of version 11.5 and later, the allowed collection protocols still in use, are:
- Check Point (OPSEC LEA)
- File
- Netflow
- ODBC
- Plugins (AWS Cloudtrail & Azure)
- SDEE
- SNMP Trap
- Syslog
- VMWare
- Windows
- Logstash
- Windows Legacy
In many cases, your log source is sending to you using a combination of Syslog and CEF definitions e.g., from which RSA NW has standard native support - unless your vendor don't deviates from standardized CEF notation.
Even if that is true, you would end up needing a combination of Syslog and a custom way to parse your results received from Syslog, that arrives in a format not already supported or for which NW does not have any Live support. In such cases, the way to solve this headache is to make use of tools available by RSA itself. They are composed of the Log Parser Tool and in the versions 11.5 and later, JSON Mappings, thich allow the parse of JSON messages arriving in the platform using a friendly UI.
In the following lines you'll know how to handle a custom implementation through the use of APIs provided by many vendors.
Creating custom log collection and parsing
Your work may involve integration of some network devices, firewalls and security SaaS tools.
I chose Python3 for quick deployment and easy maintenance. Python practitioners will find the code a little "disgraceful" but the aim is to avoid friction among team members whose experience with Python is little or nonexistent.
This type of integration is very simple:
-
Develop your API integration using any type of programming language of your choice.
-
Generate a log file, where each line represents an event collected via the API, and save the file in the path below, using Custom FileSpec :
/var/netwitness/logcollector/upload/<source_name> -
Once events are being received in the platform, proceed to generate the parser file.
-
After the parser is generated, extracted and installed, adjust and add custom meta keys as needed either by using this tool for discovery, or these steps.
a. Parsers can be restarted without reinitializing the entire application, see here for details.
b. After adjusting your index-concentrator-custom.xml and table-map-custom.xml a restart is needed in each service.
That's it. I've tried to be as brief as possible. You can find a lot of resources below in the refs. In the next topic I'll provide examples of parsers I've already made, feel free to use it as you want, hope it guides you in your parser journey. Peace be upon the traveler!
Example Parsers
Tenable.io
Example parser for Container Security using Python and a credential for authentication with the API.
/opt/parsers/tenableio/creds/access_creds.json
{"access":"access-key here","secret":"secret-key here"}
/opt/parsers/tenableio/container-security.py
import pprint, json, datetime, time, os.path
from multiprocessing.context import assert_spawning
from tenable.io import TenableIO
# JSON PrettyPrinter Identation properties
pp = pprint.PrettyPrinter(indent=4)
# Credentials load and definition
with open("/opt/parsers/tenableio/creds/access_creds.json", "r") as read_file:
data_json = json.load(read_file)
access_key = data_json["access"]
secret_key = data_json["secret"]
tio = TenableIO(access_key, secret_key)
# Path to send logs to
path_dir = "/var/netwitness/logcollector/upload/tenableio/tenableio/"
filepath = os.path.join(path_dir, "container-security.log")
# |-------------------------------------|
# |------------- Tenable.IO ------------|
# |------ Container Security Logs ------|
# |-------------------------------------|
# Check for problems
def has_problems(x, y):
if x > 0 or y > 0:
return 1
else:
return 0
# Main function
def doit():
# Define `Now`
now_time = datetime.datetime.now().replace(microsecond=0).isoformat()
# Grab current image lists
images_list = tio.cs.images.list(return_json=True)
# Create "events" array of events
events = []
# If there are any images, proceed to next step...
if len(images_list["items"]) > 0:
for item in images_list["items"]:
# Print all images found - for Fast Debug only [Turn ON/OFF]
#
# print("repo: " + item["repo_name"] + "name: " + item["name"] + " tag: " + item["tag"] + " number_of_malware: " + str(item.number_of_malware) + " number_of_vulns: " + str(item.number_of_vulns))
# Check to see if any malware or vulnerabilities were found during the scan, if so, keep going...
eval_problems = has_problems(item.number_of_malware, item.number_of_vulns)
if eval_problems == 1:
# Predefine keys to use in "Reports" and "Image Details"
image_keys = {"repo_name": "", "name": "", "tag": ""}
image_keys["repo_name"] = item["repo_name"]
image_keys["name"] = item["name"]
image_keys["tag"] = item["tag"]
# Get image details
details = tio.cs.images.details(image_keys["repo_name"], image_keys["name"], image_keys["tag"])
# Image details information grab
digest = details["digest"]
last_scanned = details["last_scanned"]
uploaded_at = details["uploaded_at"]
risk_score = details.risk_score
status = details["status"]
num_vulns = details.number_of_vulns
num_malware = details.number_of_malware
# Generate reports for the images that contains vulns or malwares
reports = tio.cs.reports.report(image_keys["repo_name"], image_keys["name"], image_keys["tag"])
# Define events counter and erase counter from previous "item"
count_probs = 0
# Clear previous message
message = ""
for report in reports["findings"]:
count_probs += 1
# Define log fields
access_complexity = report["nvd_finding"]["access_complexity"]
access_vector = report["nvd_finding"]["access_vector"]
auth = report["nvd_finding"]["auth"]
availability_impact = report["nvd_finding"]["availability_impact"]
cfd_impact = report["nvd_finding"]["confidentiality_impact"]
cve = report["nvd_finding"]["cve"]
cvss_score = report["nvd_finding"]["cvss_score"]
cvss_vector = report["nvd_finding"]["cvss_vector"]
integrity_impact = report["nvd_finding"]["integrity_impact"]
modified_date = report["nvd_finding"]["modified_date"]
published_date = report["nvd_finding"]["published_date"]
snyk_id = report["nvd_finding"]["snyk_id"]
pkg_name = report["packages"][0]["name"]
pkg_ver = report["packages"][0]["version"]
pkg_type = report["packages"][0]["type"]
# Create log message and append to "events" bucket
message = str(now_time) + " customername.cloud.tenable.com, application=\"container-security\"" + ", " + "image_name=\"" + image_keys["name"] + "\", repo_name=\"" + image_keys["repo_name"] + "\", tag=\"" + image_keys["tag"] + "\", digest=\"" + digest + "\", risk_score=\"" + str(risk_score) + "\", status=\"" + status + "\", last_scanned=\"" + last_scanned + "\", uploaded_at=\"" + uploaded_at + "\", number_of_malware=\"" + str(num_malware) + "\", number_of_vulns=\"" + str(num_vulns) + "\", actual_vuln=\"" + str(count_probs) + "\", actual_malware=\"" + str(num_malware) + "\", access_complexity=\"" + access_complexity + "\", access_vector=\"" + access_vector + "\", auth=\"" + auth + "\", availability_impact=\"" + availability_impact + "\", cfd_impact=\"" + cfd_impact + "\", cve=\"" + cve + "\", cvss_score=\"" + cvss_score + "\", cvss_vector=\"" + cvss_vector + "\", integrity_impact=\"" + integrity_impact + "\", modified_date=\"" + modified_date + "\", published_date=\"" + published_date + "\", snyk_id=\"" + snyk_id + "\", pkg_name=\"" + pkg_name + "\", pkg_ver=\"" + pkg_ver + "\", pkg_type=\"" + pkg_type + "\""
events.insert(-1, message)
# write events to the log file
for i in range(len(events)):
with open(filepath, "a+") as file_log:
file_log.seek(0)
data = file_log.read(100)
if len(data) > 0 :
file_log.write("\n")
file_log.write(events[i])
print(len(events))
if __name__ == "__main__":
doit()
TrendMicro
Example parser for Cloud App using Python and a credential token for authentication with the API and the eventsource type specification.
/etc/netwitness/ng/logcollection/content/collection/file/trendcloudapp.xml
<?xml version="1.0" encoding="UTF-8"?>
<typespec>
<name>trendcloudapp</name>
<type>file</type>
<prettyName>Trend Micro Cloud App Security</prettyName>
<version>1.0</version>
<author>administrator</author>
<description>FileCollection specification for eventsource type "Trend Micro Cloud App Security"</description>
<device>
<name>trendcloudapp</name>
</device>
<configuration>
</configuration>
<collection>
<file>
<parserId>trendcloudapp</parserId>
<processorType>generic</processorType>
<dataStartLine>1</dataStartLine>
<fieldDelim>,</fieldDelim>
<idField></idField>
<lineDelim>\n</lineDelim>
<transformPrefixTag>trendcloudapp</transformPrefixTag>
<transformReplaceFieldDelim>0</transformReplaceFieldDelim>
<transformPrefixFilename>0</transformPrefixFilename>
<transformMultipleDelimiterAsOne>0</transformMultipleDelimiterAsOne>
<transformReplacementFieldDelim></transformReplacementFieldDelim>
</file>
</collection>
</typespec>
/opt/tmcas/dlp-event-parser.py
###########################################################
#### ####
#### Trend Micro Cloud App - DLP events ####
#### ####
#### ver.1.0 ####
###########################################################
import requests, json, datetime, time, os.path, re
import dateutil.parser as dp
from datetime import timedelta
# path for testing
path_dir = "/var/netwitness/logcollector/upload/trendcloudapp/trendcloudapp/"
filepath = os.path.join(path_dir, "dlp-events.log")
# define services to be processed
services = ["exchange","teams","sharepoint","onedrive"]
def doit():
# open token file:
with open("token.json", "r") as read_file:
data_json = json.load(read_file)
token = data_json["access"]
# create 'next page link' signaler
nxtlink = 0
url_maker_nxtlink = []
# format time interval and define the 'start' and 'end' ranges
now_log_time = datetime.datetime.now().replace(microsecond=0).isoformat(' ')
now_time = datetime.datetime.now().replace(microsecond=0).isoformat()
previous_time = datetime.datetime.now() - timedelta(hours=1, minutes=0)
since = previous_time.strftime("%Y-%m-%dT%H:%M:%S.000%z") + 'Z'
until = now_time + '.000Z'
# Start request message parameters
headers = {
'Authorization': 'Bearer ' + token,
'cache-control': 'no-cache',
}
params = (
('start', since),
('end', until),
('limit', '200'),
)
for sname in services:
# create event bucket and message var
events = []
message = ""
# create the iterable URL and pass it to the API request
url_maker = "https://api.tmcas.trendmicro.com/siem/v1/security_events?service=" + sname + "&event=dlp"
print(url_maker)
response = requests.get(url_maker, headers=headers, params=params)
# work the deserialized response
current_incidents = json.loads(response.text)
# iterate in the response and format the log line message
if (len(current_incidents["security_events"]) > 0):
if (current_incidents["next_link"] != ''):
nxtlink = 1
url_maker_nxtlink.insert(-1, current_incidents["next_link"])
print(len(current_incidents["security_events"]))
for index in range(len(current_incidents["security_events"])):
for key in current_incidents["security_events"][index]:
if key == "event":
event_name = current_incidents["security_events"][index][key]
elif key == "message":
detection_time = current_incidents["security_events"][index][key]["detection_time"]
file_name = current_incidents["security_events"][index][key]["file_name"]
location = current_incidents["security_events"][index][key]["location"]
triggered_policy_name = current_incidents["security_events"][index][key]["triggered_policy_name"]
triggered_security_filter = current_incidents["security_events"][index][key]["triggered_security_filter"]
action = current_incidents["security_events"][index][key]["action"]
action_result = current_incidents["security_events"][index][key]["action_result"]
violator = current_incidents["security_events"][index][key]["violator"]
triggered_dlp_template = current_incidents["security_events"][index][key]["triggered_dlp_template"]
violating_content = current_incidents["security_events"][index][key]["violating_content"]
mail_message_id = current_incidents["security_events"][index][key]["mail_message_id"]
mail_message_sender = current_incidents["security_events"][index][key]["mail_message_sender"]
mail_message_recipient = current_incidents["security_events"][index][key]["mail_message_recipient"]
mail_message_submit_time = current_incidents["security_events"][index][key]["mail_message_submit_time"]
mail_message_delivery_time = current_incidents["security_events"][index][key]["mail_message_delivery_time"]
mail_message_subject = current_incidents["security_events"][index][key]["mail_message_subject"]
# log message creation
# convert ISO8601 time format to unix timestamp
dp_detectetime = dp.parse(detection_time)
ut_detectedtime = int(time.mktime(dp_detectetime.timetuple()))
ut_detectedtime = str(ut_detectedtime)
dp_mail_submittime = dp.parse(mail_message_submit_time)
ut_mail_submittime = int(time.mktime(dp_mail_submittime.timetuple()))
ut_mail_submittime = str(ut_mail_submittime)
dp_mail_deliverytime = dp.parse(mail_message_delivery_time)
ut_mail_deliverytime = int(time.mktime(dp_mail_deliverytime.timetuple()))
ut_mail_deliverytime = str(ut_mail_deliverytime)
# extract mail_message_recipient's emails
if len(mail_message_recipient) > 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
str_mail_message_recipient = str_mail_message_recipient.replace('\"', '')
recipients_mail = re.findall(r'[a-z0-9.\-+_]+@[a-z0-9.\-+_]+\.[a-z]+', str_mail_message_recipient)
str_mail_message_recipient = ' '.join([str(item) for item in recipients_mail])
elif len(mail_message_recipient) == 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
# extract mail_message_sender's emails
if len(mail_message_sender) > 1:
str_mail_message_sender = ' '.join([str(item) for item in mail_message_sender])
str_mail_message_sender = str_mail_message_sender.replace('\"', '')
senders_mail = re.findall(r'[a-z0-9.\-+_]+@[a-z0-9.\-+_]+\.[a-z]+', str_mail_message_sender)
str_mail_message_sender = ' '.join([str(item) for item in senders_mail])
elif len(mail_message_sender) == 1 | len(mail_message_sender) == 0 :
str_mail_message_sender = ' '.join([str(item) for item in mail_message_sender])
# extract triggered_dlp_template's message
if len(triggered_dlp_template) > 1:
str_triggered_dlp_template = ' '.join([str(item) for item in triggered_dlp_template])
str_triggered_dlp_template = str_triggered_dlp_template.replace('\"', '')
elif len(triggered_dlp_template) == 1 | len(triggered_dlp_template) == 0 :
str_triggered_dlp_template = ' '.join([str(item) for item in triggered_dlp_template])
str_triggered_dlp_template = str_triggered_dlp_template.replace('\"', '')
# clean up dangerous characters
str_mail_message_recipient = str_mail_message_recipient.replace(' ', ';')
str_triggered_dlp_template = triggered_dlp_template.replace(',', ';')
str_violating_content = violating_content.replace(',', ';')
str_file_name = file_name.replace(',', ';')
str_mail_message_subject = mail_message_subject.replace(',', ';')
str_mail_message_sender = mail_message_sender.replace('\"', '')
# main message creation
message = str(now_log_time) + " tmcas.customername.com, application=\"tmcas-" + sname + "-"+ event_name + "\", detectedutc=\"" + ut_detectedtime + "\", file_name=\"" + str_file_name + "\", violator=\"" + violator + "\", location=\"" + location + "\", triggered_policy_name=\"" + triggered_policy_name + "\", triggered_security_filter=\"" + triggered_security_filter + "\", action=\"" + action + "\", action_result=\"" + action_result + "\", mail_message_id=\"" + mail_message_id + "\", mail_message_sender=\"" + str_mail_message_sender + "\", mail_message_recipient=\"" + str_mail_message_recipient + "\", mail_message_submit_time=\"" + ut_mail_submittime + "\", mail_message_delivery_time=\"" + ut_mail_deliverytime + "\", mail_message_subject=\"" + str_mail_message_subject + "\", triggered_dlp_template=\"" + str_triggered_dlp_template + "\", violating_content=\"" + str_violating_content + "\""
events.insert(-1, message)
# write events to the log file
for i in range(len(events)):
with open(filepath, "a+") as file_log:
file_log.seek(0)
data = file_log.read(100)
if len(data) > 0 :
file_log.write("\n")
file_log.write(events[i])
print(len(events))
# check if there are a new page of events for being processed, then do it!
if (nxtlink == 1):
for url in url_maker_nxtlink:
# restate event bucket and message var
events = []
message = ""
print(url)
print(len(url_maker_nxtlink))
response = requests.get(url, headers=headers, params=params)
# work the deserialized response
current_incidents = json.loads(response.text)
# iterate in the response and format the log line message
if (len(current_incidents["security_events"]) > 0):
if (current_incidents["next_link"] != ''):
url_maker_nxtlink.insert(-1, current_incidents["next_link"])
print(len(current_incidents["security_events"]))
for index in range(len(current_incidents["security_events"])):
for key in current_incidents["security_events"][index]:
if key == "event":
event_name = current_incidents["security_events"][index][key]
elif key == "message":
detection_time = current_incidents["security_events"][index][key]["detection_time"]
file_name = current_incidents["security_events"][index][key]["file_name"]
location = current_incidents["security_events"][index][key]["location"]
triggered_policy_name = current_incidents["security_events"][index][key]["triggered_policy_name"]
triggered_security_filter = current_incidents["security_events"][index][key]["triggered_security_filter"]
action = current_incidents["security_events"][index][key]["action"]
action_result = current_incidents["security_events"][index][key]["action_result"]
violator = current_incidents["security_events"][index][key]["violator"]
triggered_dlp_template = current_incidents["security_events"][index][key]["triggered_dlp_template"]
violating_content = current_incidents["security_events"][index][key]["violating_content"]
mail_message_id = current_incidents["security_events"][index][key]["mail_message_id"]
mail_message_sender = current_incidents["security_events"][index][key]["mail_message_sender"]
mail_message_recipient = current_incidents["security_events"][index][key]["mail_message_recipient"]
mail_message_submit_time = current_incidents["security_events"][index][key]["mail_message_submit_time"]
mail_message_delivery_time = current_incidents["security_events"][index][key]["mail_message_delivery_time"]
mail_message_subject = current_incidents["security_events"][index][key]["mail_message_subject"]
# log message creation
# convert ISO8601 time format to unix timestamp
dp_detectetime = dp.parse(detection_time)
ut_detectedtime = int(time.mktime(dp_detectetime.timetuple()))
ut_detectedtime = str(ut_detectedtime)
dp_mail_submittime = dp.parse(mail_message_submit_time)
ut_mail_submittime = int(time.mktime(dp_mail_submittime.timetuple()))
ut_mail_submittime = str(ut_mail_submittime)
dp_mail_deliverytime = dp.parse(mail_message_delivery_time)
ut_mail_deliverytime = int(time.mktime(dp_mail_deliverytime.timetuple()))
ut_mail_deliverytime = str(ut_mail_deliverytime)
# extract mail_message_recipient's emails
if len(mail_message_recipient) > 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
str_mail_message_recipient = str_mail_message_recipient.replace('\"', '')
recipients_mail = re.findall(r'[a-z0-9.\-+_]+@[a-z0-9.\-+_]+\.[a-z]+', str_mail_message_recipient)
str_mail_message_recipient = ' '.join([str(item) for item in recipients_mail])
elif len(mail_message_recipient) == 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
# extract mail_message_sender's emails
if len(mail_message_sender) > 1:
str_mail_message_sender = ' '.join([str(item) for item in mail_message_sender])
str_mail_message_sender = str_mail_message_sender.replace('\"', '')
senders_mail = re.findall(r'[a-z0-9.\-+_]+@[a-z0-9.\-+_]+\.[a-z]+', str_mail_message_sender)
str_mail_message_sender = ' '.join([str(item) for item in senders_mail])
elif len(mail_message_sender) == 1 | len(mail_message_sender) == 0 :
str_mail_message_sender = ' '.join([str(item) for item in mail_message_sender])
# extract triggered_dlp_template's message
if len(triggered_dlp_template) > 1:
str_triggered_dlp_template = ' '.join([str(item) for item in triggered_dlp_template])
str_triggered_dlp_template = str_triggered_dlp_template.replace('\"', '')
elif len(triggered_dlp_template) == 1 | len(triggered_dlp_template) == 0 :
str_triggered_dlp_template = ' '.join([str(item) for item in triggered_dlp_template])
str_triggered_dlp_template = str_triggered_dlp_template.replace('\"', '')
# clean up dangerous characters
str_mail_message_recipient = str_mail_message_recipient.replace(' ', ';')
str_triggered_dlp_template = triggered_dlp_template.replace(',', ';')
str_violating_content = violating_content.replace(',', ';')
str_file_name = file_name.replace(',', ';')
str_mail_message_subject = mail_message_subject.replace(',', ';')
str_mail_message_sender = mail_message_sender.replace('\"', '')
# main message creation
message = str(now_log_time) + " tmcas.customername.com, application=\"tmcas-" + sname + "-"+ event_name + "\", detectedutc=\"" + ut_detectedtime + "\", file_name=\"" + str_file_name + "\", violator=\"" + violator + "\", location=\"" + location + "\", triggered_policy_name=\"" + triggered_policy_name + "\", triggered_security_filter=\"" + triggered_security_filter + "\", action=\"" + action + "\", action_result=\"" + action_result + "\", mail_message_id=\"" + mail_message_id + "\", mail_message_sender=\"" + str_mail_message_sender + "\", mail_message_recipient=\"" + str_mail_message_recipient + "\", mail_message_submit_time=\"" + ut_mail_submittime + "\", mail_message_delivery_time=\"" + ut_mail_deliverytime + "\", mail_message_subject=\"" + str_mail_message_subject + "\", triggered_dlp_template=\"" + str_triggered_dlp_template + "\", violating_content=\"" + str_violating_content + "\""
events.insert(-1, message)
# write events to the log file
for i in range(len(events)):
with open(filepath, "a+") as file_log:
file_log.seek(0)
data = file_log.read(100)
if len(data) > 0 :
file_log.write("\n")
file_log.write(events[i])
print(len(events))
if __name__ == "__main__":
while True:
doit()
time.sleep(300)
/opt/tmcas/ransomware-event-parser.py
###########################################################
#### ####
#### Trend Micro Cloud App - Ransomware events ####
#### ####
#### ver.1.0 ####
###########################################################
import requests, json, datetime, time, os.path, re
import dateutil.parser as dp
from datetime import timedelta
# path for testing
path_dir = "/var/netwitness/logcollector/upload/trendcloudapp/trendcloudapp/"
filepath = os.path.join(path_dir, "ransomware-events.log")
# define services to be processed
services = ["exchange","teams","sharepoint","onedrive"]
def doit():
# open token file:
with open("token.json", "r") as read_file:
data_json = json.load(read_file)
token = data_json["access"]
# create 'next page link' signaler
nxtlink = 0
url_maker_nxtlink = []
# format time interval and define the 'start' and 'end' ranges
now_log_time = datetime.datetime.now().replace(microsecond=0).isoformat(' ')
now_time = datetime.datetime.now().replace(microsecond=0).isoformat()
previous_time = datetime.datetime.now() - timedelta(hours=1, minutes=0)
since = previous_time.strftime("%Y-%m-%dT%H:%M:%S.000%z") + 'Z'
until = now_time + '.000Z'
# Start request message parameters
headers = {
'Authorization': 'Bearer ' + token,
'cache-control': 'no-cache',
}
params = (
('start', since),
('end', until),
('limit', '200'),
)
for sname in services:
# create event bucket and message var
events = []
message = ""
# create the iterable URL and pass it to the API request
url_maker = "https://api.tmcas.trendmicro.com/siem/v1/security_events?service=" + sname + "&event=ransomware"
print(url_maker)
response = requests.get(url_maker, headers=headers, params=params)
# work the deserialized response
current_incidents = json.loads(response.text)
# iterate in the response and format the log line message
if (len(current_incidents["security_events"]) > 0):
if (current_incidents["next_link"] != ''):
nxtlink = 1
url_maker_nxtlink.insert(-1, current_incidents["next_link"])
print(len(current_incidents["security_events"]))
for index in range(len(current_incidents["security_events"])):
for key in current_incidents["security_events"][index]:
if key == "event":
event_name = current_incidents["security_events"][index][key]
elif key == "message":
detection_time = current_incidents["security_events"][index][key]["detection_time"]
mail_message_recipient = current_incidents["security_events"][index][key]["mail_message_recipient"]
scan_type = current_incidents["security_events"][index][key]["scan_type"]
affected_user = current_incidents["security_events"][index][key]["affected_user"]
location = current_incidents["security_events"][index][key]["location"]
triggered_policy_name = current_incidents["security_events"][index][key]["triggered_policy_name"]
triggered_security_filter = current_incidents["security_events"][index][key]["triggered_security_filter"]
action = current_incidents["security_events"][index][key]["action"]
action_result = current_incidents["security_events"][index][key]["action_result"]
mail_message_id = current_incidents["security_events"][index][key]["mail_message_id"]
mail_message_sender = current_incidents["security_events"][index][key]["mail_message_sender"]
mail_message_submit_time = current_incidents["security_events"][index][key]["mail_message_submit_time"]
mail_message_delivery_time = current_incidents["security_events"][index][key]["mail_message_delivery_time"]
mail_message_subject = current_incidents["security_events"][index][key]["mail_message_subject"]
mail_message_file_name = current_incidents["security_events"][index][key]["mail_message_file_name"]
ransomware_name = current_incidents["security_events"][index][key]["ransomware_name"]
# log message creation
# convert ISO8601 time format to unix timestamp
dp_detectetime = dp.parse(detection_time)
ut_detectedtime = int(time.mktime(dp_detectetime.timetuple()))
ut_detectedtime = str(ut_detectedtime)
dp_mail_submittime = dp.parse(mail_message_submit_time)
ut_mail_submittime = int(time.mktime(dp_mail_submittime.timetuple()))
ut_mail_submittime = str(ut_mail_submittime)
dp_mail_deliverytime = dp.parse(mail_message_delivery_time)
ut_mail_deliverytime = int(time.mktime(dp_mail_deliverytime.timetuple()))
ut_mail_deliverytime = str(ut_mail_deliverytime)
# extract mail_message_recipient's emails
if len(mail_message_recipient) > 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
str_mail_message_recipient = str_mail_message_recipient.replace('\"', '')
recipients_mail = re.findall(r'[a-z0-9.\-+_]+@[a-z0-9.\-+_]+\.[a-z]+', str_mail_message_recipient)
str_mail_message_recipient = ' '.join([str(item) for item in recipients_mail])
elif len(mail_message_recipient) == 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
# clean up dangerous characters
str_mail_message_recipient = str_mail_message_recipient.replace(' ', ';')
str_mail_message_file_name = mail_message_file_name.replace(',', ';')
str_mail_message_subject = mail_message_subject.replace(',', ';')
str_mail_message_sender = mail_message_sender.replace('\"', '')
# main message creation
message = str(now_log_time) + " tmcas.customername.com, application=\"tmcas-" + sname + "-"+ event_name + "\", detectedutc=\"" + ut_detectedtime + "\", scan_type=\"" + scan_type + "\", affected_user=\"" + affected_user + "\", location=\"" + location + "\", triggered_policy_name=\"" + triggered_policy_name + "\", triggered_security_filter=\"" + triggered_security_filter + "\", action=\"" + action + "\", action_result=\"" + action_result + "\", mail_message_id=\"" + mail_message_id + "\", mail_message_sender=\"" + str_mail_message_sender + "\", mail_message_recipient=\"" + str_mail_message_recipient + "\", mail_message_submit_time=\"" + ut_mail_submittime + "\", mail_message_delivery_time=\"" + ut_mail_deliverytime + "\", mail_message_subject=\"" + str_mail_message_subject + "\", mail_message_file_name=\"" + str_mail_message_file_name + "\", ransomware_name=\"" + ransomware_name + "\""
events.insert(-1, message)
# write events to the log file
for i in range(len(events)):
with open(filepath, "a+") as file_log:
file_log.seek(0)
data = file_log.read(100)
if len(data) > 0 :
file_log.write("\n")
file_log.write(events[i])
print(len(events))
# check if there are a new page of events for being processed, then do it!
if (nxtlink == 1):
for url in url_maker_nxtlink:
# restate event bucket and message var
events = []
message = ""
print(url)
print(len(url_maker_nxtlink))
response = requests.get(url, headers=headers, params=params)
# work the deserialized response
current_incidents = json.loads(response.text)
# iterate in the response and format the log line message
if (len(current_incidents["security_events"]) > 0):
if (current_incidents["next_link"] != ''):
url_maker_nxtlink.insert(-1, current_incidents["next_link"])
print(len(current_incidents["security_events"]))
for index in range(len(current_incidents["security_events"])):
for key in current_incidents["security_events"][index]:
if key == "event":
event_name = current_incidents["security_events"][index][key]
elif key == "message":
detection_time = current_incidents["security_events"][index][key]["detection_time"]
mail_message_recipient = current_incidents["security_events"][index][key]["mail_message_recipient"]
scan_type = current_incidents["security_events"][index][key]["scan_type"]
affected_user = current_incidents["security_events"][index][key]["affected_user"]
location = current_incidents["security_events"][index][key]["location"]
triggered_policy_name = current_incidents["security_events"][index][key]["triggered_policy_name"]
triggered_security_filter = current_incidents["security_events"][index][key]["triggered_security_filter"]
action = current_incidents["security_events"][index][key]["action"]
action_result = current_incidents["security_events"][index][key]["action_result"]
mail_message_id = current_incidents["security_events"][index][key]["mail_message_id"]
mail_message_sender = current_incidents["security_events"][index][key]["mail_message_sender"]
mail_message_submit_time = current_incidents["security_events"][index][key]["mail_message_submit_time"]
mail_message_delivery_time = current_incidents["security_events"][index][key]["mail_message_delivery_time"]
mail_message_subject = current_incidents["security_events"][index][key]["mail_message_subject"]
mail_message_file_name = current_incidents["security_events"][index][key]["mail_message_file_name"]
ransomware_name = current_incidents["security_events"][index][key]["ransomware_name"]
# log message creation
# convert ISO8601 time format to unix timestamp
dp_detectetime = dp.parse(detection_time)
ut_detectedtime = int(time.mktime(dp_detectetime.timetuple()))
ut_detectedtime = str(ut_detectedtime)
dp_mail_submittime = dp.parse(mail_message_submit_time)
ut_mail_submittime = int(time.mktime(dp_mail_submittime.timetuple()))
ut_mail_submittime = str(ut_mail_submittime)
dp_mail_deliverytime = dp.parse(mail_message_delivery_time)
ut_mail_deliverytime = int(time.mktime(dp_mail_deliverytime.timetuple()))
ut_mail_deliverytime = str(ut_mail_deliverytime)
# extract mail_message_recipient's emails
if len(mail_message_recipient) > 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
str_mail_message_recipient = str_mail_message_recipient.replace('\"', '')
recipients_mail = re.findall(r'[a-z0-9.\-+_]+@[a-z0-9.\-+_]+\.[a-z]+', str_mail_message_recipient)
str_mail_message_recipient = ' '.join([str(item) for item in recipients_mail])
elif len(mail_message_recipient) == 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
# clean up dangerous characters
str_mail_message_recipient = str_mail_message_recipient.replace(' ', ';')
str_mail_message_file_name = mail_message_file_name.replace(',', ';')
str_mail_message_subject = mail_message_subject.replace(',', ';')
str_mail_message_sender = mail_message_sender.replace('\"', '')
# main message creation
message = str(now_log_time) + " tmcas.customername.com, application=\"tmcas-" + sname + "-"+ event_name + "\", detectedutc=\"" + ut_detectedtime + "\", scan_type=\"" + scan_type + "\", affected_user=\"" + affected_user + "\", location=\"" + location + "\", triggered_policy_name=\"" + triggered_policy_name + "\", triggered_security_filter=\"" + triggered_security_filter + "\", action=\"" + action + "\", action_result=\"" + action_result + "\", mail_message_id=\"" + mail_message_id + "\", mail_message_sender=\"" + str_mail_message_sender + "\", mail_message_recipient=\"" + str_mail_message_recipient + "\", mail_message_submit_time=\"" + ut_mail_submittime + "\", mail_message_delivery_time=\"" + ut_mail_deliverytime + "\", mail_message_subject=\"" + str_mail_message_subject + "\", mail_message_file_name=\"" + str_mail_message_file_name + "\", ransomware_name=\"" + ransomware_name + "\""
events.insert(-1, message)
# write events to the log file
for i in range(len(events)):
with open(filepath, "a+") as file_log:
file_log.seek(0)
data = file_log.read(100)
if len(data) > 0 :
file_log.write("\n")
file_log.write(events[i])
print(len(events))
if __name__ == "__main__":
while True:
doit()
time.sleep(300)
/opt/tmcas/securityrisk-event-parser.py
###########################################################
#### ####
#### Trend Micro Cloud App - Security Risk events ####
#### ####
#### ver.1.0 ####
###########################################################
import requests, json, datetime, time, os.path, re
import dateutil.parser as dp
from datetime import timedelta
# path for testing
path_dir = "/var/netwitness/logcollector/upload/trendcloudapp/trendcloudapp/"
filepath = os.path.join(path_dir, "securityrisk-events.log")
# define services to be processed
services = ["exchange","teams","sharepoint","onedrive"]
def doit():
# open token file:
with open("token.json", "r") as read_file:
data_json = json.load(read_file)
token = data_json["access"]
# create 'next page link' signaler
nxtlink = 0
url_maker_nxtlink = []
# format time interval and define the 'start' and 'end' ranges
now_log_time = datetime.datetime.now().replace(microsecond=0).isoformat(' ')
now_time = datetime.datetime.now().replace(microsecond=0).isoformat()
previous_time = datetime.datetime.now() - timedelta(hours=0, minutes=30)
since = previous_time.strftime("%Y-%m-%dT%H:%M:%S.000%z") + 'Z'
until = now_time + '.000Z'
# Start request message parameters
headers = {
'Authorization': 'Bearer ' + token,
'cache-control': 'no-cache',
}
params = (
('start', since),
('end', until),
('limit', '200'),
)
for sname in services:
# create event bucket and message var
events = []
message = ""
# create the iterable URL and pass it to the API request
url_maker = "https://api.tmcas.trendmicro.com/siem/v1/security_events?service=" + sname + "&event=securityrisk"
print(url_maker)
response = requests.get(url_maker, headers=headers, params=params)
# work the deserialized response
current_incidents = json.loads(response.text)
print(current_incidents)
# iterate in the response and format the log line message
print(current_incidents["next_link"])
if (len(current_incidents["security_events"]) > 0):
if (current_incidents["next_link"] != ''):
nxtlink = 1
url_maker_nxtlink.insert(-1,current_incidents["next_link"])
print(len(current_incidents["security_events"]))
for index in range(len(current_incidents["security_events"])):
for key in current_incidents["security_events"][index]:
if key == "event":
event_name = current_incidents["security_events"][index][key]
elif key == "message":
detection_time = current_incidents["security_events"][index][key]["detection_time"]
mail_message_recipient = current_incidents["security_events"][index][key]["mail_message_recipient"]
scan_type = current_incidents["security_events"][index][key]["scan_type"]
affected_user = current_incidents["security_events"][index][key]["affected_user"]
location = current_incidents["security_events"][index][key]["location"]
triggered_policy_name = current_incidents["security_events"][index][key]["triggered_policy_name"]
triggered_security_filter = current_incidents["security_events"][index][key]["triggered_security_filter"]
action = current_incidents["security_events"][index][key]["action"]
action_result = current_incidents["security_events"][index][key]["action_result"]
mail_message_id = current_incidents["security_events"][index][key]["mail_message_id"]
mail_message_sender = current_incidents["security_events"][index][key]["mail_message_sender"]
mail_message_submit_time = current_incidents["security_events"][index][key]["mail_message_submit_time"]
mail_message_delivery_time = current_incidents["security_events"][index][key]["mail_message_delivery_time"]
mail_message_subject = current_incidents["security_events"][index][key]["mail_message_subject"]
mail_message_file_name = current_incidents["security_events"][index][key]["mail_message_file_name"]
security_risk_name = current_incidents["security_events"][index][key]["security_risk_name"]
detected_by = current_incidents["security_events"][index][key]["detected_by"]
risk_level = current_incidents["security_events"][index][key]["risk_level"]
# log message creation
# convert ISO8601 time format to unix timestamp
dp_detectetime = dp.parse(detection_time)
ut_detectedtime = int(time.mktime(dp_detectetime.timetuple()))
ut_detectedtime = str(ut_detectedtime)
dp_mail_submittime = dp.parse(mail_message_submit_time)
ut_mail_submittime = int(time.mktime(dp_mail_submittime.timetuple()))
ut_mail_submittime = str(ut_mail_submittime)
dp_mail_deliverytime = dp.parse(mail_message_delivery_time)
ut_mail_deliverytime = int(time.mktime(dp_mail_deliverytime.timetuple()))
ut_mail_deliverytime = str(ut_mail_deliverytime)
# extract mail_message_recipient's emails
if len(mail_message_recipient) > 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
str_mail_message_recipient = str_mail_message_recipient.replace('\"', '')
recipients_mail = re.findall(r'[a-z0-9.\-+_]+@[a-z0-9.\-+_]+\.[a-z]+', str_mail_message_recipient)
str_mail_message_recipient = ' '.join([str(item) for item in recipients_mail])
str_mail_message_recipient = str_mail_message_recipient.replace(' ', ';')
elif len(mail_message_recipient) == 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
str_mail_message_file_name = mail_message_file_name.replace(',', ';')
str_mail_message_subject = mail_message_subject.replace(',', ';')
str_mail_message_sender = mail_message_sender.replace('\"', '')
# main message creation
message = str(now_log_time) + " tmcas.customername.com, application=\"tmcas-" + sname + "-"+ event_name + "\", detectedutc=\"" + ut_detectedtime + "\", scan_type=\"" + scan_type + "\", affected_user=\"" + affected_user + "\", location=\"" + location + "\", triggered_policy_name=\"" + triggered_policy_name + "\", triggered_security_filter=\"" + triggered_security_filter + "\", action=\"" + action + "\", action_result=\"" + action_result + "\", mail_message_id=\"" + mail_message_id + "\", mail_message_sender=\"" + str_mail_message_sender + "\", mail_message_recipient=\"" + str_mail_message_recipient + "\", mail_message_submit_time=\"" + ut_mail_submittime + "\", mail_message_delivery_time=\"" + ut_mail_deliverytime + "\", mail_message_subject=\"" + str_mail_message_subject + "\", mail_message_file_name=\"" + str_mail_message_file_name + "\", security_risk_name=\"" + security_risk_name + "\", detected_by=\"" + detected_by + "\", risk_level=\"" + risk_level + "\""
events.insert(-1, message)
# write events to the log file
for i in range(len(events)):
with open(filepath, "a+") as file_log:
file_log.seek(0)
data = file_log.read(100)
if len(data) > 0 :
file_log.write("\n")
file_log.write(events[i])
print(len(events))
# check if there are a new page of events for being processed, then do it!
if (nxtlink == 1):
for url in url_maker_nxtlink:
# restate event bucket and message var
events = []
message = ""
print(url)
print(len(url_maker_nxtlink))
response = requests.get(url, headers=headers, params=params)
# work the deserialized response
current_incidents = json.loads(response.text)
# iterate in the response and format the log line message
if (len(current_incidents["security_events"]) > 0):
if (current_incidents["next_link"] != ''):
url_maker_nxtlink.insert(-1, current_incidents["next_link"])
print(len(current_incidents["security_events"]))
for index in range(len(current_incidents["security_events"])):
for key in current_incidents["security_events"][index]:
if key == "event":
event_name = current_incidents["security_events"][index][key]
elif key == "message":
detection_time = current_incidents["security_events"][index][key]["detection_time"]
mail_message_recipient = current_incidents["security_events"][index][key]["mail_message_recipient"]
scan_type = current_incidents["security_events"][index][key]["scan_type"]
affected_user = current_incidents["security_events"][index][key]["affected_user"]
location = current_incidents["security_events"][index][key]["location"]
triggered_policy_name = current_incidents["security_events"][index][key]["triggered_policy_name"]
triggered_security_filter = current_incidents["security_events"][index][key]["triggered_security_filter"]
action = current_incidents["security_events"][index][key]["action"]
action_result = current_incidents["security_events"][index][key]["action_result"]
mail_message_id = current_incidents["security_events"][index][key]["mail_message_id"]
mail_message_sender = current_incidents["security_events"][index][key]["mail_message_sender"]
mail_message_submit_time = current_incidents["security_events"][index][key]["mail_message_submit_time"]
mail_message_delivery_time = current_incidents["security_events"][index][key]["mail_message_delivery_time"]
mail_message_subject = current_incidents["security_events"][index][key]["mail_message_subject"]
mail_message_file_name = current_incidents["security_events"][index][key]["mail_message_file_name"]
risk_level = current_incidents["security_events"][index][key]["risk_level"]
security_risk_name = current_incidents["security_events"][index][key]["security_risk_name"]
detected_by = current_incidents["security_events"][index][key]["detected_by"]
risk_level = current_incidents["security_events"][index][key]["risk_level"]
# log message creation
# convert ISO8601 time format to unix timestamp
dp_detectetime = dp.parse(detection_time)
ut_detectedtime = int(time.mktime(dp_detectetime.timetuple()))
ut_detectedtime = str(ut_detectedtime)
dp_mail_submittime = dp.parse(mail_message_submit_time)
ut_mail_submittime = int(time.mktime(dp_mail_submittime.timetuple()))
ut_mail_submittime = str(ut_mail_submittime)
dp_mail_deliverytime = dp.parse(mail_message_delivery_time)
ut_mail_deliverytime = int(time.mktime(dp_mail_deliverytime.timetuple()))
ut_mail_deliverytime = str(ut_mail_deliverytime)
# extract mail_message_recipient's emails
if len(mail_message_recipient) > 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
str_mail_message_recipient = str_mail_message_recipient.replace('\"', '')
recipients_mail = re.findall(r'[a-z0-9.\-+_]+@[a-z0-9.\-+_]+\.[a-z]+', str_mail_message_recipient)
str_mail_message_recipient = ' '.join([str(item) for item in recipients_mail])
str_mail_message_recipient = str_mail_message_recipient.replace(' ', ';')
elif len(mail_message_recipient) == 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
# clean up dangerous characters
str_mail_message_file_name = mail_message_file_name.replace(',', ';')
str_mail_message_subject = mail_message_subject.replace(',', ';')
str_mail_message_sender = mail_message_sender.replace('\"', '')
# main message creation
message = str(now_log_time) + " tmcas.customername.com, application=\"tmcas-" + sname + "-"+ event_name + "\", detectedutc=\"" + ut_detectedtime + "\", scan_type=\"" + scan_type + "\", affected_user=\"" + affected_user + "\", location=\"" + location + "\", triggered_policy_name=\"" + triggered_policy_name + "\", triggered_security_filter=\"" + triggered_security_filter + "\", action=\"" + action + "\", action_result=\"" + action_result + "\", mail_message_id=\"" + mail_message_id + "\", mail_message_sender=\"" + str_mail_message_sender + "\", mail_message_recipient=\"" + str_mail_message_recipient + "\", mail_message_submit_time=\"" + ut_mail_submittime + "\", mail_message_delivery_time=\"" + ut_mail_deliverytime + "\", mail_message_subject=\"" + str_mail_message_subject + "\", mail_message_file_name=\"" + str_mail_message_file_name + "\", security_risk_name=\"" + security_risk_name + "\", detected_by=\"" + detected_by + "\", risk_level=\"" + risk_level + "\""
events.insert(-1, message)
# write events to the log file
for i in range(len(events)):
with open(filepath, "a+") as file_log:
file_log.seek(0)
data = file_log.read(100)
if len(data) > 0 :
file_log.write("\n")
file_log.write(events[i])
print(len(events))
if __name__ == "__main__":
while True:
doit()
time.sleep(300)
/opt/tmcas/virtualanalyzer-event-parser.py
###########################################################
#### ####
#### Trend Micro Cloud App - Virtual Analyzer events ####
#### ####
#### ver.1.0 ####
###########################################################
import requests, json, datetime, time, os.path, re
import dateutil.parser as dp
from datetime import timedelta
# path for testing
path_dir = "/var/netwitness/logcollector/upload/trendcloudapp/trendcloudapp/"
filepath = os.path.join(path_dir, "virtualanalyzer-events.log")
# define services to be processed
services = ["exchange","teams","sharepoint","onedrive"]
def doit():
# open token file:
with open("token.json", "r") as read_file:
data_json = json.load(read_file)
token = data_json["access"]
# create 'next page link' signaler
nxtlink = 0
url_maker_nxtlink = []
# format time interval and define the 'start' and 'end' ranges
now_log_time = datetime.datetime.now().replace(microsecond=0).isoformat(' ')
now_time = datetime.datetime.now().replace(microsecond=0).isoformat()
previous_time = datetime.datetime.now() - timedelta(hours=1, minutes=0)
since = previous_time.strftime("%Y-%m-%dT%H:%M:%S.000%z") + 'Z'
until = now_time + '.000Z'
# Start request message parameters
headers = {
'Authorization': 'Bearer ' + token,
'cache-control': 'no-cache',
}
params = (
('start', since),
('end', until),
('limit', '200'),
)
for sname in services:
# create event bucket and message var
events = []
message = ""
# create the iterable URL and pass it to the API request
url_maker = "https://api.tmcas.trendmicro.com/siem/v1/security_events?service=" + sname + "&event=virtualanalyzer"
print(url_maker)
response = requests.get(url_maker, headers=headers, params=params)
# work the deserialized response
current_incidents = json.loads(response.text)
# iterate in the response and format the log line message
if (len(current_incidents["security_events"]) > 0):
if (current_incidents["next_link"] != ''):
nxtlink = 1
url_maker_nxtlink.insert(-1, current_incidents["next_link"])
print(len(current_incidents["security_events"]))
for index in range(len(current_incidents["security_events"])):
for key in current_incidents["security_events"][index]:
if key == "event":
event_name = current_incidents["security_events"][index][key]
elif key == "message":
detection_time = current_incidents["security_events"][index][key]["detection_time"]
mail_message_recipient = current_incidents["security_events"][index][key]["mail_message_recipient"]
scan_type = current_incidents["security_events"][index][key]["scan_type"]
affected_user = current_incidents["security_events"][index][key]["affected_user"]
location = current_incidents["security_events"][index][key]["location"]
triggered_policy_name = current_incidents["security_events"][index][key]["triggered_policy_name"]
triggered_security_filter = current_incidents["security_events"][index][key]["triggered_security_filter"]
action = current_incidents["security_events"][index][key]["action"]
action_result = current_incidents["security_events"][index][key]["action_result"]
mail_message_id = current_incidents["security_events"][index][key]["mail_message_id"]
mail_message_sender = current_incidents["security_events"][index][key]["mail_message_sender"]
mail_message_submit_time = current_incidents["security_events"][index][key]["mail_message_submit_time"]
mail_message_delivery_time = current_incidents["security_events"][index][key]["mail_message_delivery_time"]
mail_message_subject = current_incidents["security_events"][index][key]["mail_message_subject"]
mail_message_file_name = current_incidents["security_events"][index][key]["mail_message_file_name"]
risk_level = current_incidents["security_events"][index][key]["risk_level"]
detection_type = current_incidents["security_events"][index][key]["detection_type"]
file_sha1 = current_incidents["security_events"][index][key]["file_sha1"]
file_sha256 = current_incidents["security_events"][index][key]["file_sha256"]
# log message creation
# convert ISO8601 time format to unix timestamp
dp_detectetime = dp.parse(detection_time)
ut_detectedtime = int(time.mktime(dp_detectetime.timetuple()))
ut_detectedtime = str(ut_detectedtime)
dp_mail_submittime = dp.parse(mail_message_submit_time)
ut_mail_submittime = int(time.mktime(dp_mail_submittime.timetuple()))
ut_mail_submittime = str(ut_mail_submittime)
dp_mail_deliverytime = dp.parse(mail_message_delivery_time)
ut_mail_deliverytime = int(time.mktime(dp_mail_deliverytime.timetuple()))
ut_mail_deliverytime = str(ut_mail_deliverytime)
# extract mail_message_recipient's emails
if len(mail_message_recipient) > 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
str_mail_message_recipient = str_mail_message_recipient.replace('\"', '')
recipients_mail = re.findall(r'[a-z0-9.\-+_]+@[a-z0-9.\-+_]+\.[a-z]+', str_mail_message_recipient)
str_mail_message_recipient = ' '.join([str(item) for item in recipients_mail])
str_mail_message_recipient = str_mail_message_recipient.replace(' ', ';')
elif len(mail_message_recipient) == 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
# clean up dangerous characters
str_mail_message_file_name = mail_message_file_name.replace(',', ';')
str_mail_message_subject = mail_message_subject.replace(',', ';')
str_mail_message_sender = mail_message_sender.replace('\"', '')
# main message creation
message = str(now_log_time) + " tmcas.customername.com, application=\"tmcas-" + sname + "-"+ event_name + "\", detectedutc=\"" + ut_detectedtime + "\", scan_type=\"" + scan_type + "\", affected_user=\"" + affected_user + "\", location=\"" + location + "\", triggered_policy_name=\"" + triggered_policy_name + "\", triggered_security_filter=\"" + triggered_security_filter + "\", action=\"" + action + "\", action_result=\"" + action_result + "\", mail_message_id=\"" + mail_message_id + "\", mail_message_sender=\"" + str_mail_message_sender + "\", mail_message_recipient=\"" + str_mail_message_recipient + "\", mail_message_submit_time=\"" + ut_mail_submittime + "\", mail_message_delivery_time=\"" + ut_mail_deliverytime + "\", mail_message_subject=\"" + str_mail_message_subject + "\", mail_message_file_name=\"" + str_mail_message_file_name + "\", risk_level=\"" + risk_level + "\", detection_type=\"" + detection_type + "\", file_sha1=\"" + file_sha1 + "\", file_sha256=\"" + file_sha256 + "\""
events.insert(-1, message)
# write events to the log file
for i in range(len(events)):
with open(filepath, "a+") as file_log:
file_log.seek(0)
data = file_log.read(100)
if len(data) > 0 :
file_log.write("\n")
file_log.write(events[i])
print(len(events))
# check if there are a new page of events for being processed, then do it!
if (nxtlink == 1):
for url in url_maker_nxtlink:
# restate event bucket and message var
events = []
message = ""
print(url)
print(len(url_maker_nxtlink))
response = requests.get(url, headers=headers, params=params)
# work the deserialized response
current_incidents = json.loads(response.text)
# iterate in the response and format the log line message
if (len(current_incidents["security_events"]) > 0):
if (current_incidents["next_link"] != ''):
url_maker_nxtlink.insert(-1, current_incidents["next_link"])
print(len(current_incidents["security_events"]))
for index in range(len(current_incidents["security_events"])):
for key in current_incidents["security_events"][index]:
if key == "event":
event_name = current_incidents["security_events"][index][key]
elif key == "message":
detection_time = current_incidents["security_events"][index][key]["detection_time"]
mail_message_recipient = current_incidents["security_events"][index][key]["mail_message_recipient"]
scan_type = current_incidents["security_events"][index][key]["scan_type"]
affected_user = current_incidents["security_events"][index][key]["affected_user"]
location = current_incidents["security_events"][index][key]["location"]
triggered_policy_name = current_incidents["security_events"][index][key]["triggered_policy_name"]
triggered_security_filter = current_incidents["security_events"][index][key]["triggered_security_filter"]
action = current_incidents["security_events"][index][key]["action"]
action_result = current_incidents["security_events"][index][key]["action_result"]
mail_message_id = current_incidents["security_events"][index][key]["mail_message_id"]
mail_message_sender = current_incidents["security_events"][index][key]["mail_message_sender"]
mail_message_submit_time = current_incidents["security_events"][index][key]["mail_message_submit_time"]
mail_message_delivery_time = current_incidents["security_events"][index][key]["mail_message_delivery_time"]
mail_message_subject = current_incidents["security_events"][index][key]["mail_message_subject"]
mail_message_file_name = current_incidents["security_events"][index][key]["mail_message_file_name"]
risk_level = current_incidents["security_events"][index][key]["risk_level"]
detection_type = current_incidents["security_events"][index][key]["detection_type"]
file_sha1 = current_incidents["security_events"][index][key]["file_sha1"]
file_sha256 = current_incidents["security_events"][index][key]["file_sha256"]
# log message creation
# convert ISO8601 time format to unix timestamp
dp_detectetime = dp.parse(detection_time)
ut_detectedtime = int(time.mktime(dp_detectetime.timetuple()))
ut_detectedtime = str(ut_detectedtime)
dp_mail_submittime = dp.parse(mail_message_submit_time)
ut_mail_submittime = int(time.mktime(dp_mail_submittime.timetuple()))
ut_mail_submittime = str(ut_mail_submittime)
dp_mail_deliverytime = dp.parse(mail_message_delivery_time)
ut_mail_deliverytime = int(time.mktime(dp_mail_deliverytime.timetuple()))
ut_mail_deliverytime = str(ut_mail_deliverytime)
# extract mail_message_recipient's emails
if len(mail_message_recipient) > 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
str_mail_message_recipient = str_mail_message_recipient.replace('\"', '')
recipients_mail = re.findall(r'[a-z0-9.\-+_]+@[a-z0-9.\-+_]+\.[a-z]+', str_mail_message_recipient)
str_mail_message_recipient = ' '.join([str(item) for item in recipients_mail])
str_mail_message_recipient = str_mail_message_recipient.replace(' ', ';')
elif len(mail_message_recipient) == 1:
str_mail_message_recipient = ' '.join([str(item) for item in mail_message_recipient])
str_mail_message_file_name = mail_message_file_name.replace(',', ';')
str_mail_message_subject = mail_message_subject.replace(',', ';')
str_mail_message_sender = mail_message_sender.replace('\"', '')
# main message creation
message = str(now_log_time) + " tmcas.customername.com, application=\"tmcas-" + sname + "-"+ event_name + "\", detectedutc=\"" + ut_detectedtime + "\", scan_type=\"" + scan_type + "\", affected_user=\"" + affected_user + "\", location=\"" + location + "\", triggered_policy_name=\"" + triggered_policy_name + "\", triggered_security_filter=\"" + triggered_security_filter + "\", action=\"" + action + "\", action_result=\"" + action_result + "\", mail_message_id=\"" + mail_message_id + "\", mail_message_sender=\"" + str_mail_message_sender + "\", mail_message_recipient=\"" + str_mail_message_recipient + "\", mail_message_submit_time=\"" + ut_mail_submittime + "\", mail_message_delivery_time=\"" + ut_mail_deliverytime + "\", mail_message_subject=\"" + str_mail_message_subject + "\", mail_message_file_name=\"" + str_mail_message_file_name + "\", risk_level=\"" + risk_level + "\", detection_type=\"" + detection_type + "\", file_sha1=\"" + file_sha1 + "\", file_sha256=\"" + file_sha256 + "\""
events.insert(-1, message)
# write events to the log file
for i in range(len(events)):
with open(filepath, "a+") as file_log:
file_log.seek(0)
data = file_log.read(100)
if len(data) > 0 :
file_log.write("\n")
file_log.write(events[i])
print(len(events))
if __name__ == "__main__":
while True:
doit()
time.sleep(300)
/opt/tmcas/token.json
{"token_type":"Bearer","expires_in":"unknown","access":"MYBEARERTOKEN"}
Example parser for XDR using Python and a credential token for authentication with the API and the eventsource type specification.
/etc/netwitness/ng/logcollection/content/collection/file/trendmicroxdr.xml
<?xml version="1.0" encoding="UTF-8"?>
<typespec>
<name>trendmicroxdr</name>
<type>file</type>
<prettyName>Trend Micro XDR</prettyName>
<version>1.0</version>
<author>administrator</author>
<description>FileCollection specification for eventsource type "Trend Micro XDR Solution"</description>
<device>
<name>trendmicroxdr</name>
</device>
<configuration>
</configuration>
<collection>
<file>
<parserId>trendmicroxdr</parserId>
<processorType>generic</processorType>
<dataStartLine>1</dataStartLine>
<fieldDelim>,</fieldDelim>
<idField></idField>
<lineDelim>\n</lineDelim>
<transformPrefixTag>trendmicroxdr</transformPrefixTag>
<transformReplaceFieldDelim>0</transformReplaceFieldDelim>
<transformPrefixFilename>0</transformPrefixFilename>
<transformMultipleDelimiterAsOne>0</transformMultipleDelimiterAsOne>
<transformReplacementFieldDelim></transformReplacementFieldDelim>
</file>
</collection>
</typespec>
/opt/tmxdr/tmxdr-attack-techniques.py
import requests, json, datetime, time, os.path, re
import dateutil.parser as dp
from datetime import timedelta
# Base URL for the requested resources
url_base = 'https://api.xdr.trendmicro.com'
url_path = '/v2.0/xdr/oat/detections'
# Path to save logs to
path_dir = "/var/netwitness/logcollector/upload/trendmicroxdr/trendmicroxdr/"
filepath = os.path.join(path_dir, "attack-techniques-events.log")
def doit():
# open token file:
with open("/opt/tmxdr/token.json", "r") as read_file:
data_json = json.load(read_file)
token = data_json["access"]
# create 'next page link' signaler
nxtlink = 0
token_nxtlink = ""
################### ADJUST TIME PARAMETERS #######################
## format time interval and define the 'start' and 'end' ranges ##
##################################################################
now_log_time = datetime.datetime.now().replace(microsecond=0).isoformat(' ')
now_time = datetime.datetime.now().replace(microsecond=0).isoformat()
previous_time = datetime.datetime.now() - timedelta(hours=0, minutes=5)
since = previous_time.strftime("%Y-%m-%dT%H:%M:%S.000%z") + 'Z'
until = now_time
# convert 'since' to timestamp format
dp_since = dp.parse(since)
ut_since = int(time.mktime(dp_since.timetuple()))
ut_since = str(ut_since)
# convert 'until' to timestamp format
dp_until = dp.parse(until)
ut_until = int(time.mktime(dp_until.timetuple()))
ut_until = str(ut_until)
# Start request message parameters
headers = {'Authorization': 'Bearer ' + token, 'Content-Type': 'application/json;charset=utf-8'}
params = (
('start', ut_since),
('end', ut_until),
('size', '10')
)
# create event bucket and message var
events = []
message = ""
# request event alerts
response = requests.get(url_base + url_path, headers=headers, params=params)
print(response.url)
# work the deserialized response
current_incidents = json.loads(response.text)
response = ""
# iterate in the response and format the log line message
if len(current_incidents["data"]["detections"]) > 0:
hasToken = current_incidents["data"].get("nextBatchToken")
if hasToken == None:
nxtlink = 0
print("Events finished")
else:
nxtlink = 1
token_nxtlink = current_incidents["data"]["nextBatchToken"]
events = []
message = ""
for index in range(len(current_incidents["data"]["detections"])):
for key in current_incidents["data"]["detections"][index]:
if key == "source":
det_source = current_incidents["data"]["detections"][index][key]
elif key == "detectionTime":
detection_time = current_incidents["data"]["detections"][index][key]
elif key == "uuid":
uuid = current_incidents["data"]["detections"][index][key]
elif key == "endpoint":
endpoint_name = current_incidents["data"]["detections"][index][key]["name"]
endpoint_guid = current_incidents["data"]["detections"][index][key]["guid"]
if len(current_incidents["data"]["detections"][index][key]["ips"]) > 1:
endpoint_ip = ';'.join([str(item) for item in current_incidents["data"]["detections"][index][key]["ips"]])
else:
endpoint_ip = ' '.join([str(item) for item in current_incidents["data"]["detections"][index][key]["ips"]])
elif key == "filters":
for filter in range(len(current_incidents["data"]["detections"][index]["filters"])):
for k in current_incidents["data"]["detections"][index]["filters"][filter]:
if k == "id":
filter_id = current_incidents["data"]["detections"][index]["filters"][filter][k]
elif k == "unique_id":
filter_uid = current_incidents["data"]["detections"][index]["filters"][filter][k]
elif k == "level":
filter_level = current_incidents["data"]["detections"][index]["filters"][filter][k]
elif k == "name":
filter_name = current_incidents["data"]["detections"][index]["filters"][filter][k]
elif k == "description":
filter_description = current_incidents["data"]["detections"][index]["filters"][filter][k]
elif k == "tactics":
if len(current_incidents["data"]["detections"][index]["filters"][filter]["tactics"]) > 1:
tactics = ' '.join([str(item) for item in current_incidents["data"]["detections"][index]["filters"][filter]["tactics"]])
tactics = tactics.replace(' ', ';')
else:
tactics = ' '.join([str(item) for item in current_incidents["data"]["detections"][index]["filters"][filter]["tactics"]])
elif k == "techniques":
if len(current_incidents["data"]["detections"][index]["filters"][filter]["techniques"]) > 1:
techniques = ';'.join([str(item) for item in current_incidents["data"]["detections"][index]["filters"][filter]["techniques"]])
else:
techniques = ' '.join([str(item) for item in current_incidents["data"]["detections"][index]["filters"][filter]["techniques"]])
elif k == "highlightedObjects":
if len(current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"]) > 1:
fld1 = []
value1 = []
highlights = []
for object in range(len(current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"])):
for kye in current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"][object]:
if kye == "field":
fieldName = current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"][object][kye]
fld1.insert(1, fieldName)
elif kye == "value":
valueName = current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"][object][kye]
isArray = isinstance(valueName, list)
if (isArray == True):
valueName = ';'.join([str(item) for item in valueName])
valueName = valueName.replace('{', '\{')
valueName = valueName.replace('}', '\}')
valueName = valueName.replace('\"', '\\\'')
else:
valueName = str(valueName)
valueName = valueName.replace('{', '\{')
valueName = valueName.replace('}', '\}')
valueName = valueName.replace('\"', '\\\'')
value1.insert(1, valueName)
for flditems in range(len(fld1)):
highlighted = fld1[flditems] +"=\"" + value1[flditems] + "\""
highlights.insert(1, highlighted)
highlightedObj = ', '.join([str(item) for item in highlights])
else:
fld1 = current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"][0]["field"]
value1 = current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"][0]["value"]
value1 = value1.replace('{', '\{')
value1 = value1.replace('}', '\}')
value1 = value1.replace('\"', '\\\'')
highlightedObj = fld1 + "=\"" + value1 + "\""
# convert ISO8601 time format to unix timestamp
dp_detectetime = dp.parse(detection_time)
ut_detectedtime = int(time.mktime(dp_detectetime.timetuple()))
ut_detectedtime = str(ut_detectedtime)
message = str(now_log_time) + " tmxdr.customername.com, application=\"trendmicroxdr-" + det_source + "\", detectedutc=\"" + ut_detectedtime + "\", uuid=\"" + uuid + "\", endpoint=\"" + endpoint_name + "\", endpoint_guid=\"" + endpoint_guid + "\", endpoint_ip=\"" + endpoint_ip + "\", filter_id=\"" + filter_id + "\", filter_uid=\"" + filter_uid + "\", filter_level=\"" + filter_level + "\", filter_name=\"" + filter_name + "\", filter_description=\"" + filter_description +"\", tactics=\"" + tactics + "\", techniques=\"" + techniques + "\", " + highlightedObj
events.insert(-1, message)
current_incidents = {}
# write events to the log file
for i in range(len(events)):
with open(filepath, "a+") as file_log:
file_log.seek(0)
data = file_log.read(100)
if len(data) > 0 :
file_log.write("\n")
file_log.write(events[i])
print(len(events))
# check if there are a new page of events for being processed, then do it!
while (nxtlink == 1):
# restate event bucket and message var
events = []
message = ""
params = (
('start', ut_since),
('end', ut_until),
('size', '10'),
('nextBatchToken', token_nxtlink),
)
headers = {'Authorization': 'Bearer ' + token, 'Content-Type': 'application/json;charset=utf-8'}
# request event alerts
response = requests.get(url_base + url_path, headers=headers, params=params)
# work the deserialized response
current_incidents = json.loads(response.text)
response = ""
# iterate in the response and format the log line message
if len(current_incidents["data"]["detections"]) > 0:
hasToken = current_incidents["data"].get("nextBatchToken")
if (hasToken == None):
nxtlink = 0
token_nxtlink = ""
print("Events finished")
else:
nxtlink = 1
token_nxtlink = current_incidents["data"]["nextBatchToken"]
for index in range(len(current_incidents["data"]["detections"])):
for key in current_incidents["data"]["detections"][index]:
if key == "source":
det_source = current_incidents["data"]["detections"][index][key]
elif key == "detectionTime":
detection_time = current_incidents["data"]["detections"][index][key]
elif key == "uuid":
uuid = current_incidents["data"]["detections"][index][key]
elif key == "endpoint":
endpoint_name = current_incidents["data"]["detections"][index][key]["name"]
endpoint_guid = current_incidents["data"]["detections"][index][key]["guid"]
if len(current_incidents["data"]["detections"][index][key]["ips"]) > 1:
endpoint_ip = ';'.join([str(item) for item in current_incidents["data"]["detections"][index][key]["ips"]])
else:
endpoint_ip = ' '.join([str(item) for item in current_incidents["data"]["detections"][index][key]["ips"]])
elif key == "filters":
for filter in range(len(current_incidents["data"]["detections"][index]["filters"])):
for k in current_incidents["data"]["detections"][index]["filters"][filter]:
if k == "id":
filter_id = current_incidents["data"]["detections"][index]["filters"][filter][k]
elif k == "unique_id":
filter_uid = current_incidents["data"]["detections"][index]["filters"][filter][k]
elif k == "level":
filter_level = current_incidents["data"]["detections"][index]["filters"][filter][k]
elif k == "name":
filter_name = current_incidents["data"]["detections"][index]["filters"][filter][k]
elif k == "description":
filter_description = current_incidents["data"]["detections"][index]["filters"][filter][k]
elif k == "tactics":
if len(current_incidents["data"]["detections"][index]["filters"][filter]["tactics"]) > 1:
tactics = ' '.join([str(item) for item in current_incidents["data"]["detections"][index]["filters"][filter]["tactics"]])
tactics = tactics.replace(' ', ';')
else:
tactics = ' '.join([str(item) for item in current_incidents["data"]["detections"][index]["filters"][filter]["tactics"]])
elif k == "techniques":
if len(current_incidents["data"]["detections"][index]["filters"][filter]["techniques"]) > 1:
techniques = ';'.join([str(item) for item in current_incidents["data"]["detections"][index]["filters"][filter]["techniques"]])
else:
techniques = ' '.join([str(item) for item in current_incidents["data"]["detections"][index]["filters"][filter]["techniques"]])
elif k == "highlightedObjects":
if len(current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"]) > 1:
fld1 = []
value1 = []
highlights = []
for object in range(len(current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"])):
for kye in current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"][object]:
if kye == "field":
fieldName = current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"][object][kye]
fld1.insert(1, fieldName)
elif kye == "value":
valueName = current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"][object][kye]
isArray = isinstance(valueName, list)
if (isArray == True):
valueName = ';'.join([str(item) for item in valueName])
valueName = valueName.replace('{', '\{')
valueName = valueName.replace('}', '\}')
valueName = valueName.replace('\"', '\\\'')
else:
valueName = str(valueName)
valueName = valueName.replace('{', '\{')
valueName = valueName.replace('}', '\}')
valueName = valueName.replace('\"', '\\\'')
value1.insert(1, valueName)
for flditems in range(len(fld1)):
highlighted = fld1[flditems] +"=\"" + value1[flditems] + "\""
highlights.insert(1, highlighted)
highlightedObj = ', '.join([str(item) for item in highlights])
else:
fld1 = current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"][0]["field"]
value1 = current_incidents["data"]["detections"][index]["filters"][filter]["highlightedObjects"][0]["value"]
value1 = value1.replace('{', '\{')
value1 = value1.replace('}', '\}')
value1 = value1.replace('\"', '\\\'')
highlightedObj = fld1 + "=\"" + value1 + "\""
# convert ISO8601 time format to unix timestamp
dp_detectetime = dp.parse(detection_time)
ut_detectedtime = int(time.mktime(dp_detectetime.timetuple()))
ut_detectedtime = str(ut_detectedtime)
message = str(now_log_time) + " tmxdr.customername.com, application=\"trendmicroxdr-" + det_source + "\", detectedutc=\"" + ut_detectedtime + "\", uuid=\"" + uuid + "\", endpoint=\"" + endpoint_name + "\", endpoint_guid=\"" + endpoint_guid + "\", endpoint_ip=\"" + endpoint_ip + "\", filter_id=\"" + filter_id + "\", filter_uid=\"" + filter_uid + "\", filter_level=\"" + filter_level + "\", filter_name=\"" + filter_name + "\", filter_description=\"" + filter_description +"\", tactics=\"" + tactics + "\", techniques=\"" + techniques + "\", " + highlightedObj
events.insert(-1, message)
current_incidents = {}
# write events to the log file
for i in range(len(events)):
with open(filepath, "a+") as file_log:
file_log.seek(0)
data = file_log.read(100)
if len(data) > 0 :
file_log.write("\n")
file_log.write(events[i])
print(len(events))
if __name__ == "__main__":
while True:
doit()
time.sleep(300)
/opt/tmcas/token.json
{"token_type":"Bearer","expires_in":"unknown","access":"MYBEARERTOKENHERE"}
References
"How to Create a Custom FileSpec for Log Collection using RSA NetWitness Platform"
"RSA NetWitness Platform “How To” Informational Videos"