Skip to main content
Microsoft Security
Photo of a male wearing a gray jacket and glasses, standing in front of a computer desk with multiple monitors.
11 min read

Using Python to unearth a goldmine of threat intelligence from leaked chat logs

Dealing with a great amount of data can be time consuming, thus using Python can be very powerful to help analysts sort information and extract the most relevant data for their investigation. The open-source tools library, MSTICPy, for example, is a Python tool dedicated to threat intelligence. It aims to help threat analysts acquire, enrich, analyze, and visualize data.

This blog provides a workflow for deeper data analysis and visualization using Python, as well as for extraction and analysis of indicators of compromise (IOCs) using MSTICPy. Data sets from the February 2022 leak of data from the ransomware-as-a-service (RaaS) coordinated operation called “Conti” is used as case study.

An interactive Jupyter notebook with related data is also available for analysts interested to do further data exploration.

This research aims to provide a view into research methodology that may help other analysts apply Python to threat intelligence. Analysts can reuse the code and continue to explore the extracted information. Additionally, it offers an out-of-the-box methodology for analyzing chat logs, extracting IOCs, and improving threat intelligence and defense process using Python.

Using Python to analyze the Conti network

On February 28, 2022, a Twitter account named @ContiLeaks (allegedly a Ukrainian researcher) began posting leaked Conti data on Twitter. The leaked data sets, which were posted in a span of several months, consisted of chat logs, source codes, and backend applications.

For this research, we focused our analysis on the chat logs, which revealed crucial information about the Conti group’s operating methods, infrastructure, and organizational structure.

Compiling and translating chat logs

The leaked chat logs are written in the Russian language. To make the analysis more accessible, we adopted the methodology published here and translated the logs to English.

The chat logs revealed that the Conti group uses the messaging application Jabber to communicate among members. Since raw Jabber logs are saved using a file per day, they can be compiled in one JSON file so they can easily be manipulated with Python. Once the data is merged, they can be translated using the deep translator library. After the logs are translated and loaded into a new file, it’s then possible to load the data into a dataframe for manipulation and exploration:

df = pd.read_json(codecs.open('translated_Log2.json', 'r', 'utf-8'))
A screenshot of a table of chat messages translated from Russian to English. The table includes details when the message was sent, who it was from, to whom it was sent, the original text in Russian, and the translated English version.
Figure 1. Translated logs

Russian slang words not properly translated by the automated process can be translated by creating a dictionary. A dictionary off a list proposed here was used in this case to correctly translate the slang:

A screenshot of Python code that creates a dictionary which can be used to translate Russian slang words. It features a list of Russian slang words and their English translation.
Figure 2. Translating slang

Analyzing the chat activity timeline

One way to get insights from chat logs is to see its timeline and check the number of discussions per day. The Bokeh library can be used to build an interactive diagram and explore the loaded dataframe.

A screenshot of Python code for exploring data using the Bokeh library. It shows code for filtering results, creating diagrams, and adding hover tools.
Figure 3. Python code for exploring discussions

Using the data from Conti chat logs generates the following diagram, which shows the volume of Jabber discussions over time:

A line graph that shows a volume of discussions within the Conti group from March 2021 to March 2022. The data shows several peaks in activity, mostly concentrated from September to December 2021.
Figure 4. Volume of discussions over time

Visualizing the data as a timeline shows some peaks of activity that align to certain events. In the case of the Conti leaks, for example:

It’s interesting that no peak in chat activity was observed within the Conti group after the first leak, which could indicate that the breach was ignored or not known by the group at that time.

Analyzing the level of user activity

When analyzing chat logs, identifying the number of users and analyzing the most active ones can provide insight into the size of the group and roles of users within it. Using Python, the list of users can be extracted and saved in a text file:

A screenshot of Python code that extracts the list of users from the Conti chat logs. It also shows code to remove duplicates from the list, concatenate the dataframe, and save the list to a text file.
Figure 5. Extracting list of users

Running the script above using the Conti chat logs yielded a list of 346 unique accounts. This list can then be used to create a graph and show which users sent the most messages.

A screenshot of Python code that creates a graph to show the list of users from the Conti chat logs with the most messages.
Figure 6. Creating a graph for users with most messages
A bar chart that compares the users from the Conti chat logs based on the number of messages they sent. The bar shows that the most active user sent as many as more than eight thousand messages.
Figure 7. Most active users in the Conti chat logs

Based on the graph, the users named defender, stern, driver, bio, and mango have the largest number of discussions. Checkpoint  published extensive research on the structure of the organization and correlated the user discussions with several roles and services like human resources, coders, crypters, offensive team, SysAdmins, and more.

Mapping the users’ connections

Another way to analyze chat log data is to visualize the users’ connection. This can be done by creating a dynamic network graph that can highlight the connections between users. The Barnes Hut algorithm and the Pyvis library can be used to visualize this data.

A screenshot of Python code that creates a dynamic network graph of the Conti chat log data using the Barnes Hut algorithm and Pyvis library.
Figure 8. Creation a dynamic graph

Dynamic visualization shows a graphical overview of the network and allows zooming into the network to closely analyze the connections within. Bigger points represent the most active users, and it’s possible to highlight a user to analyze their connections. Additionally, the hovering tool shows which other users a specific user had conversations with.

A screenshot of a dynamic visualization showing a graphical overview of the Conti network. It shows users as points in the graph, all connected by lines that represent conversations between them.
Figure 9. Conti user network overview
A screenshot of a list of connections to the user named "Stern" from the Conti network. The graphical overview of the entire Conti network is shown on the background of the list.
Figure 10. Connections to user ‘Stern’

Searching for other topics of interest

Since reading data sets can be time-consuming, a simple search engine can be built to search for specific strings in the chat logs or to filter for topics of interest. For the Conti leak data, examples of these include Bitcoin, usernames, malware names, exploits, and CVEs, to name a few.

The following code snippet provides a simple search engine using the TextSearch library:

A screenshot of Python code that creates a search engine using the TextSearch library from Github. The code presents configuration options for the search engine widget and filters for search results.
Figure 11. Search engine using Python

Using MSTICPy to extract and analyze IOCs

Besides processing chat logs to analyze user activity and connections, Python can also be used to extract and analyze threat intelligence. This section shows how the MSTICPy library can be used to extract IOCs and how it can be used for additional threat hunting and intelligence.

Extracting IOCs

MSTICPy is a Python library used for threat investigation and threat hunting. The library can connect to several threat intelligence providers, as well as Microsoft tools like Microsoft Sentinel. It can be used to query logs and to enrich data. It’s particularly convenient for analyzing IOCs and adding more threat contextualization.

After installing MSTICPy, the first thing to do is to initialize the notebook. This allows the loading of several modules that can be used to extract and enrich the data. External resources like VirusTotal or OTX can also be added by configuring msticpyconfig.yaml and adding the API keys.

The IoCExtract module from MSTICPy offers a convenient way to extract IOCs using predefined regex. The code automatically extracts IOCs such as DNS, URLs, IP addresses, and hashes and then reports them in a new dataframe.

A screenshot of Python code that prepares the dataframe for IOC extraction. It presents code to remove "None" value from the dataframe, as well as to initiate the IOC extractor.
Figure 12. Passing the dataframe to the module for extraction
A screenshot of a sample table listing down IOC patterns found in the Conti chat logs. It includes the following data fields: IOC type, observable, source index, and input.
Figure 13. Sample of extracted IOCs

A regex can be added to filter specific IOCs from those extracted by the IOC extraction module by default. For example, the regex below extracts Bitcoin addresses from the Conti chat logs:

A screenshot of Python code that adds a regex in the IOCExtract module of MSTICPy. This specific regex extracts Bitcoin addresses from the Conti chat logs.
Figure 14. Extracting Bitcoin addresses and adding regex
A screenshot of a table showing the Bitcoin addresses extracted from the Conti chat logs. The table includes the following data fields: IOCtype, observable, source index, and input.
Figure 15. Sample of extracted Bitcoin addresses

After extracting IOCs, the dataframe can be cleaned to remove false positives as well as duplicate data. The final dataframe from the processed Conti chat logs contains the following unique IOC count, (these IOCs require additional analysis as not all of them are considered malicious):

URLDNSIPV4BitcoinMD5SHA-256
1,13747431717510616

Investigating IP addresses

 The threat intel lookup module TILookup in MSTICPy can be used to get more information on IOCs such as IP addresses. In the case of the Conti leak, 317 unique IP addresses were identified. Not all these IOCs are malicious but can reveal more relevant information.

The configuration file can be specified to load the TILookup module, along with other threat intelligence providers such as VirusTotal, GreyNoise, and OTX.

A screenshot of Python code that loads the threat intel lookup module in MSTICPy. It also presents code that loads other threat intelligence providers such as VirusTotal, GreyNoise, and OTX, and filters the IOCs by type.
Figure 16. Threat intel provider within MSTICPy

Running the module generates a new dataframe with more context for every IP address provided.

A screenshot of a table generated from running the threat intel lookup module. The table presents a list of IP addresses extracted from the Conti chat logs with related threat intelligence data.
Figure 17. Sample of IP addresses enriched with additional info

The module also allows to request information for a single observable.

A screenshot of Python code that extracts threat intelligence data on a single observable through the threat intel lookup module of MSTICPy.
Figure 18. Extracting information for a single observable
A screenshot of the table generated from running the threat intel lookup module. It shows threat intelligence data from GreyNoise, OTX, and VirusTotal on a particular IP address.
Figure 19. Additional threat context for one IP address

The browser provided by MSTICPy can also be used to explore the IOCs previously enriched. The interactive Jupyter notebook includes this view of the IOCs.

A screenshot of the browser provided by MSTICPy that can be used to explore IOCs extracted from the Conti chat logs. The browser shows threat intelligence details related to a selected IP address.
Figure 20. IOC browser provided by MSTICPy

In addition, MSTICPy has an embedded module that looks up the geolocation of IP addresses using Maxmind, which can be used to create a map of the IP addresses previously extracted.

A screenshot of Python code that looks up the geolocation of IP addresses. It also presents code that creates a map using the generated geolocation data.
Figure 21. Generating the IP geolocation map
A world map with the geolocation of all IPs extracted from the Conti leaks marked with red pins. The image shows that the location of IPs are concentrated in Europe and the US.
Figure 22. Geolocation of IPs extracted from the Conti leaks

Investigating URLs

Extracted URLs from IOC lists can provide details about targets, tools used to exchange information, and the infrastructure used to deploy attacks. A total of 1,137 unique URLs were extracted from the Conti leak dataset, but not all of them are usable for threat intelligence. The following code snippet shows how to filter for URLs.

A screenshot of Python code that filters for URLs among the IOC list.
Figure 23. Filtering the IOCs for URLs
A screenshot of the table generated by filtering URLs from the IOC list. The table includes the following data fields: IOC type, observable, source index, and input.
Figure 24. Sample of URLs extracted

A filter can be created to get details on executables, DLLs, ZIP files, and other files related to the extracted URLs. This can provide interesting insights and can be extracted for further research.

A screenshot of Python code that filters for specific file types related to extracted URLs. The code searches for URLs with .exe, .dll, .jpg, .zip, .7z, .rar, and .png files.
Figure 25. Filtering URLs for specific file formats
A screenshot of a table generated from filtering for URLs related to specific file formats. The table features the following data fields: IOC type, observable, source index, and input.
Figure 26. Sample of URLs delivering extracted file

Using the same technique for filtering, .onion URLs can also be identified from the URL list. This proved particularly useful in this case, since the Conti group used the Tor network for some of their infrastructure.

A screenshot of a table generated by filtering for .onion URLs from the Conti chat log IOCs. The table presents the following data fields: IOC type, observable, source index, and input.
Figure 27. Sample of extracted .onion URLs

Pivoting extracted IOCs using VirusTotal

The use of the pivot function within the MSTICPy library allows enrichment of data and discovery of additional infrastructure and IOC. This is particularly useful for threat intelligence and threat actor tracking. The next sections demonstrate the use of the VirusTotal module VTlookupV3 in MSTICPy to obtain intelligence about an IP address extracted from the Conti leak dataset that was used to deliver additional malware.

The following code initiates the VTlookupV3 in MSTICPy:

A screenshot of Python code that initiates the VirusTotal module in MSTICPy.
Figure 28. Configuring the VirusTotal module in MSTICPy

The VirusTotal module can be used to get data related to a particular IOC. The code below searches for files downloaded from a particular IP address from the Conti leak dataset:

A screenshot of Python code that uses the VirusTotal module in MSTICPy to look up files downloaded from a specific IP address.
Figure 29. Getting files downloaded from one IP address

The results show that the IP address 109[.]230[.]199[.]73 delivers several strains of malware.

A screenshot of a table generated from extracting the hashes of files downloaded from a specific IP address.
Figure 30. Hashes related to IP 109[.]230[.]199[.]73

The VirusTotal module can then be used to pivot and extract more information about these hashes. The table below shows information about the first hash on the list:

 Attributes
authentihash0d10a35c1bed8d5a4516a2e704d43f10d47ffd2aabd9ce9e04fb3446f62168bf
creation_date1624910154
crowdsourced_ids_results[{[TRUNCATED]’alert_context’: [{‘dest_ip’: ‘8.8.8.8’, ‘dest_port’: 53}, {‘dest_ip’: ‘193.204.114.232’, ‘dest_port’: 123}], ‘rule_url’: ‘https://www.snort.org/downloads/#rule-downloads’, ‘rule_source’: ‘Snort registered user ruleset’, ‘rule_id’: ‘1:527’}, {‘rule_category’: ‘not-suspicious’, ‘alert_severity’: ‘low’, ‘rule_msg’: ‘TAG_LOG_PKT’, ‘rule_raw’: ‘alert ( gid:2; sid:1; rev:1; msg:”TAG_LOG_PKT”; metadata:rule-type preproc; classtype:not-suspicious; )’, ‘alert_context’: [{‘dest_ip’: ‘107.181.161.197’, ‘dest_port’: 443}], ‘rule_url’: ‘https://www.snort.org/downloads/#rule-downloads’, ‘rule_source’: ‘Snort registered user ruleset’, ‘rule_id’: ‘2:1’}]
crowdsourced_ids_stats{‘info’: 0, ‘high’: 0, ‘medium’: 2, ‘low’: 1}
downloadableTRUE
exiftool{‘MIMEType’: ‘application/octet-stream’, ‘Subsystem’: ‘Windows GUI’, ‘MachineType’: ‘AMD AMD64’, ‘TimeStamp’: ‘2021:06:28 19:55:54+00:00’, ‘FileType’: ‘Win64 DLL’, ‘PEType’: ‘PE32+’, ‘CodeSize’: ‘115712’, ‘LinkerVersion’: ‘14.16’, ‘ImageFileCharacteristics’: ‘Executable, Large address aware, DLL’, ‘FileTypeExtension’: ‘dll’, ‘InitializedDataSize’: ‘69632’, ‘SubsystemVersion’: ‘6.0’, ‘ImageVersion’: ‘0.0’, ‘OSVersion’: ‘6.0’, ‘EntryPoint’: ‘0x139c4’, ‘UninitializedDataSize’: ‘0’}
first_submission_date1624917754
last_analysis_date16365918529
last_analysis_results{ [TRUNCATED] ‘20211110’}, ‘Tencent’: {‘category’: ‘undetected’, ‘engine_name’: ‘Tencent’, ‘engine_version’: ‘1.0.0.1’, ‘result’: None, ‘method’: ‘blacklist’, ‘engine_update’: ‘20211111’}, ‘Ad-Aware’: {‘category’: ‘malicious’, Edition’: {‘category’: ‘malicious’, ‘engine_name’: ‘McAfee-GW-Edition’, ‘engine_version’: ‘v2019.1.2+3728’, ‘result’: ‘RDN/CobaltStrike’, ‘method’: ‘blacklist’, ‘engine_update’: ‘20211110’}, ‘Trapmine’: {‘category’: ‘type-unsupported’, ‘engine_name’: ‘Trapmine’, ‘engine_version’: ‘3.5.0.1023’, ‘result’: None, ‘method’: ‘blacklist’, ‘engine_update’: ‘20200727’}, ‘CMC’: {‘category’: ‘undetected’, ‘engine_name’: ‘CMC’, ‘engine_version’: ‘2.10.2019.1’, ‘result’: None, ‘method’: ‘blacklist’, ‘engine_update’: ‘20211026’}, ‘Sophos’: {‘category’: ‘malicious’, ‘engine_name’: ‘Sophos’, ‘engine_version’: ‘1.4.1.0’, ‘result’:
last_analysis_stats{‘harmless’: 0, ‘type-unsupported’: 6, ‘suspicious’: 0, ‘confirmed-timeout’: 1, ‘timeout’: 0, ‘failure’: 0, ‘malicious’: 47, ‘undetected’: 19}
last_modification_date1646895757
last_submission_date1624917754
magicPE32+ executable for MS Windows (DLL) (GUI) Mono/.Net assembly
md555646b7df1d306b0414d4c8b3043c283
meaningful_name197.dll
names[197.dll, iduD2A1.tmp]
pe_info[TRUNCATED] {‘exports’: [‘StartW’, ‘7c908697e85da103e304d57e0193d4cf’}, {‘name’: ‘.rsrc’, ‘chi2’: 51663.55, ‘virtual_address’: 196608, ‘entropy’: 5.81, ‘raw_size’: 1536, ‘flags’: ‘r’, ‘virtual_size’: 1128, ‘md5’:, ‘GetStringTypeW’, ‘RtlUnwindEx’, ‘GetOEMCP’, ‘TerminateProcess’, ‘GetModuleHandleExW’, ‘IsValidCodePage’, ‘WriteFile’, ‘CreateFileW’, ‘FindClose’, ‘TlsGetValue’, ‘GetFileType’, ‘TlsSetValue’, ‘HeapAlloc’, ‘GetCurrentThreadId’, ‘SetLastError’, ‘LeaveCriticalSection’]}], ‘entry_point’: 80324}
popular_threat_classification{‘suggested_threat_label’: ‘trojan.bulz/shelma’, ‘popular_threat_category’: [{‘count’: 22, ‘value’: ‘trojan’}, {‘count’: 6, ‘value’: ‘downloader’}, {‘count’: 2, ‘value’: ‘dropper’}], ‘popular_threat_name’: [{‘count’: 6, ‘value’: ‘bulz’}, {‘count’: 6, ‘value’: ‘shelma’}, {‘count’: 3, ‘value’: ‘cobaltstrike’}]}
reputation0
sandbox_verdicts{‘Zenbox’: {‘category’: ‘malicious’, ‘sandbox_name’: ‘Zenbox’, ‘malware_classification’: [‘MALWARE’, ‘TROJAN’, ‘EVADER’]}, ‘C2AE’: {‘category’: ‘undetected’, ‘sandbox_name’: ‘C2AE’, ‘malware_classification’: [‘UNKNOWN_VERDICT’]}, ‘Yomi Hunter’: {‘category’: ‘malicious’, ‘sandbox_name’: ‘Yomi Hunter’, ‘malware_classification’: [‘MALWARE’]}, ‘Lastline’: {‘category’: ‘malicious’, ‘sandbox_name’: ‘Lastline’, ‘malware_classification’: [‘MALWARE’]}}
sha1ddf0214fbf92240bc60480a37c9c803e3ad06321
sha256cf0a85f491146002a26b01c8aff864a39a18a70c7b5c579e96deda212bfeec58
sigma_analysis_stats{‘high’: 0, ‘medium’: 1, ‘critical’: 1, ‘low’: 0}
sigma_analysis_summary{‘Sigma Integrated Rule Set (GitHub)’: {‘high’: 0, ‘medium’: 0, ‘critical’: 1, ‘low’: 0}, ‘SOC Prime Threat Detection Marketplace’: {‘high’: 0, ‘medium’: 1, ‘critical’: 0, ‘low’: 0}}
size181248
ssdeep3072:fck3rwbtOsN4X1JmKSol6LZVZgBPruYgr3Ig/XZO9:fck3rwblqPgokNgBPr9gA
tags[assembly, invalid-rich-pe-linker-version, detect-debug-environment, long-sleeps, 64bits, pedll]
times_submitted1
tlshT110049E14B2A914FBEE6A82B984935611B07174624338DFEF03A4C375DE0E7E15A3EF25
total_votes{‘harmless’: 0, ‘malicious’: 0}
trid[{‘file_type’: ‘Win64 Executable (generic)’, ‘probability’: 48.7}, {‘file_type’: ‘Win16 NE executable (generic)’, ‘probability’: 23.3}, {‘file_type’: ‘OS/2 Executable (generic)’, ‘probability’: 9.3}, {‘file_type’: ‘Generic Win/DOS Executable’, ‘probability’: 9.2}, {‘file_type’: ‘DOS Executable Generic’, ‘probability’: 9.2}]
type_descriptionWin32 DLL
type_extensiondll
type_tagpedll
unique_sources1
Vhash115076651d155d15555az43=z55

The results indicate that the hash is a Cobalt Strike loader, which means that Conti affiliates also use the penetration testing tool as part of their infrastructure during their operation.

In addition, the VirusTotal module can also provide details such as detection rate, type, description, and other information related to the hashes. The code snippet below generates the list of domains to which the hashes connect to.

A screenshot of Python code that generates the list of domains specific hashes connect to using the VirusTotal module.
Figure 31. Getting contacted domains
A screenshot of a table generated from extracting domains to which certain hashes connected to using the VirusTotal module.
Figure 32. Additional domains retrieved from previously extracted hashes

Doing this kind of analysis on the Conti leak data or similar data sets can lead to the discovery of possibly related domains that were not in the initial data sets.

Conclusion

This blog outlines how Python can be used to find valuable threat intelligence from data sets such as chat logs. It also presents details on how processing data using the MSTICPy library can be useful for enriching and hunting within environments, as well as collecting additional threat context. The interactive notebook provides additional code snippets that can also be used to continue log exploration.

The types of information extracted in this blog provides insights into the various elements of the criminal ecosystem that were coordinating their activities. Threat intelligence from research like this informs products and services like Microsoft 365 Defender, translating knowledge into real-world protection for customers. More importantly, the methodology described in this blog can be adapted to specific threat intelligence services, and the broader community is invited to use it for further analysis, enrichment of data, and intelligence sharing for the benefit of all.

Thomas Roccia
Microsoft 365 Defender Research Team

References