A Beginner’s Guide to Integrating Threat Intelligence
In my last post, I outlined the value of automating the use of Indicators of Compromise. It occurred to me, after I had written the post, that I had started in the middle. I took it as a given that most have a source of threat intelligence, then talked about how to make more effective use of it, focusing on the end goal of automating as much as possible.
The problem with that approach is that there are a great many organizations out there that still do not make use of any sort of threat intelligence in their operations. Others may have a source of threat intelligence, but are not making use of it, or are not getting the value they could.
For some, it is simply a lack of understanding of how to choose a threat intelligence source or how to integrate that information into existing environments.
For this post, I am going to rectify that, outlining the basics of what to consider when selecting a source of threat intelligence and providing an outline of what steps are needed to integrate that data. I will also try to point out the choices that must be made and the considerations that should be kept in mind during the process.
Selecting Threat Intelligence Sources
Choosing a source for threat intelligence is critical. You need to select a source that is appropriate for your needs and your team’s capabilities. You can choose from free/open source feeds or you may purchase a feed from one of the several dozen vendors in the market today. There are well over a hundred free or open source intelligence feeds available. Many of the open source feeds get their indicators from the same sources and report on the same indicators, creating large areas of overlap and duplication of data, which must be managed.
There are dozens of paid feeds available as well. Each has their own areas of focus, and costs vary widely. Although the quality of paid feeds is often high, they can have a much narrower focus, requiring multiple feeds to get the breadth desired. Be aware--the cost of subscribing to multiple feeds can add up quickly. The challenge of managing integrations from multiple vendors must also be considered.
A third option is also available: threat intelligence platforms. Typically, these platform vendors offer feeds of their own original research as well as curating a large number of open source or other free feeds. The curation, enrichment, and deduplication that these services provide can raise the quality of “free” threat intelligence feeds considerably. These platforms are often packaged with a well-developed API (Application Program Interface) or other tool that simplifies integration of their feeds. Threat intelligence platforms also provide a knowledge base that analysts can use to do research and gain contextual information on indicators spotted in your environment. Though they can be expensive, these platform providers can offer a valuable stepping stone for organizations just getting started with threat intelligence.
Finally, when choosing sources, one of the most important considerations should be the methods supported for collecting and ingesting those feeds. Some are much more manual than others. A few offer integration agents that provide indicators as downloads of text or database files, while others support STIIX or TAXII for collection. Others simply post lists to a web page and allow users to scrape the page via scripts. Some feed vendors, along with the platform vendors mentioned earlier, offer a flexible, well-documented API. Whatever the collection and integration methods offered, make sure these methods can be supported in your current technology stack before you purchase anything.
Integrating Threat Feeds
Once you have selected your feed sources, it’s time to work out how to integrate this data. In this post, I am focusing on integration with SIEM or centralized log management systems. This is usually the first step for users, since it requires only one integration and can be used with logs from many different security technologies.
As I mentioned in my previous post, there are other types of integrations available as well. Depending on vendor support, there are direct integrations available for firewalls, proxies, DNS, EDR, web gateways, and IPS. Additional custom integrations are possible for any technology that supports making API calls.
When integrating with a SIEM or log management solution, the basic premise is to take a list of indicators from the threat intelligence feed, then check those indicators against the log messages coming from the various security technologies in your environment.
The three things you need to make this work are a way to ingest the indicators themselves, the logs that contain the information against which you wish to compare, and the SIEM resources (rules, reports, lookup tables) needed to make the comparisons.
The method you use to ingest indicators will vary according to your selected source of intelligence. If you are using primarily open source feeds, you may need to build a system to collect and process indicators into a list you can use for comparison. This approach may require scripts to fetch the feed data from a web page or to extract indicators from a local database after collection. If you select a paid feed or a platform provider, they may provide their own tools to collect and ingest the indicators they offer.
A word of warning: be sure that the vendor you selected has a means of updating and/or expiring indicators and that your system can support these updates. Without it, your indicator list will grow forever and your indicators, which change over time, will become “stale” and inaccurate. When it comes to threat intelligence, outdated or inaccurate information is worse than no information at all.
Once you have the indicators ingested, you need to examine the logs from your security devices to determine if they capture any information you can use to compare against indicators. The most common types of indicators in threat feeds are IP addresses, domain names, URLs, and malware hashes. Technologies that usually log this type of information include firewalls, IDS, DNS, proxies, EDR systems, web gateways, and mail servers.
It may sound obvious to some, but in order to compare against the indicators, your logs must contain at least one of those types of information. If your device does not include the required information by default, don’t give up. Many security vendors provide more detailed levels of logging that may include relevant information and some allow you to customize the logs, so you can build your log messages to include just what you need.
In most cases, the information you compare against must be parsed. What that means is that it can’t be part of a big blob of text in a log message. The information must be isolated and put into its own field. Parsing methods vary by vendor. Some vendors even parse messages automatically. Consult your SIEM vendor’s documentation for details.
Once we have our list of indicators and the parsed events with the information we need to compare them against, we must set up a mechanism for doing the comparison and a means of alerting analysts when a match exists. Some SIEM vendors include that mechanism out of the box. Others require you to build it yourself within their system. In general, the mechanism is a lookup table of some sort (every vendor calls them something different) and one or more rules to do the comparison. As an example, the rule might tell the system to compare the source and destination IP of every event passing through the firewall and check for matches against our indicator list. Domain names may also be useful when compared against an organization’s DNS or email message tracking logs. EDR and mail gateway systems may capture malware hashes that can be used for comparison.
Notifying analysts of a match is the final step in the integration. You can create something real-time like email alerts for each match but, depending on your sources and the volume of events you produce, that may result in a flood of notifications. For most shops, it makes more sense to start with a daily report of some sort that details the matches that occurred. This will give you a sense of how many matches you are getting and provide an easy way for analysts to follow up on the most interesting. Remember, matches do not necessarily represent attacks against your network. Investigation and vetting of these matches is required. If the list of matches is too big, try focusing it on the connections to and from critical resources or look at the highest severity indicators, however the vendor defines that. Some feed vendors will allow you to set thresholds in their system, so that low threat or low confidence indicators are never included in the first place.
This post is an introduction to integrating threat intelligence feeds into your environment. There is much more to learn and I encourage you to explore some of the links referenced at the bottom of this article.
Once you have a handle on the basics, begin to iterate. Much like threat intelligence gathering itself, you must continually review and improve your systems and analysis over time. For example, you may opt to enrich indicators as they come in, adding information that will be useful to analysts right into the log events themselves. Or you could subscribe to a threat intelligence knowledge base that provides context and speeds investigations. You may eventually choose to the route I outlined in my last post, automating responses to certain events that meet the appropriate criteria. Whatever enhancements you build into your process, the ongoing maintenance keeps your defenses up-to-date, which is essential to protecting your business.