Streaming a new class of data into Splunk – Introducing the Splunk App for Stream

Last year in December, we announced the acquisition of Cloudmeter – a company with technology that captures data directly from the network traffic – a rapidly growing source of big data.

Today, I’m stoked to announce the general availability of the Splunk App for Stream v6.0, which stems from that acquisition.

So, why is wire data (data from the network) important? Wire data has the benefit of capturing all data in real-time – it is the communication vehicle for applications and systems to talk to each other, making it a very authoritative source of critical information. It serves a broad range of analytics across different use cases; it is non-intrusive with no impact to workloads and it can be collected without the need for instrumentation and tagging of applications.

The Splunk App for Stream passively captures and sends wire data into the Splunk platform. We’ve always been able to capture wire data. What we’re introducing with the Splunk App for Stream is a simple and elegant mechanism to capture the data. Let me explain why I think this approach is unique and absolutely fantastic.

Delivered as a software solution: This matters because it is often challenging to get wire data from on-premise services and applications e.g. web/email/application services and this problem is compounded in cloud implementations. There are no taps or span ports that can be leveraged. This capability to rapidly deploy wire data collection as software from end points delivers real time network visibility that is otherwise unavailable from cloud implementations and traditional datacenters. For instance, a lot of data compromise happens at end points, which can now be easily monitored or captured on-the-fly. Operationally, IT teams have better control of what is happening in public/hybrid Cloud infrastructures with the ability to deploy this software across all end points in any Cloud service easily. Pretty huge – don’t you think?

Flexibility in deployment: The App can be deployed as an ultra-light, non-intrusive agent to tap into network streams, which clearly makes it cloud friendly (already spoke about this above). Alternatively, the App can also be deployed as an appliance that sits on the mirror/SPAN ports of hardware switches and parse and collect data as it arrives.

Data collection customizability and data volume control: Wire data is voluminous. So, you may be concerned about what this means to your Splunk instance/license. To address this, we’ve included some pretty cool features in the App that provides you a lot of flexibility in configuring and customizing what you want to capture. With the powerful interface, you can define fine-grained filters on protocols and attributes, customize streams and create filters on streams and even aggregate data on the fly. You can whitelist/blacklist IP addresses and subnets. This provides you the ability to capture what is most critical, from endpoints or subnets most important. And of course, control data volumes.

Interface driven deployment and scale-out: The App, being a software solution, can be deployed where you want it when you want it. Let’s say you have some kind of infiltration or malware attack and you want to quickly collect data for forensics. With this App, you can pretty much do this on-the-fly through the interface. You can also manage the deployment of your App across your network with the Splunk deployment server.

Enhances OI with correlated insights: Wire data definitely has a lot of potential, especially around Splunk’s core use-cases – App Mgmt, IT Operations, Security and Business Analytics. When combined with other machine data such as logs, events and metrics, you can gain in-depth insights into performance, availability and usage and enable end-to-end insights across your critical business processes, systems and applications. And Splunk is uniquely positioned to do this.

We have proven beyond doubt that we’re the leading Big Data platform for Operational Intelligence. With the addition of wire data, this only elevates our status. Bernd Harzog, analyst from The Virtualization Practice and CEO and founder of APM Experts, wrote a really nice article about this release and how we’ve now extended the capabilities of the Splunk platform with the addition of wire data. Thanks Bernd.

Wondering how to get started. This is free folks! This App can be downloaded on Splunk Apps right away.

Before I sign off, I also want to introduce you to the Streams Examples App. In order to get you started on how to explore wire data, captured by the Splunk App for Stream, we’ve also authored a Stream Examples App that you can download from Splunk Apps. This App contain searches, examples and instructions for how to enable several use cases using data captured by the Splunk App for Stream. It will cover scenarios like looking for security relevant conversations, looking at a Web Shopping Cart for funnel analysis, using full payload data to track Shopping Cart revenue, analyzing SIP conversations, looking at Application and Database performance metrics, and more. Once you install the App for Stream, I highly recommend you download this as well – it will accelerate the value you get from the data captured by the App for Stream.

Streaming a new class of data into Splunk – Introducing the Splunk App for Stream | Splunk (2024)

References