Anomaly detection with event data in the Internet of Things
Everything all right with your devices?
Today, devices generate more data than social networks. Each device can send data several times per second, and with millions of connected devices, a typical data processing platform might be required to deal with billions of such incoming events every day. Even though processing this amount of data is obviously a considerable technological challenge, it is clear that the device data itself – even when stored in a preprocessed form – is not actionable. To get actionable insights, the collected data must be analyzed.
One type of task that can be effectively tackled with data analysis is anomaly detection. Its goal is to find unusual behavior that differs significantly from what has been observed before or from what is expected. In this white paper, we describe our approach, highlight some observations made during various projects, and have a look at the components that a system for data analysis typically includes:
- Data preprocessing: This component module is designed to solve many problems such as data cleansing and the generation of domain-specific features.
- Data analysis: The main job of this component is to find anomalies in the input data. The challenge is to choose an appropriate data mining algorithm and to fine-tune its parameters.
- Data visualization: Here the main task is to visualize the result for the end user as well as to provide a means of visual analysis. The challenge is to choose visual techniques that are appropriate for the task being solved and the problem domain.