Leveraging Big Data Concepts to Handle Disparate Data Sets

Big data can overwhelm water utilities. Smart grid solutions provide a holistic view into data.

Data abounds at water and wastewater utilities. So much so that operators would be overwhelmed if they had to sift through and interpret all of the data. It has been found that utilities often use less than 40% of the data they generate, leaving more than 60% untouched.

Raja Kadiyala

Raja Kadiyala

Unfortunately, operational patterns and insight is often lost by not holistically analyzing the complete set of data. When considering the complete data set available from maintenance activities in the CMMS, to real-time SCADA data, to regulatory data in the LIMS, to security data in the video surveillance system, to meter data in the CIS or AMR, it readily becomes a rather large task to assimilate all the data. In the end, we want to avoid inundating the operators with raw data, but rather have solutions in place to link data streams, interpret them, and then provide the operators insight and actionable information with which to respond to the condition.

With the problem at hand defined, let’s take a step back and see if we can utilize solutions from other areas. It is certainly the case that nearly every discipline from sports and advertising to public health and science rely on data-driven analysis for decision-making. Tag-lined the “Age of Big Data,” we are becoming more and more reliant on data-driven evidence and analysis for nearly every decision we make. Data is not only becoming more available to the general public, but also more understandable, thanks to computing resources and advanced algorithms for analytics. Some fundamental elements of Big Data processing are the ability to find relationships across a multitude of data, process the data into understandable ‘chunks’ that better map to business needs and can be readily consumed by humans, and to provide visualizations for more direct interpretation and understanding.

A key element in processing the Big Data within a water utility was the development of Event Detection Systems (EDSs). An EDS performs real-time analytics on a data stream to determine whether there is anomalous behavior. This analysis allows operators to focus on their daily activities, while the EDS is performing the analytical number-crunching and alerting the operators when needed.

Through the implementation of numerous Smart Grid solutions, we have been able to leverage Big Data methodologies to address the issues in dealing with disparate data sets. These solutions provide real-time integration across multiple systems, perform data analytics, and visualize the information with spatial and temporal context to provide insight and actionable information for operators. The real-time integration enables a concept that I refer to as the value of now. There is certain information that has a shelf life, much like items at a grocery store. The more time it takes for an operator to be notified of that information, the less value it has. As an example, notifying an operator of water quality issues a week after it happens does not provide much value, nor does providing information on an impending pump failure after the pump fails. By analyzing all the data at hand, patterns can be established to provide the necessary insight.

A side-benefit of gathering and monitoring all of the component data in real time is the ability to mine this data for details that aid in day-to-day operations. These benefits range from optimal filter media replacement based on true media loading to detection of oxidation (which will ultimately lead to premature pipe replacement). The savings realized through these discovered issues has been shown to outweigh the implementation cost of these solutions.

Information is knowledge, and with the help of Smart Grid technology, water utilities can more effectively utilize the abundance of data collected by their systems and turn the 60% of untouched data into information that yields operational efficiencies and cost savings. Dashboards that display relevant and contextual information and underlying data give water utility managers answers to complex issues across their water networks so they can make informed decisions. Better managing and monitoring water using a variety of data sources allows utilities to detect issues in real-time, mitigate risks, address security issues, and streamline operations.

Raja Kadiyala is currently serving as a Senior Technology Fellow and is the Global Service Leader for Intelligent Water Solutions Service Team for CH2M HILL.