Why would you use outdated IT infrastructure as part of a plan to modernize your water system infrastructure? Of course the answer is: you wouldn’t. Or at least, you shouldn’t. But that is what many utilities – water, gas, and electric – are doing when they use standard relational data management technology (invented around 1970) to tackle the immense challenges of handling data from today’s generation of smart meters and sensors.
For years customer data from water meters was recorded once a month, and consequently the demands on data systems were modest. At those rates standard relational database management systems (RDBMS) based technology was sufficient. Data volumes increase dramatically when the transaction rate increases from once a month to once every 15 minutes. That’s 3000 times more data. Experience shows that existing data management infrastructure won’t be adequate at these increased rates.
For instance, a 10 million meter network that transitions from monthly readings to readings every 15 minutes results in load of about 30 billion transactions a month. Data management infrastructure that worked well at lower volumes will likely fail to scale up smoothly to support these new data volumes and transaction rates. Faster hardware can’t solve the problem, as the inefficiencies of the relational database model become a serious hindrance to meeting service levels. Application and query performance is often degraded to the point it becomes intolerable.
Meters are only one of several sources of time-stamped data that utilities are now collecting. Many organizations have thousands of sensors being deployed throughout their systems to monitor flow, pressure, temperature, and other statistics. Some use cases require this data be recorded in intervals under a minute. How do you handle all this data cost effectively, without over spending on expensive server upgrades – that may or may not solve performance problems?
Optimizing for Time Series Data
Because data from smart meters and sensors is generated in consistent formats that include time stamps, a data management system can be optimized to store, retrieve and analyze it with respect to time intervals.
For example, IBM Informix software includes built in TimeSeries data management in addition to standard relational data management capabilities. One client comparing Informix TimeSeries to an existing Oracle Database in a one million meter proof of technology, saw data load times go from more that seven hours to under 20 minutes; report generation times go from 2-7 hours to 1-11 minutes; and the storage space required for the TimeSeries database to be about one third the amount required for the relational database. The business case that led the client to switch to Informix was based on the storage cost savings alone.
Published benchmarks have shown that the Informix TimeSeries technology provides significant performance and price advantages over industry standard database technology when applied to large smart meter systems. How significant? Five times as fast on hardware that costs one fifth as much.