OwnerJuly 2015 to presentEl Cerrito

Finding best available technologies for meeting energy needs today and tomorrow: energy efficiency, demand response,, solar, wind, electric vehicles, biofuels and smart grid. It’s all the innovations that make the energy we use more secure, clean, and affordable. The energy world's best hopes lie in what's happening in the digital realm, especially in data analytics.

Friday, October 23, 2015

Smart Meter Data Management

Now that utilities have millions of smart meters in place collecting data in near-to-real-time, the question of how to manage terabytes and petabytes of information will need to be solved. The software killer apps for Smart Grid operations at both utility and regional grid levels will be found in data analytics solutions.

Navigate this Report
Back to Consumer Index

Back to Smart IT Index
1. Background

2. Acronyms/Definitions
3. Business Case
4. Benefits
5. Risks/Issues
6. Success Factors
7. Companies/Organizations
8. Links

  • In May 2011, total U.S. smart meter installations passed the 20-million meter mark. In 2013, U.S. electric utilities had 51,924,502 advanced (smart) metering infrastructure (AMI) installations.  About 89% were residential customer installations.. Now that utilities have millions of smart meters in place collecting data in near-to-real-time, the next question is be how to manage all that information.

  • There are a number of devices in addition to smart meters being used in the energy and utilities industry to collect data, including line default detectors, sagometers which generate 12 readings per hour at 50 bytes per read and storage devices such as batteries that produce 100 byte reads per hour. A synchrophasor- a phasor measurement unit that tracks electrical waves across the power grid to monitor the health of the system (see my blog article Phase Synchronizer) - takes readings sixty times a second. This adds up to 494 megabytes a day,  176 gigabytes of data a year per synchrophaser. Together, these devices create an astronomical amount of data.

  • To meet that need, utilities have turned to a number of companies offering meter data management services – software that can integrate the new flood of data with utility systems for billing, customer account management and a host of new uses that are expected to emerge.

  • There will be Terabytes, (1024 GB) and Petabytes (1024 TB) of information to manage. Merely moving from monthly kilwatt-hour reads to hourly interval meter reads increases data handling requirements more than 730 times. There is also additional rich information available in smart meters including: amps, volts, watts, vars, total harmonic distortion, and momentary interruptions.

  • As an early example, in Austin Energy's Smart Grid 1.0 , phase one roll-out which included 500,000 meters, the data storage went from 20 TB to 200 TB, with disaster recovery redundancy. This is for 15 minute sampling, and first stage (home-oriented) integration. Ignoring shorter sampling frequencies (resulting in much higher data storage) necessary for some Smart Grid functionality, this presents a model of about 400 MB per meter per year. A lot of utilities don't have resources to manage the type of volume of smart meter information.

  • With Time of Use pricing and user charge recovery for power generated, a sizable subset of this data will no longer be simply transient and used in the aggregate. Individual elements will need to be captured and tagged for later retrieval over whatever period is chosen by regulators as appropriate for looking back.

  • Analysis may be done on an extremely large scale so algorithms must be automated.

Uploaded by on Jan 25, 2010
Utilities face a coming wave of data beyond anything they have ever experienced. This one-hour Webinar will bring in some of the worlds most experienced professionals to explain how to prepare for, and benefit from, the coming data surge. For more go to

2. Acronyms/Definitions
  1. Access Control - The goal of access control is to prevent the unauthorized use of HAN resources. Access control includes resource control; for example, preventing logon to local HAN Devices. For the purposes of smart grid standards, access control is not concerned with denying physical access. Access control is applied to an entity based on an identity or an authorization. An identity may represent an actual user, a process with its own identity (e.g., a program making a remote access connection), or a number of users represented by single identity (e.g., role-based access control).

  2. Accountability – A special type of non-repudiation, in that the accountability security service is basically holding each network entity responsible for its actions on that network.

  3. Audit Functionality - A critical element of the layered defense strategy for a system. Audit contributes to user and device accountability by recording security critical user actions while using HAN services. Audit also contributes to domain boundary enforcement services by recording activities of HAN services related to proper operation of security critical functions. In addition to auditing users and system activities, audit must be able to monitor the status of audit data to ensure its integrity and accuracy.

  4. Anonymity - A service which prevents disclosure of information which leads to the identification of the source or end-user of the information.

  5. Confidentiality - The security services, which prevent unauthorized disclosure of data (both stored and communicated). Confidentiality services prevent disclosure of data in transit and data at rest. Confidentiality services also include anonymity. Because of its role in limiting authorized disclosure of information, confidentiality services are closely linked with access control services.

  6. Data Logging – Compiling an hour-by-hour record of energy use.

  7. Green Button - a feature that allows residential and commercial customers to download detailed energy-use information in a standardized format to better manage electricity consumption and cost.

    Green Button is an industry-led effort that responds to a White House call-to-action: provide electricity customers with easy access to their energy usage data in a consumer-friendly and computer-friendly format via a "Green Button" on electric utilities' website. Green Button is based on a common technical standard developed in collaboration with a public-private partnership supported by the Commerce Department's National Institute of Standards and Technology. Voluntary adoption of a consensus standard by utilities across the Nation allows software developers and other entrepreneurs to leverage a sufficiently large market to support the creation of innovative applications that can help consumers make the most of their energy usage information. Initially launched in January, utilities committed to provide Green Button capability to nearly 12 million households in 2012. Two utilities - Pacific Gas & Electric and San Diego Gas & Electric - have implemented live functionality on their websites. Numerous companies are already developing Web and smartphone applications and services for businesses and consumers that can use Green Button data to help consumers choose the most economical rate plan for their use patterns; deliver customized energy-efficiency tips; provide easy-to-use tools to size and finance rooftop solar panels; and conduct virtual energy audits that can cut costs for building owners and speed the initiation of retrofits.

  8. Grid Metadata - Data about the data that provides context, such as network models and topology that show what is connected to what in what order) Grid metadata is famously inaccurate as much as 20% to 50% inaccurate. We can fix some of the metadata problems through technology. But much of the solution depends on people. The biggest metadata challenge is "built versus as operated." A substation might be built one way, then changed and operated another. With the smart grid bringing constant additions and upgrades to all parts of the system, the situation has become very fluid and dynamic. Not only that, but with dynamic feeder circuit switching, topology changes can occur quite quickly under either normal or stress conditions on the grid.

  9. MDMS – Meter Data Management System - Performs long term data storage and management for the vast quantities of data that are now being delivered by smart metering systems. This data consists primarily of usage data and events that are imported from the head end servers that manage the data collection in Advanced Metering Infrastructure (see my AMI article) or Automatic meter reading (AMR) systems. An MDM system will typically import the data, then validate, cleanse and process it before making it available for billing and analysis.

  10. MIU – Meter Interface Unit - Captures and stores usage data over a period of time

  11. MRE - Meter Reading Export - A file that contains completed meter reading route information that is transmitted from the meter reading host processor back to the mainframe computer.

  12. MRI - Meter Reading Import - A file that contains route information that is transmitted from the utility mainframe computer to the meter reading host processor. This file contains customer and meter input records for each account to be processed. When the MRI file is processed into routes, blank customer and meter output records are created for each account. As the meter reader collects meter reading information during the day, these output records are filled in, uploaded to the IHP, and the MRE file is created.

  13. Registration - The registration and authenticating requirements are used in conjunction with most other security services. That is, the first step of most security services is to determine the identities of one or more of the parties participating in an action. A trusted identity must be used for access control decisions and to provide accountability evidence. Knowing the identity of an entity and the existence of a peer relationship is also fundamental to establishing communication with confidentiality and integrity. If the identity of the peer in a secure communications path is not properly established, it leaves open the possibility that an unauthorized principal (an adversary) could masquerade as an authorized principal, exposing the data to disclosure or manipulation

  14. VEE - Validation, Editing, and Estimation of meter data.

Meter Data Management Flowchart including WiMax, 3G and Mesh Network paths.   Source: AusNet Services

3. Business Case
  • The need for higher speed enterprise communications and Smart Grid opens Pandora’s Box of data. Huge quantities of otherwise untapped data will be available. However, this data won’t be reach its full potential without the ability to collect, store and provide data independent analytics
  • Meter Data Management Features
    • Connections to meter systems
    • Built in out-of-the-box validations
    • Critical Validations
    • Usage Validations
    • Error Handling
    • Estimations
    • Custom Business Rules
    • Meter Data Access
    • Meter Inventory Management
    • Revenue Protection
    • Common Meter Data Repository
    • Supports Open Technologies
    • Auditing
    • Reporting
  • Application Areas
    • Balance Analysis
    • Profitability Analysis
    • Product Development
    • Flexible Portfolio Analysis
    • Demand Side Management
    • Energy Settlement
    • Unbilled Revenue
    WiMax and 3G Roll-out View  Source: AusNet Service and Bitstew

    4. Benefits
    • Interoperability - Collaboration between companies for energy delivery - The re-shaping of utilities’ organizational boundaries is a growing, global trend, and these new models require information systems with a higher degree of flexibility. A common platform integrates process and information flow between companies and supports the unique operations and information needs of each entity.
    • Comparative Context to Motivate People to Act - If all I know is my own energy consumption then I don’t know if that is good or bad. I reduced my usage by 10%, but still could be using more than average neighbor.
    • Improved Customer Service – Processing smart meter interval data as well as events such as new customer registrations, and communicating them in real‐time or as needed to market participants enables higher levels of customer service for fundamental operations such as customer service requests, connection/disconnection and billing.
    • Conservation - Encourages behavioral change among customers by helping them understand how their energy consumption levels impacts not only their wallets, but also the environment.
    • Asset Management - Ability to match asset parameters with observed readings and expected load patterns for effective asset utilization (increased reliability), perform condition-based and predictive maintenance (postpones expensive replacements), plan future enhancements and grid optimizations (enables decision making). Including real-time status/events and GIS data with this enables visualizations that provide additional insights for managing distributed grid assets.
    • Demand Response - Analyze customer load profiles, consumption patterns, demographic and weather information to match customers with the DR programs they are likely to participate in and to design new DR programs that are better aligned to customer demand patterns. Also determine the success of DR programs in load shedding and load shifting by analyzing customer behaviors and patterns during DR events. Being able to do this can help utilities manage and predict load better and reduce the need to purchase/generate costly ‘peak’ power.
    • Identify Revenue Leakage (Theft) - Ability to correlate reads, alarms and customer profile information to detect theft and revenue leakage. For example, use frequent power off/power on meter alarms with consecutive reading gaps and/or static reads from the premise to flag a potential theft event.
    • Target Education - Use consumer usage patterns and post DR event analysis to provide targeted education to customers on energy efficiency and conservation goals.
    • Dynamic pricing - Use customer consumption patterns, weather and demographic data to segment customers and offer dynamic pricing programs and tailored energy efficiency initiatives.
    • Distributed generation management - Ability to analyze the combined effect of existing generation resources and distributed energy resources (such as intermittent renewable power sources) to determine when/if peaking generation plants or virtual power plants need to be brought online. Using this information in near real-time provides the ability to switch between distributed generation sources based on demand so that utilities achieve the lowest cost of power while reducing the carbon footprint.

    ICT Fault View - Source: AusNet and Bit Stew 

    5. Risks/Issues
    • Data Storage - The massive amount of data generated by smart-grid technology could itself pose a practical problem. Right now, a utility with five million meters has about 30,000 devices for monitoring the grid. As the smart grid develops, that number could increase a thousandfold, with each device conveying a thousand times as much information as one of its counterparts does now, says Erik Udstuen, a general manager at GE Fanuc Intelligent Platforms. Though so much data may be difficult to process, it could also create opportunities for entrepreneurs to develop new monitoring applications, especially if open standards are developed.
    • Missing Data - One of the triggers for the Bakersfield Smart Meter controversy was the story of when the power went out at Tim Vanderhorst's house in east Bakersfield for almost six hours on April 20. When the lights came back on, was his computer showing that his electricity usage had tripled during that time When a SmartMeter does not transmitting a signal as it should, the utility's computer system automatically filled in the blanks with data patterned after his past power usage, or that of customers like him. While this glitch did not affect billing, it undermined consumer confidence in the system. According to TURN, this does not instill confidence in a system designed to give customers accurate and timely information about their own power usage
    • Insecurity of Existing Meter Communication Standards - In C12.18 the password is sent unencrypted. This is not a problem for a point-to-point connection from a handheld device to an optical port on a meter, but when data is transmitted over the Internet, it’s a serious security problem. A hacker with a computer loaded with freely available packet sniffing software can look at the packet and see the password. C12.21 has an alternative authentication mechanism, which provides for encrypted authentication, but the encryption is only used for authentication and all subsequent data reads and writes are done unencrypted. With a point-to-point connection via dialup telephone lines, this is not a problem because intercepting such communications is very difficult, but if the data is transmitted over the Internet, we once again have a problem because hackers can easily intercept the data. In both of these cases, the problem created is that a hacker, having successfully monitored the login, can now freely interfere with the meter and the automated meter reading (AMR) system. C12.22 will enable data encryption without requiring it, so no additional communications resources are used until encryption is actually used.
    • Data Integration - Utility mergers, open energy markets, and the integration of alternative supply into the grid are driving many utilities to adopt new business models in the transition to Smart Grid. With the responsibility of managing operations that span companies, borders and regulatory bodies, utilities are often required to link multiple CIS, logistics and AMI systems, each with differing process flows and data models.
    • Data Retention - What Smart Grid data to keep, what to archive, how long to archive the data, which data should be used for analytics, and which data needs to be maintained to meet security, data privacy and legal requirements. Failure to meet data security, privacy and legal retention requirements can lead to costly fines. That said, keeping massive quantities of data for long periods of time also comes at a cost.
    • Physical Infrastructure - IT eqiupment that is put in the field needs to be hardened to protect it from the electro-magnetic environment in which it will operate.

    Mesh Pilot Overlap - Source: AusNet Service and Bit Stew

    6. Success Factors
    • Expand data storage capabilities - The types and volume of data associated with Smart Grid use will mean a new need to bring Internet-style data centers into the complex mesh of utility control systems.
    • Data Analysis - A rigorous understanding of the importance of each piece of data to Smart Grid business processes.
    • Creative use of Data Storage Technologies - including virtualization, data de-duplication, multi-tiered archiving, and data encryption.

    ICT Dashboard  -  Source: AusNet and Bit Stew

    7. Companies/Organizations
    1. Aclara Software - Wellesley, MA - Aclara MDMS -  TWACS® (Two-Way Automatic Communications System) and eTWACS® utility networks and its iiDEAS® (Intelligent Infrastructure: Data Efficiency, Analytics and Services) operational data management platform, Aclara takes AMI beyond the meter with a comprehensive set network and software tools that provide control and visibility across your power-distribution network.

    2. Bitstew - Burnaby, ON, Canada -  By uniting centrally managed data sources with connected devices across a utility’s infrastructure, Bit Stew’s MIx Core platform and MIx Director application combine real-time data management, analytics and rich visualizations. The Mlx solutions help utilities increase awareness, discover new business insights, improve operations and asset performance. Information is unified into a common data model, providing an enterprise-wide foundation for deploying powerful applications that give control back to the operator and enable utilities to unlock the potential of the Industrial Internet of Thing

    3. Siemens, Berlin and Munich, Germany - In December 2011, eMeter, San Mateo, CA was acquired by Siemens (NYSE: SI)

      The eMeter platform is being used in a large number of smart grid deployments around the world. The firm will join the Smart Grid Division of the Siemens Infrastructure Group within Siemens Industry in the United States.

      Siemens Energy, which was an investor in eMeter, said that eMeter's software will be an integral piece to its product line aimed at utilities. "Ever-increasing demand for solutions to improve the effectiveness of the smart grid for cities and utilities makes this acquisition even more important. eMeter is renowned for its superior software and services capabilities that enable electric, gas and water utilities to realize the full benefits of the smart grid," Jan Mrosik, CEO of the Smart Grid division of the Siemens Infrastructure & Cities Sector said in a statement.

      eMeter's software is designed to collect data from two-way meters and feed it into utilities' back-end business systems. The venture capital-backed company is headed by former Oracle executive and Veritas CEO Gary Bloom.

      eMeter is also building analysis software that processes reams of data two-way meters and other grid equipment for business reports and analytical applications. The company this year introduced a cloud version of its EnergyIP meter data management software with Verizon.

      EMeter  received more than $70 million in venture capital funding from Sequoia Capital, Foundation Capital, and others.

    4. EnergyICT a subsidiary of Elster (NYSE: ELT) Customers include DTE Energy (2.8 million electric meters) and NSTAR (Netherlands).

    5. Ferranti Computer Systems - Antwerp, Belgium - Product: MECOMS 

    6. Itron - Liberty Lake, WA, (Nasdaq: ITRI ) - Itron Enterprise Edition Meter Data Management is an enterprise-wide data management solution for interval, register and event data for residential and C&I customers. It is a scalable, open-architecture system that manages data from many different collection systems. It also provides secure, accurate, reliable data to a wide array of utility billing and analysis systems.  Several big customers in the US, including Dominion (C&I customer base), Georgia Systems Operations Corp., Pepco Holdings, Seattle City Light, and Xcel Energy (over 5 million meters).

    7. Metersence - Ottawa Ontario - Meter Sense Over 20 utilities in North America, including Peterborough Distribution (30,000 smart meters in Ontario, Canada), Tillsonburg Hydro (Ontario, Canada), and Groton Utilities (Connecticut).

    8. Oracle – Redwood Shores, CA, (Nasdaq: ORCL ) - Oracle Utilities Meter Data Management is a commercial off-the-shelf application with built in out-of-the-box functionality to support the loading, validation, editing, and estimation (VEE) of meter data. It is designed for the highest levels of automation and scalability to meet the current and future needs of utilities. Bought utility meter data management software company Lodestar Corp. in 2007. and SPL WorldGroup which makes revenue and operations management software for utilities in 2006. Over 15 MDM deployments, including US multi-utility Modesto Irrigation District (100,000 smart meters), Lakeland Electric (US; 125,000 smart meters), Acea Distribuzione (Italy).

    9. OSIsoft San Leandro, CA Product: PI - System Consumer Energy (AMI pilot), Xcel Energy (Boulder SmartGrid City pilot).

    10. SAP - SAP software helps optimize warehouse and workforce processes for device installation and meter maintenance. Streamlined planning and execution processes help operations managers install meters and devices more quickly and with less effort. With on-time receipt of high-quality data from field devices, teams in marketing, sales, and customer service can create personalized offers that attract customers,  gain higher satisfaction levels and stronger revenue streams.

    11. Schneider Electric  - In 2011, Telvent  and its Meter Data Management products were acquired by the Schneider Electric group.     Schneider Electric’s Conductor MDM™ Solution boosts traditional meter data management functionality with robust data analytics and operational data storage/management capabilities – making it a comprehensive solution

    Business Operations Dashboard - Source: AusNet and Bit Stew

    8. Links
    1. Big Data, the Next Frontier for Innovation, Competition and Productivity - McKinsey Global Institute May 2011 - Looks at the vast amount of enterprise information that exists, and the challenges that organizations will face in trying to manage it. The report explores topics such as the state of digital data and how organizations can use large data sets to create value.
    2. – Meter Data Management
    3. Greentech Media - Meter Data Management

    Smart Governance

    Smart Meter Data Management
    friday, october 23, 2015
    Now that utilities have millions of smart meters in place collecting data in near-to-real-time, the question of how to manage terabytes and petabytes of information will need to be solved. The software killer apps for Smart Grid operations at both utility and regional grid levels will be found in data analytics solutions. 

    True Cost of IT
    COMING SOON, september, 2015
    Business management and IT have difficulty on proper allocation strategy of IT costs 

    IT Governance

    Data Governance
     wednesday, august 26, 2015
    A business process like Finance and HR

    Project Governance

    Cloud Governance

    Mobile Governance

    Big Data Governance
    coming soon, AUGUST, 2015
    As data has become less transactional and more interactional, the volume of that data has wildly increased. 

    Change Management

    Project Management

    Enterprise Architecture


    Curriculum Vitae Page

    Mark Miner
    529 Seaview Drive
    El Cerrito, CA 94530
    (510) 219-4064

    Skills - Here's my skills cloud from my friends on Linkedin

    Advanced Energy Economy Classes

    Neural Energy Consulting

    July 2015 – Present (4 months) El Cerrito

    Finding best available technologies for meeting energy needs today and tomorrow: energy efficiency, demand response,, solar, wind, electric vehicles, biofuels and smart grid. It’s all the innovations that make the energy we use more secure, clean, and affordable.

    Miner Energy Consulting

    May 2013 – July 2015 (2 years 3 months) El Cerrito, CA

    HERS Rating Service with Greenpoint Rating, T24 Energy Modeling and Project Prioritization and Management.

    The initial target market will be local home owners, local small business owners, contractors and real estate agents, Future services could include: training, supply chain analysis.

    Environmental Quality Committee
    El Cerrito
    April 2008 - Present ( 7 years, 7 months)

    The EQC goals are to:
    1. Promote community and individual action and engagement
    2. Involve learners of all ages
    3. Help businesses and residents be more sustainable and environmentally friendly
    4. Inform and champion the City’s environmental policies and ordinances
    5. Reduce greenhouse gas emissions
    6. Change material use patterns by encouraging Reduction, Reuse, Recycling & Composting
    7. Champion energy efficiency, renewable energy generation, and efficient transportation
    8. Minimize hazardous chemical use and human/environmental exposures to toxics
    9. Protect & create open green space, maintain and restore natural areas with an emphasis on native ecosystems, and encourage a network of human/wildlife friendly green habitat throughout developed areas
    10. Support sustainable water use, healthy creeks, and green infrastructure

    Chair Environmental Quality Committee

    El Cerrito
    Jan 2013 -  Apr 2015 (2 years 3 months)


    Recurring activities:
    • Organized semi-monthly Green Team volunteer work parties (including clean-ups and invasive plant removal), to help the City keep priority areas and waterways clear of trash, and maintain public spaces and parks.
    • Recognized volunteer efforts an annual volunteer appreciation party
    • Held numerous successful environmentally themed film and lecture events with guest speakers and audience interaction (approximately 2-4 per year)
    • Hillside Festival and Music for Madera now recurring events
    • After EQC lobbied for Community Choice Aggregation (CCA) options over several years, Council joined MCE as recommended
    • Assisted MCE with CCA outreach (energy film, Earth day and July 4th tabling)
    • Met our goal in the very successful $100k private fundraising campaign to help acquire the Madera open space connection for the Hillside Natural Area
    •  Reviewed and provided feedback on the Urban Greening Plan 
    • “Bee Haven” Pollinator Friendly Community ordinance 
    • Recommendation to Council to designate Priority Conservation Areas
    • Fundraising toward purchase of the Madera Open Space. Cosponsored Hillside Fest and the Music for Madera Festiva
    • Green living workshops on residential home energy efficiency and drought resiliency
    • Input to the San Pablo Avenue Specific Plan, (Adopted 2014)
    •  Participated in the Ohlone Greenway reopening celebration and an Earth Hour event
    • Supported policy development and hosted public forums for El Cerrito Climate Action Plan 
    • Supported policy development and hosted public forums for Single Use Plastic Bags and EPS Foam Take Out Containers ordinances

    IT Governance Director 

    APL Limited
    2003 –  (4 years)

    • Responsible for IT project portfolio including strategic alignment, financial performance, project management, risk diversification, and resource utilization. 
    • Developed and implemented company wide project management standards 
    • Deployed and managed Primavera project and portfolio management software 
    • Ensured that IT investments adhered to the architecture standards. 
    • Established an IT governance structure to review major investment decisions and track progress 
    • Developed an IT communication program, targeted to the appropriate level, to educate end-users on the key components of our IT systems and strategies. Closed gaps in employee satisfaction with communications.

    IT Project Director

    APL Limited
    2000 - 2003  (3 years)

    • Implemented CRM System. Delivered the first application utilizing the new web based Enterprise Architecture. Built cross-functional teams to address and solve many difficult issues. Managed implementation of Enterprise Architecture.

    • Managed New Horizons IT Strategy Project

    Previous Positions

    APL Limited
     –  (17 years)
    • Information Exchange Director – Developed and implemented e-commerce strategies that allowed customers to perform core business transactions with APL electronically.

    • Market Research Director – developed business plans for new markets

    • Marketing Manager – managed worldwide annual sales budget process and developed market plans giving direction to regional and country sales management

    • Sales Manager - Developed and implemented small shipments sales strategies including partnership with a national LTL trucking company

    • Marketing Information Manager - Owner of large data warehouse supporting several hundred users

    Success Stories -