I recently read an excellent summary of why cloud data councils are so valuable for anyone trying to tap the scalability and cost effectiveness of cloud computing. As Anil Nagaraj, Associate Vice President of Digital Engineering within Cognizant’s Artificial Intelligence & Analytics Practice, describes it, these councils work best when they focus on security and compliance, democratizing data access, collaborating across the business, sharing best practices, and creating and managing processes, workflows and metrics.

I see the power of such councils in my work moving large enterprise workloads to the cloud at Cloudera. Three or four years ago, only about 15% of our clients had such a council. Now, 30% to 40% do. Here are three areas where I believe data councils can provide the most value and deliver the greatest benefits.  

  1. Creating a common data model
    You can’t access, analyze or manage data you don’t know you have or can’t find. That’s why the first job of a cloud data council is to create a common data model that spans on-premise systems and multiple cloud platforms, organizing data elements and standardizing on how they relate to one another.

    While most organizations have an internal data catalog, they lack one for their cloud data, making it difficult to reliably identify the data or manage it. A cloud data council can gather representatives of all the enterprise’s users to catalog which data is most important to them, how it is organized and used by various business systems and common definitions for it. Such a common data model is the foundation for other critical work, such as ensuring the proper security, lineage and management of data. It’s also essential for eliminating data silos so enterprises can get the most comprehensive, real-time view of their businesses, customers and markets.

  2. Emphasizing security, compliance and audit
    A data council can help ensure that an organization’s in-house security accompanies their data to the cloud. From our experience, eight out of 10 banks have production data in on-premise systems that is hosted by multiple cloud providers. If the third column in a flat field of a database is a Social Security number, regardless of where the database sits, the bank still needs to properly secure and control access to that information.

    Just as cloud councils need to create common data models, they must create uniform security, identity management, access and audit across on-prem and cloud platforms. The good news is, there are open-source solutions that provide a common federated security model across both environments.

    Sometimes the data council needs to proactively help outside stakeholders adapt to the cloud. Two years ago, a large German bank asked us to help speed its move to the cloud but found European regulations governing IT operations didn’t consider the location of the workloads. The bank’s chief information security officer and chief data officer helped drive the effort to bring these regulations up to date.

  3. Scrutinizing lineage; applying governance
    Data is only as valuable as its accuracy and timeliness. You don’t want to make business decisions or financial trades on day-old data. Just as with a data model and security, your data council needs to extend your in-house lineage and governance processes and standards to the cloud.

    This is where your common data model can help by, for example, identifying the metadata that shows who created and modified data, and how old it is. It can also help you extend to the cloud the same processes used in-house to determine which tiers of storage should be used, or when to dispose of old data.

Best Practices for Full Data Council Benefits

To deliver these benefits, a data council must be a driver of innovation rather than a bureaucratic barrier to progress – and be recognized as such. Here are some of the ways we’ve seen data councils implement proper controls while meeting business challenges.

  • Include both business and technology stakeholders to get the input you need to agree on common access patterns, security, data governance and a data catalog.
  • Show quantifiable results. For example:
    • At one major car maker, the data council helped find and remove geographic details in the data it received from its vehicles that would have labeled the car maker as a terrorist organization to regulators in one of its markets.
    • At an oil producer, a data council helped identify and give priority routing to the data it needed for monitoring wellheads, diagnosing problems and shutting down the wells within the time needed to meet regulatory requirements.
    • Data councils can also reduce the up to 50% of IT budgets spent on “shadow” IT by individual business groups by offering effective solutions that meet both local needs and corporate requirements for security, governance and auditability.
  • Construct the right technology infrastructure. Based on our experience, this should include:
    • Automated data management and data warehouses that eliminate data and workflow silos to provide democratized, secure data access to speed analytics, reduce management costs and improve decision making.
    • The ability to ingest and analyze petabytes of data to provide predictive threat detection and historical investigation, using AI to identify threats not found via traditional security approaches. Users should be able to effortlessly explore and tag data through an intuitive search-based interface and use consolidated metadata and rich custom tags and comments to easily track, classify and locate data to comply with business governance and compliance rules. Also look for comprehensive native encryption and key management.
    • The ability for customers to track data lineage across any cloud and transient clusters. Support is needed for big data platforms, with data discovery, continuous optimization, audit and lineage across any cloud and transient clusters, metadata management and policy enforcement.

With a cloud data council providing the proper data models and guidance, and the proper supporting infrastructure, organizations have a much higher likelihood of getting the greatest benefit from the cloud at the lowest cost.

jamie engesser

Jamie Engesser

Jamie Engesser is the Senior Vice President of Cloud and Field Specialists at Cloudera. With more than 25 years of professional experience... Read more