With today’s unprecedented velocity of change in data and technologies, the single version of the truth (SVOT) is sometimes looked upon as a nice to have. But SVOT is not optional. Despite the difficulties in constructing a single version of the truth, it is now needed more than ever to keep CEOs happy and to stay compliant with new regulations like the GDPR.

Originally published at https://www.eckerson.com/articles/single-version-of-the-truth-not-optional

Summary

With the increased ease of gaining access to servers in data centers across the world has come the need for supporting globally distributed data storage. With the first wave of cloud era databases the ability to replicate information geographically came at the expense of transactions and familiar query languages. To address these shortcomings the engineers at Cockroach Labs have built a globally distributed SQL database with full ACID semantics in Cockroach DB. In this episode Peter Mattis, the co-founder and VP of Engineering at Cockroach Labs, describes the architecture that underlies the database, the challenges they have faced along the way, and the ways that you can use it in your own environments today.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute. For complete visibility into the health of your pipeline, including deployment tracking, and powerful alerting driven by machine-learning, DataDog has got you covered. With their monitoring, metrics, and log collection agent, including extensive integrations and distributed tracing, you’ll have everything you need to find and fix performance bottlenecks in no time. Go to dataengineeringpodcast.com/datadog today to start your free 14 day trial and get a sweet new T-Shirt. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch. Your host is Tobias Macey and today I’m interviewing Peter Mattis about CockroachDB, the SQL database for global cloud services

Interview

Introduction How did you get involved in the area of data management? What was the motivation for creating CockroachDB and building a business around it? Can you describe the architecture of CockroachDB and how it supports distributed ACID transactions?

What are some of the tradeoffs that are necessary to allow for georeplicated data with distributed transactions? What are some of the problems that you have had to work around in the RAFT protocol to provide reliable operation of the clustering mechanism?

Go is an unconventional language for building a database. What are the pros and cons of that choice? What are some of the common points of confusion that users of CockroachDB have when operating or interacting with it?

What are the edge cases and failure modes that users should be aware of?

I know that your SQL syntax is PostGreSQL compatible, so is it possible to use existing ORMs unmodified with CockroachDB?

What are some examples of extensions that are specific to CockroachDB?

What are some of the most interesting uses of CockroachDB that you have seen? When is CockroachDB the wrong choice? What do you have planned for the future of CockroachDB?

Contact Info

Peter

LinkedIn petermattis on GitHub @petermattis on Twitter

Cockroach Labs

@CockroackDB on Twitter Website cockroachdb on GitHub

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

CockroachDB Cockroach Labs SQL Google Bigtable Spanner NoSQL RDBMS (Relational Database Management System) “Big Iron” (colloquial term for mainframe computers) RAFT Consensus Algorithm Consensus MVCC (Multiversion Concurrency Control) Isolation Etcd GDPR Golang C++ Garbage Collection Metaprogramming Rust Static Linking Docker Kubernetes CAP Theorem PostGreSQL ORM (Object Relational Mapping) Information Schema PG Catalog Interleaved Tables Vertica Spark Change Data Capture

The intro and outro music is from The Hug by The Freak Fandan

Put this in your pipe and smoke it: all of the tracking we try to do of people is actually technology designed to track content. And, even that tracking of content was a hacked-together repurposing of a system designed to deliver content. In other words, we've got layers of fiction upon fiction that we're trying to muddle through (and, often, ignore) as an industry. The result? A ridiculous level of inefficiency whereby brands overspend to ineffectively reach their target audiences with direct response messages, and well-intended intermediaries grow their bank accounts. Ugh! On this episode, the gang invited Sergio Maldonado from PrivacyCloud (and, by day, from Sweetspot Intelligence) to chat about the broken environment we're operating in, as well as how GDPR and financial considerations may just force us onto a path of shaking it up! For complete show notes, including links to items mentioned in this episode and a transcript of the show, visit the show page.

Summary

Building an ETL pipeline is a common need across businesses and industries. It’s easy to get one started but difficult to manage as new requirements are added and greater scalability becomes necessary. Rather than duplicating the efforts of other engineers it might be best to use a hosted service to handle the plumbing so that you can focus on the parts that actually matter for your business. In this episode CTO and co-founder of Alooma, Yair Weinberger, explains how the platform addresses the common needs of data collection, manipulation, and storage while allowing for flexible processing. He describes the motivation for starting the company, how their infrastructure is architected, and the challenges of supporting multi-tenancy and a wide variety of integrations.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute. For complete visibility into the health of your pipeline, including deployment tracking, and powerful alerting driven by machine-learning, DataDog has got you covered. With their monitoring, metrics, and log collection agent, including extensive integrations and distributed tracing, you’ll have everything you need to find and fix performance bottlenecks in no time. Go to dataengineeringpodcast.com/datadog today to start your free 14 day trial and get a sweet new T-Shirt. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. Your host is Tobias Macey and today I’m interviewing Yair Weinberger about Alooma, a company providing data pipelines as a service

Interview

Introduction How did you get involved in the area of data management? What is Alooma and what is the origin story? How is the Alooma platform architected?

I want to go into stream VS batch here What are the most challenging components to scale?

How do you manage the underlying infrastructure to support your SLA of 5 nines? What are some of the complexities introduced by processing data from multiple customers with various compliance requirements?

How do you sandbox user’s processing code to avoid security exploits?

What are some of the potential pitfalls for automatic schema management in the target database? Given the large number of integrations, how do you maintain the

What are some challenges when creating integrations, isn’t it simply conforming with an external API?

For someone getting started with Alooma what does the workflow look like? What are some of the most challenging aspects of building and maintaining Alooma? What are your plans for the future of Alooma?

Contact Info

LinkedIn @yairwein on Twitter

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

Alooma Convert Media Data Integration ESB (Enterprise Service Bus) Tibco Mulesoft ETL (Extract, Transform, Load) Informatica Microsoft SSIS OLAP Cube S3 Azure Cloud Storage Snowflake DB Redshift BigQuery Salesforce Hubspot Zendesk Spark The Log: What every software engineer should know about real-time data’s unifying abstraction by Jay Kreps RDBMS (Relational Database Management System) SaaS (Software as a Service) Change Data Capture Kafka Storm Google Cloud PubSub Amazon Kinesis Alooma Code Engine Zookeeper Idempotence Kafka Streams Kubernetes SOC2 Jython Docker Python Javascript Ruby Scala PII (Personally Identifiable Information) GDPR (General Data Protection Regulation) Amazon EMR (Elastic Map Reduce) Sequoia Capital Lightspeed Investors Redis Aerospike Cassandra MongoDB

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast

The role of the IT solutions is to enforce the correct handling of personal data using processes developed by the establishment. Each element of the solution stack must address the objectives as appropriate to the data that it handles. Typically, personal data exists either in the form of structured data (like databases) or unstructured data (like files, text, documents, and so on.). This IBM Redbooks publication specifically deals with unstructured data and storage systems used to host unstructured data. For unstructured data storage in particular, some key attributes enable the overall solution to support compliance with the EU General Data Protection Regulation (GDPR). Because personal data subject to GDPR is commonly stored in an unstructured data format, a scale out file system like IBM Spectrum Scale provides essential functions to support GDPR requirements. This paper highlights some of the key compliance requirements and explains how IBM Spectrum Scale helps to address them.

Summary

The information about how data is acquired and processed is often as important as the data itself. For this reason metadata management systems are built to track the journey of your business data to aid in analysis, presentation, and compliance. These systems are frequently cumbersome and difficult to maintain, so Octopai was founded to alleviate that burden. In this episode Amnon Drori, CEO and co-founder of Octopai, discusses the business problems he witnessed that led him to starting the company, how their systems are able to provide valuable tools and insights, and the direction that their product will be taking in the future.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 200Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute. For complete visibility into the health of your pipeline, including deployment tracking, and powerful alerting driven by machine-learning, DataDog has got you covered. With their monitoring, metrics, and log collection agent, including extensive integrations and distributed tracing, you’ll have everything you need to find and fix performance bottlenecks in no time. Go to dataengineeringpodcast.com/datadog today to start your free 14 day trial and get a sweet new T-Shirt. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. Your host is Tobias Macey and today I’m interviewing Amnon Drori about OctopAI and the benefits of metadata management

Interview

Introduction How did you get involved in the area of data management? What is OctopAI and what was your motivation for founding it? What are some of the types of information that you classify and collect as metadata? Can you talk through the architecture of your platform? What are some of the challenges that are typically faced by metadata management systems? What is involved in deploying your metadata collection agents? Once the metadata has been collected what are some of the ways in which it can be used? What mechanisms do you use to ensure that customer data is segregated?

How do you identify and handle sensitive information during the collection step?

What are some of the most challenging aspects of your technical and business platforms that you have faced? What are some of the plans that you have for OctopAI going forward?

Contact Info

Amnon

LinkedIn @octopai_amnon on Twitter

OctopAI

@OctopaiBI on Twitter Website

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

OctopAI Metadata Metadata Management Data Integrity CRM (Customer Relationship Management) ERP (Enterprise Resource Planning) Business Intelligence ETL (Extract, Transform, Load) Informatica SAP Data Governance SSIS (SQL Server Integration Services) Vertica Airflow Luigi Oozie GDPR (General Data Privacy Regulation) Root Cause Analysis

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast

Moe sat down for a chat with privacy and GDPR expert Aurélie Pols to dive in to some of the questions that, at times, get treated as peripheral in the run-up to new regulations, but that seem like they are fairly fundamental when it comes to understanding the rationale and drivers behind those regulations: what does the Holocaust have to do with GDPR? Is GDPR something that was simply dreamed up in Europe, or are there roots in other countries (teaser: Eleanor Roosevelt). Is GDPR inherently anti-business? It's a quick chat but, hopefully, will give you some deeper perspective on the subject!

For the second year in a row for the podcast -- but the first appearance since Moe joined the crew -- we headed to the Hunguest Grandhotel Galya outside Budapest for Superweek, one of the most unique conference experiences in the digital analytics industry: comfortably isolated over an hour outside of Budapest in a beautiful setting, it's a temporary community of, for, and by the analyst. With sessions ranging from GDPR to machine learning to attribution to media analytics, the spaces before, between, and after the presentations were extended discussions with great people on a wide range of topics. The "fireside chat" on Wednesday evening was a recording of the podcast with a live audience, where we had attendees to share tips and ideas that we found particularly intriguing. And had quite a bit of fun along the way. For complete show notes, including links to items mentioned in this episode and a transcript of the show, visit the show page.

"In the midst of chaos, there is also opportunity". GDPR is coming and being prepared is not an option, it’s a requirement. Prolet will walk through the challenges presented by the new regulation for all of us, from in-company struggles to winning the battle with, and for the user.

Are you a data subject? If you're a person, then you better believe you are! And, so is every person who visits your website. And, if you are in the EU, or you have visitors from the EU, then May 25th, 2018, is a day you should be keeping a close eye on and preparing for now! On this episode, Jodi Daniels of Red Clover Advisors joins Moe and Tim to talk all things General Data Privacy Regulation (aka, GDPR). Give it a listen and pick up delightful cocktail party openers like, "Hey, do you know how to tell someone isn't from the EU? They reference PII." That's not just a delightful witticism -- it's actually important to understand the distinction between PII and personal data! For complete show notes, including links to items mentioned in this episode and a transcript of the show, visit the show page.

The updated second edition of the bestselling guide to the changes your organisation needs to make to comply with the EU GDPR. “The clear language of the guide and the extensive explanations, help to explain the many doubts that arise reading the articles of the Regulation.” Giuseppe G. Zorzino The EU General Data Protection Regulation (GDPR) will supersede the 1995 EU Data Protection Directive (DPD) and all EU member states’ national laws based on it – including the UK Data Protection Act 1998 – in May 2018. All organisations – wherever they are in the world – that process the personal data of EU residents must comply with the Regulation. Failure to do so could result in fines of up to €20 million or 4% of annual global turnover. This book provides a detailed commentary on the GDPR, explains the changes you need to make to your data protection and information security regimes, and tells you exactly what you need to do to avoid severe financial penalties. Product overview Now in its second edition, EU GDPR – An Implementation and Compliance Guide is a clear and comprehensive guide to this new data protection law, explaining the Regulation, and setting out the obligations of data processors and controllers in terms you can understand. Topics covered include: The role of the data protection officer (DPO) – including whether you need one and what they should do. Risk management and data protection impact assessments (DPIAs), including how, when and why to conduct a DPIA. Data subjects’ rights, including consent and the withdrawal of consent; subject access requests and how to handle them; and data controllers’ and processors’ obligations. International data transfers to “third countries” – including guidance on adequacy decisions and appropriate safeguards; the EU-US Privacy Shield; international organisations; limited transfers; and Cloud providers. How to adjust your data protection processes to transition to GDPR compliance, and the best way of demonstrating that compliance. A full index of the Regulation to help you find the articles and stipulations relevant to your organisation. New for the second edition: Additional definitions. Further guidance on the role of the DPO. Greater clarification on data subjects’ rights. Extra guidance on data protection impact assessments. More detailed information on subject access requests (SARs). Clarification of consent and the alternative lawful bases for processing personal data. New appendix: implementation FAQ. The GDPR will have a significant impact on organisational data protection regimes around the world. EU GDPR – An Implementation and Compliance Guide shows you exactly what you need to do to comply with the new law.

A concise introduction to EU GDPR and EU-US Privacy Shield

The EU General Data Protection Regulation will unify data protection and simplify the use of personal data across the EU when it comes into force in May 2018.

It will also apply to every organization in the world that processes personal information of EU residents.

US organizations that process EU residents' personal data will be able to comply with the GDPR via the EU-US Privacy Shield (the successor to the Safe Harbor framework), which permits international data transfers of EU data to US organizations that self-certify that they have met a number of requirements.

EU GDPR & EU-US Privacy Shield – A Pocket Guide provides an essential introduction to this new data protection law, explaining the Regulation and setting out the compliance obligations for US organizations in handling data of EU citizens, including guidance on the EU-US Privacy Shield.

Product overview

EU GDPR & EU-US Privacy Shield – A Pocket Guide sets out:

A brief history of data protection and national data protection laws in the EU (such as the UK DPA, German BDSG and French LIL). The terms and definitions used in the GDPR, including explanations. The key requirements of the GDPR, including: Which fines apply to which Articles; The six principles that should be applied to any collection and processing of personal data; The Regulation’s applicability; Data subjects’ rights; Data protection impact assessments (DPIAs); The role of the data protection officer (DPO) and whether you need one; Data breaches, and the notification of supervisory authorities and data subjects; Obligations for international data transfers. How to comply with the Regulation, including: Understanding your data, and where and how it is used (e.g. Cloud suppliers, physical records); The documentation you need to maintain (such as statements of the information you collect and process, records of data subject consent, processes for protecting personal data); The “appropriate technical and organizational measures” you need to take to ensure your compliance with the Regulation. The history and principles of the EU-US Privacy Shield, and an overview of what organizations must do to comply. A full index of the Regulation, enabling you to find relevant Articles quickly and easily.

An in-depth guide to the changes your organization needs to make to comply with the EU GDPR.

The EU General Data Protection Regulation (GDPR) will supersede the 1995 EU Data Protection Directive (DPD) and all EU member states’ national laws based on it – including the UK Data Protection Act 1998 – in May 2018.

All organizations – wherever they are in the world – that process the personally identifiable information (PII) of EU residents must comply with the Regulation. Failure to do so could result in fines of up to €20 million or 4% of annual global turnover.

US organizations that process EU residents’ personal data can comply with the GDPR via the EU-US Privacy Shield, which replaced the EU-US Safe Harbor framework in 2016. The Privacy Shield is based on the DPD, and will likely be updated once the GDPR is applied in May 2018.

This book provides a detailed commentary on the GDPR, explains the changes you need to make to your data protection and information security regimes, and tells you exactly what you need to do to avoid severe financial penalties.

Product overview

EU GDPR – An Implementation and Compliance Guide is a clear and comprehensive guide to this new data protection law, explaining the Regulation, and setting out the obligations of data processors and controllers in terms you can understand.

Topics covered include:

The role of the data protection officer (DPO) – including whether you need one and what they should do. Risk management and data protection impact assessments (DPIAs), including how, when and why to conduct a DPIA. Data subjects’ rights, including consent and the withdrawal of consent; subject access requests and how to handle them; and data controllers’ and processors’ obligations. International data transfers to “third countries” – including guidance on adequacy decisions and appropriate safeguards; the EU-US Privacy Shield; international organizations; limited transfers; and Cloud providers. How to adjust your data protection processes to transition to GDPR compliance, and the best way of demonstrating that compliance. A full index of the Regulation to help you find the articles and stipulations relevant to your organization.

The GDPR will have a significant impact on organizational data protection regimes around the world. EU GDPR – An implementation and Compliance Guide shows you exactly what you need to do to comply with the new law.

About the authors

IT Governance is a leading global provider of IT governance, risk management, and compliance expertise, and we pride ourselves on our ability to deliver a broad range of integrated, high-quality solutions that meet the real-world needs of our international client base.

Our privacy team – led by Alan Calder, Richard Campo, and Adrian Ross – has substantial experience in privacy, data protection, compliance, and information security. This experience, and our understanding of the background and drivers for the GDPR, are combined in this manual to provide the world’s first guide to implementing the new data protection regulation.