talk-data.com

01:00pm-01:10pm ET - Welcome, Cloud/Serverless Introduction Wayne Scarano, sga.com, Founder, AWS Strategy/Architecture/Security/Instructor and AWS Community Builder

01:10pm-02:00pm ET - AWS and Serverless technologies in production at Sophos/SophosLabs James Wilson, Director of X-Ops Business Development Guy Davies, Principal Cloud Architect Marc McGinnis, Principal Cloud Architect–SophosLabs

Sophos and SophosLabs built solutions supporting our products and services which leverage AWS for almost 15 years. Two of these solutions provide data to our products:

1. The Sophos Updating Services provide products with the latest version of their software and data. We will look at how the AWS serverless services are used to provide a reliable service for our almost 600 thousand customers.

2. SophosLabs Intelix (specifically Cloud Lookups) provides the latest threat intelligence from SophosLabs and Sophos AI to every deployment of Sophos products worldwide, providing answers to trillions of queries per month. We will look at how the system was developed, cloud services used, and how we monitor usage and cost in a highly scalable system.

James Wilson –Director of X-Ops Business Development James joined Sophos in 2019 and is the Product Director for SophosLabs Intelix. James works with several teams at Sophos to increase the adoption of SophosLabs Intelix across Sophos products, OEM partners, and through AWS Marketplace. Prior to Sophos, James worked at a range of companies (from startups to multinationals) to help build and deliver products that add real value to their users. Guy Davies –Principal Cloud Architect Guy has been at Sophos for over 18 years. Guy has worked with every product team to build out best in class cloud native and serverless solutions. Guy is a passionate advocate of serverless, cloud-native architecture let's stand on the shoulders of giants and let someone else do the heavy lifting for us! Marc McGinnis –Principal Cloud Architect –SophosLabs Marc has been a cloud architect in cyber security companies for over 15 years. Marc works with the SophosLabs platform team to ensure that the cloud services are built to leverage the best qualities of cloud especially in terms of availability, scalability to provide better cybersecurity outcomes for our customers.

Summary

The responsibilities of a data scientist and a data engineer often overlap and occasionally come to cross purposes. Despite these challenges it is possible for the two roles to work together effectively and produce valuable business outcomes. In this episode Will McGinnis discusses the opinions that he has gained from experience on how data teams can play to their strengths to the benefit of all.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at dataengineeringpodcast.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. You can help support the show by checking out the Patreon page which is linked from the site. To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers A few announcements:

There is still time to register for the O’Reilly Strata Conference in San Jose, CA March 5th-8th. Use the link dataengineeringpodcast.com/strata-san-jose to register and save 20% The O’Reilly AI Conference is also coming up. Happening April 29th to the 30th in New York it will give you a solid understanding of the latest breakthroughs and best practices in AI for business. Go to dataengineeringpodcast.com/aicon-new-york to register and save 20% If you work with data or want to learn more about how the projects you have heard about on the show get used in the real world then join me at the Open Data Science Conference in Boston from May 1st through the 4th. It has become one of the largest events for data scientists, data engineers, and data driven businesses to get together and learn how to be more effective. To save 60% off your tickets go to dataengineeringpodcast.com/odsc-east-2018 and register.

Your host is Tobias Macey and today I’m interviewing Will McGinnis about the relationship and boundaries between data engineers and data scientists

Interview

Introduction How did you get involved in the area of data management? The terms “Data Scientist” and “Data Engineer” are fluid and seem to have a different meaning for everyone who uses them. Can you share how you define those terms? What parallels do you see between the relationships of data engineers and data scientists and those of developers and systems administrators? Is there a particular size of organization or problem that serves as a tipping point for when you start to separate the two roles into the responsibilities of more than one person or team? What are the benefits of splitting the responsibilities of data engineering and data science?

What are the disadvantages?

What are some strategies to ensure successful interaction between data engineers and data scientists? How do you view these roles evolving as they become more prevalent across companies and industries?

Contact Info

Website wdm0006 on GitHub @willmcginniser on Twitter LinkedIn

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

Blog Post: Tendencies of Data Engineers and Data Scientists Predikto Categorical Encoders DevOps SciKit-Learn

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast