Netflix Data Engineering Open Forum: A Summary of Key Sessions

The inaugural Data Engineering Open Forum at Netflix brought together data engineers from various industries to discuss the latest trends, challenges, and opportunities in the field. This event, held on April 18th, 2024, at Netflix’s Los Gatos office, showcased the innovative ways data engineers are leveraging technology to solve complex problems and drive business outcomes.

• The forum featured presentations from Netflix’s data engineering team and industry experts, providing insights into how they are tackling real-world challenges using cutting-edge technologies.
• These presentations explored various topics, including the use of machine learning for data remediation, the application of generative AI in data modeling, the importance of real-time data delivery, and the evolution of data quality strategies.
• The forum provided a platform for knowledge sharing, collaboration, and networking among data engineers, fostering a sense of community and driving innovation in the field.

Opening Remarks: Setting the Stage for Innovation at Netflix

Max Schmeiser, Vice President of Studio and Content Data Science & Engineering at Netflix, welcomed attendees to the first-ever Data Engineering Open Forum. He highlighted the critical role that data engineers play in enabling data-driven decision-making at Netflix and fostering a culture of innovation.

• Schmeiser emphasized the importance of collaboration and knowledge sharing in driving progress in the field of data engineering.
• He set the stage for the diverse range of topics that would be covered throughout the forum, emphasizing the significance of real-world applications and practical solutions.

Machine Learning Powered Auto Remediation: A Case Study from Netflix

Stephanie Vezich Tamayo, Senior Machine Learning Engineer at Netflix, and Binbing Hou, Senior Software Engineer at Netflix, presented a case study on how Netflix is using machine learning to automate error remediation in its data platform.

• The presentation focused on the evolution of Netflix’s approach to error classification and remediation, from a rule-based classifier to a machine learning-powered system.
• They highlighted the challenges of scaling a rule-based classifier in a large and complex data platform, emphasizing the need for automation and improved accuracy.
• The presentation showcased how machine learning is being used to enhance Netflix’s data platform, improving operational efficiency and reducing the burden on engineers.

Generative AI for Enterprise Data Modeling: Automating Data Architecture

Jide Ogunjobi, Founder and CTO at Context Data, explored the potential of generative AI for automating data modeling and architecture within enterprises.

• Ogunjobi emphasized the challenges of managing and querying data across diverse systems in large organizations, highlighting the need for efficient data modeling and architecture.
• He presented the concept of an intelligent agent that can automatically discover, map, and query enterprise data, leveraging generative AI techniques to automate the data modeling process.
• The presentation highlighted the potential of generative AI to revolutionize data modeling and empower data engineers to work more efficiently and effectively.

Real-Time Delivery of Impressions at Scale: Ensuring User Experience

Tulika Bhatt, Senior Data Engineer at Netflix, shared how Netflix manages impression data at scale, ensuring real-time delivery to power its recommendation algorithms and enhance user experience.

• Bhatt discussed the massive scale of impression data generated by Netflix, highlighting the challenges of processing and delivering this data in real-time.
• She showcased the innovative solutions that Netflix has developed to manage this high-volume, real-time data requirement, balancing scalability and cost.
• The presentation demonstrated the importance of real-time data processing in delivering a seamless and personalized user experience, particularly in the context of Netflix’s recommendation system.

Reflections on Building a Data Platform From the Ground Up: Navigating GDPR

Jessica Larson, Data Engineer and author of “Snowflake Access Control,” shared her experiences in building a new data platform in a post-GDPR world, emphasizing the importance of data privacy and regulatory compliance.

• Larson highlighted the significant differences between pre-GDPR and post-GDPR data platform development, focusing on the increased emphasis on data security and privacy.
• She discussed the challenges of balancing performance and cost with the need to prioritize sensitive data protection and regulatory compliance.
• The presentation provided valuable insights for data engineers working in a regulated environment, outlining best practices for building secure and compliant data platforms.

Unbundling the Data Warehouse: The Case for Independent Storage

Jason Reid, Co-founder and Head of Product at Tabular, explored the concept of unbundling the data warehouse, arguing for the benefits of independent storage in terms of performance, governance, and flexibility.

• Reid discussed the pros and cons of both bundled and unbundled data warehouse architectures, highlighting the trade-offs in terms of performance, cost, and flexibility.
• He explored how the trend of data warehouse unbundling is shaping the data engineering landscape, emphasizing the growing demand for modular and flexible data solutions.
• The presentation provided a thought-provoking discussion on the future of data warehousing, highlighting the potential of independent storage to meet the evolving needs of data engineers and businesses.

Data Quality Score: Evolving the Data Quality Strategy at Airbnb

Clark Wright, Staff Analytics Engineer at Airbnb, shared how Airbnb has developed a data quality score to improve the accuracy and reliability of its data, leading to more informed decision-making.

• Wright discussed the importance of data quality in driving business outcomes, emphasizing the need for a robust data quality strategy.
• He outlined Airbnb’s journey towards establishing a data quality score, highlighting the challenges and successes in implementing this new approach.
• The presentation provided a practical case study on how a data quality score can be implemented and used to improve the overall quality of data, leading to better insights and decision-making.

Data Productivity at Scale: Addressing Challenges in Data Pipeline Development

Iaroslav Zeigerman, Co-Founder and Chief Architect at Tobiko Data, presented SQLMesh, an open-source project designed to address challenges faced by data practitioners in developing and managing data pipelines at scale.

• Zeigerman discussed the shortcomings of existing tooling for data pipeline development, highlighting the need for more efficient and reliable solutions.
• He presented SQLMesh as a platform for streamlining data pipeline development, offering features for data versioning, testing, and deployment.
• The presentation emphasized the importance of data productivity in enabling data engineers to work more efficiently and effectively, ultimately driving faster innovation and better business outcomes.

Conclusion: A Call for Continued Collaboration and Innovation in Data Engineering

The Data Engineering Open Forum at Netflix served as a valuable platform for knowledge sharing, collaboration, and networking among data engineers. The event showcased the latest advancements in data engineering, highlighting the use of machine learning, real-time data processing, generative AI, and data quality strategies.

As the field of data engineering continues to evolve, the need for collaboration and innovation is paramount. The insights shared at the forum provide a glimpse into the future of data engineering, inspiring data engineers to embrace new technologies and work together to solve complex challenges.