Senior Staff Engineer, Machine Learning Platforms Lead at Wayfair
Ian is the lead of the Machine Learning Platforms team at Wayfair. He spent most of his seven years at Wayfair working on bridging the gap between experimental data science and software engineering workstreams for use-cases spanning Wayfair’s business areas. Over the past two years, Ian led his team in building internal model storage, model serving, and feature library platforms that help data scientists focus on modeling and experimentation while providing engineers the tools to build reliable systems around them.
WATCH LIVE: November 18 @ 3:35PM – 4:05PM ET
Building an ML platform one problem at a time
This talk will cover how Wayfair built a set of in-house platforms to enable 150+ machine learning practitioners and engineers who partner with them to deliver wins across Wayfair’s business domains. Our journey was organic and each step of the way was motivated by a specific problem. We will talk about how:
– The need for reliable source of truth inspired the creation of an internal event logging and transcription tool (Scribe)
– Unreliable and open access filesystems led to a model storage and versioning solution (Mesa Hub)
– The challenges with scaling monolithic applications led to our model serving solution (Sweetwater)
– Difficulty matching feature values between training and prediction inspired our feature store (Crucible)
At each step, Wayfair made conscious decisions to build internal tools instead of using off-the-shelf solutions. The talk will cover the reasoning behind it as well as why others might not want to do the same. Attendees will leave the session understanding the challenges that come with building production ML systems at Wayfair scale and how those problems changed over time. They will also learn about how we drove adoption for new platforms by acting like a start-up within a large company. Finally, they will gain appreciation for flexibility gained by tackling complex workflows by building platform components that solve one problem at a time instead of tackling everything at once.