Josh Wills - Statistician, Former Director of Data Engineering at Slack #51

Biography
Episode Summary
Key Milestones
Additional Resources

Biography

Josh Wills is a former engineer and Statistician on Slack’s Search, Learning, and Intelligence Team helping to build the company’s production search and machine learning infrastructure. He’s a recovering manager, having most recently built and led Slack’s data engineering team and before that the data science and engineering teams at Cloudera and Google.

Josh is a member of the Apache Software Foundation, the founder of the Apache Crunch project, and a co-author of O’Reilly’s Advanced Analytics with Spark. In May of 2012, he tweeted a pithy definition of a data scientist as someone who is better at statistics than any software engineer and better at software engineering than any statistician, and his Twitter mentions have never been the same.

Click Here –> For more information about tech careers

Episode Summary

“Statistics in school, at least when I was there, was very much focused on the problem. ‘How do we extract as much information as we can from a relatively small subset of data.’ Statistics at Google and Facebook was much more concerned with ‘we are drowning in data, we have an overwhelming amount of data. How do we not delude ourselves?‘”

“If you want to be a mentee, there’s no work that you’re too good to do. Like the best learning average opportunities of my life came from situations where I was more than willing to roll up my sleeves and do things that are not sexy and are not cool.”

“I didn’t go back to engineering and I never stopped engineering. I was always engineering while I was doing this management stuff on the side. Recruiting and hiring, I enjoy. And I very much enjoy mentoring people. But I do not personally care about or tolerate sitting through a sprint review or doing a budget planning meeting or, god help me, sitting through a promotion process.”

—Josh Wills

In this episode we’ll cover:

What are the differences in studying statistics in school versus working as a statistician at Google or Facebook?
Why Josh chose to be a high-level individual contributor (IC) over management at Slack?
How Josh worked a full-time job at IBM while getting his master’s degree.

Key Milestones

[2:05] – Currently Josh is taking time off. He was previously director of data engineering at Slack. He also has experience working at Cloudera, Google, Indeed and IBM. Josh studied Math at Duke.
[4:35] – Josh got introduced to computer science by his interest in neuroscience in college. He ended up applying for and winning an NSF REU (research experience for undergraduates) at Carnegie Melon. He slowly learned how to do basic programming in MATLAB.
[7:05] – Josh decided to get his masters degree while working full-time as a software engineer at IBM. It was a stressful time for him and he later joined a startup Zilliant which was run by one of his professors. This is where he got his feet wet as a statistician.
[11:11] – What are the differences in studying statistics in school versus working as a statistician at Google or Facebook? In school, it was all about extracting information from a small subset of data, while at Google and Facebook it was all about not getting overwhelmed by massive amounts of data available.
[13:39]- Josh brings up an article by Andrej Karpathy about the data-centric perspective of programming – https://medium.com/@karpathy/software-2-0-a64152b37c35
[15:23] – Josh takes us through his decision to get into management at Google, Cloudera and a director of data engineering at slack. He enjoyed recruiting and mentoring, but couldn’t bare sitting through budget meetings or a promotion process. He decided to primarily stick to engineering.
[17:25] – Josh and Grant talk about the book ‘The Manager’s Path’ by Camille Fournier. Link is below.
[19:14] – Josh recalls 3 of his biggest mentors throughout his career.
[23:53] – How does Josh evaluate the state DevOps today? And why is it so hard to get it right?
[26:05] – What is A/B testing? Why is it so important in making the right decisions for a company?
[32:24] – Learning how to read code as well as books is an important skill to attain. Josh shares some resources to maximize your reading.