Shopify's Data Science & Engineering Foundations

Shopify’s Data team uses these foundational approaches to data warehousing and analysis empowering us to deliver the best results for our ecosystem.

Shopify logo
Shopify
•
Marc-Olivier Arsenault
•10 min read•intermediate•
--
•View Original
SQL

Overview

The article discusses Shopify's Data Science & Engineering foundations, emphasizing the importance of structured data management and collaboration within the organization. It outlines key approaches such as modelled data, data consistency, rigorous ETL processes, and the significance of communication and collaboration in delivering insights.

What You'll Learn

1

How to structure data for better collaboration across teams

2

Why rigorous ETL processes enhance data accuracy and trust

3

How to leverage vetted data points for consistent decision-making

4

When to implement peer review processes in data projects

Prerequisites & Requirements

  • Understanding of data warehousing concepts
  • Familiarity with Spark and Presto(optional)

Key Questions Answered

What are the key benefits of modelled data in data warehousing?
The key benefits of modelled data include eliminating the need to understand raw data structures, ensuring data compatibility between teams, and facilitating faster and more collaborative work. This structured approach allows data teams to efficiently deliver insights to stakeholders.
How does Shopify ensure data consistency and open access?
Shopify maintains a single data modelling platform built on Spark, accessible to all employees. This openness allows anyone to query data, enhancing discoverability and ensuring that all teams can leverage existing data for their needs.
What role does peer review play in data projects at Shopify?
Peer review is integral to Shopify's data processes, with all work being reviewed by at least two other data scientists. This practice enhances data accuracy and builds trust across the organization, ensuring high standards are met in analytics and reporting.
Why is deep product understanding important for data analysis?
Deep product understanding allows data teams to focus on metrics that matter for merchants, guiding stakeholders to ask the right questions and design impactful experiments. This expertise is cultivated by dividing teams into sub-teams focused on specific product areas.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Backend
Spark
Used as the foundation for Shopify's data modelling platform.
Backend
Presto
Serves as the query engine for accessing modelled data across the company.

Key Actionable Insights

1
Implement a modelled data approach to streamline data access and collaboration.
By adopting a standardized data modelling philosophy, teams can work more efficiently and understand each other's data models, leading to quicker insights and better decision-making.
2
Establish rigorous ETL processes to enhance data quality and trust.
Unit testing data pipeline jobs can prevent errors and ensure data accuracy, which is crucial for maintaining stakeholder confidence in analytics.
3
Utilize vetted data points for reliable decision-making.
By storing vetted data points with context and ensuring they remain consistent over time, teams can make informed decisions based on accurate historical data.
4
Encourage peer review in all data-related work.
Involving multiple reviewers in data projects can significantly improve the quality of outputs and foster a culture of trust in data across the organization.

Common Pitfalls

1
Failing to communicate insights effectively can lead to misunderstandings.
When data scientists present findings without context or clear recommendations, stakeholders may misinterpret the results, leading to poor decision-making.

Related Concepts

Data Warehousing
Etl Processes
Data Modeling
Collaboration In Data Teams