Inside Data@Scale 2015

Ganapathy Krishnamoorthy

Visit the post for more.

Overview

The article discusses the Data@Scale 2015 event, where engineers from various leading technology companies gathered to address the challenges of scaling data storage and processing. Presentations covered topics such as distributed SQL query engines, data visualization, real-time log analysis, and cloud-based NoSQL databases.

What You'll Learn

1

How to utilize Presto for interactive SQL queries on big data sources

2

Why effective data visualization is critical for user engagement

3

How to implement real-time log indexing and search with LogLens

4

When to use Azure DocumentDB for JSON document storage at scale

5

How to design scalable storage systems like Backblaze Vaults

Key Questions Answered

What is Presto and how is it used for big data queries?

Presto is an open-source distributed SQL query engine optimized for ad-hoc analysis at interactive speed. It is designed to run interactive analytic queries against large data sources, providing a reliable engine for end users.

What are the key components of Tableau's data visualization architecture?

Tableau's data visualization architecture includes components like the Tableau Data Engine and Data Server, which are essential for generating visualizations efficiently. Recent performance improvements in Tableau 9 address challenges posed by increasing data volumes and complexity.

How does LogLens improve log search and visualization?

LogLens is a service from Twitter that provides real-time indexing, search, and visualization of service logs. It enhances the user experience by simplifying the process of finding patterns in logs generated by services across hundreds of machines.

What challenges did Dropbox face with Firefly's design?

Dropbox's Firefly system was designed to manage over one billion adds and edits daily, presenting challenges in organizing a vast number of files. The system is built to be private, scalable, and fast, addressing these complexities effectively.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Backend

Presto

An open-source distributed SQL query engine for big data analysis.

Frontend

Tableau

A data visualization tool that processes and visualizes large datasets.

Backend

Loglens

A service for real-time indexing and visualization of logs.

Database

Azure Documentdb

A cloud-based NoSQL database for storing JSON documents.

Backend

Firefly

An indexing and search system for managing user-generated content in Dropbox.

Backend

Zippydb

A key/value memory cache service used by Facebook.

Storage

Backblaze Vaults

A distributed storage system for file availability across multiple servers.

Key Actionable Insights

1
Implementing a distributed SQL query engine like Presto can significantly enhance data analysis capabilities for large datasets.
Presto's design allows for interactive queries, making it suitable for businesses that require quick insights from big data sources.

2
Utilizing advanced data visualization techniques can greatly improve user engagement and decision-making.
As data volumes grow, ensuring that visualizations remain interactive and responsive is crucial for maintaining user interest and effectiveness.

3
Real-time log analysis tools like LogLens can streamline the process of identifying issues in distributed systems.
By providing indexing and visualization capabilities, LogLens helps engineers quickly find patterns and troubleshoot problems in service logs.