Overview
The article discusses Coral, an open-sourced SQL translation, analysis, and rewrite engine developed at LinkedIn for modern data lakehouses. It highlights how Coral enhances the accessibility and expressibility of Dali views, integrates with various compute engines like Presto and Spark, and supports complex data operations.
What You'll Learn
1
How to integrate Coral with Presto for querying Hive views
2
Why Dali views are more agile and portable than traditional views
3
How to manage user-defined functions (UDFs) in Dali views
Key Questions Answered
What is Coral and how does it enhance data lakehouses?
Coral is a SQL translation, analysis, and rewrite engine that improves the accessibility and expressibility of Dali views in data lakehouses. It allows for seamless integration with various compute engines, enabling users to query complex views efficiently while maintaining data governance and lineage.
How do Dali views differ from traditional views?
Dali views are designed to be portable and agile, meaning their schemas can evolve with underlying data changes. They also support user-defined functions as first-class citizens, allowing for more complex data manipulations compared to traditional views.
What is the view development lifecycle at LinkedIn?
The view development lifecycle involves creating Git projects for view SQL files, using Gradle for UDF dependencies, and following a standard code review process. Once checked in, the view artifacts are published and registered in the Dali Catalog, streamlining the view creation process.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Coral
Used as a SQL translation and rewrite engine for data lakehouses.
Backend
Presto
Integrated with Coral to enable querying complex Hive views.
Backend
Spark
Utilizes Coral for rewriting views into Spark SQL.
Backend
Pig
Uses Coral to generate executable Pig Latin from Dali views.
Key Actionable Insights
1Utilize Coral to enhance the portability of your SQL views across different engines.By leveraging Coral's capabilities, you can ensure that your views remain functional across various compute engines like Presto and Spark, reducing the need for engine-specific adaptations.
2Implement user-defined functions (UDFs) as part of your Dali views to enhance data processing.Incorporating UDFs allows for more complex data transformations and calculations, making your views more powerful and flexible.
3Adopt the view development lifecycle practices outlined in the article for better collaboration and efficiency.Following a structured lifecycle for view development can streamline the process, improve code quality, and facilitate easier maintenance of data views.
Common Pitfalls
1
Failing to manage schema evolution can lead to downstream data issues.
Without proper schema evolution rules, changes in underlying tables may cause views to break or produce incorrect results, affecting data consumers.
Related Concepts
Data Lakehouses
SQL Translation
User-defined Functions
Data Governance