Overview
The article features an interview with Felix Cheung, Data Platform Engineering Manager at Uber, discussing the advantages of open source software in private enterprise. It highlights Felix's contributions to various Apache projects and his role in Uber's open source initiatives, emphasizing the importance of community collaboration in software development.
What You'll Learn
1
How to leverage open source projects for building scalable data infrastructure
2
Why community collaboration is essential in open source software development
3
When to contribute back to open source projects after making improvements
Key Questions Answered
What is Felix Cheung's background in engineering?
Felix Cheung has been working with Big Data open source projects for about six years, starting with improving reliability in his previous company's infrastructure. His experience with frameworks like Apache Spark led him to Uber, where he now leads the Core Data team.
How does Uber integrate open source with its Big Data platform?
Uber uses several Apache Software Foundation projects, extending them to meet architectural needs. Changes are validated internally and then upstreamed, ensuring that improvements benefit the wider community while adhering to the Apache v2 license.
What are some notable contributions Felix has made to open source projects?
Felix has collaborated with LinkedIn’s Core Data Infrastructure team and the Apache Spark community to enhance scalability and reliability. He has worked on design proposals for disaggregating data shuffle and improving ML model serving in Apache Spark.
What newer projects is Felix excited about?
Felix is enthusiastic about XGBoost, a popular machine learning library for distributed gradient boosting. His team is investing in it, contributing to its development and enhancing its capabilities for large-scale distributed training.
Key Statistics & Figures
Data processed by Uber's infrastructure
over 100 petabytes
This data powers Uber’s services globally.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Apache Spark
Used for enhancing scalability and reliability in data processing.
Machine Learning
Xgboost
A library for optimized distributed gradient boosting, used for large-scale training.
Backend
Apache Hudi
Software developed at Uber and donated to the Apache Software Foundation.
Backend
Apache Kafka
Part of the infrastructure for processing data.
Backend
Apache Hadoop
Integral to Uber's data processing infrastructure.
Key Actionable Insights
1Engage with the open source community by contributing to projects that align with your expertise.This not only helps improve the software but also builds your professional network and enhances your skills.
2Consider using Apache projects as foundational tools in your data infrastructure.These projects provide robust solutions that can be adapted to meet specific business needs, ensuring reliability and scalability.
3Participate in local meetups and conferences to share experiences and learn from others in the open source community.This fosters collaboration and innovation, which are crucial for staying updated with the latest developments in technology.
Common Pitfalls
1
Failing to contribute back to open source projects after making improvements can lead to missed opportunities for collaboration.
This can happen when developers focus solely on their internal needs without considering the broader community benefits.
Related Concepts
Open Source Software Development
Big Data Technologies
Community Collaboration In Software Engineering