Overview
The article discusses how challenges in engineering can foster innovation, particularly through the experiences of Suja Viswesan at LinkedIn. It highlights the importance of adapting to new environments, automating processes, and scaling teams while maintaining a focus on organizational growth.
What You'll Learn
1
How to automate production review processes using tools like Dr. Elephant
2
Why scaling engineering teams requires a focus on organizational structure
3
How to identify and address operational challenges in a growing tech environment
Prerequisites & Requirements
- Understanding of big data platforms and automation tools(optional)
- Experience in engineering team management
Key Questions Answered
How did Suja Viswesan adapt to her new role at LinkedIn?
Suja Viswesan adapted to her new role by leading the Hadoop Development team and quickly learning to manage a service instead of an enterprise product. She focused on identifying outages proactively and collaborated closely with her team to navigate the challenges of a major Hadoop upgrade.
What automation projects did Suja lead at LinkedIn?
Suja led the development of two key automation projects: Byte-Ray, a byte code analysis tool that helps identify incompatibility issues during Hadoop upgrades, and Dr. Elephant, which automates the production review process, significantly reducing the time from weeks to minutes.
What challenges did Suja face while scaling her team?
Suja faced challenges in scaling her team due to organic growth and the addition of teams under her management. She realized the need to delegate responsibilities and restructure the team into three pillars to maintain efficiency and clarity of mission.
How does LinkedIn support career growth for engineers?
LinkedIn fosters career growth by encouraging discussions about career advancement and providing support through mentorship and resources. Suja's experiences highlight how leaders at LinkedIn helped her navigate transitions and scale her responsibilities effectively.
Technologies & Tools
Backend
Hadoop
Used as the primary big data platform for various projects and automation tools.
Automation Tool
Dr. Elephant
Automates the production review process for Hadoop/Spark workloads.
Automation Tool
Byte-ray
Analyzes byte code to identify incompatibility issues during Hadoop upgrades.
Workflow Scheduler
Azkaban
Integrated with Byte-Ray to manage user workflows and detect issues.
Key Actionable Insights
1Implement automation tools like Dr. Elephant to streamline production reviews.By automating the review process, teams can significantly reduce the time required for project promotions, allowing for faster deployment and increased efficiency.
2Focus on organizational structure to manage growing teams effectively.As teams expand, establishing clear roles and responsibilities helps prevent bottlenecks and ensures that all team members are aligned with the organization's goals.
3Encourage a culture of ownership among engineers for better resource management.By delegating ownership of specific projects or components, engineers are more likely to take responsibility for their work, leading to improved performance and innovation.
Common Pitfalls
1
Failing to adapt to a new organizational structure can lead to inefficiencies.
When teams grow, it's essential to reassess roles and responsibilities. Not doing so can result in missed meetings and slowed hiring processes, ultimately hindering team performance.
Related Concepts
Big Data Platforms
Automation In Engineering
Team Management And Scaling
Career Development In Tech