Overview
Netflix has announced the open-source release of two web applications, Scumblr and Sketchy, aimed at enhancing security teams' capabilities to monitor internet discussions and capture website content. Scumblr allows for customizable searches across various platforms, while Sketchy provides reliable screenshot and text scraping functionalities.
What You'll Learn
1
How to set up Scumblr for automated internet searches
2
Why integrating Sketchy with Scumblr enhances security monitoring
3
How to leverage Workflowable for custom workflows in Scumblr
4
When to use Sketchy for capturing screenshots of potentially harmful sites
Prerequisites & Requirements
- Familiarity with Ruby on Rails and web application development
- Basic understanding of Redis and Sidekiq for background tasks(optional)
Key Questions Answered
What is Scumblr and how does it help security teams?
Scumblr is a Ruby on Rails web application that enables security teams to search the internet for relevant content and discussions. It allows for customizable searches across various platforms and can run searches manually or automatically, helping teams stay informed about potential threats.
What components are used in the architecture of Scumblr?
Scumblr utilizes Ruby on Rails 4.0.9, a backend database for storing results, Redis with Sidekiq for background tasks, Workflowable for managing workflows, and Sketchy for capturing screenshots. This architecture supports efficient searching and processing of internet content.
How does Sketchy improve the functionality of Scumblr?
Sketchy integrates with Scumblr to automatically generate screenshots of identified results, providing a snapshot of web pages at the time they were discovered. This feature allows security analysts to review potentially malicious sites without directly visiting them, enhancing safety.
What challenges does Sketchy address in capturing screenshots?
Sketchy addresses challenges such as capturing screenshots from AJAX-heavy sites and dealing with issues like cut-off images and SSL problems. It uses PhantomJS for lazy rendering to ensure accurate captures, making it reliable for various web content.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Ruby On Rails
Used to build the Scumblr web application.
Database
Redis
Used for managing background tasks in Scumblr.
Background Processing
Sidekiq
Used in conjunction with Redis for background job processing.
Backend
Python
Used to build the Sketchy application.
Web Framework
Flask
Framework used to serve the Sketchy application.
Screenshot Capture
Phantomjs
Used by Sketchy to capture screenshots of web pages.
Task Management
Celery
Used in Sketchy to manage jobs and schedule tasks.
Key Actionable Insights
1Implement Scumblr to automate your security team's internet monitoring efforts. By setting up recurring searches, your team can proactively identify potential threats without manual intervention.This automation allows security teams to focus on analyzing results rather than spending time on searching, which can significantly enhance their efficiency.
2Utilize Sketchy to safely capture screenshots of suspicious websites. This tool allows analysts to preview content without the risk of visiting potentially harmful sites directly.By isolating the screenshot capture process, Sketchy mitigates the risk of exposure to malicious actions while providing valuable insights into web content.
3Take advantage of the Workflowable gem to create custom workflows in Scumblr. This feature enables tailored responses based on the type of search results, improving your incident response strategy.Custom workflows can streamline the process of handling different types of findings, ensuring that your security team can act swiftly and effectively.
Common Pitfalls
1
One common pitfall is underestimating the complexity of capturing screenshots from AJAX-heavy sites. Many developers may assume that traditional screenshot tools will suffice, but they often fail to render dynamic content correctly.
To avoid this, using a tool like Sketchy that incorporates lazy rendering techniques is crucial for ensuring accurate captures of such sites.
Related Concepts
Web Application Security Monitoring
Automated Search And Analysis Tools
Dynamic Content Rendering Challenges