How we built enterprise search to be secure and private

Many don’t know that “Slack” is in fact a backronym—it stands for “Searchable Log of all Communication and Knowledge”. And these days, it’s not just a searchable log: with Slack AI, Slack is now an intelligent log, leveraging the latest in generative AI to securely surface powerful, time-saving insights. We built Slack AI from the…

Slack logo
Slack
•
Ian Hoffman
•7 min read•advanced•
--
•View Original
AWSChefOAuthPythonRetrieval Augmented GenerationTypeScript

Overview

The article discusses the development of Slack's enterprise search functionality, emphasizing its security and privacy features. It outlines the principles guiding the architecture, including data handling and user permissions, while integrating external applications like Google Drive and GitHub.

What You'll Learn

1

How to implement secure enterprise search that respects user privacy

2

Why using real-time data fetching enhances data accuracy in search results

3

When to apply OAuth for user authorization in external data access

Prerequisites & Requirements

  • Understanding of OAuth and data privacy principles
  • Familiarity with API integrations(optional)

Key Questions Answered

How does Slack ensure customer data remains secure during enterprise search?
Slack ensures customer data remains secure by never storing external source data in its databases and only fetching real-time data that users can access. This approach adheres to strict privacy principles, ensuring that data does not leave Slack's trust boundary.
What principles guide the architecture of Slack's enterprise search?
The architecture of Slack's enterprise search is guided by principles such as never storing external data, maintaining up-to-date permissions, requiring explicit user consent for external data access, and adhering to the principle of least privilege.
Why is real-time data fetching preferred over indexing in Slack's search?
Real-time data fetching is preferred because it ensures that the data returned to users is always current and reflects the latest permissions. This method eliminates the risk of stale data, enhancing the reliability of search results.
How does Slack handle user permissions for external data sources?
Slack requires that users and admins explicitly grant access to external data sources, allowing them to control which data is accessible. This is facilitated through OAuth, ensuring that only authorized actions are performed.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Backend
AWS
Used to host closed-source large language models in a secure environment.
Security
Oauth
Facilitates user authorization for accessing external data sources.

Key Actionable Insights

1
Implement a federated search approach to ensure data privacy and security.
By using real-time data fetching instead of storing external data, you can enhance the security of your application and ensure compliance with privacy regulations.
2
Utilize OAuth for managing user permissions effectively.
This allows users to maintain control over their data access, ensuring that only authorized actions are performed, which is crucial for maintaining trust.
3
Regularly review and update permissions for external data sources.
This practice helps in maintaining data integrity and security, especially as user roles and access needs evolve over time.

Common Pitfalls

1
Failing to implement proper user consent mechanisms for accessing external data can lead to privacy violations.
Without clear consent, users may feel their data is being misused, which can damage trust and lead to compliance issues.
2
Storing external data instead of fetching it in real-time can result in outdated information being presented to users.
This can mislead users and affect decision-making, highlighting the importance of real-time data access.