General Architecture Overview
Key Components
- LanceDB architecture is deployed in an AKS cluster within its own VPC
- Client applications connect to the cluster securely using Azure Private Link
- AKS cluster is granted Azure Blob Storage read/write permissions using Azure Workload Identity
- Azure EventHub can be used as the message queue by LanceDB Enterprise for internal message communication (alternative: self-hosted Kafka cluster in AKS)
Read Path Architecture
Read Path Flow
- Client Application sends query request through Private Link
- Query Nodes receive and process the request
- Plan Executors optimize and execute the query using distributed data cache to speed up read queries
- Azure Blob Storage stores data and indices in Lance, while Plan Executors maintain distributed cache for performance
Write Path Architecture
Write Path Flow
Query nodes write data and indices synchronously to Azure Blob Storage in Lance data format while asynchronously sending data modification events to Azure EventHub (or self-hosted Kafka cluster). These write events are processed by the Lance Agent, which launches indexing pods or data optimization pods to optimize data for better read performance.Deployment Options
Storage Architecture Support
Deployment Models
LanceDB Enterprise supports three deployment models on Azure:1. Fully Managed Service
- Infrastructure and storage in LanceDB’s Azure account
- Complete management by LanceDB team
- Simplest setup for customers
2. BYOC (Bring Your Own Cloud)
- Infrastructure and storage in customer’s Azure account
- Fully Managed by LanceDB
- Full control over data residency
3. Hybrid - Bring Your Own Container
- Infrastructure in LanceDB’s account
- Storage containers in customer’s account
For private deployments, high performance at extreme scale, or if you have strict security requirements, contact us about LanceDB Enterprise.