242 lines
8.6 KiB
Markdown
242 lines
8.6 KiB
Markdown
# LightRAG Production System
|
|
|
|
A production-grade RAG system using LightRAG with GPU-accelerated OCR, multi-database storage, and advanced document processing.
|
|
|
|
## System Overview
|
|
|
|
This system provides:
|
|
- **GPU-accelerated OCR** using PaddleOCR with RTX 4070 Super
|
|
- **Multi-database storage** (Redis, Neo4j, Qdrant, PostgreSQL)
|
|
- **Advanced document processing** for PDF, images, Office documents
|
|
- **Snowflake Arctic Embed** for embeddings
|
|
- **DeepSeek API** for LLM functionality
|
|
- **Jina Reranker** for search optimization
|
|
|
|
## Quick Start
|
|
|
|
### Prerequisites
|
|
|
|
1. **Install Ollama** and pull required models:
|
|
```bash
|
|
ollama pull snowflake-arctic-embed2:latest
|
|
ollama pull jina-reranker-v2:latest
|
|
ollama pull mistral-nemo:latest
|
|
```
|
|
|
|
2. **Ensure databases are running**:
|
|
- Redis (Memurai) on `localhost:6379`
|
|
- Neo4j on `bolt://localhost:7687` (user: `neo4j`, password: `jleu1212`)
|
|
- Qdrant on `http://localhost:6333`
|
|
- PostgreSQL on `localhost:5432` (database: `rag_anything`, user: `jleu3482`, password: `jleu1212`)
|
|
|
|
3. **Install Python dependencies**:
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
### Running the System
|
|
|
|
#### Method 1: Using zrun.bat (Windows)
|
|
```bash
|
|
zrun.bat
|
|
```
|
|
|
|
#### Method 2: Manual Start
|
|
```bash
|
|
cd LightRAG-main
|
|
python -m lightrag.api.lightrag_server --port 3015 --host 0.0.0.0 --working-dir rag_storage --input-dir ../inputs --key jleu1212 --auto-scan-at-startup --llm-binding openai --embedding-binding ollama --rerank-binding jina
|
|
```
|
|
|
|
### Access the Web UI
|
|
|
|
Open your browser to: http://localhost:3015/webui/
|
|
|
|
**Login Credentials:**
|
|
- Username: `jleu3482`
|
|
- Password: `jleu1212`
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables (`.env`)
|
|
|
|
Key configurations:
|
|
- **LLM**: DeepSeek API with GPU acceleration
|
|
- **Embeddings**: Snowflake Arctic Embed 2 (1024 dimensions)
|
|
- **Reranker**: Jina Reranker v2
|
|
- **Storage**: Redis + Neo4j + Qdrant + PostgreSQL
|
|
- **OCR**: PaddleOCR with GPU acceleration
|
|
|
|
### Performance Settings
|
|
- **Chunk Size**: 1200 tokens
|
|
- **Chunk Overlap**: 100 tokens
|
|
- **Max Parallel Insert**: 4 operations
|
|
- **Max Graph Nodes**: 1000 nodes
|
|
- **GPU Acceleration**: Enabled for OCR and embeddings
|
|
|
|
## Document Processing
|
|
|
|
### Enhanced Processing Pipeline with Dependency Isolation
|
|
|
|
The system now features a sophisticated document processing pipeline with complete dependency isolation between OCR and image classification components:
|
|
|
|
#### Processing Flow:
|
|
1. **Text-First Extraction**: All file types attempt text extraction first
|
|
2. **Image Extraction**: Extract images from documents (PDF, DOCX, images)
|
|
3. **OCR Processing**: PaddleOCR with GPU acceleration for text extraction from images
|
|
4. **Image Classification**: OpenCLIP in isolated environment for image content classification
|
|
5. **Metadata Integration**: Classification results included in searchable content
|
|
6. **Vector Embedding**: Snowflake Arctic Embed 2
|
|
7. **Multi-Database Storage**: Neo4j for knowledge graphs + Qdrant for vector search
|
|
|
|
#### Dependency Isolation Architecture:
|
|
- **PaddleOCR**: Runs in main environment with GPU-only operation
|
|
- **OpenCLIP**: Runs in isolated virtual environment (`openclip_gpu_env`) to avoid dependency conflicts
|
|
- **Persistent Classifier**: Fast GPU classifier with batch processing (16.6x performance improvement)
|
|
|
|
#### Supported File Types
|
|
- **PDF** (text-based and scanned): PyMuPDF + PaddleOCR + Tabula table extraction
|
|
- **Images**: JPEG, PNG, BMP, TIFF, GIF, WebP (OCR with table detection)
|
|
- **Office Documents**: DOC, DOCX, PPT, PPTX, XLS, XLSX (native table extraction)
|
|
- **Text Files**: TXT, MD (text pattern table detection)
|
|
- **HTML**: BeautifulSoup4 (table extraction from HTML tables)
|
|
|
|
### Table Extraction Capabilities
|
|
|
|
LightRAG now features enhanced table recognition with a hybrid approach for optimal speed and accuracy:
|
|
|
|
#### Table Extraction Methods
|
|
1. **Tabula Integration** (Digital PDFs):
|
|
- Extracts tables from PDFs with text layers using Tabula library
|
|
- Supports both lattice (bordered) and stream (borderless) table detection
|
|
- Non-AI approach with excellent accuracy for digital PDFs
|
|
- Fast processing with direct PDF parsing
|
|
|
|
2. **Enhanced OCR Heuristics** (Scanned Documents):
|
|
- Advanced layout analysis of OCR bounding boxes
|
|
- Adaptive row grouping based on text height and vertical alignment
|
|
- Column clustering with dynamic threshold detection
|
|
- Header row detection and table boundary validation
|
|
- Non-AI approach optimized for scanned documents and images
|
|
|
|
3. **Text Pattern Detection** (Simple Tables):
|
|
- Detects pipe (`|`) and tab-separated tables in text content
|
|
- Identifies table-like structures in plain text documents
|
|
|
|
#### Hybrid Processing Strategy
|
|
- **Digital PDFs**: Tabula extraction first, fallback to text pattern detection
|
|
- **Scanned PDFs**: OCR with enhanced heuristic table detection
|
|
- **Images**: OCR-based table extraction only
|
|
- **Office Documents**: Native table extraction from DOCX/XLSX formats
|
|
|
|
#### Performance Characteristics
|
|
- **Non-AI Methods**: All table extraction methods are non-AI, ensuring fast processing
|
|
- **Speed**: Tabula extraction is near-instant for digital PDFs; OCR heuristics add minimal overhead
|
|
- **Accuracy**: High accuracy for digital PDFs with Tabula; good accuracy for scanned documents with enhanced heuristics
|
|
- **Integration**: Extracted tables are included in searchable content and metadata
|
|
|
|
#### Configuration
|
|
- Tabula is automatically used when available (requires `tabula-py>=2.8.0`)
|
|
- Enhanced OCR heuristics are enabled by default in the optimized OCR processor
|
|
- Table extraction can be disabled via configuration if needed
|
|
|
|
### Image Classification Features
|
|
- **Object Detection**: Identifies objects in images (e.g., "a photo of a bee")
|
|
- **Content Understanding**: Classifies document elements (screenshots, charts, diagrams)
|
|
- **Search Integration**: Classification metadata included in indexed content
|
|
- **High Accuracy**: 100% confidence bee detection in test documents
|
|
|
|
#### Test Document
|
|
Use `test.docx` containing a bee image to verify the complete classification workflow:
|
|
```bash
|
|
python test.py
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
### Health Check
|
|
```bash
|
|
curl http://localhost:3015/api/health
|
|
```
|
|
|
|
### System Status
|
|
```bash
|
|
curl http://localhost:3015/health
|
|
```
|
|
|
|
### Document Upload
|
|
```bash
|
|
curl -X POST -F "file=@document.pdf" http://localhost:3015/api/upload -H "X-API-Key: jleu1212"
|
|
```
|
|
|
|
### Search
|
|
```bash
|
|
curl -X POST http://localhost:3015/api/search -H "Content-Type: application/json" -H "X-API-Key: jleu1212" -d '{"query": "your search query"}'
|
|
```
|
|
|
|
## GPU Configuration
|
|
|
|
The system is optimized for NVIDIA RTX 4070 Super with:
|
|
- CUDA 11.8
|
|
- cuDNN 8.6.0
|
|
- PaddlePaddle GPU version
|
|
- PyTorch with CUDA support
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
1. **OCR not working**: Check CUDA installation and PaddleOCR GPU support
|
|
2. **Database connection errors**: Verify all databases are running
|
|
3. **Model loading errors**: Ensure Ollama models are downloaded
|
|
4. **GPU memory issues**: Reduce batch sizes in configuration
|
|
|
|
### Logs
|
|
Check `lightrag.log` for detailed processing logs and error information.
|
|
|
|
## Production Validation
|
|
|
|
The system has been validated with comprehensive document processing:
|
|
|
|
### Core Functionality
|
|
- ✅ OCR PDF upload and indexing
|
|
- ✅ Multi-database storage (Redis, Neo4j, Qdrant, PostgreSQL)
|
|
- ✅ GPU-accelerated PaddleOCR with RTX 4070 Super
|
|
- ✅ DeepSeek API integration with regional restriction fix
|
|
- ✅ Snowflake Arctic Embed embeddings (1024 dimensions)
|
|
- ✅ Web UI authentication and document management
|
|
- ✅ Search functionality with OCR content
|
|
|
|
### Advanced Features
|
|
- ✅ Complete dependency isolation between PaddleOCR and OpenCLIP
|
|
- ✅ Image classification with OpenCLIP in isolated environment
|
|
- ✅ Bee image detection at 100% confidence
|
|
- ✅ Classification metadata included in searchable content
|
|
- ✅ Persistent GPU classifier with batch processing
|
|
- ✅ Text-first extraction for all file types
|
|
|
|
### Test Documents
|
|
- Use `ocr.pdf` to test OCR functionality
|
|
- Use `test.docx` containing bee image to test complete classification workflow:
|
|
```bash
|
|
python test.py
|
|
```
|
|
|
|
### Bee Search Verification
|
|
The system successfully indexes and searches for image classification content:
|
|
- Search for "bee" returns documents with bee images
|
|
- Search for "photo of a bee" returns relevant content
|
|
- Image classification metadata is searchable in LightRAG
|
|
|
|
## Performance Optimization
|
|
|
|
- Enable GPU acceleration for OCR and embeddings
|
|
- Use parallel processing for document ingestion
|
|
- Optimize chunk sizes for your document types
|
|
- Monitor database performance and connection pools
|
|
|
|
## Security
|
|
|
|
- API key authentication for upload and search
|
|
- Secure database connections
|
|
- Input validation and sanitization
|
|
- Rate limiting on API endpoints |