Test: Added auto-commit documentation and test file

GITEA_SETUP_AND_AUTOCOMMIT.md

@@ -0,0 +1,203 @@
+# Gitea Setup and Auto-Commit Documentation
+
+## Overview
+
+This document describes the Gitea repository setup and auto-commit functionality implemented for the LightRAG project. The system includes:
+
+1. **Document Download Endpoint**: Clickable hyperlinks for document references in search results
+2. **Gitea Repository**: Complete version control setup with automatic commits
+3. **Auto-Commit Script**: Script to automatically commit and push changes
+
+## Repository Details
+
+- **URL**: https://git.mtrcompute.com/jleu3482/railseek6
+- **Clone URL**: http://localhost:8467/jleu3482/railseek6.git
+- **Credentials**: 
+  - Username: `jleu3482`
+  - Password: `jleu1212`
+- **Initial Commit**: "Initial commit: LightRAG project with document download and auto-commit"
+
+## Document Download Implementation
+
+### API Endpoint
+Added to `LightRAG-main/lightrag/api/routers/document_routes.py`:
+```python
+@router.get("/documents/{doc_id}/download")
+async def download_document(doc_id: str):
+    """
+    Download a document by ID.
+    Returns the document file for download with appropriate headers.
+    """
+    # Implementation details...
+```
+
+### Web UI Integration
+Modified `LightRAG-main/webui/index.html` to make document references clickable:
+- Added JavaScript to convert document filenames to clickable download links
+- Each reference now includes a download button with the URL: `/api/documents/{doc_id}/download`
+
+## Gitea Setup Process
+
+### 1. User Registration
+Created Gitea user `jleu3482` with email `slclabs@gmail.com` and password `jleu1212`.
+
+### 2. Repository Creation
+Created repository `railseek6` via Gitea API:
+```bash
+curl -u jleu3482:jleu1212 -X POST https://git.mtrcompute.com/api/v1/user/repos \
+  -H "Content-Type: application/json" \
+  -d '{"name": "railseek6", "description": "LightRAG project with document download and auto-commit functionality", "private": false}'
+```
+
+### 3. Local Git Configuration
+```bash
+git init
+git config user.name "jleu3482"
+git config user.email "slclabs@gmail.com"
+git remote add origin http://localhost:8467/jleu3482/railseek6.git
+git add -A
+git commit -m "Initial commit: LightRAG project with document download and auto-commit"
+git push -u origin master
+```
+
+## Auto-Commit Script
+
+### File: `auto_commit_final.py`
+
+This script provides automatic commit and push functionality for future changes.
+
+### Usage
+```bash
+# Basic usage (auto-generates commit message)
+python auto_commit_final.py
+
+# With custom commit message
+python auto_commit_final.py "Fixed OCR processing issue"
+
+# For major changes (recommended)
+python auto_commit_final.py "Major: Implemented GPU acceleration for image classification"
+```
+
+### Features
+1. **Automatic Change Detection**: Checks `git status` for changes
+2. **Add All Changes**: Automatically stages all modified files
+3. **Commit with Message**: Uses provided message or generates timestamp-based message
+4. **Push to Remote**: Pushes to Gitea repository
+5. **Error Handling**: Includes fallback mechanisms for push failures
+6. **Git Configuration**: Ensures proper user.name and user.email are set
+
+### Integration with Development Workflow
+
+For major changes, run the auto-commit script:
+```bash
+python auto_commit_final.py "Description of major changes made"
+```
+
+The script will:
+1. Detect all changed files
+2. Stage them for commit
+3. Commit with the provided message
+4. Push to the Gitea repository
+5. Display the latest commits for verification
+
+## Testing the Complete Workflow
+
+### 1. Document Download Test
+1. Upload a document to LightRAG
+2. Perform a search query
+3. Check that document references in results are clickable
+4. Click a reference to download the document
+
+### 2. Auto-Commit Test
+1. Make a change to any file
+2. Run `python auto_commit_final.py "Test commit"`
+3. Verify the commit appears at: https://git.mtrcompute.com/jleu3482/railseek6
+
+### 3. Repository Verification
+```bash
+# Check repository status
+git status
+
+# View commit history
+git log --oneline -5
+
+# Verify remote connection
+git remote -v
+```
+
+## Troubleshooting
+
+### Git Push Authentication Issues
+If `git push` fails due to authentication:
+1. Update remote URL with credentials:
+   ```bash
+   git remote set-url origin http://jleu3482:jleu1212@localhost:8467/jleu3482/railseek6.git
+   ```
+
+2. Or use the auto-commit script which includes credential fallback.
+
+### Gitea API Access
+For programmatic access to Gitea:
+```python
+import requests
+response = requests.get(
+    "https://git.mtrcompute.com/api/v1/user/repos",
+    auth=("jleu3482", "jleu1212")
+)
+```
+
+### Large Repository Handling
+The repository contains 42,417 files (2.6 GB). Git operations may take time:
+- Initial push: ~1 minute
+- Subsequent commits: Faster due to delta compression
+
+## Best Practices
+
+### Commit Messages
+- Use descriptive messages that explain what changed
+- Format: `"Category: Brief description"`
+- Examples:
+  - `"Feature: Added document download endpoint"`
+  - `"Fix: Resolved OCR timeout issue"`
+  - `"Refactor: Optimized image classification pipeline"`
+
+### When to Commit
+- After implementing a new feature
+- After fixing a bug
+- After significant refactoring
+- Before major changes that could break functionality
+
+### Repository Maintenance
+- Regularly pull updates if working in team
+- Use branches for experimental features
+- Keep commit history clean and meaningful
+
+## Files Created/Modified
+
+### Modified Files
+1. `LightRAG-main/lightrag/api/routers/document_routes.py` - Added download endpoint
+2. `LightRAG-main/webui/index.html` - Added clickable document references
+
+### Created Scripts
+1. `auto_commit_final.py` - Main auto-commit script
+2. `register_gitea_user.py` - User registration helper
+3. `create_gitea_repo.py` - Repository creation helper
+4. `test_gitea_login.py` - Login verification
+
+## Future Enhancements
+
+1. **Git Hooks**: Pre-commit hooks for code quality checks
+2. **CI/CD Integration**: Automated testing on push
+3. **Branch Protection**: Require reviews for main branch
+4. **Issue Tracking**: Link commits to Gitea issues
+5. **Release Management**: Tag releases for version control
+
+## Conclusion
+
+The Gitea repository setup provides:
+- Complete version control for the LightRAG project
+- Clickable document download links for users
+- Automated commit workflow for developers
+- Centralized code management with self-hosted Gitea
+
+All major changes to the project should now be committed using the auto-commit script to maintain a complete history of development.

auto_commit_final.py

@@ -1,186 +1,130 @@
 #!/usr/bin/env python3
 """
-Final auto-commit solution that tries multiple methods:
-1. First try: Use standard git if available
-2. Second try: Use Go-Git with proper environment
-3. Third try: Use Gitea API directly
+Auto-commit script for LightRAG project.
+Usage: python auto_commit_final.py "Commit message describing changes"
 """
 
-import os
-import sys
 import subprocess
+import sys
+import os
 from datetime import datetime
 
-def check_git_in_path():
-    """Check if git is available in PATH"""
+def run_command(cmd, cwd=None):
+    """Run a shell command and return output."""
     try:
-        result = subprocess.run(
-            ["git", "--version"],
-            capture_output=True,
-            text=True
-        )
-        return result.returncode == 0
-    except:
-        return False
-
-def check_gogit():
-    """Check if Go-Git is available"""
-    git_exe = r"C:\Program Files\gitea\git.exe"
-    return os.path.exists(git_exe)
-
-def run_standard_git(description):
-    """Commit using standard git in PATH"""
-    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
-    commit_message = f"{description}\n\nAuto-commit at: {timestamp}"
-    
-    commands = [
-        ["git", "add", "."],
-        ["git", "commit", "-m", commit_message],
-        ["git", "push", "origin", "main"]
-    ]
-    
-    print("Using standard Git...")
-    for cmd in commands:
-        print(f"Running: {' '.join(cmd)}")
-        try:
-            result = subprocess.run(cmd, capture_output=True, text=True)
-            if result.returncode != 0:
-                print(f"Error: {result.stderr}")
-                return False
-        except Exception as e:
-            print(f"Exception: {e}")
-            return False
-    
-    return True
-
-def run_gogit(description):
-    """Commit using Go-Git"""
-    git_exe = r"C:\Program Files\gitea\git.exe"
-    gitea_dir = os.path.dirname(git_exe)
-    
-    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
-    commit_message = f"{description}\n\nAuto-commit at: {timestamp}"
-    
-    print("Using Go-Git...")
-    
-    # Try with gogit.bat wrapper if it exists
-    if os.path.exists("gogit.bat"):
-        print("Using gogit.bat wrapper...")
-        commands = [
-            ["gogit.bat", "add", "."],
-            ["gogit.bat", "commit", "-m", commit_message],
-            ["gogit.bat", "push", "origin", "main"]
-        ]
-    else:
-        # Try direct git.exe with environment
-        print(f"Using git.exe directly from {gitea_dir}...")
-        env = os.environ.copy()
-        env["GITEA_WORK_DIR"] = gitea_dir
-        env["GITEA_CUSTOM_DIR"] = os.path.join(gitea_dir, "custom")
-        
-        commands = [
-            [git_exe, "add", "."],
-            [git_exe, "commit", "-m", commit_message],
-            [git_exe, "push", "origin", "main"]
-        ]
-    
-    for cmd in commands:
-        print(f"Running: {' '.join(cmd)}")
-        try:
-            result = subprocess.run(
-                cmd,
-                capture_output=True,
-                text=True,
-                cwd=gitea_dir if "gogit.bat" not in cmd[0] else None
-            )
-            print(f"Output: {result.stdout}")
-            if result.returncode != 0:
-                print(f"Error: {result.stderr}")
-        except Exception as e:
-            print(f"Exception: {e}")
-    
-    return True
-
-def run_gitea_api(description):
-    """Commit using Gitea API"""
-    print("Using Gitea API...")
-    try:
-        result = subprocess.run(
-            [sys.executable, "final_gitea_commit.py", description],
-            capture_output=True,
-            text=True
-        )
-        print(result.stdout)
-        if result.returncode == 0:
-            return True
-        else:
-            print(f"API Error: {result.stderr}")
-            return False
+        result = subprocess.run(cmd, shell=True, capture_output=True, text=True, cwd=cwd)
+        return result.returncode, result.stdout, result.stderr
     except Exception as e:
-        print(f"Exception running API: {e}")
+        return 1, "", str(e)
+
+def auto_commit(commit_message=None):
+    """Perform automatic commit and push to Gitea."""
+    
+    # Get commit message from command line or generate one
+    if commit_message:
+        message = commit_message
+    elif len(sys.argv) > 1:
+        message = sys.argv[1]
+    else:
+        # Generate a timestamp-based message
+        timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+        message = f"Auto-commit: {timestamp}"
+    
+    print(f"Auto-commit starting with message: {message}")
+    print("=" * 60)
+    
+    # Step 1: Check git status
+    print("1. Checking git status...")
+    code, out, err = run_command("git status --porcelain")
+    if code != 0:
+        print(f"Error checking git status: {err}")
         return False
     
-def main():
-    if len(sys.argv) < 2:
-        print("Usage: python auto_commit_final.py \"Description of changes\"")
-        print("Example: python auto_commit_final.py \"Added document download endpoint\"")
-        sys.exit(1)
+    if not out.strip():
+        print("No changes to commit.")
+        return True
     
-    description = sys.argv[1]
+    print(f"Changes detected:\n{out}")
     
-    print("=" * 60)
-    print("AUTO-COMMIT SYSTEM")
-    print("=" * 60)
-    print(f"Commit: {description}")
-    print(f"Time: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
-    print()
+    # Step 2: Add all changes
+    print("\n2. Adding all changes...")
+    code, out, err = run_command("git add -A")
+    if code != 0:
+        print(f"Error adding changes: {err}")
+        return False
+    print("Changes added.")
     
-    # Check available methods
-    has_standard_git = check_git_in_path()
-    has_gogit = check_gogit()
+    # Step 3: Commit
+    print(f"\n3. Committing with message: '{message}'")
+    code, out, err = run_command(f'git commit -m "{message}"')
+    if code != 0:
+        print(f"Error committing: {err}")
+        return False
+    print(f"Commit successful: {out.strip()}")
     
-    print("Available methods:")
-    print(f"  Standard Git: {'✓' if has_standard_git else '✗'}")
-    print(f"  Go-Git: {'✓' if has_gogit else '✗'}")
-    print(f"  Gitea API: ✓ (always available)")
-    print()
+    # Step 4: Push to remote
+    print("\n4. Pushing to remote repository...")
+    code, out, err = run_command("git push origin master")
+    if code != 0:
+        print(f"Error pushing: {err}")
         
-    # Try methods in order of preference
-    success = False
+        # Try with credentials in URL
+        print("Trying with credentials...")
+        remote_url = "http://jleu3482:jleu1212@localhost:8467/jleu3482/railseek6.git"
+        code, out, err = run_command(f'git push {remote_url} master')
+        if code != 0:
+            print(f"Push failed: {err}")
+            return False
     
-    if has_standard_git:
-        print("Trying standard Git first...")
-        success = run_standard_git(description)
-        if success:
-            print("✓ Success with standard Git")
+    print("Push successful!")
     
-    if not success and has_gogit:
-        print("\nStandard Git failed, trying Go-Git...")
-        success = run_gogit(description)
-        if success:
-            print("✓ Success with Go-Git")
+    # Step 5: Show git log
+    print("\n5. Latest commit:")
+    code, out, err = run_command("git log --oneline -3")
+    if code == 0:
+        print(out)
     
-    if not success:
-        print("\nBoth Git methods failed, using Gitea API...")
-        success = run_gitea_api(description)
-        if success:
-            print("✓ Success with Gitea API")
+    print("\n" + "=" * 60)
+    print("Auto-commit completed successfully!")
+    return True
 
-    print()
-    print("=" * 60)
-    print("RESULT")
-    print("=" * 60)
-    if success:
-        print(f"✅ SUCCESS: Changes committed to repository")
-        print(f"📁 Repository: https://git.mtrcompute.com/jleu3482/railseek6")
+def setup_git_config():
+    """Ensure git config is properly set."""
+    print("Checking git configuration...")
+    
+    # Set user name if not set
+    code, out, err = run_command("git config user.name")
+    if not out.strip():
+        run_command('git config user.name "jleu3482"')
+        print("Set user.name to jleu3482")
+    
+    # Set user email if not set
+    code, out, err = run_command("git config user.email")
+    if not out.strip():
+        run_command('git config user.email "slclabs@gmail.com"')
+        print("Set user.email to slclabs@gmail.com")
+    
+    # Set remote URL
+    remote_url = "http://localhost:8467/jleu3482/railseek6.git"
+    code, out, err = run_command("git remote get-url origin")
+    if code != 0 or not out.strip():
+        run_command(f'git remote add origin {remote_url}')
+        print(f"Set remote origin to {remote_url}")
     else:
-        print(f"❌ FAILED: Could not commit changes")
-        print("Please check:")
-        print("1. Git installation and PATH")
-        print("2. Gitea credentials")
-        print("3. Network connection")
-    
-    print("=" * 60)
+        print(f"Remote origin already set to: {out.strip()}")
 
 if __name__ == "__main__":
-    main()
+    print("LightRAG Auto-Commit Script")
+    print("=" * 60)
+    
+    # Setup git config
+    setup_git_config()
+    
+    # Run auto-commit
+    success = auto_commit()
+    
+    if success:
+        sys.exit(0)
+    else:
+        print("Auto-commit failed!")
+        sys.exit(1)

create_gitea_repo.py

@@ -0,0 +1,82 @@
+import requests
+import re
+import json
+
+def create_gitea_repository():
+    session = requests.Session()
+    
+    # Login first
+    login_url = "https://git.mtrcompute.com/user/login"
+    response = session.get(login_url)
+    
+    # Extract CSRF token
+    csrf_token = None
+    match = re.search(r'csrfToken:\s*\'([^\']+)\'', response.text)
+    if match:
+        csrf_token = match.group(1)
+    
+    if not csrf_token:
+        print("Could not find CSRF token")
+        return False
+    
+    # Login
+    data = {
+        'user_name': 'jleu3482',
+        'password': 'jleu1212',
+        'csrf_token': csrf_token
+    }
+    
+    headers = {
+        'Content-Type': 'application/x-www-form-urlencoded',
+        'Referer': login_url
+    }
+    
+    response = session.post(login_url, data=data, headers=headers, allow_redirects=False)
+    
+    if response.status_code not in [302, 303]:
+        print(f"Login failed. Status: {response.status_code}")
+        return False
+    
+    print("Login successful!")
+    
+    # Now create repository via API
+    api_url = "https://git.mtrcompute.com/api/v1/user/repos"
+    
+    repo_data = {
+        'name': 'railseek6',
+        'description': 'LightRAG project with document download and auto-commit functionality',
+        'private': False,
+        'auto_init': False,
+        'gitignores': '',
+        'license': '',
+        'readme': '',
+        'default_branch': 'master'
+    }
+    
+    # Get CSRF token for API (might need to be in headers)
+    response = session.get("https://git.mtrcompute.com/api/v1/user")
+    print(f"User API status: {response.status_code}")
+    
+    # Try to create repo
+    headers = {
+        'Content-Type': 'application/json',
+        'X-Csrf-Token': csrf_token
+    }
+    
+    response = session.post(api_url, json=repo_data, headers=headers)
+    
+    print(f"Create repo status: {response.status_code}")
+    print(f"Create repo response: {response.text}")
+    
+    if response.status_code == 201:
+        print("Repository created successfully!")
+        return True
+    elif response.status_code == 409:
+        print("Repository already exists")
+        return True
+    else:
+        print("Failed to create repository")
+        return False
+
+if __name__ == "__main__":
+    create_gitea_repository()

make_commit.bat

@@ -0,0 +1,2 @@
+@echo off
+git commit -m "Initial commit: LightRAG project with document download endpoint and auto-commit system"

register_gitea_user.py

@@ -0,0 +1,57 @@
+import requests
+import re
+
+def register_gitea_user():
+    # First get the CSRF token from the registration page
+    session = requests.Session()
+    
+    # Get registration page
+    url = "https://git.mtrcompute.com/user/sign_up"
+    response = session.get(url)
+    
+    # Extract CSRF token
+    csrf_token = None
+    match = re.search(r'csrfToken:\s*\'([^\']+)\'', response.text)
+    if match:
+        csrf_token = match.group(1)
+    else:
+        # Try another pattern
+        match = re.search(r'name="csrf_token"\s+value="([^"]+)"', response.text)
+        if match:
+            csrf_token = match.group(1)
+    
+    if not csrf_token:
+        print("Could not find CSRF token")
+        return False
+    
+    print(f"CSRF Token: {csrf_token}")
+    
+    # Prepare registration data
+    data = {
+        'user_name': 'jleu3482',
+        'email': 'slclabs@gmail.com',
+        'password': 'jleu1212',
+        'retype': 'jleu1212',
+        'csrf_token': csrf_token
+    }
+    
+    # Submit registration
+    headers = {
+        'Content-Type': 'application/x-www-form-urlencoded',
+        'Referer': url
+    }
+    
+    response = session.post(url, data=data, headers=headers, allow_redirects=False)
+    
+    print(f"Status Code: {response.status_code}")
+    print(f"Headers: {response.headers}")
+    
+    if response.status_code == 302:
+        print("Registration successful! Redirecting...")
+        return True
+    else:
+        print(f"Registration failed. Response: {response.text[:500]}")
+        return False
+
+if __name__ == "__main__":
+    register_gitea_user()

test_gitea_login.py

@@ -0,0 +1,58 @@
+import requests
+import re
+
+def test_gitea_login():
+    session = requests.Session()
+    
+    # First get login page and CSRF token
+    login_url = "https://git.mtrcompute.com/user/login"
+    response = session.get(login_url)
+    
+    # Extract CSRF token
+    csrf_token = None
+    match = re.search(r'csrfToken:\s*\'([^\']+)\'', response.text)
+    if match:
+        csrf_token = match.group(1)
+    else:
+        match = re.search(r'name="csrf_token"\s+value="([^"]+)"', response.text)
+        if match:
+            csrf_token = match.group(1)
+    
+    if not csrf_token:
+        print("Could not find CSRF token")
+        return False
+    
+    print(f"CSRF Token: {csrf_token}")
+    
+    # Login
+    data = {
+        'user_name': 'jleu3482',
+        'password': 'jleu1212',
+        'csrf_token': csrf_token
+    }
+    
+    headers = {
+        'Content-Type': 'application/x-www-form-urlencoded',
+        'Referer': login_url
+    }
+    
+    response = session.post(login_url, data=data, headers=headers, allow_redirects=False)
+    
+    print(f"Login Status Code: {response.status_code}")
+    print(f"Login Headers: {response.headers}")
+    
+    if response.status_code == 302:
+        print("Login successful!")
+        
+        # Try to access user repos API
+        api_url = "https://git.mtrcompute.com/api/v1/user/repos"
+        response = session.get(api_url)
+        print(f"API Status: {response.status_code}")
+        print(f"API Response: {response.text[:500]}")
+        return True
+    else:
+        print(f"Login failed. Response: {response.text[:500]}")
+        return False
+
+if __name__ == "__main__":
+    test_gitea_login()