ModerateWinGuy/BDFR_Web
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(UI): initial working frontend UI

Browse Source
This commit is contained in:
ModerateWinGuy
2025-10-09 17:12:55 +13:00
parent e8972cae38
commit 9e61d18bf6
26 changed files with 6885 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bdfr/API_README.md
+430
View File
@@ -0,0 +1,430 @@
# BDFR API Layer
The BDFR API layer provides a direct integration interface for the web interface, eliminating the need for subprocess console parsing. It wraps the existing BDFR core classes and provides a clean API with structured progress callbacks.
## Overview
The API layer consists of several key components:
- **BDFRManager**: Main API interface class
- **ProgressCallback**: Abstract base class for progress notifications
- **ProgressEvent**: Structured progress event data
- **DownloadType**: Enumeration of supported download types
- **DownloadStatus**: Enumeration of download statuses
## Quick Start
### Basic Usage
```python
from bdfr.api import BDFRManager, LoggingCallback
# Create a manager
manager = BDFRManager("./downloads")
# Download from a subreddit
download_id = manager.download_subreddit(
"python",
limit=50,
sort="hot",
no_dupes=True
)
# Check status
status = manager.get_download_status(download_id)
print(f"Progress: {status['progress']}%")
```
### Advanced Usage with Custom Callbacks
```python
from bdfr.api import BDFRManager, ProgressCallback, ProgressEvent
class WebSocketCallback(ProgressCallback):
def __init__(self, websocket):
self.websocket = websocket
async def on_progress(self, event: ProgressEvent):
await self.websocket.send_json(event.to_dict())
async def on_error(self, event: ProgressEvent):
await self.websocket.send_json(event.to_dict())
async def on_completed(self, event: ProgressEvent):
await self.websocket.send_json(event.to_dict())
# Use custom callbacks
callbacks = [LoggingCallback(), WebSocketCallback(ws)]
download_id = manager.download_subreddit("technology", limit=100, progress_callbacks=callbacks)
```
## API Reference
### BDFRManager
The main API class that manages downloads and provides the primary interface.
#### Constructor
```python
BDFRManager(download_directory: Optional[Union[str, Path]] = None)
```
- `download_directory`: Base directory for downloads. Defaults to current directory.
#### Methods
##### `create_download()`
Create a new download operation.
```python
create_download(
download_type: DownloadType,
name: str,
config: Optional[Configuration] = None,
progress_callbacks: Optional[List[ProgressCallback]] = None
) -> str
```
Returns a download ID for tracking the operation.
##### `start_download()`
Start a download operation.
```python
start_download(download_id: str) -> bool
```
Returns `True` if started successfully.
##### `get_download_status()`
Get the current status of a download.
```python
get_download_status(download_id: str) -> Optional[Dict[str, Any]]
```
Returns download status information or `None` if not found.
##### `cancel_download()`
Cancel a running download.
```python
cancel_download(download_id: str) -> bool
```
Returns `True` if cancelled successfully.
##### `list_downloads()`
List all active downloads.
```python
list_downloads() -> Dict[str, Dict[str, Any]]
```
Returns dictionary of download information keyed by download ID.
##### `cleanup_completed()`
Clean up old completed downloads.
```python
cleanup_completed(max_age_seconds: int = 3600) -> int
```
Returns number of downloads cleaned up.
#### Convenience Methods
##### `download_subreddit()`
Download content from a subreddit.
```python
download_subreddit(
subreddit_name: str,
limit: Optional[int] = None,
sort: str = "hot",
time_filter: str = "all",
no_dupes: bool = False,
progress_callbacks: Optional[List[ProgressCallback]] = None
) -> str
```
##### `download_user()`
Download content from a user.
```python
download_user(
username: str,
limit: Optional[int] = None,
submitted: bool = True,
upvoted: bool = False,
saved: bool = False,
no_dupes: bool = False,
progress_callbacks: Optional[List[ProgressCallback]] = None
) -> str
```
##### `archive_subreddit()`
Archive subreddit data (metadata only).
```python
archive_subreddit(
subreddit_name: str,
format_type: str = "json",
limit: Optional[int] = None,
progress_callbacks: Optional[List[ProgressCallback]] = None
) -> str
```
##### `clone_subreddit()`
Clone subreddit (both download and archive).
```python
clone_subreddit(
subreddit_name: str,
limit: Optional[int] = None,
format_type: str = "json",
no_dupes: bool = False,
progress_callbacks: Optional[List[ProgressCallback]] = None
) -> str
```
### ProgressCallback
Abstract base class for implementing progress callbacks.
#### Methods
##### `on_progress(event: ProgressEvent)`
Called when progress is made.
##### `on_error(event: ProgressEvent)`
Called when an error occurs.
##### `on_completed(event: ProgressEvent)`
Called when download is completed.
### ProgressEvent
Represents a progress event with structured data.
#### Attributes
- `event_type`: "progress", "status", "error", or "completed"
- `download_id`: Unique identifier for the download
- `message`: Human-readable message
- `progress`: Progress percentage (0-100)
- `data`: Additional structured data
- `timestamp`: When the event occurred
#### Methods
##### `to_dict() -> Dict[str, Any]`
Convert to dictionary for JSON serialization.
### DownloadType
Enumeration of supported download types:
- `SUBREDDIT`: Download from subreddit
- `USER`: Download from user
- `MULTIREDDIT`: Download from multireddit
- `SUBMISSIONS`: Download specific submissions
- `ARCHIVE`: Archive only (no downloads)
- `CLONE`: Both download and archive
### DownloadStatus
Enumeration of download statuses:
- `QUEUED`: Download is queued but not started
- `RUNNING`: Download is in progress
- `COMPLETED`: Download completed successfully
- `FAILED`: Download failed with an error
- `CANCELLED`: Download was cancelled
- `PAUSED`: Download is paused
## Configuration
The API uses the existing BDFR `Configuration` class. You can pass a custom configuration to `create_download()` or use the convenience methods with their specific parameters.
### Common Configuration Options
- `limit`: Maximum number of posts to process
- `sort`: Sort method (hot, top, new, controversial, rising)
- `time`: Time filter (all, hour, day, week, month, year)
- `no_dupes`: Avoid duplicate downloads
- `make_hard_links`: Create hard links for duplicates
- `format`: Archive format (json, xml, yaml)
## Web Interface Integration
### FastAPI Integration Example
```python
from fastapi import FastAPI, WebSocket
from bdfr.api import get_bdfr_manager, WebSocketCallback
app = FastAPI()
manager = get_bdfr_manager("./downloads")
@app.post("/api/download/subreddit")
async def download_subreddit(subreddit: str, limit: int = 10):
download_id = manager.download_subreddit(subreddit, limit=limit)
return {"download_id": download_id}
@app.websocket("/ws/progress/{download_id}")
async def progress_websocket(websocket: WebSocket, download_id: str):
await websocket.accept()
class FastAPICallback(WebSocketCallback):
def __init__(self):
super().__init__(None)
async def on_progress(self, event: ProgressEvent):
if event.download_id == download_id:
await websocket.send_json(event.to_dict())
# Add callback to existing download or create new one
# (Implementation depends on your specific needs)
```
### Real-time Progress Updates
The API provides structured progress events that can be easily serialized to JSON for web clients:
```python
# Example progress event
{
"event_type": "progress",
"download_id": "123e4567-e89b-12d3-a456-426614174000",
"message": "Downloaded submission abc123 from r/python",
"progress": 45.2,
"data": {
"items_processed": 12,
"items_found": 25,
"current_item": "abc123",
"phase": "downloading_submission"
},
"timestamp": "2023-12-07T10:30:45.123456"
}
```
## Error Handling
The API provides comprehensive error handling:
- **Download errors**: Network issues, authentication problems, etc.
- **Configuration errors**: Invalid parameters, missing files, etc.
- **System errors**: Disk space, permissions, etc.
All errors are captured and reported through the progress callback system with detailed error information.
## Threading and Concurrency
The API is designed to be thread-safe and supports concurrent downloads:
- Each download runs in its own thread
- Progress callbacks are async-safe
- Multiple downloads can run simultaneously
- Thread-safe status tracking
## Logging
The API integrates with Python's logging system:
```python
import logging
# Configure logging
logging.basicConfig(level=logging.INFO)
# Use LoggingCallback for automatic log output
callbacks = [LoggingCallback("my_app")]
```
## Examples
See `bdfr/examples/api_usage.py` for comprehensive examples including:
- Basic usage
- Custom callbacks
- Web integration
- Error handling
- User downloads
- Archive operations
## Migration from Subprocess
### Before (Subprocess)
```python
import subprocess
import json
# Start BDFR via subprocess
proc = subprocess.Popen([
"python", "-m", "bdfr", "download",
"--subreddit", "python",
"--limit", "50",
"./downloads"
], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# Parse console output for progress
while True:
line = proc.stdout.readline().decode().strip()
if not line:
break
# Parse progress from console output...
```
### After (API)
```python
from bdfr.api import BDFRManager, LoggingCallback
# Use API directly
manager = BDFRManager("./downloads")
download_id = manager.download_subreddit("python", limit=50)
# Get structured progress updates
status = manager.get_download_status(download_id)
print(f"Progress: {status['progress']}%")
```
## Benefits
1. **No subprocess overhead**: Direct integration with BDFR core
2. **Structured progress**: Rich progress events instead of console parsing
3. **Better error handling**: Detailed error information and stack traces
4. **Thread-safe**: Concurrent downloads with proper synchronization
5. **Web-friendly**: JSON-serializable progress events
6. **Extensible**: Custom progress callbacks for different integrations
7. **Maintainable**: Clean separation of concerns
## Requirements
- Python 3.7+
- Existing BDFR installation
- Dependencies: `praw`, `requests`, and other BDFR dependencies
## Installation
The API layer is included with BDFR and requires no additional installation. Simply import and use:
```python
from bdfr.api import BDFRManager
bdfr/api.py
+1290
View File
File diff suppressed because it is too large Load Diff
bdfr/archiver.py
+5 -1
View File
@@ -31,6 +31,7 @@ class Archiver(RedditConnector):
def download(self):
for generator in self.reddit_lists:
submission = None
try:
for submission in generator:
try:
@@ -50,7 +51,10 @@ class Archiver(RedditConnector):
except prawcore.PrawcoreException as e:
logger.error(f"Submission {submission.id} failed to be archived due to a PRAW exception: {e}")
except prawcore.PrawcoreException as e:
logger.error(f"The submission after {submission.id} failed to download due to a PRAW exception: {e}")
if submission is not None:
logger.error(f"The submission after {submission.id} failed to download due to a PRAW exception: {e}")
else:
logger.error(f"Download failed due to a PRAW exception: {e}")
logger.debug("Waiting 60 seconds to continue")
sleep(60)
bdfr/cloner.py
+5 -1
View File
@@ -20,6 +20,7 @@ class RedditCloner(RedditDownloader, Archiver):
def download(self):
for generator in self.reddit_lists:
submission = None
try:
for submission in generator:
try:
@@ -28,6 +29,9 @@ class RedditCloner(RedditDownloader, Archiver):
except prawcore.PrawcoreException as e:
logger.error(f"Submission {submission.id} failed to be cloned due to a PRAW exception: {e}")
except prawcore.PrawcoreException as e:
logger.error(f"The submission after {submission.id} failed to download due to a PRAW exception: {e}")
if submission is not None:
logger.error(f"The submission after {submission.id} failed to download due to a PRAW exception: {e}")
else:
logger.error(f"Download failed due to a PRAW exception: {e}")
logger.debug("Waiting 60 seconds to continue")
sleep(60)
bdfr/connector.py
+12
View File
@@ -386,6 +386,18 @@ class RedditConnector(metaclass=ABCMeta):
generators.append(self.reddit_instance.redditor(user).saved(limit=self.args.limit))
except prawcore.PrawcoreException as e:
logger.error(f"User {user} failed to be retrieved due to a PRAW exception: {e}")
# Detect HTTP 429 (rate limiting) and propagate as a hard failure so the UI can show 'failed'
TooManyRequests = getattr(prawcore.exceptions, "TooManyRequests", None)
is_rate_limited = False
if TooManyRequests is not None and isinstance(e, TooManyRequests):
is_rate_limited = True
elif (hasattr(e, "response") and getattr(e.response, "status_code", None) == 429) or "429" in str(e):
is_rate_limited = True
if is_rate_limited:
logger.error("Received HTTP 429 (rate limited). Propagating error to fail the download.")
raise
logger.debug("Waiting 60 seconds to continue")
sleep(60)
return generators
bdfr/downloader.py
+1 -1
View File
@@ -84,7 +84,7 @@ class RedditDownloader(RedditConnector):
except prawcore.PrawcoreException as e:
logger.error(f"Submission {submission.id} failed to download due to a PRAW exception: {e}")
except prawcore.PrawcoreException as e:
submission_id = last_submission_id or "unknown"
submission_id = last_submission_id if last_submission_id is not None else "unknown"
logger.error(f"The submission after {submission_id} failed to download due to a PRAW exception: {e}")
logger.debug("Waiting 60 seconds to continue")
sleep(60)
bdfr/examples/api_usage.py
+409
View File
@@ -0,0 +1,409 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
BDFR API Usage Examples
This file demonstrates how to use the BDFR API layer for direct integration
with the web interface, eliminating the need for subprocess console parsing.
"""
import asyncio
import logging
import sys
from pathlib import Path
from typing import List
# Add the parent directory to the path so we can import bdfr
sys.path.insert(0, str(Path(__file__).parent.parent))
from bdfr.api import (
BDFRManager,
DownloadType,
DownloadStatus,
ProgressEvent,
ProgressCallback,
LoggingCallback,
get_bdfr_manager
)
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='[%(asctime)s] %(levelname)s: %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
logger = logging.getLogger(__name__)
class WebSocketCallback(ProgressCallback):
"""Example callback that simulates WebSocket updates"""
def __init__(self, websocket_id: str = "demo"):
self.websocket_id = websocket_id
async def on_progress(self, event: ProgressEvent):
"""Send progress update to WebSocket"""
print(f"📊 [{self.websocket_id}] Progress: {event.message}")
if event.progress is not None:
print(f" Progress: {event.progress:.1f}%")
if event.data:
print(f" Data: {event.data}")
async def on_error(self, event: ProgressEvent):
"""Send error update to WebSocket"""
print(f"❌ [{self.websocket_id}] ERROR: {event.message}")
if event.data.get('exception'):
print(f" Exception: {event.data['exception']}")
async def on_completed(self, event: ProgressEvent):
"""Send completion update to WebSocket"""
status_icon = "✅" if event.event_type == "completed" else "⚠️"
print(f"{status_icon} [{self.websocket_id}] {event.message}")
if event.data:
print(f" Final stats: {event.data}")
class DatabaseCallback(ProgressCallback):
"""Example callback that saves progress to a database"""
def __init__(self, db_connection_string: str = "sqlite:///progress.db"):
self.db_connection = db_connection_string
async def on_progress(self, event: ProgressEvent):
"""Save progress to database"""
# In a real implementation, you would save to your database
print(f"💾 [DB] Saved progress for {event.download_id}: {event.progress}%")
async def on_error(self, event: ProgressEvent):
"""Save error to database"""
print(f"💾 [DB] Saved error for {event.download_id}: {event.message}")
async def on_completed(self, event: ProgressEvent):
"""Save completion to database"""
print(f"💾 [DB] Saved completion for {event.download_id}")
async def example_basic_usage():
"""Basic usage example"""
print("🚀 Basic BDFR API Usage Example")
print("=" * 50)
# Create a BDFR manager
manager = BDFRManager("./downloads")
# Create a download for a subreddit
download_id = manager.download_subreddit(
"python", # subreddit name
limit=10, # download 10 posts
sort="hot", # sort by hot
no_dupes=True # avoid duplicates
)
print(f"📋 Created download with ID: {download_id}")
# Monitor progress
while True:
status = manager.get_download_status(download_id)
if not status:
print("❌ Download not found!")
break
print(f"📊 Status: {status['status']} | Progress: {status['progress']:.1f}%")
if status['status'] in ['completed', 'failed', 'cancelled']:
print(f"🏁 Download finished with status: {status['status']}")
break
await asyncio.sleep(2) # Check every 2 seconds
return download_id
async def example_advanced_usage():
"""Advanced usage with custom callbacks"""
print("\n🎯 Advanced BDFR API Usage Example")
print("=" * 50)
# Create custom callbacks
callbacks = [
LoggingCallback("web_interface"),
WebSocketCallback("user_123"),
DatabaseCallback()
]
# Create manager with custom download directory
manager = BDFRManager("./custom_downloads")
# Download from multiple subreddits
subreddits = ["programming", "learnprogramming", "Python"]
download_ids = []
for subreddit in subreddits:
download_id = manager.create_download(
DownloadType.SUBREDDIT,
subreddit,
progress_callbacks=callbacks
)
# Start the download
manager.start_download(download_id)
download_ids.append(download_id)
print(f"📋 Started download {download_id} for r/{subreddit}")
# Monitor all downloads
while download_ids:
active_downloads = []
for download_id in download_ids[:]: # Copy list to avoid modification during iteration
status = manager.get_download_status(download_id)
if not status:
print(f"❌ Download {download_id} not found")
download_ids.remove(download_id)
continue
print(f"📊 {download_id}: {status['status']} ({status['progress']:.1f}%)")
if status['status'] in ['completed', 'failed', 'cancelled']:
print(f"🏁 Download {download_id} finished")
download_ids.remove(download_id)
else:
active_downloads.append(download_id)
if not active_downloads:
break
await asyncio.sleep(3) # Check every 3 seconds
return len(download_ids) == 0 # Return success status
async def example_user_download():
"""Example of downloading user content"""
print("\n👤 User Download Example")
print("=" * 50)
manager = get_bdfr_manager() # Use default manager
# Download user's submitted posts
download_id = manager.download_user(
"testuser", # username
limit=25, # 25 posts
submitted=True,
upvoted=False,
saved=False
)
print(f"📋 Created user download: {download_id}")
# Check status periodically
for _ in range(10): # Check for up to 20 seconds
status = manager.get_download_status(download_id)
if not status:
print("❌ Download not found")
break
print(f"📊 Status: {status['status']} | Progress: {status['progress']:.1f}%")
if status['status'] in ['completed', 'failed']:
break
await asyncio.sleep(2)
return download_id
async def example_archive_operation():
"""Example of archiving subreddit data"""
print("\n📚 Archive Operation Example")
print("=" * 50)
manager = BDFRManager("./archives")
# Archive subreddit data (metadata only)
download_id = manager.archive_subreddit(
"dataisbeautiful",
format_type="json",
limit=50
)
print(f"📋 Created archive operation: {download_id}")
# Monitor progress
while True:
status = manager.get_download_status(download_id)
if not status:
print("❌ Archive not found")
break
print(f"📊 Archive status: {status['status']} | Progress: {status['progress']:.1f}%")
if status['status'] in ['completed', 'failed']:
print(f"🏁 Archive finished with status: {status['status']}")
break
await asyncio.sleep(2)
return download_id
async def example_web_integration():
"""Example showing how to integrate with a web application"""
print("\n🌐 Web Integration Example")
print("=" * 50)
# Simulate a web application using the API
class MockWebApp:
def __init__(self):
self.manager = BDFRManager("./web_downloads")
self.active_sessions = {}
async def handle_download_request(self, user_id: str, subreddit: str, limit: int):
"""Handle a download request from the web interface"""
# Create custom callback for this user
callback = WebSocketCallback(f"ws_{user_id}")
# Create and start download
download_id = self.manager.download_subreddit(
subreddit,
limit=limit,
progress_callbacks=[callback]
)
# Track for this user session
if user_id not in self.active_sessions:
self.active_sessions[user_id] = []
self.active_sessions[user_id].append(download_id)
return {
"success": True,
"download_id": download_id,
"message": f"Started download of r/{subreddit} (limit: {limit})"
}
async def get_user_downloads(self, user_id: str):
"""Get all downloads for a user"""
if user_id not in self.active_sessions:
return []
downloads = []
for download_id in self.active_sessions[user_id]:
status = self.manager.get_download_status(download_id)
if status:
downloads.append(status)
return downloads
async def cancel_user_download(self, user_id: str, download_id: str):
"""Cancel a specific download for a user"""
if user_id in self.active_sessions and download_id in self.active_sessions[user_id]:
success = self.manager.cancel_download(download_id)
if success:
self.active_sessions[user_id].remove(download_id)
return {"success": True, "message": "Download cancelled"}
else:
return {"success": False, "message": "Failed to cancel download"}
return {"success": False, "message": "Download not found for user"}
# Simulate web app usage
app = MockWebApp()
# Simulate user requests
user_id = "user123"
# User starts a download
result1 = await app.handle_download_request(user_id, "technology", 20)
print(f"User request result: {result1}")
# User starts another download
result2 = await app.handle_download_request(user_id, "science", 15)
print(f"User request result: {result2}")
# Check user's downloads
user_downloads = await app.get_user_downloads(user_id)
print(f"User has {len(user_downloads)} active downloads:")
for download in user_downloads:
print(f" - {download['id']}: {download['status']} ({download['progress']:.1f}%)")
# Cancel one download
if user_downloads:
cancel_result = await app.cancel_user_download(user_id, user_downloads[0]['id'])
print(f"Cancel result: {cancel_result}")
return len(user_downloads)
async def example_error_handling():
"""Example of error handling"""
print("\n⚠️ Error Handling Example")
print("=" * 50)
manager = BDFRManager("./test_downloads")
# Try to download from a non-existent subreddit
download_id = manager.download_subreddit(
"this_subreddit_does_not_exist",
limit=5
)
print(f"📋 Created download for non-existent subreddit: {download_id}")
# Monitor for error
for _ in range(5): # Check for up to 10 seconds
status = manager.get_download_status(download_id)
if not status:
print("❌ Download disappeared")
break
print(f"📊 Status: {status['status']}")
if status['status'] == 'failed':
print(f"🏁 Download failed as expected: {status.get('error', 'Unknown error')}")
break
await asyncio.sleep(2)
return download_id
async def main():
"""Run all examples"""
print("🎯 BDFR API Examples")
print("=" * 60)
print("This demonstrates the new BDFR API layer for direct web integration")
print()
try:
# Run basic example
await example_basic_usage()
# Run advanced example
await example_advanced_usage()
# Run user download example
await example_user_download()
# Run archive example
await example_archive_operation()
# Run web integration example
await example_web_integration()
# Run error handling example
await example_error_handling()
print("\n🎉 All examples completed!")
except KeyboardInterrupt:
print("\n⏹️ Examples interrupted by user")
except Exception as e:
print(f"\n❌ Error running examples: {e}")
import traceback
traceback.print_exc()
if __name__ == "__main__":
# Run the examples
asyncio.run(main())