# BDFR API Layer The BDFR API layer provides a direct integration interface for the web interface, eliminating the need for subprocess console parsing. It wraps the existing BDFR core classes and provides a clean API with structured progress callbacks. ## Overview The API layer consists of several key components: - **BDFRManager**: Main API interface class - **ProgressCallback**: Abstract base class for progress notifications - **ProgressEvent**: Structured progress event data - **DownloadType**: Enumeration of supported download types - **DownloadStatus**: Enumeration of download statuses ## Quick Start ### Basic Usage ```python from bdfr.api import BDFRManager, LoggingCallback # Create a manager manager = BDFRManager("./downloads") # Download from a subreddit download_id = manager.download_subreddit( "python", limit=50, sort="hot", no_dupes=True ) # Check status status = manager.get_download_status(download_id) print(f"Progress: {status['progress']}%") ``` ### Advanced Usage with Custom Callbacks ```python from bdfr.api import BDFRManager, ProgressCallback, ProgressEvent class WebSocketCallback(ProgressCallback): def __init__(self, websocket): self.websocket = websocket async def on_progress(self, event: ProgressEvent): await self.websocket.send_json(event.to_dict()) async def on_error(self, event: ProgressEvent): await self.websocket.send_json(event.to_dict()) async def on_completed(self, event: ProgressEvent): await self.websocket.send_json(event.to_dict()) # Use custom callbacks callbacks = [LoggingCallback(), WebSocketCallback(ws)] download_id = manager.download_subreddit("technology", limit=100, progress_callbacks=callbacks) ``` ## API Reference ### BDFRManager The main API class that manages downloads and provides the primary interface. #### Constructor ```python BDFRManager(download_directory: Optional[Union[str, Path]] = None) ``` - `download_directory`: Base directory for downloads. Defaults to current directory. #### Methods ##### `create_download()` Create a new download operation. ```python create_download( download_type: DownloadType, name: str, config: Optional[Configuration] = None, progress_callbacks: Optional[List[ProgressCallback]] = None ) -> str ``` Returns a download ID for tracking the operation. ##### `start_download()` Start a download operation. ```python start_download(download_id: str) -> bool ``` Returns `True` if started successfully. ##### `get_download_status()` Get the current status of a download. ```python get_download_status(download_id: str) -> Optional[Dict[str, Any]] ``` Returns download status information or `None` if not found. ##### `cancel_download()` Cancel a running download. ```python cancel_download(download_id: str) -> bool ``` Returns `True` if cancelled successfully. ##### `list_downloads()` List all active downloads. ```python list_downloads() -> Dict[str, Dict[str, Any]] ``` Returns dictionary of download information keyed by download ID. ##### `cleanup_completed()` Clean up old completed downloads. ```python cleanup_completed(max_age_seconds: int = 3600) -> int ``` Returns number of downloads cleaned up. #### Convenience Methods ##### `download_subreddit()` Download content from a subreddit. ```python download_subreddit( subreddit_name: str, limit: Optional[int] = None, sort: str = "hot", time_filter: str = "all", no_dupes: bool = False, progress_callbacks: Optional[List[ProgressCallback]] = None ) -> str ``` ##### `download_user()` Download content from a user. ```python download_user( username: str, limit: Optional[int] = None, submitted: bool = True, upvoted: bool = False, saved: bool = False, no_dupes: bool = False, progress_callbacks: Optional[List[ProgressCallback]] = None ) -> str ``` ##### `archive_subreddit()` Archive subreddit data (metadata only). ```python archive_subreddit( subreddit_name: str, format_type: str = "json", limit: Optional[int] = None, progress_callbacks: Optional[List[ProgressCallback]] = None ) -> str ``` ##### `clone_subreddit()` Clone subreddit (both download and archive). ```python clone_subreddit( subreddit_name: str, limit: Optional[int] = None, format_type: str = "json", no_dupes: bool = False, progress_callbacks: Optional[List[ProgressCallback]] = None ) -> str ``` ### ProgressCallback Abstract base class for implementing progress callbacks. #### Methods ##### `on_progress(event: ProgressEvent)` Called when progress is made. ##### `on_error(event: ProgressEvent)` Called when an error occurs. ##### `on_completed(event: ProgressEvent)` Called when download is completed. ### ProgressEvent Represents a progress event with structured data. #### Attributes - `event_type`: "progress", "status", "error", or "completed" - `download_id`: Unique identifier for the download - `message`: Human-readable message - `progress`: Progress percentage (0-100) - `data`: Additional structured data - `timestamp`: When the event occurred #### Methods ##### `to_dict() -> Dict[str, Any]` Convert to dictionary for JSON serialization. ### DownloadType Enumeration of supported download types: - `SUBREDDIT`: Download from subreddit - `USER`: Download from user - `MULTIREDDIT`: Download from multireddit - `SUBMISSIONS`: Download specific submissions - `ARCHIVE`: Archive only (no downloads) - `CLONE`: Both download and archive ### DownloadStatus Enumeration of download statuses: - `QUEUED`: Download is queued but not started - `RUNNING`: Download is in progress - `COMPLETED`: Download completed successfully - `FAILED`: Download failed with an error - `CANCELLED`: Download was cancelled - `PAUSED`: Download is paused ## Configuration The API uses the existing BDFR `Configuration` class. You can pass a custom configuration to `create_download()` or use the convenience methods with their specific parameters. ### Common Configuration Options - `limit`: Maximum number of posts to process - `sort`: Sort method (hot, top, new, controversial, rising) - `time`: Time filter (all, hour, day, week, month, year) - `no_dupes`: Avoid duplicate downloads - `make_hard_links`: Create hard links for duplicates - `format`: Archive format (json, xml, yaml) ## Web Interface Integration ### FastAPI Integration Example ```python from fastapi import FastAPI, WebSocket from bdfr.api import get_bdfr_manager, WebSocketCallback app = FastAPI() manager = get_bdfr_manager("./downloads") @app.post("/api/download/subreddit") async def download_subreddit(subreddit: str, limit: int = 10): download_id = manager.download_subreddit(subreddit, limit=limit) return {"download_id": download_id} @app.websocket("/ws/progress/{download_id}") async def progress_websocket(websocket: WebSocket, download_id: str): await websocket.accept() class FastAPICallback(WebSocketCallback): def __init__(self): super().__init__(None) async def on_progress(self, event: ProgressEvent): if event.download_id == download_id: await websocket.send_json(event.to_dict()) # Add callback to existing download or create new one # (Implementation depends on your specific needs) ``` ### Real-time Progress Updates The API provides structured progress events that can be easily serialized to JSON for web clients: ```python # Example progress event { "event_type": "progress", "download_id": "123e4567-e89b-12d3-a456-426614174000", "message": "Downloaded submission abc123 from r/python", "progress": 45.2, "data": { "items_processed": 12, "items_found": 25, "current_item": "abc123", "phase": "downloading_submission" }, "timestamp": "2023-12-07T10:30:45.123456" } ``` ## Error Handling The API provides comprehensive error handling: - **Download errors**: Network issues, authentication problems, etc. - **Configuration errors**: Invalid parameters, missing files, etc. - **System errors**: Disk space, permissions, etc. All errors are captured and reported through the progress callback system with detailed error information. ## Threading and Concurrency The API is designed to be thread-safe and supports concurrent downloads: - Each download runs in its own thread - Progress callbacks are async-safe - Multiple downloads can run simultaneously - Thread-safe status tracking ## Logging The API integrates with Python's logging system: ```python import logging # Configure logging logging.basicConfig(level=logging.INFO) # Use LoggingCallback for automatic log output callbacks = [LoggingCallback("my_app")] ``` ## Examples See `bdfr/examples/api_usage.py` for comprehensive examples including: - Basic usage - Custom callbacks - Web integration - Error handling - User downloads - Archive operations ## Migration from Subprocess ### Before (Subprocess) ```python import subprocess import json # Start BDFR via subprocess proc = subprocess.Popen([ "python", "-m", "bdfr", "download", "--subreddit", "python", "--limit", "50", "./downloads" ], stdout=subprocess.PIPE, stderr=subprocess.PIPE) # Parse console output for progress while True: line = proc.stdout.readline().decode().strip() if not line: break # Parse progress from console output... ``` ### After (API) ```python from bdfr.api import BDFRManager, LoggingCallback # Use API directly manager = BDFRManager("./downloads") download_id = manager.download_subreddit("python", limit=50) # Get structured progress updates status = manager.get_download_status(download_id) print(f"Progress: {status['progress']}%") ``` ## Benefits 1. **No subprocess overhead**: Direct integration with BDFR core 2. **Structured progress**: Rich progress events instead of console parsing 3. **Better error handling**: Detailed error information and stack traces 4. **Thread-safe**: Concurrent downloads with proper synchronization 5. **Web-friendly**: JSON-serializable progress events 6. **Extensible**: Custom progress callbacks for different integrations 7. **Maintainable**: Clean separation of concerns ## Requirements - Python 3.7+ - Existing BDFR installation - Dependencies: `praw`, `requests`, and other BDFR dependencies ## Installation The API layer is included with BDFR and requires no additional installation. Simply import and use: ```python from bdfr.api import BDFRManager