BDFR_Web/bdfr/API_README.md

BDFR API Layer

The BDFR API layer provides a direct integration interface for the web interface, eliminating the need for subprocess console parsing. It wraps the existing BDFR core classes and provides a clean API with structured progress callbacks.

Overview

The API layer consists of several key components:

• BDFRManager: Main API interface class
• ProgressCallback: Abstract base class for progress notifications
• ProgressEvent: Structured progress event data
• DownloadType: Enumeration of supported download types
• DownloadStatus: Enumeration of download statuses

Quick Start

Basic Usage

from bdfr.api import BDFRManager, LoggingCallback

# Create a manager
manager = BDFRManager("./downloads")

# Download from a subreddit
download_id = manager.download_subreddit(
    "python",
    limit=50,
    sort="hot",
    no_dupes=True
)

# Check status
status = manager.get_download_status(download_id)
print(f"Progress: {status['progress']}%")

Advanced Usage with Custom Callbacks

from bdfr.api import BDFRManager, ProgressCallback, ProgressEvent

class WebSocketCallback(ProgressCallback):
    def __init__(self, websocket):
        self.websocket = websocket

    async def on_progress(self, event: ProgressEvent):
        await self.websocket.send_json(event.to_dict())

    async def on_error(self, event: ProgressEvent):
        await self.websocket.send_json(event.to_dict())

    async def on_completed(self, event: ProgressEvent):
        await self.websocket.send_json(event.to_dict())

# Use custom callbacks
callbacks = [LoggingCallback(), WebSocketCallback(ws)]
download_id = manager.download_subreddit("technology", limit=100, progress_callbacks=callbacks)

API Reference

BDFRManager

The main API class that manages downloads and provides the primary interface.

Constructor

BDFRManager(download_directory: Optional[Union[str, Path]] = None)

• download_directory: Base directory for downloads. Defaults to current directory.

Methods

create_download()

Create a new download operation.

create_download(
    download_type: DownloadType,
    name: str,
    config: Optional[Configuration] = None,
    progress_callbacks: Optional[List[ProgressCallback]] = None
) -> str

Returns a download ID for tracking the operation.

start_download()

Start a download operation.

start_download(download_id: str) -> bool

Returns True if started successfully.

get_download_status()

Get the current status of a download.

get_download_status(download_id: str) -> Optional[Dict[str, Any]]

Returns download status information or None if not found.

cancel_download()

Cancel a running download.

cancel_download(download_id: str) -> bool

Returns True if cancelled successfully.

list_downloads()

List all active downloads.

list_downloads() -> Dict[str, Dict[str, Any]]

Returns dictionary of download information keyed by download ID.

cleanup_completed()

Clean up old completed downloads.

cleanup_completed(max_age_seconds: int = 3600) -> int

Returns number of downloads cleaned up.

Convenience Methods

download_subreddit()

Download content from a subreddit.

download_subreddit(
    subreddit_name: str,
    limit: Optional[int] = None,
    sort: str = "hot",
    time_filter: str = "all",
    no_dupes: bool = False,
    progress_callbacks: Optional[List[ProgressCallback]] = None
) -> str

download_user()

Download content from a user.

download_user(
    username: str,
    limit: Optional[int] = None,
    submitted: bool = True,
    upvoted: bool = False,
    saved: bool = False,
    no_dupes: bool = False,
    progress_callbacks: Optional[List[ProgressCallback]] = None
) -> str

archive_subreddit()

Archive subreddit data (metadata only).

archive_subreddit(
    subreddit_name: str,
    format_type: str = "json",
    limit: Optional[int] = None,
    progress_callbacks: Optional[List[ProgressCallback]] = None
) -> str

clone_subreddit()

Clone subreddit (both download and archive).

clone_subreddit(
    subreddit_name: str,
    limit: Optional[int] = None,
    format_type: str = "json",
    no_dupes: bool = False,
    progress_callbacks: Optional[List[ProgressCallback]] = None
) -> str

ProgressCallback

Abstract base class for implementing progress callbacks.

Methods

on_progress(event: ProgressEvent)

Called when progress is made.

on_error(event: ProgressEvent)

Called when an error occurs.

on_completed(event: ProgressEvent)

Called when download is completed.

ProgressEvent

Represents a progress event with structured data.

Attributes

• event_type: "progress", "status", "error", or "completed"
• download_id: Unique identifier for the download
• message: Human-readable message
• progress: Progress percentage (0-100)
• data: Additional structured data
• timestamp: When the event occurred

Methods

to_dict() -> Dict[str, Any]

Convert to dictionary for JSON serialization.

DownloadType

Enumeration of supported download types:

• SUBREDDIT: Download from subreddit
• USER: Download from user
• MULTIREDDIT: Download from multireddit
• SUBMISSIONS: Download specific submissions
• ARCHIVE: Archive only (no downloads)
• CLONE: Both download and archive

DownloadStatus

Enumeration of download statuses:

• QUEUED: Download is queued but not started
• RUNNING: Download is in progress
• COMPLETED: Download completed successfully
• FAILED: Download failed with an error
• CANCELLED: Download was cancelled
• PAUSED: Download is paused

Configuration

The API uses the existing BDFR Configuration class. You can pass a custom configuration to create_download() or use the convenience methods with their specific parameters.

Common Configuration Options

• limit: Maximum number of posts to process
• sort: Sort method (hot, top, new, controversial, rising)
• time: Time filter (all, hour, day, week, month, year)
• no_dupes: Avoid duplicate downloads
• make_hard_links: Create hard links for duplicates
• format: Archive format (json, xml, yaml)

Web Interface Integration

FastAPI Integration Example

from fastapi import FastAPI, WebSocket
from bdfr.api import get_bdfr_manager, WebSocketCallback

app = FastAPI()
manager = get_bdfr_manager("./downloads")

@app.post("/api/download/subreddit")
async def download_subreddit(subreddit: str, limit: int = 10):
    download_id = manager.download_subreddit(subreddit, limit=limit)
    return {"download_id": download_id}

@app.websocket("/ws/progress/{download_id}")
async def progress_websocket(websocket: WebSocket, download_id: str):
    await websocket.accept()

    class FastAPICallback(WebSocketCallback):
        def __init__(self):
            super().__init__(None)

        async def on_progress(self, event: ProgressEvent):
            if event.download_id == download_id:
                await websocket.send_json(event.to_dict())

    # Add callback to existing download or create new one
    # (Implementation depends on your specific needs)

Real-time Progress Updates

The API provides structured progress events that can be easily serialized to JSON for web clients:

# Example progress event
{
    "event_type": "progress",
    "download_id": "123e4567-e89b-12d3-a456-426614174000",
    "message": "Downloaded submission abc123 from r/python",
    "progress": 45.2,
    "data": {
        "items_processed": 12,
        "items_found": 25,
        "current_item": "abc123",
        "phase": "downloading_submission"
    },
    "timestamp": "2023-12-07T10:30:45.123456"
}

Error Handling

The API provides comprehensive error handling:

• Download errors: Network issues, authentication problems, etc.
• Configuration errors: Invalid parameters, missing files, etc.
• System errors: Disk space, permissions, etc.

All errors are captured and reported through the progress callback system with detailed error information.

Threading and Concurrency

The API is designed to be thread-safe and supports concurrent downloads:

• Each download runs in its own thread
• Progress callbacks are async-safe
• Multiple downloads can run simultaneously
• Thread-safe status tracking

Logging

The API integrates with Python's logging system:

import logging

# Configure logging
logging.basicConfig(level=logging.INFO)

# Use LoggingCallback for automatic log output
callbacks = [LoggingCallback("my_app")]

Examples

See bdfr/examples/api_usage.py for comprehensive examples including:

• Basic usage
• Custom callbacks
• Web integration
• Error handling
• User downloads
• Archive operations

Migration from Subprocess

Before (Subprocess)

import subprocess
import json

# Start BDFR via subprocess
proc = subprocess.Popen([
    "python", "-m", "bdfr", "download",
    "--subreddit", "python",
    "--limit", "50",
    "./downloads"
], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

# Parse console output for progress
while True:
    line = proc.stdout.readline().decode().strip()
    if not line:
        break
    # Parse progress from console output...

After (API)

from bdfr.api import BDFRManager, LoggingCallback

# Use API directly
manager = BDFRManager("./downloads")
download_id = manager.download_subreddit("python", limit=50)

# Get structured progress updates
status = manager.get_download_status(download_id)
print(f"Progress: {status['progress']}%")

Benefits

1. No subprocess overhead: Direct integration with BDFR core
2. Structured progress: Rich progress events instead of console parsing
3. Better error handling: Detailed error information and stack traces
4. Thread-safe: Concurrent downloads with proper synchronization
5. Web-friendly: JSON-serializable progress events
6. Extensible: Custom progress callbacks for different integrations
7. Maintainable: Clean separation of concerns

Requirements

• Python 3.7+
• Existing BDFR installation
• Dependencies: praw, requests, and other BDFR dependencies

Installation

The API layer is included with BDFR and requires no additional installation. Simply import and use:

from bdfr.api import BDFRManager