# Scheduled Downloads Implementation Plan ## Overview This document outlines the implementation plan for adding scheduled download functionality to the BDFR Web Interface. Users will be able to configure downloads that run automatically on a daily schedule, perfect for keeping up with new content from their favorite subreddits or users. ## Requirements Summary - **Database**: SQLite with SQLAlchemy ORM - **Scheduling**: Daily frequency (runs every 24 hours) - **UI Approach**: Simple - checkbox in Advanced Options + management section on main page - **Time Filter**: Automatically set to "last day" for daily runs - **Duplicate Handling**: Works with existing no-dupes functionality - **Deployment**: Docker container environment - **Execution Model**: **Sequential only - one task at a time, queued execution** ## Architecture ### 1. Database Schema #### ScheduledTask Table ```python class ScheduledTask: id: UUID (Primary Key) name: str # User-friendly name for the task enabled: bool # Whether task is active # Download Configuration source_type: str # "subreddit" or "user" source_name: str # Name of subreddit or username download_mode: str # "download", "archive", or "clone" # Filter Options limit: int sort: str # "hot", "top", "new", etc. time_filter: str # Always "day" for daily tasks min_score: int (optional) no_dupes: bool # Always true for scheduled tasks simple_check: bool # Scheduling schedule_frequency: str # "daily" (extensible for future: "weekly", "custom") run_time: time # Time of day to run (e.g., "02:00:00") timezone: str # User's timezone (default: UTC) # Metadata created_at: datetime updated_at: datetime last_run_at: datetime (nullable) next_run_at: datetime # Authentication auth_state: str (nullable) # For authenticated downloads ``` #### TaskExecutionHistory Table ```python class TaskExecutionHistory: id: UUID (Primary Key) task_id: UUID (Foreign Key -> ScheduledTask) # Execution Details started_at: datetime completed_at: datetime (nullable) status: str # "success", "failed", "running", "queued" # Results items_found: int items_downloaded: int error_message: str (nullable) # Link to download download_id: str # Links to active_downloads tracking ``` ### 2. Backend Components #### File Structure ``` web_interface/ ├── app/ │ ├── __init__.py │ ├── main.py (existing) │ ├── auth.py (existing) │ ├── database.py (NEW - SQLAlchemy setup) │ ├── models.py (NEW - DB models) │ ├── scheduler.py (NEW - APScheduler + Queue integration) │ ├── task_queue.py (NEW - Sequential task queue manager) │ └── scheduled_tasks.py (NEW - Task management logic) ├── data/ │ └── scheduled_tasks.db (SQLite database - created at runtime) └── requirements.txt (UPDATE - add dependencies) ``` #### Dependencies to Add ```txt sqlalchemy>=2.0.0 alembic>=1.12.0 # For database migrations apscheduler>=3.10.0 # For task scheduling ``` #### API Endpoints **Scheduled Tasks CRUD:** - `POST /api/scheduled-tasks` - Create new scheduled task - `GET /api/scheduled-tasks` - List all scheduled tasks - `GET /api/scheduled-tasks/{task_id}` - Get specific task details - `PUT /api/scheduled-tasks/{task_id}` - Update task configuration - `DELETE /api/scheduled-tasks/{task_id}` - Delete task - `POST /api/scheduled-tasks/{task_id}/toggle` - Enable/disable task - `POST /api/scheduled-tasks/{task_id}/run-now` - Trigger immediate execution (adds to queue) **Task History & Queue:** - `GET /api/scheduled-tasks/{task_id}/history` - Get execution history - `GET /api/scheduled-tasks/history/recent` - Get recent executions across all tasks - `GET /api/scheduled-tasks/queue` - Get current task queue status ### 3. Sequential Task Queue System #### Task Queue Manager (`task_queue.py`) **Core Concept**: Only one scheduled download can run at a time. When multiple tasks are triggered (either by schedule or "Run Now"), they are queued and executed sequentially. ```python import asyncio from typing import Optional, List, Dict from datetime import datetime import logging logger = logging.getLogger(__name__) class TaskQueue: """ Manages sequential execution of scheduled download tasks. Ensures only one task runs at a time. """ def __init__(self): self.queue: asyncio.Queue = asyncio.Queue() self.current_task: Optional[str] = None # Current task_id being executed self.is_processing: bool = False self.worker_task: Optional[asyncio.Task] = None async def add_task(self, task_id: str, priority: int = 0): """ Add a task to the queue. Args: task_id: UUID of the scheduled task priority: 0 = scheduled (normal), 1 = manual "Run Now" (higher priority) """ await self.queue.put({ 'task_id': task_id, 'priority': priority, 'queued_at': datetime.now() }) logger.info(f"Task {task_id} added to queue (priority={priority}, queue_size={self.queue.qsize()})") # Start worker if not already running if not self.is_processing: await self.start_worker() async def start_worker(self): """Start the queue worker if not already running""" if self.worker_task is None or self.worker_task.done(): self.worker_task = asyncio.create_task(self._process_queue()) logger.info("Queue worker started") async def _process_queue(self): """Process tasks from queue sequentially""" self.is_processing = True logger.info("Queue worker processing started") while True: try: # Wait for next task (with timeout to allow graceful shutdown) try: task_info = await asyncio.wait_for( self.queue.get(), timeout=60.0 ) except asyncio.TimeoutError: # Check if queue is empty if self.queue.empty(): logger.info("Queue empty, worker stopping") break continue task_id = task_info['task_id'] self.current_task = task_id logger.info(f"Executing task {task_id} from queue (queue_size={self.queue.qsize()})") # Execute the task (this will block until download completes) try: await execute_scheduled_task(task_id) logger.info(f"Task {task_id} completed successfully") except Exception as e: logger.error(f"Task {task_id} failed: {e}") finally: self.current_task = None self.queue.task_done() except Exception as e: logger.error(f"Queue worker error: {e}") self.is_processing = False logger.info("Queue worker stopped") def get_queue_status(self) -> Dict: """Get current queue status""" return { 'current_task': self.current_task, 'queue_size': self.queue.qsize(), 'is_processing': self.is_processing } async def stop(self): """Stop the queue worker gracefully""" logger.info("Stopping queue worker...") if self.worker_task and not self.worker_task.done(): # Wait for current task to complete await self.worker_task logger.info("Queue worker stopped") # Global queue instance task_queue = TaskQueue() ``` ### 4. Scheduler Service (Docker-Aware with Sequential Queue) #### APScheduler Configuration ```python from apscheduler.schedulers.asyncio import AsyncIOScheduler from apscheduler.triggers.cron import CronTrigger from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore import pytz # Use database job store for persistence across container restarts jobstores = { 'default': SQLAlchemyJobStore(url='sqlite:///data/scheduler_jobs.db') } # Configure scheduler for Docker container scheduler = AsyncIOScheduler( jobstores=jobstores, timezone=pytz.UTC, # Container runs in UTC job_defaults={ 'coalesce': True, # Combine multiple missed executions into one 'max_instances': 1, # Only one instance of each job at a time 'misfire_grace_time': 3600 # Allow up to 1 hour late execution } ) # Add job for each enabled task def schedule_task(task: ScheduledTask): """ Schedule a task to be added to the queue at specified time. Note: This doesn't execute the task directly, it queues it. """ user_tz = pytz.timezone(task.timezone) hour, minute = task.run_time.hour, task.run_time.minute scheduler.add_job( func=queue_scheduled_task, # Add to queue, not execute directly trigger=CronTrigger(hour=hour, minute=minute, timezone=user_tz), args=[task.id], id=str(task.id), replace_existing=True ) logger.info(f"Scheduled task {task.id} for {hour:02d}:{minute:02d} {task.timezone}") async def queue_scheduled_task(task_id: str): """ Called by scheduler at the configured time. Adds task to queue rather than executing immediately. """ logger.info(f"Scheduler triggered for task {task_id}, adding to queue") await task_queue.add_task(task_id, priority=0) # Normal priority for scheduled tasks ``` #### Task Execution Flow with Queue ```mermaid graph TD A[Scheduler triggers at scheduled time] --> B[Add task to queue] B --> C{Is queue worker running?} C -->|No| D[Start queue worker] C -->|Yes| E[Task waits in queue] D --> F[Worker picks next task from queue] E --> F F --> G[Load task from DB] G --> H[Check if enabled] H -->|Disabled| I[Skip, mark in history] H -->|Enabled| J[Create execution history] J --> K[Set status = 'running'] K --> L[Build download parameters] L --> M[Set time_filter=day, no_dupes=true] M --> N[Call BDFR API - BLOCKS until complete] N --> O[Wait for download to finish] O --> P[Update execution history] P --> Q[Update last_run_at] Q --> R[Worker picks next task] R -->|Queue empty| S[Worker idles/stops] R -->|More tasks| F style N fill:#ffcccc style O fill:#ffcccc note1[Note: Worker blocks here until download completes] ``` ### 5. Integration with BDFR API (Sequential Execution) ```python async def execute_scheduled_task(task_id: str): """ Execute a scheduled download task. This function BLOCKS until the download is complete, ensuring sequential execution. """ # Load task from database task = get_scheduled_task(task_id) if not task.enabled: logger.info(f"Skipping disabled task {task_id}") # Still record in history that it was skipped execution = create_execution_history(task_id) execution.status = 'skipped' execution.completed_at = datetime.now(pytz.UTC) save_execution_history(execution) return # Create execution history record execution = create_execution_history(task_id) try: logger.info(f"Executing scheduled task {task_id}: {task.source_type}/{task.source_name}") # Build download parameters kwargs = { 'limit': task.limit, 'sort': task.sort, 'time_filter': 'day', # Always "day" for daily scheduled tasks 'no_dupes': True, # Always enabled for scheduled tasks 'simple_check': task.simple_check, 'auth_state': task.auth_state } # Create download using existing API # This returns immediately with a download_id download_id = await create_download_with_bdfr_api( download_type=task.source_type, name=task.source_name, **kwargs ) logger.info(f"Task {task_id} started as download {download_id}") # Track download execution.download_id = download_id execution.status = 'running' save_execution_history(execution) # **CRITICAL: Wait for download to complete before returning** # This ensures the next queued task doesn't start until this one finishes await wait_for_download_completion(download_id) # Check final status download_status = bdfr_manager.get_download_status(download_id) if download_status and download_status['status'] == 'completed': execution.status = 'success' execution.items_found = download_status.get('items_found', 0) execution.items_downloaded = download_status.get('items_processed', 0) logger.info(f"Task {task_id} completed successfully") else: execution.status = 'failed' execution.error_message = download_status.get('error', 'Unknown error') logger.error(f"Task {task_id} failed: {execution.error_message}") execution.completed_at = datetime.now(pytz.UTC) save_execution_history(execution) # Update task timestamps task.last_run_at = datetime.now(pytz.UTC) task.next_run_at = calculate_next_run(task) save_scheduled_task(task) except Exception as e: logger.error(f"Task {task_id} execution error: {e}") execution.status = 'failed' execution.error_message = str(e) execution.completed_at = datetime.now(pytz.UTC) save_execution_history(execution) raise async def wait_for_download_completion(download_id: str, timeout: int = 3600): """ Wait for a download to complete. Polls the download status until it's no longer running. Args: download_id: The download to wait for timeout: Maximum seconds to wait (default 1 hour) """ start_time = datetime.now() check_interval = 5 # Check every 5 seconds while True: # Check if timeout exceeded elapsed = (datetime.now() - start_time).total_seconds() if elapsed > timeout: logger.error(f"Download {download_id} timed out after {timeout}s") raise TimeoutError(f"Download exceeded timeout of {timeout}s") # Check download status status = bdfr_manager.get_download_status(download_id) if not status: logger.warning(f"Download {download_id} status not found") break download_status = status.get('status', 'unknown') # Check if download is finished (completed, failed, or cancelled) if download_status in ['completed', 'failed', 'cancelled']: logger.info(f"Download {download_id} finished with status: {download_status}") break # Still running, wait before checking again await asyncio.sleep(check_interval) ``` ### 6. API Endpoints with Queue Support ```python @app.post("/api/scheduled-tasks/{task_id}/run-now") async def run_task_now(task_id: str): """ Manually trigger a scheduled task to run now. Adds it to the queue with high priority. """ task = get_scheduled_task(task_id) if not task: raise HTTPException(status_code=404, detail="Task not found") # Add to queue with priority (goes ahead of scheduled tasks) await task_queue.add_task(task_id, priority=1) queue_status = task_queue.get_queue_status() return { "message": f"Task {task_id} added to queue", "queue_position": queue_status['queue_size'], "currently_running": queue_status['current_task'], "status": "queued" if queue_status['current_task'] else "starting" } @app.get("/api/scheduled-tasks/queue") async def get_queue_status(): """Get current task queue status""" status = task_queue.get_queue_status() # Get details of current task if any current_task_info = None if status['current_task']: task = get_scheduled_task(status['current_task']) if task: current_task_info = { 'id': task.id, 'name': task.name, 'source': f"{task.source_type}/{task.source_name}" } return { 'queue_size': status['queue_size'], 'is_processing': status['is_processing'], 'current_task': current_task_info } ``` ### 7. Frontend Implementation #### UI Modifications to [`index.html`](web_interface/templates/index.html:1) **Queue Status Indicator (add to header):** ```html ``` **Advanced Options Section - Add Checkbox:** ```html ``` **New Section - Scheduled Tasks List:** ```html

📅 Scheduled Downloads

📅

No scheduled tasks yet

Check "Schedule Daily Run" when creating a download to set up automated daily downloads.

Note: Scheduled tasks run one at a time to prevent server overload.

``` **Task Card Template:** ```html

{task_name}

{source_type}: {source_name} ⏰ Runs daily at {run_time}
Last Run: {last_run_at || "Never"}
Next Run: {next_run_at}
Mode: {download_mode}
``` #### JavaScript Modifications to [`app.js`](web_interface/static/js/app.js:1) **Add queue status polling:** ```javascript async loadScheduledTasks() { const response = await fetch('/api/scheduled-tasks'); const tasks = await response.json(); this.renderScheduledTasks(tasks); // Also update queue status this.updateQueueStatus(); } async updateQueueStatus() { try { const response = await fetch('/api/scheduled-tasks/queue'); const status = await response.json(); // Update queue badge const queueBadge = document.getElementById('queueStatusBadge'); const queueText = document.getElementById('queueBadgeText'); if (status.queue_size > 0 || status.current_task) { queueBadge.style.display = 'flex'; queueText.textContent = `Queue: ${status.queue_size}${status.current_task ? ' (1 running)' : ''}`; } else { queueBadge.style.display = 'none'; } // Highlight currently running task card document.querySelectorAll('.task-card').forEach(card => { const taskId = card.dataset.taskId; const queueInfo = card.querySelector('.task-queue-info'); if (status.current_task && status.current_task.id === taskId) { card.classList.add('task-running'); queueInfo.textContent = '⚡ Currently Running'; queueInfo.style.display = 'inline'; } else { card.classList.remove('task-running'); queueInfo.style.display = 'none'; } }); } catch (error) { console.error('Failed to update queue status:', error); } } async runTaskNow(taskId) { const response = await fetch(`/api/scheduled-tasks/${taskId}/run-now`, { method: 'POST' }); if (response.ok) { const result = await response.json(); if (result.queue_position > 0) { this.showSuccess(`Task added to queue. Position: ${result.queue_position}`); } else { this.showSuccess('Task execution starting...'); } this.updateQueueStatus(); } else { this.showError('Failed to queue task'); } } // Poll queue status every 10 seconds startQueuePolling() { setInterval(() => { if (document.querySelectorAll('.task-card').length > 0) { this.updateQueueStatus(); } }, 10000); } ``` ### 8. Implementation Steps 1. **Backend Foundation** (Steps 4-5) - Add SQLAlchemy and APScheduler to [`requirements.txt`](web_interface/requirements.txt:1) - Create [`database.py`](web_interface/app/database.py) with Docker-aware paths - Create [`models.py`](web_interface/app/models.py) with ScheduledTask and TaskExecutionHistory models - Initialize database on app startup 2. **Task Queue System** (Step 7) - Create [`task_queue.py`](web_interface/app/task_queue.py) with sequential queue manager - Implement queue worker with blocking execution - Add queue status tracking and reporting 3. **Scheduler Service** (Step 7 continued) - Create [`scheduler.py`](web_interface/app/scheduler.py) with Docker-aware APScheduler - Integrate with task queue (scheduler adds to queue, doesn't execute directly) - Implement `wait_for_download_completion()` to block until download finishes - Add scheduler lifecycle hooks to [`main.py`](web_interface/app/main.py:1) 4. **API Endpoints** (Step 6) - Create [`scheduled_tasks.py`](web_interface/app/scheduled_tasks.py) with CRUD operations - Add routes to [`main.py`](web_interface/app/main.py:1) - Implement task toggle, delete, and run-now (with queue) - Add queue status endpoint 5. **Frontend - Form** (Step 8) - Add "Run Daily" checkbox to Advanced Options in [`index.html`](web_interface/templates/index.html:1) - Add conditional schedule configuration fields - Update form submission logic in [`app.js`](web_interface/static/js/app.js:1) - Auto-detect browser timezone 6. **Frontend - Management** (Step 9) - Add Scheduled Tasks section to [`index.html`](web_interface/templates/index.html:1) - Add queue status indicator - Implement task card rendering with queue status - Add toggle, delete, and run-now functions to [`app.js`](web_interface/static/js/app.js:1) - Add queue status polling 7. **Integration** (Steps 10-11) - Connect scheduler to BDFR API via [`create_download_with_bdfr_api()`](web_interface/app/main.py:373) - Implement automatic `time_filter="day"` for scheduled tasks - Add execution history tracking - Ensure sequential execution with proper blocking 8. **Testing** (Step 14) - Test task creation, editing, deletion - Test queue functionality (multiple tasks, sequential execution) - Test "Run Now" adds to queue correctly - Test priority (manual tasks run before scheduled) - Test container restart persistence - Verify only one task runs at a time ## Key Features ### Sequential Execution (NEW) - **Task Queue**: All scheduled downloads go through a FIFO queue - **Blocking Execution**: Each task blocks until its download completes - **No Concurrency**: Only one download runs at a time, preventing system overload - **Priority System**: Manual "Run Now" tasks get priority over scheduled tasks - **Queue Status**: Users can see queue size and currently running task ### Docker-Specific Features - **Persistent Storage**: SQLite database and downloads persist via volume mounts - **Container Restarts**: APScheduler with job store survives restarts, queue rebuilds on startup - **Timezone Handling**: User timezone stored, converted to UTC for container execution - **Logging**: Structured logging for container environment - **Health Checks**: Scheduler and queue status included in health endpoint ### Automatic Configuration for Scheduled Tasks - **time_filter**: Always set to "day" - ensures only last 24 hours of content - **no_dupes**: Always enabled - prevents re-downloading same content - **Timezone handling**: Store user timezone, convert to UTC for execution, display in user timezone - **Sequential execution**: Guaranteed one-at-a-time processing ### Smart Duplicate Prevention When a scheduled task runs: 1. BDFR checks existing hashes (if no_dupes enabled) 2. Only downloads new content from last 24 hours 3. Skips content already downloaded in previous runs ### Execution Tracking - Every run creates a history record - Tracks success/failure status - Records items found vs. items downloaded - Links to the actual download progress for real-time monitoring - Shows queue position and current task status ### User Experience - Simple checkbox to schedule any download - Visual indication of enabled/disabled tasks - Queue status badge shows pending tasks - Currently running task highlighted - Next run time displayed in user's local timezone - One-click to add task to queue immediately - Easy enable/disable without deleting task - Queue position shown when manually running tasks ## Sequential Execution Examples ### Scenario 1: Multiple Scheduled Tasks ``` 02:00 AM - Task A triggers, added to queue 02:00 AM - Task B triggers, added to queue 02:00 AM - Task C triggers, added to queue Execution Order: 1. Task A starts, downloads 100 posts (takes 15 minutes) 2. Task B starts at 02:15 AM, downloads 50 posts (takes 8 minutes) 3. Task C starts at 02:23 AM, downloads 75 posts (takes 12 minutes) 4. All complete by 02:35 AM ``` ### Scenario 2: Manual "Run Now" During Scheduled Task ``` 02:00 AM - Task A starts (scheduled, downloading...) 02:10 AM - User clicks "Run Now" on Task B 02:10 AM - Task B added to queue with priority Execution Order: 1. Task A continues running (started first) 2. Task B waits in queue 3. Task A completes at 02:15 AM 4. Task B starts immediately at 02:15 AM (priority over other scheduled tasks) ``` ### Scenario 3: Container Restart During Execution ``` 02:00 AM - Task A starts downloading 02:10 AM - Container restarts (Docker update, etc.) 02:10 AM - Container comes back up 02:10 AM - Task A marked as "failed" with "interrupted" message 02:10 AM - Scheduled tasks reload, Task A will retry at next scheduled time (tomorrow 02:00 AM) 02:10 AM - Other pending tasks start processing from queue ``` ## Future Enhancements (Not in Initial Implementation) 1. **Parallel Execution**: Optional setting to allow N tasks at once (requires more resources) 2. **Smart Scheduling**: Stagger start times automatically if many tasks at same time 3. **Queue Priorities**: User-configurable priority levels for tasks 4. **Retry Logic**: Auto-retry failed tasks with exponential backoff 5. **Additional Frequencies**: Weekly, custom intervals 6. **Notification System**: Email/webhook notifications on completion/failure 7. **Advanced Filters**: Score thresholds, content type filters 8. **Task Templates**: Save and reuse task configurations 9. **Execution History Page**: Dedicated page for detailed history with charts 10. **Bulk Operations**: Enable/disable/delete multiple tasks at once 11. **Export/Import**: Backup and restore scheduled tasks ## Conclusion This implementation provides a robust scheduled downloads system with **guaranteed sequential execution**, designed specifically for Docker deployment in resource-constrained environments. The queue-based approach ensures: - ✅ Only one download at a time (no resource contention) - ✅ Fair task ordering (FIFO with priority support) - ✅ Data persistence across container restarts - ✅ Reliable scheduling with APScheduler - ✅ Proper timezone handling (user TZ -> container UTC) - ✅ Simple, clean UI - ✅ Integration with existing BDFR API - ✅ Smart defaults (daily schedule, time_filter="day", no_dupes=true) - ✅ Easy management (enable/disable/delete/run now) - ✅ Container-aware logging and health checks - ✅ Transparent queue status for users The sequential execution model is perfect for: - Single-user home servers - Docker containers with limited CPU/memory - Preventing Reddit API rate limits - Ensuring reliable, predictable downloads - Avoiding file system contention The system is production-ready for Docker deployment and extensible for future enhancements like parallel execution if needed.