BDFR_Web/web_interface/SCHEDULED_DOWNLOADS.md

Scheduled Downloads Feature

Overview

The BDFR Web Interface now supports scheduled downloads, allowing users to configure downloads that run automatically on a daily basis. This feature is designed to work seamlessly in Docker containers and ensures sequential execution to prevent system overload.

Features

1. Daily Scheduling

• Tasks run once per day at a user-specified time
• Time is specified in the user's local timezone and automatically converted to UTC for container execution
• Automatically sets time_filter="day" to only download content from the last 24 hours
• Always enables no_dupes=True to avoid re-downloading existing content

2. Sequential Execution

• Tasks are processed one at a time through a queue system
• Manual "Run Now" tasks have priority over scheduled tasks
• Queue status is displayed in real-time

3. Persistent Storage

• SQLite database stores task configurations
• APScheduler job store ensures tasks persist across container restarts
• Execution history tracked for each task

4. Full Task Management

• Create, enable/disable, and delete scheduled tasks
• Run tasks manually on-demand
• View last run and next scheduled run times

Architecture

Backend Components

1. Database Models (web_interface/app/models.py)

• ScheduledTask: Stores task configuration
  • Fields: name, source (subreddit/user), schedule, timezone, etc.
  • Automatically sets time_filter="day" and no_dupes=True
• TaskExecutionHistory: Tracks each execution
  • Fields: task_id, status, items downloaded, errors, etc.

2. Task Queue (web_interface/app/task_queue.py)

• TaskQueue class manages sequential execution
• Priority queue: 0=scheduled, 1=manual
• Blocks until each download completes before starting the next
• Thread-safe using asyncio

3. Scheduler Service (web_interface/app/scheduler.py)

• APScheduler with SQLAlchemy job store for persistence
• Functions:
  • schedule_task(): Creates cron job
  • queue_scheduled_task(): Adds task to queue (called by scheduler)
  • execute_scheduled_task(): Executes download and waits for completion
  • wait_for_download_completion(): Polls every 5 seconds until done

4. API Endpoints (web_interface/app/scheduled_tasks.py)

POST   /api/scheduled-tasks          - Create task
GET    /api/scheduled-tasks          - List all tasks
GET    /api/scheduled-tasks/{id}     - Get specific task
PUT    /api/scheduled-tasks/{id}     - Update task
DELETE /api/scheduled-tasks/{id}     - Delete task
POST   /api/scheduled-tasks/{id}/toggle - Enable/disable
POST   /api/scheduled-tasks/{id}/run-now - Queue immediately
GET    /api/scheduled-tasks/{id}/history - Execution history
GET    /api/scheduled-tasks/queue/status - Queue status

Frontend Components

1. HTML (web_interface/templates/index.html)

• "Run Daily" checkbox in Advanced Options
• Schedule configuration fields (task name, run time)
• Scheduled Downloads section with task cards
• Queue status badge

2. JavaScript (web_interface/static/js/app.js)

• loadScheduledTasks(): Fetches and renders tasks
• createScheduledTask(): Creates new scheduled task
• toggleTask(), deleteTask(), runTaskNow(): Task management
• updateQueueStatus(): Polls queue every 10 seconds
• Auto-detects browser timezone via Intl.DateTimeFormat()

3. CSS (web_interface/static/css/style.css)

• Task card styling with hover effects
• Status badges (enabled/disabled)
• Schedule options panel
• Queue status badge

Usage Guide

Creating a Scheduled Download

1. Configure Download Settings

   • Select download mode (Download/Archive/Clone)
   • Choose source type (Subreddit/User)
   • Enter source name
   • Set limit, sort, and other options

2. Enable Scheduling

   • Check "Run Daily" in Advanced Options
   • Enter a task name (e.g., "Daily Python Posts")
   • Select run time (24-hour format, in your local timezone)

3. Submit

   • Click "Start Download" to create the scheduled task
   • Task appears in the Scheduled Downloads section
   • First run scheduled for the specified time

Managing Scheduled Tasks

Each task card shows:

• Task name and source
• Download mode
• Schedule (daily at X time)
• Last run and next run times
• Status (Enabled/Disabled)

Actions:

• Disable/Enable: Toggle task on/off without deleting
• Run Now: Add task to queue immediately (higher priority)
• Delete: Remove task permanently

Queue System

• Queue status badge shows number of tasks waiting
• Tasks execute one at a time to prevent overload
• Manual "Run Now" tasks have priority over scheduled tasks
• Download progress appears in Progress section

Docker Deployment

Volume Mounts Required

volumes:
  - ./downloads:/downloads          # Downloaded files
  - ./data:/app/data                # Database and job store

Database Location

• SQLite: /app/data/scheduled_tasks.db
• APScheduler job store: Same database

Timezone Handling

• User specifies time in their local timezone
• Frontend auto-detects timezone via JavaScript
• Backend converts to UTC for container execution
• Cron jobs run at correct local time regardless of container timezone

Technical Details

Sequential Execution Flow

1. APScheduler triggers at scheduled time
2. Scheduler calls queue_scheduled_task(task_id)
3. Task added to queue with priority 0
4. Queue worker picks up task
5. execute_scheduled_task() called
6. Downloads via existing BDFR API
7. wait_for_download_completion() polls every 5s
8. Once complete, queue processes next task
9. Execution history recorded

Time Filter Logic

For scheduled tasks:

• time_filter is automatically set to "day"
• This filters Reddit API to only return posts from last 24 hours
• Combined with daily scheduling, ensures only new content downloaded
• Prevents re-downloading old content

Duplicate Prevention

For scheduled tasks:

• no_dupes is automatically enabled
• Uses existing BDFR duplicate detection
• Checks file hashes or URLs (depending on simple_check setting)
• Skips files that already exist

Testing Checklist

Basic Functionality

• Create scheduled task via "Run Daily" checkbox
• Task appears in Scheduled Downloads section
• Task name, source, and schedule displayed correctly
• Enable/disable toggle works
• Delete removes task

Execution

• Manual "Run Now" triggers download immediately
• Download progress appears in Progress section
• Task completes successfully
• Execution history recorded

Sequential Processing

• Queue multiple tasks via "Run Now"
• Tasks execute one at a time (not concurrent)
• Queue badge shows correct count
• Manual tasks execute before scheduled tasks

Persistence

• Restart container/server
• Tasks still present after restart
• Scheduled jobs still execute at correct time
• Execution history preserved

Timezone Handling

• Create task with different timezone
• Task runs at correct local time
• Next run time displays in user's timezone

Edge Cases

• Create task with invalid source name
• Disable task, verify it doesn't run
• Enable disabled task
• Delete task while it's running
• Run same task multiple times quickly

Troubleshooting

Tasks Not Running

1. Check container logs for scheduler errors
2. Verify /app/data volume is mounted
3. Check database file permissions
4. Verify APScheduler is running (scheduler.running())

Queue Stuck

1. Check task_queue status in logs
2. Verify WebSocket connection for progress updates
3. Restart container to reset queue

Timezone Issues

1. Verify browser timezone detection in DevTools
2. Check conversion in scheduler logs
3. Ensure container has correct UTC time

Database Issues

1. Check /app/data/scheduled_tasks.db exists
2. Verify write permissions
3. Use SQLite browser to inspect tables
4. Check for migration errors in logs

Future Enhancements

Potential improvements:

• Weekly scheduling option
• Custom time filters (last 3 days, last week, etc.)
• Email notifications on completion/failure
• Retry logic for failed tasks
• Task templates for quick setup
• Bulk operations (enable/disable multiple tasks)
• Advanced schedule expressions (cron syntax)
• Export/import task configurations
• Task execution statistics and charts
• Pause/resume queue

API Examples

Create Task

curl -X POST http://localhost:8000/api/scheduled-tasks \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Daily Python Posts",
    "source_type": "subreddit",
    "source_name": "python",
    "download_mode": "download",
    "limit": 50,
    "sort": "hot",
    "run_time": "02:00",
    "timezone": "Pacific/Auckland",
    "enabled": true
  }'

List Tasks

curl http://localhost:8000/api/scheduled-tasks

Toggle Task

curl -X POST http://localhost:8000/api/scheduled-tasks/1/toggle

Run Task Now

curl -X POST http://localhost:8000/api/scheduled-tasks/1/run-now

Get Queue Status

curl http://localhost:8000/api/scheduled-tasks/queue/status

Files Modified/Created

Created:

• web_interface/app/database.py - Database configuration
• web_interface/app/models.py - ORM models
• web_interface/app/task_queue.py - Queue manager
• web_interface/app/scheduler.py - Scheduler service
• web_interface/app/scheduled_tasks.py - API endpoints
• web_interface/SCHEDULED_DOWNLOADS.md - This file

Modified:

• web_interface/requirements.txt - Added dependencies
• web_interface/app/main.py - Integrated scheduler
• web_interface/templates/index.html - Added UI elements
• web_interface/static/js/app.js - Added JavaScript functions
• web_interface/static/css/style.css - Added styles

Dependencies Added

sqlalchemy>=2.0.0
alembic>=1.12.0
apscheduler>=3.10.0
pytz>=2023.3