BDFR_Web/web_interface/SCHEDULED_DOWNLOADS.md

# Scheduled Downloads Feature

## Overview

The BDFR Web Interface now supports scheduled downloads, allowing users to configure downloads that run automatically on a daily basis. This feature is designed to work seamlessly in Docker containers and ensures sequential execution to prevent system overload.

## Features

### 1. **Daily Scheduling**
- Tasks run once per day at a user-specified time
- Time is specified in the user's local timezone and automatically converted to UTC for container execution
- Automatically sets `time_filter="day"` to only download content from the last 24 hours
- Always enables `no_dupes=True` to avoid re-downloading existing content

### 2. **Sequential Execution**
- Tasks are processed one at a time through a queue system
- Manual "Run Now" tasks have priority over scheduled tasks
- Queue status is displayed in real-time

### 3. **Persistent Storage**
- SQLite database stores task configurations
- APScheduler job store ensures tasks persist across container restarts
- Execution history tracked for each task

### 4. **Full Task Management**
- Create, enable/disable, and delete scheduled tasks
- Run tasks manually on-demand
- View last run and next scheduled run times

## Architecture

### Backend Components

#### 1. **Database Models** (`web_interface/app/models.py`)
- `ScheduledTask`: Stores task configuration
  - Fields: name, source (subreddit/user), schedule, timezone, etc.
  - Automatically sets time_filter="day" and no_dupes=True
- `TaskExecutionHistory`: Tracks each execution
  - Fields: task_id, status, items downloaded, errors, etc.

#### 2. **Task Queue** (`web_interface/app/task_queue.py`)
- `TaskQueue` class manages sequential execution
- Priority queue: 0=scheduled, 1=manual
- Blocks until each download completes before starting the next
- Thread-safe using asyncio

#### 3. **Scheduler Service** (`web_interface/app/scheduler.py`)
- APScheduler with SQLAlchemy job store for persistence
- Functions:
  - `schedule_task()`: Creates cron job
  - `queue_scheduled_task()`: Adds task to queue (called by scheduler)
  - `execute_scheduled_task()`: Executes download and waits for completion
  - `wait_for_download_completion()`: Polls every 5 seconds until done

#### 4. **API Endpoints** (`web_interface/app/scheduled_tasks.py`)
```
POST   /api/scheduled-tasks          - Create task
GET    /api/scheduled-tasks          - List all tasks
GET    /api/scheduled-tasks/{id}     - Get specific task
PUT    /api/scheduled-tasks/{id}     - Update task
DELETE /api/scheduled-tasks/{id}     - Delete task
POST   /api/scheduled-tasks/{id}/toggle - Enable/disable
POST   /api/scheduled-tasks/{id}/run-now - Queue immediately
GET    /api/scheduled-tasks/{id}/history - Execution history
GET    /api/scheduled-tasks/queue/status - Queue status
```

### Frontend Components

#### 1. **HTML** (`web_interface/templates/index.html`)
- "Run Daily" checkbox in Advanced Options
- Schedule configuration fields (task name, run time)
- Scheduled Downloads section with task cards
- Queue status badge

#### 2. **JavaScript** (`web_interface/static/js/app.js`)
- `loadScheduledTasks()`: Fetches and renders tasks
- `createScheduledTask()`: Creates new scheduled task
- `toggleTask()`, `deleteTask()`, `runTaskNow()`: Task management
- `updateQueueStatus()`: Polls queue every 10 seconds
- Auto-detects browser timezone via `Intl.DateTimeFormat()`

#### 3. **CSS** (`web_interface/static/css/style.css`)
- Task card styling with hover effects
- Status badges (enabled/disabled)
- Schedule options panel
- Queue status badge

## Usage Guide

### Creating a Scheduled Download

1. **Configure Download Settings**
   - Select download mode (Download/Archive/Clone)
   - Choose source type (Subreddit/User)
   - Enter source name
   - Set limit, sort, and other options

2. **Enable Scheduling**
   - Check "Run Daily" in Advanced Options
   - Enter a task name (e.g., "Daily Python Posts")
   - Select run time (24-hour format, in your local timezone)

3. **Submit**
   - Click "Start Download" to create the scheduled task
   - Task appears in the Scheduled Downloads section
   - First run scheduled for the specified time

### Managing Scheduled Tasks

Each task card shows:
- Task name and source
- Download mode
- Schedule (daily at X time)
- Last run and next run times
- Status (Enabled/Disabled)

**Actions:**
- **Disable/Enable**: Toggle task on/off without deleting
- **Run Now**: Add task to queue immediately (higher priority)
- **Delete**: Remove task permanently

### Queue System

- Queue status badge shows number of tasks waiting
- Tasks execute one at a time to prevent overload
- Manual "Run Now" tasks have priority over scheduled tasks
- Download progress appears in Progress section

## Docker Deployment

### Volume Mounts Required

```yaml
volumes:
  - ./downloads:/downloads          # Downloaded files
  - ./data:/app/data                # Database and job store
```

### Database Location
- SQLite: `/app/data/scheduled_tasks.db`
- APScheduler job store: Same database

### Timezone Handling
- User specifies time in their local timezone
- Frontend auto-detects timezone via JavaScript
- Backend converts to UTC for container execution
- Cron jobs run at correct local time regardless of container timezone

## Technical Details

### Sequential Execution Flow

1. APScheduler triggers at scheduled time
2. Scheduler calls `queue_scheduled_task(task_id)`
3. Task added to queue with priority 0
4. Queue worker picks up task
5. `execute_scheduled_task()` called
6. Downloads via existing BDFR API
7. `wait_for_download_completion()` polls every 5s
8. Once complete, queue processes next task
9. Execution history recorded

### Time Filter Logic

For scheduled tasks:
- `time_filter` is automatically set to "day"
- This filters Reddit API to only return posts from last 24 hours
- Combined with daily scheduling, ensures only new content downloaded
- Prevents re-downloading old content

### Duplicate Prevention

For scheduled tasks:
- `no_dupes` is automatically enabled
- Uses existing BDFR duplicate detection
- Checks file hashes or URLs (depending on simple_check setting)
- Skips files that already exist

## Testing Checklist

### Basic Functionality
- [ ] Create scheduled task via "Run Daily" checkbox
- [ ] Task appears in Scheduled Downloads section
- [ ] Task name, source, and schedule displayed correctly
- [ ] Enable/disable toggle works
- [ ] Delete removes task

### Execution
- [ ] Manual "Run Now" triggers download immediately
- [ ] Download progress appears in Progress section
- [ ] Task completes successfully
- [ ] Execution history recorded

### Sequential Processing
- [ ] Queue multiple tasks via "Run Now"
- [ ] Tasks execute one at a time (not concurrent)
- [ ] Queue badge shows correct count
- [ ] Manual tasks execute before scheduled tasks

### Persistence
- [ ] Restart container/server
- [ ] Tasks still present after restart
- [ ] Scheduled jobs still execute at correct time
- [ ] Execution history preserved

### Timezone Handling
- [ ] Create task with different timezone
- [ ] Task runs at correct local time
- [ ] Next run time displays in user's timezone

### Edge Cases
- [ ] Create task with invalid source name
- [ ] Disable task, verify it doesn't run
- [ ] Enable disabled task
- [ ] Delete task while it's running
- [ ] Run same task multiple times quickly

## Troubleshooting

### Tasks Not Running
1. Check container logs for scheduler errors
2. Verify `/app/data` volume is mounted
3. Check database file permissions
4. Verify APScheduler is running (`scheduler.running()`)

### Queue Stuck
1. Check task_queue status in logs
2. Verify WebSocket connection for progress updates
3. Restart container to reset queue

### Timezone Issues
1. Verify browser timezone detection in DevTools
2. Check conversion in scheduler logs
3. Ensure container has correct UTC time

### Database Issues
1. Check `/app/data/scheduled_tasks.db` exists
2. Verify write permissions
3. Use SQLite browser to inspect tables
4. Check for migration errors in logs

## Future Enhancements

Potential improvements:
- [ ] Weekly scheduling option
- [ ] Custom time filters (last 3 days, last week, etc.)
- [ ] Email notifications on completion/failure
- [ ] Retry logic for failed tasks
- [ ] Task templates for quick setup
- [ ] Bulk operations (enable/disable multiple tasks)
- [ ] Advanced schedule expressions (cron syntax)
- [ ] Export/import task configurations
- [ ] Task execution statistics and charts
- [ ] Pause/resume queue

## API Examples

### Create Task
```bash
curl -X POST http://localhost:8000/api/scheduled-tasks \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Daily Python Posts",
    "source_type": "subreddit",
    "source_name": "python",
    "download_mode": "download",
    "limit": 50,
    "sort": "hot",
    "run_time": "02:00",
    "timezone": "Pacific/Auckland",
    "enabled": true
  }'
```

### List Tasks
```bash
curl http://localhost:8000/api/scheduled-tasks
```

### Toggle Task
```bash
curl -X POST http://localhost:8000/api/scheduled-tasks/1/toggle
```

### Run Task Now
```bash
curl -X POST http://localhost:8000/api/scheduled-tasks/1/run-now
```

### Get Queue Status
```bash
curl http://localhost:8000/api/scheduled-tasks/queue/status
```

## Files Modified/Created

### Created:
- `web_interface/app/database.py` - Database configuration
- `web_interface/app/models.py` - ORM models
- `web_interface/app/task_queue.py` - Queue manager
- `web_interface/app/scheduler.py` - Scheduler service
- `web_interface/app/scheduled_tasks.py` - API endpoints
- `web_interface/SCHEDULED_DOWNLOADS.md` - This file

### Modified:
- `web_interface/requirements.txt` - Added dependencies
- `web_interface/app/main.py` - Integrated scheduler
- `web_interface/templates/index.html` - Added UI elements
- `web_interface/static/js/app.js` - Added JavaScript functions
- `web_interface/static/css/style.css` - Added styles

## Dependencies Added

```
sqlalchemy>=2.0.0
alembic>=1.12.0
apscheduler>=3.10.0
pytz>=2023.3