feat(UI): added docker image and config

2025-10-10 11:15:28 +13:00
parent 9e61d18bf6
commit 9f5a25fcf5
23 changed files with 3287 additions and 18 deletions
@@ -0,0 +1,319 @@
+# Scheduled Downloads Feature
+
+## Overview
+
+The BDFR Web Interface now supports scheduled downloads, allowing users to configure downloads that run automatically on a daily basis. This feature is designed to work seamlessly in Docker containers and ensures sequential execution to prevent system overload.
+
+## Features
+
+### 1. **Daily Scheduling**
+- Tasks run once per day at a user-specified time
+- Time is specified in the user's local timezone and automatically converted to UTC for container execution
+- Automatically sets `time_filter="day"` to only download content from the last 24 hours
+- Always enables `no_dupes=True` to avoid re-downloading existing content
+
+### 2. **Sequential Execution**
+- Tasks are processed one at a time through a queue system
+- Manual "Run Now" tasks have priority over scheduled tasks
+- Queue status is displayed in real-time
+
+### 3. **Persistent Storage**
+- SQLite database stores task configurations
+- APScheduler job store ensures tasks persist across container restarts
+- Execution history tracked for each task
+
+### 4. **Full Task Management**
+- Create, enable/disable, and delete scheduled tasks
+- Run tasks manually on-demand
+- View last run and next scheduled run times
+
+## Architecture
+
+### Backend Components
+
+#### 1. **Database Models** (`web_interface/app/models.py`)
+- `ScheduledTask`: Stores task configuration
+  - Fields: name, source (subreddit/user), schedule, timezone, etc.
+  - Automatically sets time_filter="day" and no_dupes=True
+- `TaskExecutionHistory`: Tracks each execution
+  - Fields: task_id, status, items downloaded, errors, etc.
+
+#### 2. **Task Queue** (`web_interface/app/task_queue.py`)
+- `TaskQueue` class manages sequential execution
+- Priority queue: 0=scheduled, 1=manual
+- Blocks until each download completes before starting the next
+- Thread-safe using asyncio
+
+#### 3. **Scheduler Service** (`web_interface/app/scheduler.py`)
+- APScheduler with SQLAlchemy job store for persistence
+- Functions:
+  - `schedule_task()`: Creates cron job
+  - `queue_scheduled_task()`: Adds task to queue (called by scheduler)
+  - `execute_scheduled_task()`: Executes download and waits for completion
+  - `wait_for_download_completion()`: Polls every 5 seconds until done
+
+#### 4. **API Endpoints** (`web_interface/app/scheduled_tasks.py`)
+```
+POST   /api/scheduled-tasks          - Create task
+GET    /api/scheduled-tasks          - List all tasks
+GET    /api/scheduled-tasks/{id}     - Get specific task
+PUT    /api/scheduled-tasks/{id}     - Update task
+DELETE /api/scheduled-tasks/{id}     - Delete task
+POST   /api/scheduled-tasks/{id}/toggle - Enable/disable
+POST   /api/scheduled-tasks/{id}/run-now - Queue immediately
+GET    /api/scheduled-tasks/{id}/history - Execution history
+GET    /api/scheduled-tasks/queue/status - Queue status
+```
+
+### Frontend Components
+
+#### 1. **HTML** (`web_interface/templates/index.html`)
+- "Run Daily" checkbox in Advanced Options
+- Schedule configuration fields (task name, run time)
+- Scheduled Downloads section with task cards
+- Queue status badge
+
+#### 2. **JavaScript** (`web_interface/static/js/app.js`)
+- `loadScheduledTasks()`: Fetches and renders tasks
+- `createScheduledTask()`: Creates new scheduled task
+- `toggleTask()`, `deleteTask()`, `runTaskNow()`: Task management
+- `updateQueueStatus()`: Polls queue every 10 seconds
+- Auto-detects browser timezone via `Intl.DateTimeFormat()`
+
+#### 3. **CSS** (`web_interface/static/css/style.css`)
+- Task card styling with hover effects
+- Status badges (enabled/disabled)
+- Schedule options panel
+- Queue status badge
+
+## Usage Guide
+
+### Creating a Scheduled Download
+
+1. **Configure Download Settings**
+   - Select download mode (Download/Archive/Clone)
+   - Choose source type (Subreddit/User)
+   - Enter source name
+   - Set limit, sort, and other options
+
+2. **Enable Scheduling**
+   - Check "Run Daily" in Advanced Options
+   - Enter a task name (e.g., "Daily Python Posts")
+   - Select run time (24-hour format, in your local timezone)
+
+3. **Submit**
+   - Click "Start Download" to create the scheduled task
+   - Task appears in the Scheduled Downloads section
+   - First run scheduled for the specified time
+
+### Managing Scheduled Tasks
+
+Each task card shows:
+- Task name and source
+- Download mode
+- Schedule (daily at X time)
+- Last run and next run times
+- Status (Enabled/Disabled)
+
+**Actions:**
+- **Disable/Enable**: Toggle task on/off without deleting
+- **Run Now**: Add task to queue immediately (higher priority)
+- **Delete**: Remove task permanently
+
+### Queue System
+
+- Queue status badge shows number of tasks waiting
+- Tasks execute one at a time to prevent overload
+- Manual "Run Now" tasks have priority over scheduled tasks
+- Download progress appears in Progress section
+
+## Docker Deployment
+
+### Volume Mounts Required
+
+```yaml
+volumes:
+  - ./downloads:/downloads          # Downloaded files
+  - ./data:/app/data                # Database and job store
+```
+
+### Database Location
+- SQLite: `/app/data/scheduled_tasks.db`
+- APScheduler job store: Same database
+
+### Timezone Handling
+- User specifies time in their local timezone
+- Frontend auto-detects timezone via JavaScript
+- Backend converts to UTC for container execution
+- Cron jobs run at correct local time regardless of container timezone
+
+## Technical Details
+
+### Sequential Execution Flow
+
+1. APScheduler triggers at scheduled time
+2. Scheduler calls `queue_scheduled_task(task_id)`
+3. Task added to queue with priority 0
+4. Queue worker picks up task
+5. `execute_scheduled_task()` called
+6. Downloads via existing BDFR API
+7. `wait_for_download_completion()` polls every 5s
+8. Once complete, queue processes next task
+9. Execution history recorded
+
+### Time Filter Logic
+
+For scheduled tasks:
+- `time_filter` is automatically set to "day"
+- This filters Reddit API to only return posts from last 24 hours
+- Combined with daily scheduling, ensures only new content downloaded
+- Prevents re-downloading old content
+
+### Duplicate Prevention
+
+For scheduled tasks:
+- `no_dupes` is automatically enabled
+- Uses existing BDFR duplicate detection
+- Checks file hashes or URLs (depending on simple_check setting)
+- Skips files that already exist
+
+## Testing Checklist
+
+### Basic Functionality
+- [ ] Create scheduled task via "Run Daily" checkbox
+- [ ] Task appears in Scheduled Downloads section
+- [ ] Task name, source, and schedule displayed correctly
+- [ ] Enable/disable toggle works
+- [ ] Delete removes task
+
+### Execution
+- [ ] Manual "Run Now" triggers download immediately
+- [ ] Download progress appears in Progress section
+- [ ] Task completes successfully
+- [ ] Execution history recorded
+
+### Sequential Processing
+- [ ] Queue multiple tasks via "Run Now"
+- [ ] Tasks execute one at a time (not concurrent)
+- [ ] Queue badge shows correct count
+- [ ] Manual tasks execute before scheduled tasks
+
+### Persistence
+- [ ] Restart container/server
+- [ ] Tasks still present after restart
+- [ ] Scheduled jobs still execute at correct time
+- [ ] Execution history preserved
+
+### Timezone Handling
+- [ ] Create task with different timezone
+- [ ] Task runs at correct local time
+- [ ] Next run time displays in user's timezone
+
+### Edge Cases
+- [ ] Create task with invalid source name
+- [ ] Disable task, verify it doesn't run
+- [ ] Enable disabled task
+- [ ] Delete task while it's running
+- [ ] Run same task multiple times quickly
+
+## Troubleshooting
+
+### Tasks Not Running
+1. Check container logs for scheduler errors
+2. Verify `/app/data` volume is mounted
+3. Check database file permissions
+4. Verify APScheduler is running (`scheduler.running()`)
+
+### Queue Stuck
+1. Check task_queue status in logs
+2. Verify WebSocket connection for progress updates
+3. Restart container to reset queue
+
+### Timezone Issues
+1. Verify browser timezone detection in DevTools
+2. Check conversion in scheduler logs
+3. Ensure container has correct UTC time
+
+### Database Issues
+1. Check `/app/data/scheduled_tasks.db` exists
+2. Verify write permissions
+3. Use SQLite browser to inspect tables
+4. Check for migration errors in logs
+
+## Future Enhancements
+
+Potential improvements:
+- [ ] Weekly scheduling option
+- [ ] Custom time filters (last 3 days, last week, etc.)
+- [ ] Email notifications on completion/failure
+- [ ] Retry logic for failed tasks
+- [ ] Task templates for quick setup
+- [ ] Bulk operations (enable/disable multiple tasks)
+- [ ] Advanced schedule expressions (cron syntax)
+- [ ] Export/import task configurations
+- [ ] Task execution statistics and charts
+- [ ] Pause/resume queue
+
+## API Examples
+
+### Create Task
+```bash
+curl -X POST http://localhost:8000/api/scheduled-tasks \
+  -H "Content-Type: application/json" \
+  -d '{
+    "name": "Daily Python Posts",
+    "source_type": "subreddit",
+    "source_name": "python",
+    "download_mode": "download",
+    "limit": 50,
+    "sort": "hot",
+    "run_time": "02:00",
+    "timezone": "Pacific/Auckland",
+    "enabled": true
+  }'
+```
+
+### List Tasks
+```bash
+curl http://localhost:8000/api/scheduled-tasks
+```
+
+### Toggle Task
+```bash
+curl -X POST http://localhost:8000/api/scheduled-tasks/1/toggle
+```
+
+### Run Task Now
+```bash
+curl -X POST http://localhost:8000/api/scheduled-tasks/1/run-now
+```
+
+### Get Queue Status
+```bash
+curl http://localhost:8000/api/scheduled-tasks/queue/status
+```
+
+## Files Modified/Created
+
+### Created:
+- `web_interface/app/database.py` - Database configuration
+- `web_interface/app/models.py` - ORM models
+- `web_interface/app/task_queue.py` - Queue manager
+- `web_interface/app/scheduler.py` - Scheduler service
+- `web_interface/app/scheduled_tasks.py` - API endpoints
+- `web_interface/SCHEDULED_DOWNLOADS.md` - This file
+
+### Modified:
+- `web_interface/requirements.txt` - Added dependencies
+- `web_interface/app/main.py` - Integrated scheduler
+- `web_interface/templates/index.html` - Added UI elements
+- `web_interface/static/js/app.js` - Added JavaScript functions
+- `web_interface/static/css/style.css` - Added styles
+
+## Dependencies Added
+
+```
+sqlalchemy>=2.0.0
+alembic>=1.12.0
+apscheduler>=3.10.0
+pytz>=2023.3