Fixed issue where file extenions not found and auth timeout
This commit is contained in:
@@ -0,0 +1,839 @@
|
||||
# Scheduled Downloads Implementation Plan
|
||||
|
||||
## Overview
|
||||
This document outlines the implementation plan for adding scheduled download functionality to the BDFR Web Interface. Users will be able to configure downloads that run automatically on a daily schedule, perfect for keeping up with new content from their favorite subreddits or users.
|
||||
|
||||
## Requirements Summary
|
||||
- **Database**: SQLite with SQLAlchemy ORM
|
||||
- **Scheduling**: Daily frequency (runs every 24 hours)
|
||||
- **UI Approach**: Simple - checkbox in Advanced Options + management section on main page
|
||||
- **Time Filter**: Automatically set to "last day" for daily runs
|
||||
- **Duplicate Handling**: Works with existing no-dupes functionality
|
||||
- **Deployment**: Docker container environment
|
||||
- **Execution Model**: **Sequential only - one task at a time, queued execution**
|
||||
|
||||
## Architecture
|
||||
|
||||
### 1. Database Schema
|
||||
|
||||
#### ScheduledTask Table
|
||||
```python
|
||||
class ScheduledTask:
|
||||
id: UUID (Primary Key)
|
||||
name: str # User-friendly name for the task
|
||||
enabled: bool # Whether task is active
|
||||
|
||||
# Download Configuration
|
||||
source_type: str # "subreddit" or "user"
|
||||
source_name: str # Name of subreddit or username
|
||||
download_mode: str # "download", "archive", or "clone"
|
||||
|
||||
# Filter Options
|
||||
limit: int
|
||||
sort: str # "hot", "top", "new", etc.
|
||||
time_filter: str # Always "day" for daily tasks
|
||||
min_score: int (optional)
|
||||
no_dupes: bool # Always true for scheduled tasks
|
||||
simple_check: bool
|
||||
|
||||
# Scheduling
|
||||
schedule_frequency: str # "daily" (extensible for future: "weekly", "custom")
|
||||
run_time: time # Time of day to run (e.g., "02:00:00")
|
||||
timezone: str # User's timezone (default: UTC)
|
||||
|
||||
# Metadata
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
last_run_at: datetime (nullable)
|
||||
next_run_at: datetime
|
||||
|
||||
# Authentication
|
||||
auth_state: str (nullable) # For authenticated downloads
|
||||
```
|
||||
|
||||
#### TaskExecutionHistory Table
|
||||
```python
|
||||
class TaskExecutionHistory:
|
||||
id: UUID (Primary Key)
|
||||
task_id: UUID (Foreign Key -> ScheduledTask)
|
||||
|
||||
# Execution Details
|
||||
started_at: datetime
|
||||
completed_at: datetime (nullable)
|
||||
status: str # "success", "failed", "running", "queued"
|
||||
|
||||
# Results
|
||||
items_found: int
|
||||
items_downloaded: int
|
||||
error_message: str (nullable)
|
||||
|
||||
# Link to download
|
||||
download_id: str # Links to active_downloads tracking
|
||||
```
|
||||
|
||||
### 2. Backend Components
|
||||
|
||||
#### File Structure
|
||||
```
|
||||
web_interface/
|
||||
├── app/
|
||||
│ ├── __init__.py
|
||||
│ ├── main.py (existing)
|
||||
│ ├── auth.py (existing)
|
||||
│ ├── database.py (NEW - SQLAlchemy setup)
|
||||
│ ├── models.py (NEW - DB models)
|
||||
│ ├── scheduler.py (NEW - APScheduler + Queue integration)
|
||||
│ ├── task_queue.py (NEW - Sequential task queue manager)
|
||||
│ └── scheduled_tasks.py (NEW - Task management logic)
|
||||
├── data/
|
||||
│ └── scheduled_tasks.db (SQLite database - created at runtime)
|
||||
└── requirements.txt (UPDATE - add dependencies)
|
||||
```
|
||||
|
||||
#### Dependencies to Add
|
||||
```txt
|
||||
sqlalchemy>=2.0.0
|
||||
alembic>=1.12.0 # For database migrations
|
||||
apscheduler>=3.10.0 # For task scheduling
|
||||
```
|
||||
|
||||
#### API Endpoints
|
||||
|
||||
**Scheduled Tasks CRUD:**
|
||||
- `POST /api/scheduled-tasks` - Create new scheduled task
|
||||
- `GET /api/scheduled-tasks` - List all scheduled tasks
|
||||
- `GET /api/scheduled-tasks/{task_id}` - Get specific task details
|
||||
- `PUT /api/scheduled-tasks/{task_id}` - Update task configuration
|
||||
- `DELETE /api/scheduled-tasks/{task_id}` - Delete task
|
||||
- `POST /api/scheduled-tasks/{task_id}/toggle` - Enable/disable task
|
||||
- `POST /api/scheduled-tasks/{task_id}/run-now` - Trigger immediate execution (adds to queue)
|
||||
|
||||
**Task History & Queue:**
|
||||
- `GET /api/scheduled-tasks/{task_id}/history` - Get execution history
|
||||
- `GET /api/scheduled-tasks/history/recent` - Get recent executions across all tasks
|
||||
- `GET /api/scheduled-tasks/queue` - Get current task queue status
|
||||
|
||||
### 3. Sequential Task Queue System
|
||||
|
||||
#### Task Queue Manager (`task_queue.py`)
|
||||
|
||||
**Core Concept**: Only one scheduled download can run at a time. When multiple tasks are triggered (either by schedule or "Run Now"), they are queued and executed sequentially.
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from typing import Optional, List, Dict
|
||||
from datetime import datetime
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class TaskQueue:
|
||||
"""
|
||||
Manages sequential execution of scheduled download tasks.
|
||||
Ensures only one task runs at a time.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self.queue: asyncio.Queue = asyncio.Queue()
|
||||
self.current_task: Optional[str] = None # Current task_id being executed
|
||||
self.is_processing: bool = False
|
||||
self.worker_task: Optional[asyncio.Task] = None
|
||||
|
||||
async def add_task(self, task_id: str, priority: int = 0):
|
||||
"""
|
||||
Add a task to the queue.
|
||||
|
||||
Args:
|
||||
task_id: UUID of the scheduled task
|
||||
priority: 0 = scheduled (normal), 1 = manual "Run Now" (higher priority)
|
||||
"""
|
||||
await self.queue.put({
|
||||
'task_id': task_id,
|
||||
'priority': priority,
|
||||
'queued_at': datetime.now()
|
||||
})
|
||||
logger.info(f"Task {task_id} added to queue (priority={priority}, queue_size={self.queue.qsize()})")
|
||||
|
||||
# Start worker if not already running
|
||||
if not self.is_processing:
|
||||
await self.start_worker()
|
||||
|
||||
async def start_worker(self):
|
||||
"""Start the queue worker if not already running"""
|
||||
if self.worker_task is None or self.worker_task.done():
|
||||
self.worker_task = asyncio.create_task(self._process_queue())
|
||||
logger.info("Queue worker started")
|
||||
|
||||
async def _process_queue(self):
|
||||
"""Process tasks from queue sequentially"""
|
||||
self.is_processing = True
|
||||
logger.info("Queue worker processing started")
|
||||
|
||||
while True:
|
||||
try:
|
||||
# Wait for next task (with timeout to allow graceful shutdown)
|
||||
try:
|
||||
task_info = await asyncio.wait_for(
|
||||
self.queue.get(),
|
||||
timeout=60.0
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
# Check if queue is empty
|
||||
if self.queue.empty():
|
||||
logger.info("Queue empty, worker stopping")
|
||||
break
|
||||
continue
|
||||
|
||||
task_id = task_info['task_id']
|
||||
self.current_task = task_id
|
||||
|
||||
logger.info(f"Executing task {task_id} from queue (queue_size={self.queue.qsize()})")
|
||||
|
||||
# Execute the task (this will block until download completes)
|
||||
try:
|
||||
await execute_scheduled_task(task_id)
|
||||
logger.info(f"Task {task_id} completed successfully")
|
||||
except Exception as e:
|
||||
logger.error(f"Task {task_id} failed: {e}")
|
||||
finally:
|
||||
self.current_task = None
|
||||
self.queue.task_done()
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Queue worker error: {e}")
|
||||
|
||||
self.is_processing = False
|
||||
logger.info("Queue worker stopped")
|
||||
|
||||
def get_queue_status(self) -> Dict:
|
||||
"""Get current queue status"""
|
||||
return {
|
||||
'current_task': self.current_task,
|
||||
'queue_size': self.queue.qsize(),
|
||||
'is_processing': self.is_processing
|
||||
}
|
||||
|
||||
async def stop(self):
|
||||
"""Stop the queue worker gracefully"""
|
||||
logger.info("Stopping queue worker...")
|
||||
if self.worker_task and not self.worker_task.done():
|
||||
# Wait for current task to complete
|
||||
await self.worker_task
|
||||
logger.info("Queue worker stopped")
|
||||
|
||||
# Global queue instance
|
||||
task_queue = TaskQueue()
|
||||
```
|
||||
|
||||
### 4. Scheduler Service (Docker-Aware with Sequential Queue)
|
||||
|
||||
#### APScheduler Configuration
|
||||
```python
|
||||
from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
from apscheduler.triggers.cron import CronTrigger
|
||||
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore
|
||||
import pytz
|
||||
|
||||
# Use database job store for persistence across container restarts
|
||||
jobstores = {
|
||||
'default': SQLAlchemyJobStore(url='sqlite:///data/scheduler_jobs.db')
|
||||
}
|
||||
|
||||
# Configure scheduler for Docker container
|
||||
scheduler = AsyncIOScheduler(
|
||||
jobstores=jobstores,
|
||||
timezone=pytz.UTC, # Container runs in UTC
|
||||
job_defaults={
|
||||
'coalesce': True, # Combine multiple missed executions into one
|
||||
'max_instances': 1, # Only one instance of each job at a time
|
||||
'misfire_grace_time': 3600 # Allow up to 1 hour late execution
|
||||
}
|
||||
)
|
||||
|
||||
# Add job for each enabled task
|
||||
def schedule_task(task: ScheduledTask):
|
||||
"""
|
||||
Schedule a task to be added to the queue at specified time.
|
||||
Note: This doesn't execute the task directly, it queues it.
|
||||
"""
|
||||
user_tz = pytz.timezone(task.timezone)
|
||||
hour, minute = task.run_time.hour, task.run_time.minute
|
||||
|
||||
scheduler.add_job(
|
||||
func=queue_scheduled_task, # Add to queue, not execute directly
|
||||
trigger=CronTrigger(hour=hour, minute=minute, timezone=user_tz),
|
||||
args=[task.id],
|
||||
id=str(task.id),
|
||||
replace_existing=True
|
||||
)
|
||||
logger.info(f"Scheduled task {task.id} for {hour:02d}:{minute:02d} {task.timezone}")
|
||||
|
||||
async def queue_scheduled_task(task_id: str):
|
||||
"""
|
||||
Called by scheduler at the configured time.
|
||||
Adds task to queue rather than executing immediately.
|
||||
"""
|
||||
logger.info(f"Scheduler triggered for task {task_id}, adding to queue")
|
||||
await task_queue.add_task(task_id, priority=0) # Normal priority for scheduled tasks
|
||||
```
|
||||
|
||||
#### Task Execution Flow with Queue
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Scheduler triggers at scheduled time] --> B[Add task to queue]
|
||||
B --> C{Is queue worker running?}
|
||||
C -->|No| D[Start queue worker]
|
||||
C -->|Yes| E[Task waits in queue]
|
||||
D --> F[Worker picks next task from queue]
|
||||
E --> F
|
||||
F --> G[Load task from DB]
|
||||
G --> H[Check if enabled]
|
||||
H -->|Disabled| I[Skip, mark in history]
|
||||
H -->|Enabled| J[Create execution history]
|
||||
J --> K[Set status = 'running']
|
||||
K --> L[Build download parameters]
|
||||
L --> M[Set time_filter=day, no_dupes=true]
|
||||
M --> N[Call BDFR API - BLOCKS until complete]
|
||||
N --> O[Wait for download to finish]
|
||||
O --> P[Update execution history]
|
||||
P --> Q[Update last_run_at]
|
||||
Q --> R[Worker picks next task]
|
||||
R -->|Queue empty| S[Worker idles/stops]
|
||||
R -->|More tasks| F
|
||||
|
||||
style N fill:#ffcccc
|
||||
style O fill:#ffcccc
|
||||
note1[Note: Worker blocks here until download completes]
|
||||
```
|
||||
|
||||
### 5. Integration with BDFR API (Sequential Execution)
|
||||
|
||||
```python
|
||||
async def execute_scheduled_task(task_id: str):
|
||||
"""
|
||||
Execute a scheduled download task.
|
||||
This function BLOCKS until the download is complete,
|
||||
ensuring sequential execution.
|
||||
"""
|
||||
# Load task from database
|
||||
task = get_scheduled_task(task_id)
|
||||
|
||||
if not task.enabled:
|
||||
logger.info(f"Skipping disabled task {task_id}")
|
||||
# Still record in history that it was skipped
|
||||
execution = create_execution_history(task_id)
|
||||
execution.status = 'skipped'
|
||||
execution.completed_at = datetime.now(pytz.UTC)
|
||||
save_execution_history(execution)
|
||||
return
|
||||
|
||||
# Create execution history record
|
||||
execution = create_execution_history(task_id)
|
||||
|
||||
try:
|
||||
logger.info(f"Executing scheduled task {task_id}: {task.source_type}/{task.source_name}")
|
||||
|
||||
# Build download parameters
|
||||
kwargs = {
|
||||
'limit': task.limit,
|
||||
'sort': task.sort,
|
||||
'time_filter': 'day', # Always "day" for daily scheduled tasks
|
||||
'no_dupes': True, # Always enabled for scheduled tasks
|
||||
'simple_check': task.simple_check,
|
||||
'auth_state': task.auth_state
|
||||
}
|
||||
|
||||
# Create download using existing API
|
||||
# This returns immediately with a download_id
|
||||
download_id = await create_download_with_bdfr_api(
|
||||
download_type=task.source_type,
|
||||
name=task.source_name,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
logger.info(f"Task {task_id} started as download {download_id}")
|
||||
|
||||
# Track download
|
||||
execution.download_id = download_id
|
||||
execution.status = 'running'
|
||||
save_execution_history(execution)
|
||||
|
||||
# **CRITICAL: Wait for download to complete before returning**
|
||||
# This ensures the next queued task doesn't start until this one finishes
|
||||
await wait_for_download_completion(download_id)
|
||||
|
||||
# Check final status
|
||||
download_status = bdfr_manager.get_download_status(download_id)
|
||||
|
||||
if download_status and download_status['status'] == 'completed':
|
||||
execution.status = 'success'
|
||||
execution.items_found = download_status.get('items_found', 0)
|
||||
execution.items_downloaded = download_status.get('items_processed', 0)
|
||||
logger.info(f"Task {task_id} completed successfully")
|
||||
else:
|
||||
execution.status = 'failed'
|
||||
execution.error_message = download_status.get('error', 'Unknown error')
|
||||
logger.error(f"Task {task_id} failed: {execution.error_message}")
|
||||
|
||||
execution.completed_at = datetime.now(pytz.UTC)
|
||||
save_execution_history(execution)
|
||||
|
||||
# Update task timestamps
|
||||
task.last_run_at = datetime.now(pytz.UTC)
|
||||
task.next_run_at = calculate_next_run(task)
|
||||
save_scheduled_task(task)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Task {task_id} execution error: {e}")
|
||||
execution.status = 'failed'
|
||||
execution.error_message = str(e)
|
||||
execution.completed_at = datetime.now(pytz.UTC)
|
||||
save_execution_history(execution)
|
||||
raise
|
||||
|
||||
async def wait_for_download_completion(download_id: str, timeout: int = 3600):
|
||||
"""
|
||||
Wait for a download to complete.
|
||||
Polls the download status until it's no longer running.
|
||||
|
||||
Args:
|
||||
download_id: The download to wait for
|
||||
timeout: Maximum seconds to wait (default 1 hour)
|
||||
"""
|
||||
start_time = datetime.now()
|
||||
check_interval = 5 # Check every 5 seconds
|
||||
|
||||
while True:
|
||||
# Check if timeout exceeded
|
||||
elapsed = (datetime.now() - start_time).total_seconds()
|
||||
if elapsed > timeout:
|
||||
logger.error(f"Download {download_id} timed out after {timeout}s")
|
||||
raise TimeoutError(f"Download exceeded timeout of {timeout}s")
|
||||
|
||||
# Check download status
|
||||
status = bdfr_manager.get_download_status(download_id)
|
||||
|
||||
if not status:
|
||||
logger.warning(f"Download {download_id} status not found")
|
||||
break
|
||||
|
||||
download_status = status.get('status', 'unknown')
|
||||
|
||||
# Check if download is finished (completed, failed, or cancelled)
|
||||
if download_status in ['completed', 'failed', 'cancelled']:
|
||||
logger.info(f"Download {download_id} finished with status: {download_status}")
|
||||
break
|
||||
|
||||
# Still running, wait before checking again
|
||||
await asyncio.sleep(check_interval)
|
||||
```
|
||||
|
||||
### 6. API Endpoints with Queue Support
|
||||
|
||||
```python
|
||||
@app.post("/api/scheduled-tasks/{task_id}/run-now")
|
||||
async def run_task_now(task_id: str):
|
||||
"""
|
||||
Manually trigger a scheduled task to run now.
|
||||
Adds it to the queue with high priority.
|
||||
"""
|
||||
task = get_scheduled_task(task_id)
|
||||
if not task:
|
||||
raise HTTPException(status_code=404, detail="Task not found")
|
||||
|
||||
# Add to queue with priority (goes ahead of scheduled tasks)
|
||||
await task_queue.add_task(task_id, priority=1)
|
||||
|
||||
queue_status = task_queue.get_queue_status()
|
||||
|
||||
return {
|
||||
"message": f"Task {task_id} added to queue",
|
||||
"queue_position": queue_status['queue_size'],
|
||||
"currently_running": queue_status['current_task'],
|
||||
"status": "queued" if queue_status['current_task'] else "starting"
|
||||
}
|
||||
|
||||
@app.get("/api/scheduled-tasks/queue")
|
||||
async def get_queue_status():
|
||||
"""Get current task queue status"""
|
||||
status = task_queue.get_queue_status()
|
||||
|
||||
# Get details of current task if any
|
||||
current_task_info = None
|
||||
if status['current_task']:
|
||||
task = get_scheduled_task(status['current_task'])
|
||||
if task:
|
||||
current_task_info = {
|
||||
'id': task.id,
|
||||
'name': task.name,
|
||||
'source': f"{task.source_type}/{task.source_name}"
|
||||
}
|
||||
|
||||
return {
|
||||
'queue_size': status['queue_size'],
|
||||
'is_processing': status['is_processing'],
|
||||
'current_task': current_task_info
|
||||
}
|
||||
```
|
||||
|
||||
### 7. Frontend Implementation
|
||||
|
||||
#### UI Modifications to [`index.html`](web_interface/templates/index.html:1)
|
||||
|
||||
**Queue Status Indicator (add to header):**
|
||||
```html
|
||||
<div id="queueStatus" class="queue-status" style="display: none;">
|
||||
<span class="queue-icon">⏳</span>
|
||||
<span id="queueText">Processing scheduled tasks...</span>
|
||||
</div>
|
||||
```
|
||||
|
||||
**Advanced Options Section - Add Checkbox:**
|
||||
```html
|
||||
<!-- After existing checkboxes in Advanced Options -->
|
||||
<label class="checkbox-label" data-tooltip="Run this download automatically every day at a specific time">
|
||||
<input type="checkbox" id="runDaily" name="run_daily">
|
||||
<span class="checkmark"></span>
|
||||
Schedule Daily Run
|
||||
</label>
|
||||
|
||||
<!-- Conditionally shown when checkbox is checked -->
|
||||
<div id="scheduleOptions" style="display: none;">
|
||||
<div class="form-group">
|
||||
<label for="scheduleName">Task Name:</label>
|
||||
<input type="text" id="scheduleName" name="schedule_name"
|
||||
placeholder="e.g., Daily r/Python downloads">
|
||||
<small class="form-help">Give this scheduled task a descriptive name</small>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="scheduleTime">Run Time:</label>
|
||||
<input type="time" id="scheduleTime" name="schedule_time" value="02:00">
|
||||
<small class="form-help">Time when task will run daily (your local time). Tasks run one at a time in order.</small>
|
||||
</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
**New Section - Scheduled Tasks List:**
|
||||
```html
|
||||
<!-- After Progress Section -->
|
||||
<section class="scheduled-section">
|
||||
<div class="section-header">
|
||||
<h2>📅 Scheduled Downloads</h2>
|
||||
<div id="queueStatusBadge" class="queue-badge" style="display: none;">
|
||||
<span class="badge-icon">⏳</span>
|
||||
<span id="queueBadgeText">Queue: 0</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="scheduledTasksList" class="scheduled-tasks-list">
|
||||
<!-- Empty state -->
|
||||
<div class="no-tasks" id="noTasksMessage">
|
||||
<div class="empty-state">
|
||||
<div class="empty-icon">📅</div>
|
||||
<p>No scheduled tasks yet</p>
|
||||
<p>Check "Schedule Daily Run" when creating a download to set up automated daily downloads.</p>
|
||||
<p><strong>Note:</strong> Scheduled tasks run one at a time to prevent server overload.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Task cards will be inserted here -->
|
||||
<div id="scheduledTasksItems"></div>
|
||||
</div>
|
||||
</section>
|
||||
```
|
||||
|
||||
**Task Card Template:**
|
||||
```html
|
||||
<div class="task-card" id="task-{task_id}" data-task-id="{task_id}">
|
||||
<div class="task-header">
|
||||
<div class="task-info">
|
||||
<h4>{task_name}</h4>
|
||||
<div class="task-meta">
|
||||
<span class="task-source">{source_type}: {source_name}</span>
|
||||
<span class="task-schedule">⏰ Runs daily at {run_time}</span>
|
||||
<span class="task-queue-info" style="display: none;">
|
||||
⏳ Queued / Currently Running
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="task-controls">
|
||||
<button class="btn-toggle" onclick="toggleTask('{task_id}')">
|
||||
{enabled ? "✓ Enabled" : "○ Disabled"}
|
||||
</button>
|
||||
<button class="btn-run-now" onclick="runTaskNow('{task_id}')">
|
||||
▶ Run Now
|
||||
</button>
|
||||
<button class="btn-delete" onclick="deleteTask('{task_id}')">
|
||||
🗑 Delete
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="task-details">
|
||||
<div class="task-stat">
|
||||
<span class="stat-label">Last Run:</span>
|
||||
<span class="stat-value">{last_run_at || "Never"}</span>
|
||||
</div>
|
||||
<div class="task-stat">
|
||||
<span class="stat-label">Next Run:</span>
|
||||
<span class="stat-value">{next_run_at}</span>
|
||||
</div>
|
||||
<div class="task-stat">
|
||||
<span class="stat-label">Mode:</span>
|
||||
<span class="stat-value">{download_mode}</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
#### JavaScript Modifications to [`app.js`](web_interface/static/js/app.js:1)
|
||||
|
||||
**Add queue status polling:**
|
||||
```javascript
|
||||
async loadScheduledTasks() {
|
||||
const response = await fetch('/api/scheduled-tasks');
|
||||
const tasks = await response.json();
|
||||
this.renderScheduledTasks(tasks);
|
||||
|
||||
// Also update queue status
|
||||
this.updateQueueStatus();
|
||||
}
|
||||
|
||||
async updateQueueStatus() {
|
||||
try {
|
||||
const response = await fetch('/api/scheduled-tasks/queue');
|
||||
const status = await response.json();
|
||||
|
||||
// Update queue badge
|
||||
const queueBadge = document.getElementById('queueStatusBadge');
|
||||
const queueText = document.getElementById('queueBadgeText');
|
||||
|
||||
if (status.queue_size > 0 || status.current_task) {
|
||||
queueBadge.style.display = 'flex';
|
||||
queueText.textContent = `Queue: ${status.queue_size}${status.current_task ? ' (1 running)' : ''}`;
|
||||
} else {
|
||||
queueBadge.style.display = 'none';
|
||||
}
|
||||
|
||||
// Highlight currently running task card
|
||||
document.querySelectorAll('.task-card').forEach(card => {
|
||||
const taskId = card.dataset.taskId;
|
||||
const queueInfo = card.querySelector('.task-queue-info');
|
||||
|
||||
if (status.current_task && status.current_task.id === taskId) {
|
||||
card.classList.add('task-running');
|
||||
queueInfo.textContent = '⚡ Currently Running';
|
||||
queueInfo.style.display = 'inline';
|
||||
} else {
|
||||
card.classList.remove('task-running');
|
||||
queueInfo.style.display = 'none';
|
||||
}
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('Failed to update queue status:', error);
|
||||
}
|
||||
}
|
||||
|
||||
async runTaskNow(taskId) {
|
||||
const response = await fetch(`/api/scheduled-tasks/${taskId}/run-now`, {
|
||||
method: 'POST'
|
||||
});
|
||||
|
||||
if (response.ok) {
|
||||
const result = await response.json();
|
||||
|
||||
if (result.queue_position > 0) {
|
||||
this.showSuccess(`Task added to queue. Position: ${result.queue_position}`);
|
||||
} else {
|
||||
this.showSuccess('Task execution starting...');
|
||||
}
|
||||
|
||||
this.updateQueueStatus();
|
||||
} else {
|
||||
this.showError('Failed to queue task');
|
||||
}
|
||||
}
|
||||
|
||||
// Poll queue status every 10 seconds
|
||||
startQueuePolling() {
|
||||
setInterval(() => {
|
||||
if (document.querySelectorAll('.task-card').length > 0) {
|
||||
this.updateQueueStatus();
|
||||
}
|
||||
}, 10000);
|
||||
}
|
||||
```
|
||||
|
||||
### 8. Implementation Steps
|
||||
|
||||
1. **Backend Foundation** (Steps 4-5)
|
||||
- Add SQLAlchemy and APScheduler to [`requirements.txt`](web_interface/requirements.txt:1)
|
||||
- Create [`database.py`](web_interface/app/database.py) with Docker-aware paths
|
||||
- Create [`models.py`](web_interface/app/models.py) with ScheduledTask and TaskExecutionHistory models
|
||||
- Initialize database on app startup
|
||||
|
||||
2. **Task Queue System** (Step 7)
|
||||
- Create [`task_queue.py`](web_interface/app/task_queue.py) with sequential queue manager
|
||||
- Implement queue worker with blocking execution
|
||||
- Add queue status tracking and reporting
|
||||
|
||||
3. **Scheduler Service** (Step 7 continued)
|
||||
- Create [`scheduler.py`](web_interface/app/scheduler.py) with Docker-aware APScheduler
|
||||
- Integrate with task queue (scheduler adds to queue, doesn't execute directly)
|
||||
- Implement `wait_for_download_completion()` to block until download finishes
|
||||
- Add scheduler lifecycle hooks to [`main.py`](web_interface/app/main.py:1)
|
||||
|
||||
4. **API Endpoints** (Step 6)
|
||||
- Create [`scheduled_tasks.py`](web_interface/app/scheduled_tasks.py) with CRUD operations
|
||||
- Add routes to [`main.py`](web_interface/app/main.py:1)
|
||||
- Implement task toggle, delete, and run-now (with queue)
|
||||
- Add queue status endpoint
|
||||
|
||||
5. **Frontend - Form** (Step 8)
|
||||
- Add "Run Daily" checkbox to Advanced Options in [`index.html`](web_interface/templates/index.html:1)
|
||||
- Add conditional schedule configuration fields
|
||||
- Update form submission logic in [`app.js`](web_interface/static/js/app.js:1)
|
||||
- Auto-detect browser timezone
|
||||
|
||||
6. **Frontend - Management** (Step 9)
|
||||
- Add Scheduled Tasks section to [`index.html`](web_interface/templates/index.html:1)
|
||||
- Add queue status indicator
|
||||
- Implement task card rendering with queue status
|
||||
- Add toggle, delete, and run-now functions to [`app.js`](web_interface/static/js/app.js:1)
|
||||
- Add queue status polling
|
||||
|
||||
7. **Integration** (Steps 10-11)
|
||||
- Connect scheduler to BDFR API via [`create_download_with_bdfr_api()`](web_interface/app/main.py:373)
|
||||
- Implement automatic `time_filter="day"` for scheduled tasks
|
||||
- Add execution history tracking
|
||||
- Ensure sequential execution with proper blocking
|
||||
|
||||
8. **Testing** (Step 14)
|
||||
- Test task creation, editing, deletion
|
||||
- Test queue functionality (multiple tasks, sequential execution)
|
||||
- Test "Run Now" adds to queue correctly
|
||||
- Test priority (manual tasks run before scheduled)
|
||||
- Test container restart persistence
|
||||
- Verify only one task runs at a time
|
||||
|
||||
## Key Features
|
||||
|
||||
### Sequential Execution (NEW)
|
||||
- **Task Queue**: All scheduled downloads go through a FIFO queue
|
||||
- **Blocking Execution**: Each task blocks until its download completes
|
||||
- **No Concurrency**: Only one download runs at a time, preventing system overload
|
||||
- **Priority System**: Manual "Run Now" tasks get priority over scheduled tasks
|
||||
- **Queue Status**: Users can see queue size and currently running task
|
||||
|
||||
### Docker-Specific Features
|
||||
- **Persistent Storage**: SQLite database and downloads persist via volume mounts
|
||||
- **Container Restarts**: APScheduler with job store survives restarts, queue rebuilds on startup
|
||||
- **Timezone Handling**: User timezone stored, converted to UTC for container execution
|
||||
- **Logging**: Structured logging for container environment
|
||||
- **Health Checks**: Scheduler and queue status included in health endpoint
|
||||
|
||||
### Automatic Configuration for Scheduled Tasks
|
||||
- **time_filter**: Always set to "day" - ensures only last 24 hours of content
|
||||
- **no_dupes**: Always enabled - prevents re-downloading same content
|
||||
- **Timezone handling**: Store user timezone, convert to UTC for execution, display in user timezone
|
||||
- **Sequential execution**: Guaranteed one-at-a-time processing
|
||||
|
||||
### Smart Duplicate Prevention
|
||||
When a scheduled task runs:
|
||||
1. BDFR checks existing hashes (if no_dupes enabled)
|
||||
2. Only downloads new content from last 24 hours
|
||||
3. Skips content already downloaded in previous runs
|
||||
|
||||
### Execution Tracking
|
||||
- Every run creates a history record
|
||||
- Tracks success/failure status
|
||||
- Records items found vs. items downloaded
|
||||
- Links to the actual download progress for real-time monitoring
|
||||
- Shows queue position and current task status
|
||||
|
||||
### User Experience
|
||||
- Simple checkbox to schedule any download
|
||||
- Visual indication of enabled/disabled tasks
|
||||
- Queue status badge shows pending tasks
|
||||
- Currently running task highlighted
|
||||
- Next run time displayed in user's local timezone
|
||||
- One-click to add task to queue immediately
|
||||
- Easy enable/disable without deleting task
|
||||
- Queue position shown when manually running tasks
|
||||
|
||||
## Sequential Execution Examples
|
||||
|
||||
### Scenario 1: Multiple Scheduled Tasks
|
||||
```
|
||||
02:00 AM - Task A triggers, added to queue
|
||||
02:00 AM - Task B triggers, added to queue
|
||||
02:00 AM - Task C triggers, added to queue
|
||||
|
||||
Execution Order:
|
||||
1. Task A starts, downloads 100 posts (takes 15 minutes)
|
||||
2. Task B starts at 02:15 AM, downloads 50 posts (takes 8 minutes)
|
||||
3. Task C starts at 02:23 AM, downloads 75 posts (takes 12 minutes)
|
||||
4. All complete by 02:35 AM
|
||||
```
|
||||
|
||||
### Scenario 2: Manual "Run Now" During Scheduled Task
|
||||
```
|
||||
02:00 AM - Task A starts (scheduled, downloading...)
|
||||
02:10 AM - User clicks "Run Now" on Task B
|
||||
02:10 AM - Task B added to queue with priority
|
||||
|
||||
Execution Order:
|
||||
1. Task A continues running (started first)
|
||||
2. Task B waits in queue
|
||||
3. Task A completes at 02:15 AM
|
||||
4. Task B starts immediately at 02:15 AM (priority over other scheduled tasks)
|
||||
```
|
||||
|
||||
### Scenario 3: Container Restart During Execution
|
||||
```
|
||||
02:00 AM - Task A starts downloading
|
||||
02:10 AM - Container restarts (Docker update, etc.)
|
||||
02:10 AM - Container comes back up
|
||||
02:10 AM - Task A marked as "failed" with "interrupted" message
|
||||
02:10 AM - Scheduled tasks reload, Task A will retry at next scheduled time (tomorrow 02:00 AM)
|
||||
02:10 AM - Other pending tasks start processing from queue
|
||||
```
|
||||
|
||||
## Future Enhancements (Not in Initial Implementation)
|
||||
|
||||
1. **Parallel Execution**: Optional setting to allow N tasks at once (requires more resources)
|
||||
2. **Smart Scheduling**: Stagger start times automatically if many tasks at same time
|
||||
3. **Queue Priorities**: User-configurable priority levels for tasks
|
||||
4. **Retry Logic**: Auto-retry failed tasks with exponential backoff
|
||||
5. **Additional Frequencies**: Weekly, custom intervals
|
||||
6. **Notification System**: Email/webhook notifications on completion/failure
|
||||
7. **Advanced Filters**: Score thresholds, content type filters
|
||||
8. **Task Templates**: Save and reuse task configurations
|
||||
9. **Execution History Page**: Dedicated page for detailed history with charts
|
||||
10. **Bulk Operations**: Enable/disable/delete multiple tasks at once
|
||||
11. **Export/Import**: Backup and restore scheduled tasks
|
||||
|
||||
## Conclusion
|
||||
|
||||
This implementation provides a robust scheduled downloads system with **guaranteed sequential execution**, designed specifically for Docker deployment in resource-constrained environments. The queue-based approach ensures:
|
||||
|
||||
- ✅ Only one download at a time (no resource contention)
|
||||
- ✅ Fair task ordering (FIFO with priority support)
|
||||
- ✅ Data persistence across container restarts
|
||||
- ✅ Reliable scheduling with APScheduler
|
||||
- ✅ Proper timezone handling (user TZ -> container UTC)
|
||||
- ✅ Simple, clean UI
|
||||
- ✅ Integration with existing BDFR API
|
||||
- ✅ Smart defaults (daily schedule, time_filter="day", no_dupes=true)
|
||||
- ✅ Easy management (enable/disable/delete/run now)
|
||||
- ✅ Container-aware logging and health checks
|
||||
- ✅ Transparent queue status for users
|
||||
|
||||
The sequential execution model is perfect for:
|
||||
- Single-user home servers
|
||||
- Docker containers with limited CPU/memory
|
||||
- Preventing Reddit API rate limits
|
||||
- Ensuring reliable, predictable downloads
|
||||
- Avoiding file system contention
|
||||
|
||||
The system is production-ready for Docker deployment and extensible for future enhancements like parallel execution if needed.
|
||||
Reference in New Issue
Block a user