diff --git a/.env.example b/.env.example index 2ae88cf..e338cdf 100644 --- a/.env.example +++ b/.env.example @@ -1,6 +1,18 @@ # BDFR Web Interface - Docker Environment Configuration # Copy this file to .env and update the values as needed +# ============================================================================ +# DOCKER USER CONFIGURATION (Important for file permissions) +# ============================================================================ +# Set these to match your host user's UID and GID to avoid permission issues +# Find your UID/GID by running: id +# On Linux/Mac: id -u (for UID) and id -g (for GID) +# On Windows with WSL: wsl id -u +# Default: 1000:1000 (common default user on Linux systems) + +PUID=1000 +PGID=1000 + # ============================================================================ # REDDIT OAUTH CONFIGURATION (REQUIRED for authenticated features) # ============================================================================ @@ -41,7 +53,7 @@ DEBUG=false # Directory where downloads will be stored (inside container) # This is mounted from ./downloads on the host -BDFR_DOWNLOAD_DIR=/downloads +BDFR_DOWNLOAD_DIR=/app/downloads # Directory for application data and databases (inside container) # This is mounted from ./data on the host diff --git a/DOCKER.md b/DOCKER.md index d3c5d1d..80f10a6 100644 --- a/DOCKER.md +++ b/DOCKER.md @@ -44,6 +44,12 @@ This guide covers running the BDFR Web Interface with full BDFR backend support Create a `.env` file in the project root with the following variables: ```env +# Docker User Configuration (Important for file permissions) +# Set these to match your host user's UID/GID to avoid permission issues +# Find your UID/GID by running: id (on Linux/Mac) or wsl id (on Windows WSL) +PUID=1000 +PGID=1000 + # Reddit OAuth Configuration (Required for authenticated features) BDFR_CLIENT_ID=your_client_id_here BDFR_CLIENT_SECRET=your_client_secret_here @@ -74,7 +80,7 @@ The Docker setup uses two volume mounts: | Host Path | Container Path | Purpose | |-----------|----------------|---------| -| `./downloads` | `/downloads` | All Reddit downloads are stored here | +| `./downloads` | `/app/downloads` | All Reddit downloads are stored here | | `./data` | `/app/data` | SQLite databases, scheduled tasks, and BDFR configuration | **Data Directory Structure:** @@ -139,22 +145,22 @@ docker exec -it bdfr-web-interface bash **Run BDFR commands:** ```bash # Download from a subreddit -bdfr download /downloads --subreddit Python -L 10 +bdfr download /app/downloads --subreddit Python -L 10 # Download from a user -bdfr download /downloads --user reddituser --submitted -L 100 +bdfr download /app/downloads --user reddituser --submitted -L 100 # Archive posts -bdfr archive /downloads --subreddit all -L 500 +bdfr archive /app/downloads --subreddit all -L 500 # Clone (download + archive) -bdfr clone /downloads --subreddit EarthPorn -L 50 +bdfr clone /app/downloads --subreddit EarthPorn -L 50 ``` **One-line BDFR commands:** ```bash # Download without entering the container -docker exec bdfr-web-interface bdfr download /downloads --subreddit Python -L 10 +docker exec bdfr-web-interface bdfr download /app/downloads --subreddit Python -L 10 # View BDFR version docker exec bdfr-web-interface bdfr --version @@ -185,7 +191,7 @@ Mount a different host directory for downloads: ```yaml volumes: - - /path/to/your/downloads:/downloads + - /path/to/your/downloads:/app/downloads - ./data:/app/data ``` @@ -251,7 +257,29 @@ docker-compose up -d ### Permission Issues -If you encounter permission errors with volumes: +If you encounter permission errors with volumes (e.g., "Permission denied" when creating directories): + +**Solution 1: Configure PUID/PGID (Recommended)** + +Set `PUID` and `PGID` in your `.env` file to match your host user: + +```bash +# Find your UID and GID +id + +# Example output: uid=1001(username) gid=1001(groupname) +# Add to .env file: +PUID=1001 +PGID=1001 +``` + +Then restart the container: +```bash +docker-compose down +docker-compose up -d +``` + +**Solution 2: Fix Host Directory Permissions** **Linux/Mac:** ```bash @@ -259,9 +287,18 @@ sudo chown -R $USER:$USER downloads data config chmod -R 755 downloads data config ``` +**TrueNAS/NAS Systems:** +When using network shares or NAS storage: +1. Find your NAS user's UID/GID (usually in user management settings) +2. Set `PUID` and `PGID` in `.env` to match your NAS user +3. Ensure the NAS user has read/write permissions on mounted shares + **Windows:** Ensure Docker Desktop has access to the drive where the project is located (Settings → Resources → File Sharing). +**Why this matters:** +The container runs as a specific user (default UID 1000). If your host directories are owned by a different user, the container won't be able to write to them. Setting `PUID` and `PGID` tells Docker to run the container as your host user, matching permissions. + ### Downloads Not Appearing 1. Check volume mounts are correct: @@ -271,7 +308,7 @@ Ensure Docker Desktop has access to the drive where the project is located (Sett 2. Verify download directory inside container: ```bash - docker exec bdfr-web-interface ls -la /downloads + docker exec bdfr-web-interface ls -la /app/downloads ``` 3. Check container logs for errors: diff --git a/DOCKER_PUBLISH.md b/DOCKER_PUBLISH.md new file mode 100644 index 0000000..59eac8f --- /dev/null +++ b/DOCKER_PUBLISH.md @@ -0,0 +1,262 @@ +# Publishing BDFR Web Interface to Docker Hub + +This guide covers how to build and push the Docker image to Docker Hub. + +## Prerequisites + +1. **Docker Hub Account**: Create one at https://hub.docker.com if you don't have one +2. **Docker installed**: Ensure Docker is running on your machine + +## Step 1: Login to Docker Hub + +Open your terminal and login: + +```bash +docker login +``` + +Enter your Docker Hub username and password when prompted. + +## Step 2: Build the Image with Proper Tagging + +Build the image with your Docker Hub username and desired repository name: + +```bash +# Replace 'yourusername' with your actual Docker Hub username +# Replace 'bdfr-web' with your desired repository name (or keep it) + +docker build -t yourusername/bdfr-web:latest . + +# You can also add version tags +docker build -t yourusername/bdfr-web:latest -t yourusername/bdfr-web:1.0.0 . +``` + +**Example:** +```bash +docker build -t danielsmith/bdfr-web:latest -t danielsmith/bdfr-web:1.0.0 . +``` + +## Step 3: Test the Image Locally (Optional but Recommended) + +Before pushing, verify the image works: + +```bash +docker run -d -p 8000:8000 \ + -v $(pwd)/downloads:/downloads \ + -v $(pwd)/data:/app/data \ + --name bdfr-test \ + yourusername/bdfr-web:latest +``` + +Visit `http://localhost:8000` to verify it works, then clean up: + +```bash +docker stop bdfr-test +docker rm bdfr-test +``` + +## Step 4: Push to Docker Hub + +Push the image to Docker Hub: + +```bash +# Push latest tag +docker push yourusername/bdfr-web:latest + +# If you created a version tag, push that too +docker push yourusername/bdfr-web:1.0.0 +``` + +## Step 5: Verify the Push + +1. Go to https://hub.docker.com +2. Navigate to your repositories +3. You should see `bdfr-web` listed +4. Click on it to see tags and details + +## Using the Published Image + +Others can now pull and use your image: + +```bash +docker pull yourusername/bdfr-web:latest +docker run -d -p 8000:8000 -v ./downloads:/downloads yourusername/bdfr-web:latest +``` + +Or using docker-compose, update [`docker-compose.yml`](docker-compose.yml): + +```yaml +services: + bdfr-web: + image: yourusername/bdfr-web:latest # Replace 'build:' section with this + # ... rest of configuration +``` + +## Automated Build with Docker Hub + +You can set up automated builds that trigger when you push to GitHub: + +1. Go to https://hub.docker.com +2. Navigate to your repository +3. Click "Builds" tab +4. Click "Configure Automated Builds" +5. Connect your GitHub account +6. Select your repository +7. Configure build rules (e.g., build on push to main branch) + +## Multi-Platform Builds (Optional) + +To build for multiple architectures (amd64, arm64, etc.): + +### Setup buildx (one-time setup) + +```bash +# Create a new builder +docker buildx create --name multiplatform --use + +# Bootstrap the builder +docker buildx inspect --bootstrap +``` + +### Build and push multi-platform image + +```bash +docker buildx build \ + --platform linux/amd64,linux/arm64 \ + -t yourusername/bdfr-web:latest \ + -t yourusername/bdfr-web:1.0.0 \ + --push \ + . +``` + +This creates images that work on both x86_64 (Intel/AMD) and ARM64 (Apple Silicon, Raspberry Pi, etc.). + +## Updating Your Published Image + +When you make changes: + +1. **Update version**: Increment version in tags (e.g., 1.0.0 → 1.0.1) +2. **Rebuild**: + ```bash + docker build -t yourusername/bdfr-web:latest -t yourusername/bdfr-web:1.0.1 . + ``` +3. **Push**: + ```bash + docker push yourusername/bdfr-web:latest + docker push yourusername/bdfr-web:1.0.1 + ``` + +## Best Practices + +### Tagging Strategy + +- `latest`: Always points to the most recent stable build +- `1.0.0`, `1.0.1`: Specific version tags for reproducibility +- `1.0`, `1`: Major/minor version tags +- `dev`: Development/unstable builds + +Example tagging: +```bash +docker build -t yourusername/bdfr-web:latest \ + -t yourusername/bdfr-web:1.0.1 \ + -t yourusername/bdfr-web:1.0 \ + -t yourusername/bdfr-web:1 . +``` + +### Repository Description + +Add a good description to your Docker Hub repository: + +1. Go to your repository on Docker Hub +2. Click "Description" tab +3. Add a README with: + - What the image does + - How to use it + - Environment variables + - Volume mounts + - Example docker-compose.yml + +### Size Optimization + +Current image uses multi-stage builds and is already optimized. To check size: + +```bash +docker images yourusername/bdfr-web +``` + +## Complete Script + +Here's a complete script to build and push (save as `publish-docker.sh`): + +```bash +#!/bin/bash +set -e + +# Configuration +DOCKER_USERNAME="yourusername" +IMAGE_NAME="bdfr-web" +VERSION="1.0.0" + +# Build +echo "Building Docker image..." +docker build -t ${DOCKER_USERNAME}/${IMAGE_NAME}:latest \ + -t ${DOCKER_USERNAME}/${IMAGE_NAME}:${VERSION} . + +# Test +echo "Testing image..." +docker run --rm ${DOCKER_USERNAME}/${IMAGE_NAME}:latest bdfr --version + +# Login (if not already logged in) +echo "Logging in to Docker Hub..." +docker login + +# Push +echo "Pushing to Docker Hub..." +docker push ${DOCKER_USERNAME}/${IMAGE_NAME}:latest +docker push ${DOCKER_USERNAME}/${IMAGE_NAME}:${VERSION} + +echo "Done! Image published to:" +echo " docker pull ${DOCKER_USERNAME}/${IMAGE_NAME}:latest" +echo " docker pull ${DOCKER_USERNAME}/${IMAGE_NAME}:${VERSION}" +``` + +Make it executable: +```bash +chmod +x publish-docker.sh +./publish-docker.sh +``` + +## Troubleshooting + +### "denied: requested access to the resource is denied" + +- Make sure you're logged in: `docker login` +- Verify your username in the image tag matches your Docker Hub username +- Check that the repository exists or that you have permissions + +### "image not found" after push + +- Wait a few minutes - Docker Hub indexing can be delayed +- Refresh the Docker Hub web page +- Try pulling: `docker pull yourusername/bdfr-web:latest` + +### Large image size + +Current image should be ~500-800MB. If larger: +- Check that `.dockerignore` is working +- Verify multi-stage build is being used +- Clean up any unnecessary files in the image + +## Private Repositories + +To make your repository private: + +1. Go to Docker Hub repository settings +2. Change visibility to "Private" +3. Users will need to login to pull: `docker login` before `docker pull` + +## Additional Resources + +- [Docker Hub Documentation](https://docs.docker.com/docker-hub/) +- [Dockerfile Best Practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) +- [Docker Buildx Documentation](https://docs.docker.com/buildx/working-with-buildx/) \ No newline at end of file diff --git a/Dockerfile b/Dockerfile index 01c9e1f..42946ab 100644 --- a/Dockerfile +++ b/Dockerfile @@ -44,6 +44,16 @@ RUN apt-get update && apt-get install -y \ COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages COPY --from=builder /usr/local/bin /usr/local/bin +# Ensure Python packages are readable by all users +# This is critical when running container with custom user via PUID/PGID +RUN chmod -R a+rX /usr/local/lib/python3.11/site-packages && \ + chmod -R a+rx /usr/local/bin + +# Copy default_config.cfg to a world-readable location +# This avoids issues with importlib.resources when running as non-root user +RUN cp /usr/local/lib/python3.11/site-packages/bdfr/default_config.cfg /tmp/bdfr_default_config.cfg && \ + chmod 644 /tmp/bdfr_default_config.cfg + # Copy web interface files COPY web_interface/app/ ./app/ COPY web_interface/templates/ ./templates/ @@ -55,19 +65,16 @@ COPY docker-entrypoint.sh /usr/local/bin/ RUN chmod +x /usr/local/bin/docker-entrypoint.sh # Create necessary directories with proper permissions -RUN mkdir -p /downloads /app/data /app/logs && \ - chmod 755 /downloads /app/data /app/logs +# Use 777 to allow any user (set by TrueNAS or docker-compose) to write +RUN mkdir -p /app/downloads /app/data /app/logs && \ + chmod -R 777 /app -# Create a non-root user for running the application -RUN useradd --create-home --shell /bin/bash bdfr && \ - chown -R bdfr:bdfr /app /downloads - -# Switch to non-root user -USER bdfr +# Note: No USER directive - container will run as the user specified by +# the orchestrator (TrueNAS, docker-compose, etc.) or root by default # Set environment variables ENV PYTHONUNBUFFERED=1 -ENV BDFR_DOWNLOAD_DIR=/downloads +ENV BDFR_DOWNLOAD_DIR=/app/downloads ENV BDFR_DATA_DIR=/app/data ENV BDFR_CONFIG_DIR=/app/data/bdfr-config ENV APPDATA=/app/data/bdfr-config diff --git a/bdfr/api.py b/bdfr/api.py index bb7dd4b..36e4452 100644 --- a/bdfr/api.py +++ b/bdfr/api.py @@ -112,7 +112,7 @@ class LoggingCallback(ProgressCallback): async def on_progress(self, event: ProgressEvent): """Log progress events""" if event.progress is not None: - self.logger.info(f"[{event.download_id}] {event.message} ({event.progress}%)") + self.logger.info(f"[{event.download_id}] {event.message} ({int(round(event.progress))}%)") else: self.logger.info(f"[{event.download_id}] {event.message}") diff --git a/bdfr/connector.py b/bdfr/connector.py index 0a1605f..f9f9dcd 100644 --- a/bdfr/connector.py +++ b/bdfr/connector.py @@ -195,16 +195,38 @@ class RedditConnector(metaclass=ABCMeta): Path(self.config_directory, "config.cfg"), Path(self.config_directory, "default_config.cfg"), ] + logger.debug(f"Config directory is: {self.config_directory}") + logger.debug(f"Checking possible config paths: {[str(p) for p in possible_paths]}") self.config_location = None for path in possible_paths: - if path.resolve().expanduser().exists(): + resolved_path = path.resolve().expanduser() + logger.debug(f"Checking if {resolved_path} exists: {resolved_path.exists()}") + if resolved_path.exists(): self.config_location = path logger.debug(f"Loading configuration from {path}") break if not self.config_location: - with importlib.resources.path("bdfr", "default_config.cfg") as path: - self.config_location = path - shutil.copy(self.config_location, Path(self.config_directory, "default_config.cfg")) + # Try to use a fallback location that avoids importlib.resources context manager issues + # when running as non-root user in Docker + fallback_config = Path("/tmp/bdfr_default_config.cfg") + if fallback_config.exists(): + logger.debug("Using fallback config from /tmp/bdfr_default_config.cfg") + shutil.copy(fallback_config, Path(self.config_directory, "default_config.cfg")) + self.config_location = Path(self.config_directory, "default_config.cfg") + else: + # Fall back to importlib.resources if no fallback is available + try: + with importlib.resources.path("bdfr", "default_config.cfg") as path: + self.config_location = path + shutil.copy(self.config_location, Path(self.config_directory, "default_config.cfg")) + except (PermissionError, OSError) as e: + logger.error(f"Failed to access default config via importlib.resources: {e}") + # Last resort: try to read from package directory directly + package_config = Path("/usr/local/lib/python3.11/site-packages/bdfr/default_config.cfg") + if package_config.exists(): + logger.debug("Using package config directly") + shutil.copy(package_config, Path(self.config_directory, "default_config.cfg")) + self.config_location = Path(self.config_directory, "default_config.cfg") if not self.config_location: raise errors.BulkDownloaderException("Could not find a configuration file to load") self.cfg_parser.read(self.config_location) diff --git a/bdfr/downloader.py b/bdfr/downloader.py index fe12bc2..cc9fcdc 100644 --- a/bdfr/downloader.py +++ b/bdfr/downloader.py @@ -153,7 +153,7 @@ class RedditDownloader(RedditConnector): if destination.exists(): # Check if we already have this file's hash if destination in self.master_hash_list.values(): - logger.debug(f"File {destination} from submission {submission.id} already exists, continuing") + logger.info(f"File {destination.name} from submission {submission.id} already exists") continue else: # File exists but not in our hash list - calculate its hash @@ -166,6 +166,7 @@ class RedditDownloader(RedditConnector): self.url_list[res.url] = existing_file_hash logger.debug(f"Added hash for existing file: {existing_file_hash}") + logger.info(f"File {destination.name} from submission {submission.id} already exists") files_processed += 1 if self.args.no_dupes: self._save_hash_list() @@ -185,7 +186,6 @@ class RedditDownloader(RedditConnector): ) return resource_hash = res.hash.hexdigest() - destination.parent.mkdir(parents=True, exist_ok=True) # Simple-check: URL-based duplicate detection (fast path) if self.args.simple_check and hasattr(res, 'url') and res.url in self.url_list: @@ -213,6 +213,9 @@ class RedditDownloader(RedditConnector): if self.args.no_dupes: self._save_hash_list() return + + # Only create folder if we're actually going to write the file (not a duplicate) + destination.parent.mkdir(parents=True, exist_ok=True) try: with destination.open("wb") as file: file.write(res.content) diff --git a/bdfr/examples/api_usage.py b/bdfr/examples/api_usage.py index 9fb47b2..42acd40 100644 --- a/bdfr/examples/api_usage.py +++ b/bdfr/examples/api_usage.py @@ -46,7 +46,7 @@ class WebSocketCallback(ProgressCallback): """Send progress update to WebSocket""" print(f"📊 [{self.websocket_id}] Progress: {event.message}") if event.progress is not None: - print(f" Progress: {event.progress:.1f}%") + print(f" Progress: {int(round(event.progress))}%") if event.data: print(f" Data: {event.data}") @@ -109,7 +109,7 @@ async def example_basic_usage(): print("❌ Download not found!") break - print(f"📊 Status: {status['status']} | Progress: {status['progress']:.1f}%") + print(f"📊 Status: {status['status']} | Progress: {int(round(status['progress']))}%") if status['status'] in ['completed', 'failed', 'cancelled']: print(f"🏁 Download finished with status: {status['status']}") @@ -162,7 +162,7 @@ async def example_advanced_usage(): download_ids.remove(download_id) continue - print(f"📊 {download_id}: {status['status']} ({status['progress']:.1f}%)") + print(f"📊 {download_id}: {status['status']} ({int(round(status['progress']))}%)") if status['status'] in ['completed', 'failed', 'cancelled']: print(f"🏁 Download {download_id} finished") @@ -203,7 +203,7 @@ async def example_user_download(): print("❌ Download not found") break - print(f"📊 Status: {status['status']} | Progress: {status['progress']:.1f}%") + print(f"📊 Status: {status['status']} | Progress: {int(round(status['progress']))}%") if status['status'] in ['completed', 'failed']: break @@ -236,7 +236,7 @@ async def example_archive_operation(): print("❌ Archive not found") break - print(f"📊 Archive status: {status['status']} | Progress: {status['progress']:.1f}%") + print(f"📊 Archive status: {status['status']} | Progress: {int(round(status['progress']))}%") if status['status'] in ['completed', 'failed']: print(f"🏁 Archive finished with status: {status['status']}") @@ -325,7 +325,7 @@ async def example_web_integration(): user_downloads = await app.get_user_downloads(user_id) print(f"User has {len(user_downloads)} active downloads:") for download in user_downloads: - print(f" - {download['id']}: {download['status']} ({download['progress']:.1f}%)") + print(f" - {download['id']}: {download['status']} ({int(round(download['progress']))}%)") # Cancel one download if user_downloads: diff --git a/bdfr/site_downloaders/base_downloader.py b/bdfr/site_downloaders/base_downloader.py index e4ac111..505691a 100644 --- a/bdfr/site_downloaders/base_downloader.py +++ b/bdfr/site_downloaders/base_downloader.py @@ -27,11 +27,23 @@ class BaseDownloader(ABC): @staticmethod def retrieve_url(url: str, cookies: dict = None, headers: dict = None) -> requests.Response: - try: - res = requests.get(url, cookies=cookies, headers=headers) - except requests.exceptions.RequestException as e: - logger.exception(e) - raise SiteDownloaderError(f"Failed to get page {url}") - if res.status_code != 200: - raise ResourceNotFound(f"Server responded with {res.status_code} to {url}") - return res + max_retries = 3 + for attempt in range(1, max_retries + 1): + try: + res = requests.get(url, cookies=cookies, headers=headers, timeout=10) + if res.status_code != 200: + logger.error(f"Attempt {attempt}: Server responded with {res.status_code} to {url}") + if attempt == max_retries: + raise ResourceNotFound(f"Server responded with {res.status_code} to {url}") + else: + return res + except requests.exceptions.SSLError as ssl_err: + logger.error(f"Attempt {attempt}: SSL error for {url}: {ssl_err}") + if attempt == max_retries: + raise SiteDownloaderError(f"SSL error after {max_retries} attempts for {url}: {ssl_err}") + except requests.exceptions.RequestException as e: + logger.error(f"Attempt {attempt}: Request error for {url}: {e}") + if attempt == max_retries: + raise SiteDownloaderError(f"Failed to get page {url} after {max_retries} attempts: {e}") + # Should not reach here + raise SiteDownloaderError(f"Failed to get page {url} after {max_retries} attempts") diff --git a/docker-compose.yml b/docker-compose.yml index 91c0bbf..54009ed 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -6,11 +6,15 @@ services: context: . dockerfile: Dockerfile container_name: bdfr-web-interface + # Note: user directive removed - container runs as user specified by orchestrator + # For TrueNAS: Set 'Custom User' in container settings + # For docker-compose: Uncomment and set user: "UID:GID" if needed + # user: "${PUID:-1000}:${PGID:-1000}" ports: - "8000:8000" volumes: # Downloads directory - all Reddit content goes here - - ./downloads:/downloads + - ./downloads:/app/downloads # Database persistence - SQLite databases, scheduled tasks, and BDFR config - ./data:/app/data environment: @@ -25,8 +29,8 @@ services: - PORT=${PORT:-8000} - DEBUG=${DEBUG:-false} - # Download Configuration - - BDFR_DOWNLOAD_DIR=/downloads + # Download Configuration (using /app/downloads for simplicity) + - BDFR_DOWNLOAD_DIR=/app/downloads - BDFR_DATA_DIR=/app/data # BDFR Configuration - stored in mounted data directory @@ -54,4 +58,4 @@ networks: # Uncomment to use named volumes instead of bind mounts # volumes: # downloads: -# data: \ No newline at end of file +# data: diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh index fd77e5c..cacdc4c 100644 --- a/docker-entrypoint.sh +++ b/docker-entrypoint.sh @@ -1,10 +1,56 @@ #!/bin/bash set -e +echo "=== BDFR Docker Container Startup Diagnostics ===" +echo "Current user: $(id)" +echo "Current user name: $(whoami)" +echo "" + +# Check Python package directory permissions +echo "Checking Python package directory permissions:" +if [ -d "/usr/local/lib/python3.11/site-packages/bdfr" ]; then + ls -la /usr/local/lib/python3.11/site-packages/bdfr/ | head -20 + echo "" + echo "Checking default_config.cfg specifically:" + ls -l /usr/local/lib/python3.11/site-packages/bdfr/default_config.cfg || echo "File not found!" + echo "Can read default_config.cfg: $(test -r /usr/local/lib/python3.11/site-packages/bdfr/default_config.cfg && echo 'YES' || echo 'NO')" +else + echo "ERROR: BDFR package directory not found!" +fi +echo "" + +# Check temp config +echo "Checking temp config:" +ls -l /tmp/bdfr_default_config.cfg 2>/dev/null || echo "Temp config not found!" +echo "Can read temp config: $(test -r /tmp/bdfr_default_config.cfg && echo 'YES' || echo 'NO')" +echo "" + # Create BDFR config directory if it doesn't exist # This needs to be done at runtime because /app/data is a volume mount +echo "Creating BDFR config directory..." mkdir -p /app/data/bdfr-config chmod 755 /app/data/bdfr-config +echo "Checking /app/data permissions:" +ls -la /app/data/ 2>/dev/null || echo "Cannot list /app/data" +echo "" + +# Copy default config if it doesn't exist +# Use the temp copy created during build to avoid permission issues with importlib.resources +if [ ! -f /app/data/bdfr-config/default_config.cfg ]; then + echo "Copying default config to /app/data/bdfr-config/..." + cp /tmp/bdfr_default_config.cfg /app/data/bdfr-config/default_config.cfg + chmod 644 /app/data/bdfr-config/default_config.cfg + echo "Config copied successfully" +else + echo "Config already exists in /app/data/bdfr-config/" +fi + +echo "Final config check:" +ls -l /app/data/bdfr-config/default_config.cfg 2>/dev/null || echo "Config not found in data dir!" +echo "" +echo "=== End of diagnostics ===" +echo "" + # Execute the main command exec "$@" \ No newline at end of file diff --git a/docker_overview.md b/docker_overview.md new file mode 100644 index 0000000..8d04022 --- /dev/null +++ b/docker_overview.md @@ -0,0 +1,157 @@ +# BDFR Web Interface + +A complete web-based interface for the Bulk Downloader for Reddit (BDFR) with real-time progress tracking and scheduled downloads. + +This tool is a frontend for the BDFR tool found here: https://github.com/Serene-Arc/bulk-downloader-for-reddit +Although I've made a few changes to allow for better repeat download persistence to avoid redownloading previously downloaded content. + +## Features + +### 🎯 **Three Download Modes** +- **Download**: Media files only (images, videos, gifs) from Reddit posts +- **Archive**: Post metadata (title, author, comments) in JSON/XML format +- **Clone**: Complete backup combining media files and metadata + +### 📍 **Multiple Source Types** +- **Subreddit Downloads**: Download content from any subreddit with customizable filters +- **User Downloads**: Download posts from specific users (submitted, upvoted, or saved content) + +### ⚡ **Real-Time Progress Tracking** +- WebSocket-based live progress updates +- Visual progress bars and status indicators +- Download queue management with cancel functionality +- Phase tracking (downloading, processing, completing) + +### 🔐 **Reddit OAuth2 Authentication** +- Secure authentication with Reddit API +- Access to private content (saved/upvoted posts) +- Session management and token refresh + +### ⚙️ **Advanced Options** +- **Duplicate Detection**: Avoid re-downloading existing files +- **Hard Link Creation**: Save disk space with file links +- **Scheduled Downloads**: Set up daily automatic downloads +- **Content Filtering**: Filter by score, time range, and sort order +- **Simple Check**: Fast URL-based duplicate detection + +### 📊 **Management Dashboard** +- Active download monitoring +- Scheduled task management +- System status indicators +- Download history and statistics + +## Quick Start with Docker Compose + +```yaml +version: '3.8' +services: + bdfr-web: + image: moderatewinguy/bdfr-web:latest + # Uncomment and set user if needed (optional for most setups) + # user: "1000:1000" # Set to your user's UID:GID (run 'id' to find yours) + ports: + - "8000:8000" + volumes: + - ./downloads:/app/downloads # Reddit content storage + - ./data:/app/data # Database persistence + environment: + - BDFR_CLIENT_ID=your_reddit_client_id + - BDFR_CLIENT_SECRET=your_reddit_client_secret + - BDFR_REDIRECT_URI=http://localhost:8000/auth/callback +``` + +Run with: +```bash +docker-compose up -d +``` + +Access at: http://localhost:8000 + +## Configuration + +### User Permissions + +The container is designed to run as any user, allowing it to match your system's permissions. + +**For TrueNAS/NAS Systems:** +- Use the "Custom User" setting in your container configuration +- Set it to the UID of the user that owns your mounted shares +- The container will automatically run as that user + +**For Docker Compose:** +If needed, uncomment the `user:` line in docker-compose.yml: +```yaml +user: "1000:1000" # Set to your user's UID:GID +``` + +Find your UID/GID with: `id` (Linux/Mac) or `wsl id` (Windows WSL) + +**The container will work with any UID/GID** - just ensure the mounted volumes are owned by the same user. + +### Environment Variables + +| Variable | Description | Required | +|----------|-------------|----------| +| `BDFR_CLIENT_ID` | Reddit OAuth2 client ID | Yes | +| `BDFR_CLIENT_SECRET` | Reddit OAuth2 client secret | Yes | +| `BDFR_REDIRECT_URI` | OAuth2 redirect URI | Yes | +| `HOST` | Server host (default: 0.0.0.0) | No | +| `PORT` | Server port (default: 8000) | No | +| `DEBUG` | Enable debug mode | No | + +## Volume Mounts + +- **`/app/downloads`**: All Reddit content is stored here +- **`/app/data`**: SQLite databases for scheduled tasks and application data + +## API Endpoints + +- `GET /` - Web interface +- `POST /api/download/subreddit` - Start subreddit download +- `POST /api/download/user` - Start user download +- `GET /api/downloads` - List active downloads +- `DELETE /api/downloads/{id}` - Cancel download +- `GET /health` - Health check +- `WS /ws/progress` - Real-time progress updates + +## Getting Reddit OAuth Credentials + +1. Go to https://www.reddit.com/prefs/apps +2. Click "Create App" or "Create Another App" +3. Select "web app" and fill in the details +4. Set redirect URI to: `http://localhost:8000/auth/callback` +5. Copy the client ID and client secret to your `.env` file + +## Use Cases + +- **Content Archiving**: Save Reddit posts and media for offline access +- **Research Data Collection**: Gather post metadata for analysis +- **Media Backup**: Download images/videos from favorite subreddits +- **Automated Downloads**: Schedule daily content collection +- **Personal Archive**: Backup your own posts and saved content + +## Supported Content Sources + +- Direct image/video links +- Imgur albums and images +- Reddit native media (images, videos, gifs) +- YouTube videos +- Gfycat animations +- And many more via yt-dlp fallback + +## Technical Details + +- **Base Image**: Python 3.11-slim +- **Web Framework**: FastAPI with WebSocket support +- **Frontend**: Vanilla JavaScript with modern CSS +- **Database**: SQLite for task scheduling +- **Authentication**: OAuth2 with Reddit API +- **File Processing**: Multi-threaded downloads with progress tracking + + +--- + +**Note**: This image includes the complete BDFR tool, so you can also run CLI commands directly: +```bash +docker exec bdfr-web bdfr download /app/downloads --subreddit python -L 50 +``` \ No newline at end of file diff --git a/web_interface/static/js/app.js b/web_interface/static/js/app.js index d910a6a..dc87426 100644 --- a/web_interface/static/js/app.js +++ b/web_interface/static/js/app.js @@ -430,7 +430,7 @@ class BDFRApp { card.className = 'progress-card'; card.id = `progress-${downloadId}`; - const typeLabel = data.type === 'subreddit' ? 'Subreddit' : 'User'; + const typeLabel = data.subreddit ? 'Subreddit' : 'User'; const targetName = data.subreddit || data.username || 'Unknown'; const itemsProcessedInit = Number.isFinite(data.items_processed) ? data.items_processed : (data.data && Number.isFinite(data.data.items_processed) ? data.data.items_processed : 0); const limitInit = Number.isFinite(data.limit) ? data.limit : (data.data && Number.isFinite(data.data.limit) ? data.data.limit : undefined); @@ -456,7 +456,7 @@ class BDFRApp {