Update CLAUDE.md to enhance project documentation and clarify soundboard system details

2025-07-19 09:40:05 +02:00
parent 0c15c3f1a9
commit 8f47f25b4b
1 changed files with 205 additions and 5 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -2,9 +2,22 @@

 This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

+## Project Overview
+
+**Soundboard V2** - A full-stack web application for managing and playing sound files with streaming capabilities.
+
+### Core Features
+- **Sound Management**: Upload, organize, and play sound files
+- **Stream Processing**: Download and process audio from YouTube, SoundCloud, and other platforms using yt-dlp
+- **Music Player**: Full-featured VLC-based music player with playlist support
+- **Real-time Updates**: WebSocket support for live player state synchronization
+- **Multi-Provider OAuth**: Google and GitHub authentication with JWT tokens
+- **Credits System**: Usage tracking with tiered subscription plans
+- **Sound Normalization**: Automatic audio processing and normalization
+
 ## Project Structure

-This is a full-stack application with separate backend (Flask) and frontend (Next.js) components:
+This is a full-stack application with separate backend (Flask) and frontend (React) components:

 - **Backend** (`backend/`): Flask application with clean architecture
  - Entry point: `main.py`
@@ -13,18 +26,20 @@ This is a full-stack application with separate backend (Flask) and frontend (Nex
  - Uses modern Python tooling (Black, Ruff, pytest)
  - CORS enabled for frontend integration
  - Python 3.12+ required
+  - Real-time features via Flask-SocketIO

 - **Frontend** (`frontend/`): React application with Vite
  - Uses React 19, TypeScript, React Router v7, and Tailwind CSS v4
  - Shadcn/ui components with consistent sidebar layout for authenticated users
  - Built with Vite for fast development and production builds
+  - Real-time updates via Socket.IO client

 ## Development Commands

 ### Backend (Flask)
 ```bash
 cd backend
-uv run python main.py         # Run the Flask development server (localhost:5000)
+uv run python main.py         # Run the Flask development server with SocketIO (localhost:5000)
 uv run pytest                 # Run tests
 uv run pytest tests/          # Run specific test directory
 uv run ruff check             # Lint code
@@ -39,6 +54,13 @@ uv run flask --app app db migrate      # Create new migration
 uv run flask --app app db upgrade      # Apply migrations
 ```

+#### Key Dependencies
+- **yt-dlp**: YouTube and streaming platform downloader
+- **python-vlc**: VLC media player Python bindings
+- **flask-socketio**: Real-time WebSocket communication
+- **ffmpeg-python**: Audio processing and normalization
+- **apscheduler**: Background task scheduling
+
 #### Environment Variables
 ```bash
 # Required for sessions and JWT
@@ -58,10 +80,10 @@ export GITHUB_CLIENT_SECRET="your_github_client_secret"
 export STREAM_MAX_CONCURRENT="2"              # Number of concurrent downloads
 ```

-### Frontend (Next.js)
+### Frontend (React/Vite)
 ```bash
 cd frontend
-bun dev                        # Start development server (preferred)
+bun dev                        # Start development server (preferred) on localhost:3000
 # Alternative package managers:
 npm run dev / yarn dev / pnpm dev
 bun run build                  # Build for production
@@ -69,6 +91,14 @@ bun run start                  # Start production server
 bun run lint                   # Run ESLint
 ```

+#### Key Dependencies
+- **@number-flow/react**: Animated number transitions
+- **socket.io-client**: Real-time WebSocket communication
+- **next-themes**: Dark/light theme support
+- **react-router**: Client-side routing (v7)
+- **lucide-react**: Icon library
+- **@radix-ui**: Headless UI components
+
 ## Code Style

 ### Backend
@@ -326,4 +356,174 @@ def ai_generation():
 {
  "error": "Insufficient credits. Required: 10, Available: 5"
 }
-```
+```
+
+## Soundboard System
+
+The application includes a comprehensive sound management system with multiple sound types and real-time playback capabilities.
+
+### Sound Types
+
+- **SDB (Soundboard)**: Traditional soundboard effects and clips stored in `sounds/soundboard/`
+- **SAY (Text-to-Speech)**: Generated speech audio files stored in `sounds/say/`
+- **STR (Stream)**: Downloaded audio from streaming platforms stored in `sounds/stream/`
+
+### Database Models
+
+#### Sound Model
+- **Core fields**: id, type, name, filename, duration, size, hash
+- **Normalization**: normalized_filename, normalized_duration, normalized_size, normalized_hash
+- **Properties**: is_normalized, is_music, is_deletable, play_count
+- **Relationships**: playlist_sounds, streams
+- **Methods**: increment_play_count(), set_normalized_info(), find_by_hash()
+
+#### Stream Model
+- **Metadata**: service, service_id, url, title, track, artist, album, genre
+- **Status tracking**: status (pending/processing/completed/failed), error
+- **Relationships**: Links to Sound model when processing completes
+- **Constraints**: Unique constraint on service + service_id
+
+#### Playlist Models
+- **Playlist**: id, name, description, genre, user_id, is_main, is_current
+- **PlaylistSound**: Junction table with playlist_id, sound_id, order
+- **Methods**: add_sound(), find_current_playlist(), find_main_playlist()
+
+### API Endpoints
+
+#### Soundboard Routes (`/api/soundboard/`)
+- `GET /sounds` - Get all soundboard sounds with type filtering
+- `POST /sounds/<id>/play` - Play a sound (costs 1 credit)
+- `POST /stop-all` - Stop all currently playing sounds
+- `POST /force-stop` - Force stop with aggressive cleanup
+- `GET /status` - Get current playback status
+
+#### Stream Routes (`/api/stream/`)
+- `POST /add-url` - Add streaming URL to processing queue
+
+#### Music Player Routes (`/api/player/`)
+- `GET /state` - Get current player state
+- `POST /play` - Start playback
+- `POST /pause` - Pause playback
+- `POST /stop` - Stop playback
+- `POST /next` - Skip to next track
+- `POST /previous` - Skip to previous track
+- `POST /seek` - Seek to position (0.0-1.0)
+- `POST /volume` - Set volume (0-100)
+- `POST /mode` - Set play mode (continuous/loop-playlist/loop-one/random/single)
+- `POST /playlist` - Load playlist into player
+- `POST /play-track` - Play track at specific index
+
+#### Admin Routes (`/api/admin/`)
+- Sound management endpoints for admin users
+- User management endpoints
+
+### Services
+
+#### StreamProcessingService
+- **Queue-based processing**: Multi-threaded download queue with configurable concurrency
+- **Metadata extraction**: Uses yt-dlp to extract video/audio metadata
+- **Download processing**: Downloads audio in opus format with thumbnails
+- **File management**: Moves files to organized directory structure
+- **Database integration**: Creates Sound entries and links to Stream records
+- **Playlist integration**: Automatically adds processed sounds to main playlist
+- **Normalization**: Triggers automatic audio normalization after download
+
+#### MusicPlayerService  
+- **VLC integration**: Uses python-vlc bindings for audio playback
+- **Playlist management**: Full playlist support with multiple play modes
+- **Real-time sync**: Background thread syncs player state with VLC
+- **Play tracking**: Tracks listening time and play counts (20% completion threshold)
+- **WebSocket updates**: Emits real-time player state via SocketIO
+- **Multi-format support**: Plays normalized and original audio files
+
+#### VLCService
+- **Process management**: Handles multiple VLC instances for sound effects
+- **Force stop capabilities**: Aggressive cleanup for stuck processes
+- **Sound tracking**: Records play counts and user interactions
+
+#### SoundNormalizerService
+- **Audio processing**: FFmpeg-based normalization with ReplayGain
+- **Two-pass processing**: Optional high-quality normalization
+- **Batch operations**: Queue-based processing for multiple files
+
+#### SocketIOService
+- **Real-time communication**: WebSocket server for live updates
+- **Player state sync**: Broadcasts player state changes to all clients
+- **Sound event notifications**: Play count updates and status changes
+
+### File Organization
+
+```
+sounds/
+├── soundboard/           # Original soundboard files
+├── say/                  # Text-to-speech files  
+├── stream/               # Downloaded streaming audio
+│   └── thumbnails/       # Video thumbnails
+├── normalized/           # Normalized versions
+│   ├── soundboard/
+│   ├── say/
+│   └── stream/
+└── temp/                 # Temporary download files
+```
+
+### Real-time Features
+
+#### WebSocket Events
+- **player_state_update**: Complete player state including current track, position, volume
+- **sound_play_count_changed**: When sound play counts are updated
+- **stream_processing_update**: Status updates for stream downloads
+
+#### Player State Structure
+```json
+{
+  "is_playing": true,
+  "current_time": 45000,
+  "duration": 180000,
+  "volume": 80,
+  "play_mode": "continuous",
+  "current_track": {
+    "id": 123,
+    "title": "Song Title",
+    "artist": "Artist Name",
+    "duration": 180000,
+    "thumbnail": "http://localhost:5000/api/sounds/str/thumbnails/image.jpg",
+    "service_url": "https://youtube.com/watch?v=...",
+    "type": "STR"
+  },
+  "current_track_index": 2,
+  "playlist": [...],
+  "playlist_id": 1
+}
+```
+
+### Stream Processing Workflow
+
+1. **URL Submission**: User submits streaming URL via `/api/stream/add-url`
+2. **Validation**: Check for duplicate URLs and extract basic metadata
+3. **Queue Addition**: Add stream to processing queue with "pending" status
+4. **Background Processing**: Worker thread processes stream:
+   - Extract full metadata with yt-dlp
+   - Download audio file (opus format preferred)
+   - Download thumbnail if available
+   - Move files to organized directory structure
+   - Create Sound database entry
+   - Link Stream to Sound
+   - Add to main playlist
+   - Trigger normalization
+5. **Status Updates**: Stream status updated throughout process
+6. **Completion**: Stream marked as "completed" with sound_id
+
+### Play Modes
+
+- **single**: Play one track and stop
+- **continuous**: Play through playlist once
+- **loop-playlist**: Repeat entire playlist
+- **loop-one**: Repeat current track
+- **random**: Random track selection
+
+### Error Handling
+
+- **Stream processing errors**: Detailed error messages stored in Stream.error
+- **VLC playback errors**: Graceful fallbacks and process cleanup
+- **File system errors**: Validation and cleanup of incomplete downloads
+- **Database errors**: Transaction rollbacks and error logging