Much base work started
Some checks failed
CI / scan_ruby (push) Has been cancelled
CI / scan_js (push) Has been cancelled
CI / lint (push) Has been cancelled
CI / test (push) Has been cancelled
CI / system-test (push) Has been cancelled

This commit is contained in:
Dan Milne
2025-10-31 14:36:14 +11:00
parent 4a35bf6758
commit 88a906064f
97 changed files with 5333 additions and 2774 deletions

File diff suppressed because it is too large Load Diff

548
docs/phases/phase_1.md Normal file
View File

@@ -0,0 +1,548 @@
# Velour Phase 1: MVP (Local Filesystem)
Phase 1 delivers a complete video library application for local files with grouping, transcoding, and playback. This is the foundation that provides immediate value.
**Architecture Note**: This phase implements a extensible MediaFile architecture using Single Table Inheritance (STI). Video inherits from MediaFile, preparing the system for future audio support in Phase 5.
## Technology Stack
### Core Components
- **Ruby on Rails 8.x** with SQLite3
- **Hotwire** (Turbo + Stimulus) for frontend
- **Solid Queue** for background jobs
- **Video.js** for video playback
- **FFmpeg** for video processing
- **Active Storage** for thumbnails/assets
- **TailwindCSS** for styling
## Database Schema (Core Models)
### Work Model
```ruby
class Work < ApplicationRecord
validates :title, presence: true
has_many :videos, dependent: :destroy
has_many :external_ids, dependent: :destroy
scope :organized, -> { where(organized: true) }
scope :unorganized, -> { where(organized: false) }
def primary_video
videos.order(created_at: :desc).first
end
end
```
### MediaFile Model (Base Class)
```ruby
class MediaFile < ApplicationRecord
# Base class for all media files using STI
include Streamable, Processable
# Common associations
belongs_to :work
belongs_to :storage_location
has_many :playback_sessions, dependent: :destroy
# Common metadata stores
store :fingerprints, accessors: [:xxhash64, :md5, :oshash, :phash]
store :media_metadata, accessors: [:duration, :codec, :bit_rate, :format]
# Common validations and methods
validates :filename, presence: true
validates :xxhash64, presence: true, uniqueness: { scope: :storage_location_id }
scope :web_compatible, -> { where(web_compatible: true) }
scope :needs_transcoding, -> { where(web_compatible: false) }
def display_title
work&.display_title || filename
end
def full_file_path
File.join(storage_location.path, filename)
end
def format_duration
# Duration formatting logic
end
end
```
### Video Model (Inherits from MediaFile)
```ruby
class Video < MediaFile
# Video-specific associations
has_many :video_assets, dependent: :destroy
# Video-specific metadata
store :video_metadata, accessors: [:width, :height, :video_codec, :audio_codec, :frame_rate]
# Video-specific methods
def resolution_label
# Resolution-based quality labeling (SD, 720p, 1080p, 4K, etc.)
end
end
```
### StorageLocation Model
```ruby
class StorageLocation < ApplicationRecord
has_many :videos, dependent: :destroy
validates :name, presence: true
validates :path, presence: true, uniqueness: true
validates :storage_type, presence: true, inclusion: { in: %w[local] }
validate :path_must_exist_and_be_readable
def accessible?
File.exist?(path) && File.readable?(path)
end
private
def path_must_exist_and_be_readable
errors.add(:path, "must exist and be readable") unless accessible?
end
end
```
## Storage Architecture (Local Only)
### Directory Structure
```bash
# User's original media directories (read-only references)
/movies/action/Die Hard (1988).mkv
/movies/anime/Your Name (2016).webm
/movies/scifi/The Matrix (1999).avi
# Velour transcoded files (same directories, web-compatible)
/movies/action/Die Hard (1988).web.mp4
/movies/anime/Your Name (2016).web.mp4
/movies/scifi/The Matrix (1999).web.mp4
# Velour managed directories
./velour_data/
├── assets/ # Active Storage for generated content
│ └── thumbnails/ # Video screenshots
└── tmp/ # Temporary processing files
└── transcodes/ # Temporary transcode storage
```
### Docker Volume Mounting with Heuristic Discovery
Users mount their video directories under `/videos` and Velour automatically discovers and categorizes them:
```yaml
# docker-compose.yml
volumes:
- /path/to/user/movies:/videos/movies:ro
- /path/to/user/tv_shows:/videos/tv:ro
- /path/to/user/documentaries:/videos/docs:ro
- /path/to/user/anime:/videos/anime:ro
- ./velour_data:/app/velour_data
```
### Automatic Storage Location Discovery
```ruby
# app/services/storage_discovery_service.rb
class StorageDiscoveryService
CATEGORIES = {
'movies' => 'Movies',
'tv' => 'TV Shows',
'tv_shows' => 'TV Shows',
'series' => 'TV Shows',
'docs' => 'Documentaries',
'documentaries' => 'Documentaries',
'anime' => 'Anime',
'cartoons' => 'Animation',
'animation' => 'Animation',
'sports' => 'Sports',
'music' => 'Music Videos',
'music_videos' => 'Music Videos',
'kids' => 'Kids Content',
'family' => 'Family Content'
}.freeze
def self.discover_and_create
base_path = '/videos'
return [] unless Dir.exist?(base_path)
discovered = []
Dir.children(base_path).each do |subdir|
dir_path = File.join(base_path, subdir)
next unless Dir.exist?(dir_path)
category = categorize_directory(subdir)
storage = StorageLocation.find_or_create_by!(
name: "#{category}: #{subdir.titleize}",
path: dir_path,
storage_type: 'local'
)
discovered << storage
end
discovered
end
def self.categorize_directory(dirname)
downcase = dirname.downcase
CATEGORIES[downcase] || CATEGORIES.find { |key, _| downcase.include?(key) }&.last || 'Other'
end
end
```
## Video Processing Pipeline
### Background Job with ActiveJob 8.1 Continuations
```ruby
class VideoProcessorJob < ApplicationJob
include ActiveJob::Statuses
def perform(video_id)
video = Video.find(video_id)
progress.update(stage: "metadata", total: 100, current: 0)
metadata = VideoMetadataExtractor.new(video.full_file_path).extract
video.update!(video_metadata: metadata)
progress.update(stage: "metadata", total: 100, current: 100)
progress.update(stage: "thumbnail", total: 100, current: 0)
generate_thumbnail(video)
progress.update(stage: "thumbnail", total: 100, current: 100)
unless video.web_compatible?
progress.update(stage: "transcode", total: 100, current: 0)
transcode_video(video)
progress.update(stage: "transcode", total: 100, current: 100)
end
progress.update(stage: "complete", total: 100, current: 100)
end
private
def generate_thumbnail(video)
# Generate thumbnail at 10% of duration
thumbnail_path = VideoTranscoder.new.extract_frame(video.full_file_path, video.duration * 0.1)
video.video_assets.create!(asset_type: "thumbnail", file: File.open(thumbnail_path))
end
def transcode_video(video)
transcoder = VideoTranscoder.new
output_path = video.full_file_path.gsub(/\.[^.]+$/, '.web.mp4')
transcoder.transcode_for_web(
input_path: video.full_file_path,
output_path: output_path,
on_progress: ->(current, total) { progress.update(current: current) }
)
video.update!(
transcoded_path: File.basename(output_path),
transcoded_permanently: true,
web_compatible: true
)
end
end
```
## Frontend Architecture
### Uninterrupted Video Playback
```erb
<!-- videos/show.html.erb -->
<turbo-frame id="video-player-frame" data-turbo-permanent>
<div class="video-container">
<video
id="video-player"
data-controller="video-player"
data-video-player-video-id-value="<%= @video.id %>"
data-video-player-start-position-value="<%= @last_position %>"
class="w-full"
controls>
<source src="<%= stream_video_path(@video) %>" type="video/mp4">
</video>
</div>
</turbo-frame>
<turbo-frame id="video-info">
<h1><%= @work.title %></h1>
<p>Duration: <%= format_duration(@video.duration) %></p>
</turbo-frame>
```
### Video Player Stimulus Controller
```javascript
// app/javascript/controllers/video_player_controller.js
import { Controller } from "@hotwired/stimulus"
export default class extends Controller {
static values = { videoId: Number, startPosition: Number }
connect() {
this.player = videojs(this.element, {
controls: true,
responsive: true,
fluid: true,
playbackRates: [0.5, 1, 1.25, 1.5, 2]
})
this.player.ready(() => {
if (this.startPositionValue > 0) {
this.player.currentTime(this.startPositionValue)
}
})
// Save position every 10 seconds
this.interval = setInterval(() => {
this.savePosition()
}, 10000)
// Save on pause
this.player.on("pause", () => this.savePosition())
}
disconnect() {
clearInterval(this.interval)
this.player.dispose()
}
savePosition() {
const position = Math.floor(this.player.currentTime())
fetch(`/videos/${this.videoIdValue}/playback-position`, {
method: "PATCH",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ position })
})
}
}
```
## Service Objects
### File Scanner Service
```ruby
class FileScannerService
def initialize(storage_location)
@storage_location = storage_location
end
def scan
return failure_result("Storage location not accessible") unless @storage_location.accessible?
video_files = find_video_files
new_videos = process_files(video_files)
success_result(new_videos)
rescue => e
failure_result(e.message)
end
private
def find_video_files
Dir.glob(File.join(@storage_location.path, "**", "*.{mp4,avi,mkv,mov,wmv,flv,webm,m4v}"))
end
def process_files(file_paths)
new_videos = []
file_paths.each do |file_path|
filename = File.basename(file_path)
next if Video.exists?(filename: filename, storage_location: @storage_location)
video = Video.create!(
filename: filename,
storage_location: @storage_location,
work: Work.find_or_create_by(title: extract_title(filename))
)
new_videos << video
VideoProcessorJob.perform_later(video.id)
end
new_videos
end
def extract_title(filename)
# Simple title extraction - can be enhanced
File.basename(filename, ".*").gsub(/[\[\(].*?[\]\)]/, "").strip
end
def success_result(videos = [])
{ success: true, videos: videos, message: "Found #{videos.length} new videos" }
end
def failure_result(message)
{ success: false, message: message }
end
end
```
### Video Transcoder Service
```ruby
class VideoTranscoder
def transcode_for_web(input_path:, output_path:, on_progress: nil)
movie = FFMPEG::Movie.new(input_path)
# Calculate progress callback
progress_callback = ->(progress) {
on_progress&.call(progress, 100)
}
movie.transcode(output_path, {
video_codec: "libx264",
audio_codec: "aac",
custom: %w[
-pix_fmt yuv420p
-preset medium
-crf 23
-movflags +faststart
-tune fastdecode
]
}, &progress_callback)
end
def extract_frame(input_path, seconds)
movie = FFMPEG::Movie.new(input_path)
output_path = "#{Rails.root}/tmp/thumbnail_#{SecureRandom.hex(8)}.jpg"
movie.screenshot(output_path, seek_time: seconds, resolution: "320x240")
output_path
end
end
```
## Routes
```ruby
Rails.application.routes.draw do
root "storage_locations#index"
resources :storage_locations, only: [:index, :show, :create, :destroy] do
member do
post :scan
end
end
resources :works, only: [:index, :show] do
resources :videos, only: [:show]
end
resources :videos, only: [] do
member do
get :stream
patch :playback_position
post :retry_processing
end
resources :playback_sessions, only: [:create]
end
# Real-time job progress
resources :jobs, only: [:show] do
member do
get :progress
end
end
end
```
## Implementation Sub-Phases
### Phase 1A: Core Foundation (Week 1-2)
1. Generate models with migrations
2. Implement model validations and associations
3. Create storage adapter pattern
4. Create FileScannerService
5. Basic UI for storage locations and video listing
### Phase 1B: Video Playback (Week 3)
1. Video streaming controller with byte-range support
2. Video.js integration with Stimulus controller
3. Playback session tracking
4. Resume functionality
### Phase 1C: Processing Pipeline (Week 4)
1. Video metadata extraction with FFmpeg
2. Background job processing with progress tracking
3. Thumbnail generation and storage
4. Processing UI with status indicators
### Phase 1D: Works & Grouping (Week 5)
1. Duplicate detection by file hash
2. Manual grouping interface
3. Works display with version selection
4. Search, filtering, and pagination
## Configuration
### Environment Variables
```bash
# Required
RAILS_ENV=development
RAILS_MASTER_KEY=your_master_key
# Video processing
FFMPEG_PATH=/usr/bin/ffmpeg
FFPROBE_PATH=/usr/bin/ffprobe
# Storage
VIDEOS_PATH=./velour_data/videos
MAX_TRANSCODE_SIZE_GB=50
# Background jobs (SolidQueue runs with defaults)
SOLID_QUEUE_PROCESSES="*:2" # 2 workers for all queues
```
### Docker Configuration
```dockerfile
FROM ruby:3.3-alpine
# Install FFmpeg and other dependencies
RUN apk add --no-cache \
ffmpeg \
imagemagick \
sqlite-dev \
nodejs \
npm
# Install Rails and other gems
COPY Gemfile* /app/
RUN bundle install
# Copy application code
COPY . /app/
# Create directories
RUN mkdir -p /app/velour_data/{assets,tmp}
EXPOSE 3000
CMD ["rails", "server", "-b", "0.0.0.0"]
```
## Testing Strategy
### Model Tests
- Model validations and associations
- Scopes and class methods
- JSON store accessor functionality
### Service Tests
- FileScannerService with sample directory
- VideoTranscoder service methods
- Background job processing
### System Tests
- Video playback functionality
- File scanning workflow
- Work grouping interface
This Phase 1 delivers a complete, useful video library application that provides immediate value while establishing the foundation for future enhancements.

486
docs/phases/phase_2.md Normal file
View File

@@ -0,0 +1,486 @@
# Velour Phase 2: Authentication & Multi-User
Phase 2 adds user management, authentication, and multi-user support while maintaining the simplicity of the core application.
## Authentication Architecture
### Rails Authentication Generators + OIDC Extension
We use Rails' built-in authentication generators as the foundation, extended with OIDC support for enterprise environments.
### User Model
```ruby
class User < ApplicationRecord
# Include default devise modules or Rails authentication
# Devise modules: :database_authenticatable, :registerable,
# :recoverable, :rememberable, :validatable
has_many :playback_sessions, dependent: :destroy
has_many :user_preferences, dependent: :destroy
validates :email, presence: true, uniqueness: true
enum role: { user: 0, admin: 1 }
def admin?
role == "admin" || email == ENV.fetch("ADMIN_EMAIL", "").downcase
end
def can_manage_storage?
admin?
end
def can_manage_users?
admin?
end
end
```
### Authentication Setup
```bash
# Generate Rails authentication
rails generate authentication
# Add OIDC support
gem 'omniauth-openid-connect'
bundle install
```
### OIDC Configuration
```ruby
# config/initializers/omniauth.rb
Rails.application.config.middleware.use OmniAuth::Builder do
provider :openid_connect, {
name: :oidc,
issuer: ENV['OIDC_ISSUER'],
client_id: ENV['OIDC_CLIENT_ID'],
client_secret: ENV['OIDC_CLIENT_SECRET'],
scope: [:openid, :email, :profile],
response_type: :code,
client_options: {
identifier: ENV['OIDC_CLIENT_ID'],
secret: ENV['OIDC_CLIENT_SECRET'],
redirect_uri: "#{ENV['RAILS_HOST']}/auth/oidc/callback"
}
}
end
```
## First User Bootstrap Flow
### Initial Setup
```ruby
# db/seeds.rb
admin_email = ENV.fetch('ADMIN_EMAIL', 'admin@velour.local')
User.find_or_create_by!(email: admin_email) do |user|
user.password = SecureRandom.hex(16)
user.role = :admin
puts "Created admin user: #{admin_email}"
puts "Password: #{user.password}"
end
```
### First Login Controller
```ruby
class FirstSetupController < ApplicationController
before_action :ensure_no_users_exist
before_action :require_admin_setup, only: [:create_admin]
def show
@user = User.new
end
def create_admin
@user = User.new(user_params)
@user.role = :admin
if @user.save
session[:user_id] = @user.id
redirect_to root_path, notice: "Admin account created successfully!"
else
render :show, status: :unprocessable_entity
end
end
private
def ensure_no_users_exist
redirect_to root_path if User.exists?
end
def require_admin_setup
redirect_to first_setup_path unless ENV.key?('ADMIN_EMAIL')
end
def user_params
params.require(:user).permit(:email, :password, :password_confirmation)
end
end
```
## User Management
### User Preferences
```ruby
class UserPreference < ApplicationRecord
belongs_to :user
store :settings, coder: JSON, accessors: [
:default_video_quality,
:auto_play_next,
:subtitle_language,
:theme
]
validates :user_id, presence: true, uniqueness: true
end
```
### Per-User Playback Sessions
```ruby
class PlaybackSession < ApplicationRecord
belongs_to :user
belongs_to :video
scope :for_user, ->(user) { where(user: user) }
scope :recent, -> { order(updated_at: :desc) }
def self.resume_position_for(video, user)
for_user(user).where(video: video).last&.position || 0
end
end
```
## Authorization & Security
### Model-Level Authorization
```ruby
# app/models/concerns/authorizable.rb
module Authorizable
extend ActiveSupport::Concern
def viewable_by?(user)
true # All videos viewable by all users
end
def editable_by?(user)
user.admin?
end
end
# Include in Video and Work models
class Video < ApplicationRecord
include Authorizable
# ... rest of model
end
```
### Controller Authorization
```ruby
class ApplicationController < ActionController::Base
before_action :authenticate_user!
before_action :set_current_user
private
def set_current_user
Current.user = current_user
end
def require_admin
redirect_to root_path, alert: "Access denied" unless current_user&.admin?
end
end
class StorageLocationsController < ApplicationController
before_action :require_admin, except: [:index, :show]
# ... rest of controller
end
```
## Updated Controllers for Multi-User
### Videos Controller
```ruby
class VideosController < ApplicationController
before_action :set_video, only: [:show, :stream, :update_position]
before_action :authorize_video
def show
@work = @video.work
@last_position = PlaybackSession.resume_position_for(@video, current_user)
# Create playback session for tracking
@playback_session = current_user.playback_sessions.create!(
video: @video,
position: @last_position
)
end
def stream
unless @video.viewable_by?(current_user)
head :forbidden
return
end
send_file @video.web_stream_path,
type: "video/mp4",
disposition: "inline",
range: request.headers['Range']
end
def update_position
current_user.playback_sessions.where(video: @video).last&.update!(
position: params[:position],
completed: params[:completed] || false
)
head :ok
end
private
def set_video
@video = Video.find(params[:id])
end
def authorize_video
head :forbidden unless @video.viewable_by?(current_user)
end
end
```
### Storage Locations Controller (Admin Only)
```ruby
class StorageLocationsController < ApplicationController
before_action :require_admin, except: [:index, :show]
before_action :set_storage_location, only: [:show, :destroy, :scan]
def index
@storage_locations = StorageLocation.accessible.order(:name)
end
def show
@videos = @storage_location.videos.includes(:work).order(:filename)
end
def create
@storage_location = StorageLocation.new(storage_location_params)
if @storage_location.save
redirect_to @storage_location, notice: 'Storage location was successfully created.'
else
render :new, status: :unprocessable_entity
end
end
def destroy
if @storage_location.videos.exists?
redirect_to @storage_location, alert: 'Cannot delete storage location with videos.'
else
@storage_location.destroy
redirect_to storage_locations_path, notice: 'Storage location was successfully deleted.'
end
end
def scan
service = FileScannerService.new(@storage_location)
result = service.scan
if result[:success]
redirect_to @storage_location, notice: result[:message]
else
redirect_to @storage_location, alert: result[:message]
end
end
private
def set_storage_location
@storage_location = StorageLocation.find(params[:id])
end
def storage_location_params
params.require(:storage_location).permit(:name, :path, :storage_type)
end
end
```
## User Interface Updates
### User Navigation
```erb
<!-- app/views/layouts/application.html.erb -->
<nav class="bg-gray-800 text-white p-4">
<div class="container mx-auto flex justify-between items-center">
<%= link_to 'Velour', root_path, class: 'text-xl font-bold' %>
<div class="flex items-center space-x-4">
<%= link_to 'Library', works_path, class: 'hover:text-gray-300' %>
<%= link_to 'Storage', storage_locations_path, class: 'hover:text-gray-300' if current_user.admin? %>
<div class="relative">
<button data-action="click->dropdown#toggle" class="flex items-center">
<%= current_user.email %>
<svg class="w-4 h-4 ml-1" fill="currentColor" viewBox="0 0 20 20">
<path fill-rule="evenodd" d="M5.293 7.293a1 1 0 011.414 0L10 10.586l3.293-3.293a1 1 0 111.414 1.414l-4 4a1 1 0 01-1.414 0l-4-4a1 1 0 010-1.414z" clip-rule="evenodd"/>
</svg>
</button>
<div data-dropdown-target="menu" class="hidden absolute right-0 mt-2 w-48 bg-white rounded-md shadow-lg py-1 z-50">
<%= link_to 'Settings', edit_user_registration_path, class: 'block px-4 py-2 text-sm text-gray-700 hover:bg-gray-100' %>
<%= link_to 'Admin Panel', admin_path, class: 'block px-4 py-2 text-sm text-gray-700 hover:bg-gray-100' if current_user.admin? %>
<%= button_to 'Sign Out', destroy_user_session_path, method: :delete, class: 'block w-full text-left px-4 py-2 text-sm text-gray-700 hover:bg-gray-100' %>
</div>
</div>
</div>
</div>
</nav>
```
### Admin Panel
```erb
<!-- app/views/admin/dashboard.html.erb -->
<div class="container mx-auto p-6">
<h1 class="text-3xl font-bold mb-6">Admin Dashboard</h1>
<div class="grid grid-cols-1 md:grid-cols-3 gap-6">
<div class="bg-white p-6 rounded-lg shadow">
<h2 class="text-xl font-semibold mb-4">Users</h2>
<p class="text-3xl font-bold text-blue-600"><%= User.count %></p>
<p class="text-gray-600">Total users</p>
<%= link_to 'Manage Users', admin_users_path, class: 'mt-2 text-blue-500 hover:text-blue-700' %>
</div>
<div class="bg-white p-6 rounded-lg shadow">
<h2 class="text-xl font-semibold mb-4">Videos</h2>
<p class="text-3xl font-bold text-green-600"><%= Video.count %></p>
<p class="text-gray-600">Total videos</p>
<%= link_to 'View Library', works_path, class: 'mt-2 text-green-500 hover:text-green-700' %>
</div>
<div class="bg-white p-6 rounded-lg shadow">
<h2 class="text-xl font-semibold mb-4">Storage</h2>
<p class="text-3xl font-bold text-purple-600"><%= StorageLocation.count %></p>
<p class="text-gray-600">Storage locations</p>
<%= link_to 'Manage Storage', storage_locations_path, class: 'mt-2 text-purple-500 hover:text-purple-700' %>
</div>
</div>
<div class="mt-8 bg-white p-6 rounded-lg shadow">
<h2 class="text-xl font-semibold mb-4">System Status</h2>
<div class="space-y-2">
<div class="flex justify-between">
<span>Background Jobs:</span>
<span class="font-mono"><%= SolidQueue::Job.count %></span>
</div>
<div class="flex justify-between">
<span>Processing Jobs:</span>
<span class="font-mono"><%= SolidQueue::Job.where(finished_at: nil).count %></span>
</div>
<div class="flex justify-between">
<span>Failed Jobs:</span>
<span class="font-mono"><%= SolidQueue::FailedExecution.count %></span>
</div>
</div>
</div>
</div>
```
## Environment Configuration
### New Environment Variables
```bash
# Authentication
ADMIN_EMAIL=admin@yourdomain.com
RAILS_HOST=https://your-velour-domain.com
# OIDC (optional)
OIDC_ISSUER=https://your-oidc-provider.com
OIDC_CLIENT_ID=your_client_id
OIDC_CLIENT_SECRET=your_client_secret
# Session management
RAILS_SESSION_COOKIE_SECURE=true
RAILS_SESSION_COOKIE_SAME_SITE=lax
```
## Testing for Phase 2
### Authentication Tests
```ruby
# test/integration/authentication_test.rb
class AuthenticationTest < ActionDispatch::IntegrationTest
test "first user can create admin account" do
User.delete_all
get first_setup_path
assert_response :success
post first_setup_path, params: {
user: {
email: "admin@example.com",
password: "password123",
password_confirmation: "password123"
}
}
assert_redirected_to root_path
assert_equal "admin@example.com", User.last.email
assert User.last.admin?
end
test "regular users cannot access admin features" do
user = users(:regular)
sign_in user
get storage_locations_path
assert_redirected_to root_path
post storage_locations_path, params: {
storage_location: {
name: "Test",
path: "/test",
storage_type: "local"
}
}
assert_redirected_to root_path
end
end
```
## Migration from Phase 1
### Database Migration
```ruby
class AddAuthenticationToUsers < ActiveRecord::Migration[7.1]
def change
create_table :users do |t|
t.string :email, null: false, index: { unique: true }
t.string :encrypted_password, null: false
t.string :role, default: 'user', null: false
t.timestamps null: false
end
create_table :user_preferences do |t|
t.references :user, null: false, foreign_key: true
t.json :settings, default: {}
t.timestamps null: false
end
# Add user_id to existing playback_sessions
add_reference :playback_sessions, :user, null: false, foreign_key: true
# Create first admin if ADMIN_EMAIL is set
if ENV.key?('ADMIN_EMAIL')
User.create!(
email: ENV['ADMIN_EMAIL'],
password: SecureRandom.hex(16),
role: 'admin'
)
end
end
end
```
Phase 2 provides a complete multi-user system while maintaining the simplicity of Phase 1. Users can have personal playback history, and administrators can manage the system through a clean interface.

773
docs/phases/phase_3.md Normal file
View File

@@ -0,0 +1,773 @@
# Velour Phase 3: Remote Sources & Import
Phase 3 extends the video library to support remote storage sources like S3, JellyFin servers, and web directories. This allows users to access and import videos from multiple locations.
## Extended Storage Architecture
### New Storage Location Types
```ruby
class StorageLocation < ApplicationRecord
has_many :videos, dependent: :destroy
validates :name, presence: true
validates :storage_type, presence: true, inclusion: { in: %w[local s3 jellyfin web] }
store :configuration, accessors: [
# S3 configuration
:bucket, :region, :access_key_id, :secret_access_key, :endpoint,
# JellyFin configuration
:server_url, :api_key, :username,
# Web directory configuration
:base_url, :auth_type, :username, :password, :headers
], coder: JSON
# Storage-type specific validations
validate :validate_s3_configuration, if: -> { s3? }
validate :validate_jellyfin_configuration, if: -> { jellyfin? }
validate :validate_web_configuration, if: -> { web? }
enum storage_type: { local: 0, s3: 1, jellyfin: 2, web: 3 }
def accessible?
case storage_type
when 'local'
File.exist?(path) && File.readable?(path)
when 's3'
s3_client&.bucket(bucket)&.exists?
when 'jellyfin'
jellyfin_client&.ping?
when 'web'
web_accessible?
else
false
end
end
def scanner
case storage_type
when 'local'
LocalFileScanner.new(self)
when 's3'
S3Scanner.new(self)
when 'jellyfin'
JellyFinScanner.new(self)
when 'web'
WebDirectoryScanner.new(self)
end
end
def streamer
case storage_type
when 'local'
LocalStreamer.new(self)
when 's3'
S3Streamer.new(self)
when 'jellyfin'
JellyFinStreamer.new(self)
when 'web'
WebStreamer.new(self)
end
end
private
def validate_s3_configuration
%w[bucket region access_key_id secret_access_key].each do |field|
errors.add(:configuration, "#{field} is required for S3 storage") if send(field).blank?
end
end
def validate_jellyfin_configuration
%w[server_url api_key].each do |field|
errors.add(:configuration, "#{field} is required for JellyFin storage") if send(field).blank?
end
end
def validate_web_configuration
errors.add(:configuration, "base_url is required for web storage") if base_url.blank?
end
end
```
## S3 Storage Implementation
### S3 Scanner Service
```ruby
class S3Scanner
def initialize(storage_location)
@storage_location = storage_location
@client = s3_client
end
def scan
return failure_result("S3 bucket not accessible") unless @storage_location.accessible?
video_files = find_video_files_in_s3
new_videos = process_s3_files(video_files)
success_result(new_videos)
rescue Aws::Errors::ServiceError => e
failure_result("S3 error: #{e.message}")
end
private
def s3_client
@s3_client ||= Aws::S3::Client.new(
region: @storage_location.region,
access_key_id: @storage_location.access_key_id,
secret_access_key: @storage_location.secret_access_key,
endpoint: @storage_location.endpoint # Optional for S3-compatible services
)
end
def find_video_files_in_s3
bucket = Aws::S3::Bucket.new(@storage_location.bucket, client: s3_client)
video_extensions = %w[.mp4 .avi .mkv .mov .wmv .flv .webm .m4v]
bucket.objects(prefix: "")
.select { |obj| video_extensions.any? { |ext| obj.key.end_with?(ext) } }
.to_a
end
def process_s3_files(s3_objects)
new_videos = []
s3_objects.each do |s3_object|
filename = File.basename(s3_object.key)
next if Video.exists?(filename: filename, storage_location: @storage_location)
video = Video.create!(
filename: filename,
storage_location: @storage_location,
work: Work.find_or_create_by(title: extract_title(filename)),
file_size: s3_object.size,
video_metadata: {
remote_url: s3_object.key,
last_modified: s3_object.last_modified
}
)
new_videos << video
VideoProcessorJob.perform_later(video.id)
end
new_videos
end
def extract_title(filename)
File.basename(filename, ".*").gsub(/[\[\(].*?[\]\)]/, "").strip
end
def success_result(videos = [])
{ success: true, videos: videos, message: "Found #{videos.length} new videos in S3" }
end
def failure_result(message)
{ success: false, message: message }
end
end
```
### S3 Streamer
```ruby
class S3Streamer
def initialize(storage_location)
@storage_location = storage_location
@client = s3_client
end
def stream(video, range: nil)
s3_object = s3_object_for_video(video)
if range
# Handle byte-range requests for seeking
range_header = "bytes=#{range}"
resp = @client.get_object(
bucket: @storage_location.bucket,
key: video.video_metadata['remote_url'],
range: range_header
)
{
body: resp.body,
status: 206, # Partial content
headers: {
'Content-Range' => "bytes #{range}/#{s3_object.size}",
'Content-Length' => resp.content_length,
'Accept-Ranges' => 'bytes',
'Content-Type' => 'video/mp4'
}
}
else
resp = @client.get_object(
bucket: @storage_location.bucket,
key: video.video_metadata['remote_url']
)
{
body: resp.body,
status: 200,
headers: {
'Content-Length' => resp.content_length,
'Content-Type' => 'video/mp4'
}
}
end
end
def presigned_url(video, expires_in: 1.hour)
signer = Aws::S3::Presigner.new(client: @client)
signer.presigned_url(
:get_object,
bucket: @storage_location.bucket,
key: video.video_metadata['remote_url'],
expires_in: expires_in.to_i
)
end
private
def s3_client
@s3_client ||= Aws::S3::Client.new(
region: @storage_location.region,
access_key_id: @storage_location.access_key_id,
secret_access_key: @storage_location.secret_access_key,
endpoint: @storage_location.endpoint
)
end
def s3_object_for_video(video)
@client.get_object(
bucket: @storage_location.bucket,
key: video.video_metadata['remote_url']
)
end
end
```
## JellyFin Integration
### JellyFin Client
```ruby
class JellyFinClient
def initialize(server_url:, api_key:, username: nil)
@server_url = server_url.chomp('/')
@api_key = api_key
@username = username
@http = Faraday.new(url: @server_url) do |faraday|
faraday.headers['X-Emby-Token'] = @api_key
faraday.adapter Faraday.default_adapter
end
end
def ping?
response = @http.get('/System/Ping')
response.success?
rescue
false
end
def libraries
response = @http.get('/Library/VirtualFolders')
JSON.parse(response.body)
end
def movies(library_id = nil)
path = library_id ? "/Users/#{user_id}/Items?ParentId=#{library_id}&IncludeItemTypes=Movie&Recursive=true" : "/Users/#{user_id}/Items?IncludeItemTypes=Movie&Recursive=true"
response = @http.get(path)
JSON.parse(response.body)['Items']
end
def tv_shows(library_id = nil)
path = library_id ? "/Users/#{user_id}/Items?ParentId=#{library_id}&IncludeItemTypes=Series&Recursive=true" : "/Users/#{user_id}/Items?IncludeItemTypes=Series&Recursive=true"
response = @http.get(path)
JSON.parse(response.body)['Items']
end
def episodes(show_id)
response = @http.get("/Shows/#{show_id}/Episodes?UserId=#{user_id}")
JSON.parse(response.body)['Items']
end
def streaming_url(item_id)
"#{@server_url}/Videos/#{item_id}/stream?Static=true&MediaSourceId=#{item_id}&DeviceId=Velour&api_key=#{@api_key}"
end
def item_details(item_id)
response = @http.get("/Users/#{user_id}/Items/#{item_id}")
JSON.parse(response.body)
end
private
def user_id
@user_id ||= begin
response = @http.get('/Users')
users = JSON.parse(response.body)
if @username
user = users.find { |u| u['Name'] == @username }
user&.dig('Id') || users.first['Id']
else
users.first['Id']
end
end
end
end
```
### JellyFin Scanner
```ruby
class JellyFinScanner
def initialize(storage_location)
@storage_location = storage_location
@client = jellyfin_client
end
def scan
return failure_result("JellyFin server not accessible") unless @storage_location.accessible?
movies = @client.movies
shows = @client.tv_shows
episodes = []
shows.each do |show|
episodes.concat(@client.episodes(show['Id']))
end
all_items = movies + episodes
new_videos = process_jellyfin_items(all_items)
success_result(new_videos)
rescue => e
failure_result("JellyFin error: #{e.message}")
end
private
def jellyfin_client
@client ||= JellyFinClient.new(
server_url: @storage_location.server_url,
api_key: @storage_location.api_key,
username: @storage_location.username
)
end
def process_jellyfin_items(items)
new_videos = []
items.each do |item|
next unless item['MediaType'] == 'Video'
title = item['Name']
year = item['ProductionYear']
work_title = year ? "#{title} (#{year})" : title
work = Work.find_or_create_by(title: work_title) do |w|
w.year = year
w.description = item['Overview']
end
video = Video.find_or_initialize_by(
filename: item['Id'],
storage_location: @storage_location
)
if video.new_record?
video.update!(
work: work,
video_metadata: {
jellyfin_id: item['Id'],
media_type: item['Type'],
runtime: item['RunTimeTicks'] ? item['RunTimeTicks'] / 10_000_000 : nil,
premiere_date: item['PremiereDate'],
community_rating: item['CommunityRating'],
genres: item['Genres']
}
)
new_videos << video
VideoProcessorJob.perform_later(video.id)
end
end
new_videos
end
def success_result(videos = [])
{ success: true, videos: videos, message: "Found #{videos.length} new videos from JellyFin" }
end
def failure_result(message)
{ success: false, message: message }
end
end
```
### JellyFin Streamer
```ruby
class JellyFinStreamer
def initialize(storage_location)
@storage_location = storage_location
@client = jellyfin_client
end
def stream(video, range: nil)
jellyfin_id = video.video_metadata['jellyfin_id']
stream_url = @client.streaming_url(jellyfin_id)
# For JellyFin, we typically proxy the stream
if range
proxy_stream_with_range(stream_url, range)
else
proxy_stream(stream_url)
end
end
private
def jellyfin_client
@client ||= JellyFinClient.new(
server_url: @storage_location.server_url,
api_key: @storage_location.api_key,
username: @storage_location.username
)
end
def proxy_stream(url)
response = Faraday.get(url)
{
body: response.body,
status: response.status,
headers: response.headers
}
end
def proxy_stream_with_range(url, range)
response = Faraday.get(url, nil, { 'Range' => "bytes=#{range}" })
{
body: response.body,
status: response.status,
headers: response.headers
}
end
end
```
## Video Import System
### Import Job with Progress Tracking
```ruby
class VideoImportJob < ApplicationJob
include ActiveJob::Statuses
def perform(video_id, destination_storage_location_id)
video = Video.find(video_id)
destination = StorageLocation.find(destination_storage_location_id)
progress.update(stage: "download", total: 100, current: 0)
# Download file from source
downloaded_file = download_video(video, destination) do |current, total|
progress.update(current: (current.to_f / total * 50).to_i) # Download is 50% of progress
end
progress.update(stage: "process", total: 100, current: 50)
# Create new video record in destination
new_video = Video.create!(
filename: video.filename,
storage_location: destination,
work: video.work,
file_size: video.file_size
)
# Copy file to destination
destination_path = File.join(destination.path, video.filename)
FileUtils.cp(downloaded_file.path, destination_path)
# Process the new video
VideoProcessorJob.perform_later(new_video.id)
progress.update(stage: "complete", total: 100, current: 100)
# Clean up temp file
File.delete(downloaded_file.path)
end
private
def download_video(video, destination, &block)
case video.storage_location.storage_type
when 's3'
download_from_s3(video, &block)
when 'jellyfin'
download_from_jellyfin(video, &block)
when 'web'
download_from_web(video, &block)
else
raise "Unsupported import from #{video.storage_location.storage_type}"
end
end
def download_from_s3(video, &block)
temp_file = Tempfile.new(['video_import', File.extname(video.filename)])
s3_client = Aws::S3::Client.new(
region: video.storage_location.region,
access_key_id: video.storage_location.access_key_id,
secret_access_key: video.storage_location.secret_access_key
)
s3_client.get_object(
bucket: video.storage_location.bucket,
key: video.video_metadata['remote_url'],
response_target: temp_file.path
) do |chunk|
yield(chunk.bytes_written, chunk.size) if block_given?
end
temp_file
end
def download_from_jellyfin(video, &block)
temp_file = Tempfile.new(['video_import', File.extname(video.filename)])
jellyfin_id = video.video_metadata['jellyfin_id']
client = JellyFinClient.new(
server_url: video.storage_location.server_url,
api_key: video.storage_location.api_key
)
stream_url = client.streaming_url(jellyfin_id)
# Download with progress tracking
uri = URI(stream_url)
Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https') do |http|
request = Net::HTTP::Get.new(uri)
http.request(request) do |response|
total_size = response['Content-Length'].to_i
downloaded = 0
response.read_body do |chunk|
temp_file.write(chunk)
downloaded += chunk.bytesize
yield(downloaded, total_size) if block_given?
end
end
end
temp_file
end
end
```
### Import UI
```erb
<!-- app/views/videos/_import_button.html.erb -->
<% if video.storage_location.remote? && current_user.admin? %>
<div data-controller="import-dialog">
<button
data-action="click->import-dialog#show"
class="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded">
Import to Local Storage
</button>
<div
data-import-dialog-target="dialog"
class="hidden fixed inset-0 bg-gray-600 bg-opacity-50 overflow-y-auto h-full w-full z-50">
<div class="relative top-20 mx-auto p-5 border w-96 shadow-lg rounded-md bg-white">
<h3 class="text-lg font-bold text-gray-900 mb-4">Import Video</h3>
<p class="text-gray-600 mb-4">
Import "<%= video.filename %>" to a local storage location for offline access and transcoding.
</p>
<div class="mb-4">
<label class="block text-gray-700 text-sm font-bold mb-2">
Destination Storage:
</label>
<select
name="destination_storage_location_id"
data-import-dialog-target="destination"
class="shadow appearance-none border rounded w-full py-2 px-3 text-gray-700 leading-tight">
<% StorageLocation.local.accessible.each do |location| %>
<option value="<%= location.id %>"><%= location.name %></option>
<% end %>
</select>
</div>
<div class="flex justify-end space-x-2">
<button
data-action="click->import-dialog#hide"
class="bg-gray-500 hover:bg-gray-700 text-white font-bold py-2 px-4 rounded">
Cancel
</button>
<button
data-action="click->import-dialog#import"
data-video-id="<%= video.id %>"
class="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded">
Import
</button>
</div>
</div>
</div>
<!-- Progress display -->
<div
data-import-dialog-target="progress"
class="hidden mt-2">
<div class="bg-blue-100 border-l-4 border-blue-500 text-blue-700 p-4">
<p class="font-bold">Importing video...</p>
<div class="mt-2">
<div class="bg-blue-200 rounded-full h-2">
<div
data-import-dialog-target="progressBar"
class="bg-blue-600 h-2 rounded-full transition-all duration-300"
style="width: 0%"></div>
</div>
<p class="text-sm mt-1" data-import-dialog-target="progressText">Starting...</p>
</div>
</div>
</div>
</div>
<% end %>
```
### Import Stimulus Controller
```javascript
// app/javascript/controllers/import_dialog_controller.js
import { Controller } from "@hotwired/stimulus"
import { get } from "@rails/request.js"
export default class extends Controller {
static targets = ["dialog", "progress", "progressBar", "progressText", "destination"]
show() {
this.dialogTarget.classList.remove("hidden")
}
hide() {
this.dialogTarget.classList.add("hidden")
}
async import(event) {
const videoId = event.target.dataset.videoId
const destinationId = this.destinationTarget.value
this.hide()
this.progressTarget.classList.remove("hidden")
try {
// Start import job
const response = await post("/videos/import", {
body: JSON.stringify({
video_id: videoId,
destination_storage_location_id: destinationId
})
})
const { jobId } = await response.json
// Poll for progress
this.pollProgress(jobId)
} catch (error) {
console.error("Import failed:", error)
this.progressTarget.innerHTML = `
<div class="bg-red-100 border-l-4 border-red-500 text-red-700 p-4">
<p class="font-bold">Import failed</p>
<p class="text-sm">${error.message}</p>
</div>
`
}
}
async pollProgress(jobId) {
const updateProgress = async () => {
try {
const response = await get(`/jobs/${jobId}/progress`)
const progress = await response.json
this.progressBarTarget.style.width = `${progress.current}%`
this.progressTextTarget.textContent = `${progress.stage}: ${progress.current}%`
if (progress.stage === "complete") {
this.progressTarget.innerHTML = `
<div class="bg-green-100 border-l-4 border-green-500 text-green-700 p-4">
<p class="font-bold">Import complete!</p>
</div>
`
setTimeout(() => {
window.location.reload()
}, 2000)
} else if (progress.stage === "failed") {
this.progressTarget.innerHTML = `
<div class="bg-red-100 border-l-4 border-red-500 text-red-700 p-4">
<p class="font-bold">Import failed</p>
<p class="text-sm">${progress.error || "Unknown error"}</p>
</div>
`
} else {
setTimeout(updateProgress, 1000)
}
} catch (error) {
console.error("Failed to get progress:", error)
setTimeout(updateProgress, 1000)
}
}
updateProgress()
}
}
```
## Environment Configuration
### New Environment Variables
```bash
# S3 Configuration
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_DEFAULT_REGION=us-east-1
# Import Settings
MAX_IMPORT_SIZE_GB=10
IMPORT_TIMEOUT_MINUTES=60
# Rate limiting
JELLYFIN_RATE_LIMIT_DELAY=1 # seconds between requests
```
### New Gems
```ruby
# Gemfile
gem 'aws-sdk-s3', '~> 1'
gem 'faraday', '~> 2.0'
gem 'httparty', '~> 0.21'
```
## Routes for Phase 3
```ruby
# Add to existing routes
resources :videos, only: [] do
member do
post :import
end
end
namespace :admin do
resources :storage_locations do
member do
post :test_connection
end
end
end
```
Phase 3 enables users to access video libraries from multiple remote sources while maintaining a unified interface. The import system allows bringing remote videos into local storage for offline access and transcoding.

636
docs/phases/phase_4.md Normal file
View File

@@ -0,0 +1,636 @@
# Velour Phase 4: Federation
Phase 4 enables federation between multiple Velour instances, allowing users to share and access video libraries across different servers while maintaining security and access controls.
## Federation Architecture
### Overview
Velour federation allows instances to:
- Share video libraries with other trusted instances
- Stream videos from remote servers
- Sync metadata and work information
- Maintain access control and authentication
### Federation Storage Location Type
```ruby
class StorageLocation < ApplicationRecord
# ... existing code ...
validates :storage_type, inclusion: { in: %w[local s3 jellyfin web velour] }
store :configuration, accessors: [
# Existing configurations...
# Velour federation configuration
:remote_instance_url, :api_key, :instance_name, :trusted_instances
], coder: JSON
enum storage_type: { local: 0, s3: 1, jellyfin: 2, web: 3, velour: 4 }
def federation_client
return nil unless velour?
@federation_client ||= VelourFederationClient.new(
instance_url: remote_instance_url,
api_key: api_key
)
end
def accessible?
case storage_type
# ... existing cases ...
when 'velour'
federation_client&.ping?
else
false
end
end
def scanner
case storage_type
# ... existing cases ...
when 'velour'
VelourFederationScanner.new(self)
else
super
end
end
def streamer
case storage_type
# ... existing cases ...
when 'velour'
VelourFederationStreamer.new(self)
else
super
end
end
end
```
## Federation API Authentication
### API Key Management
```ruby
class ApiKey < ApplicationRecord
belongs_to :user
has_many :federation_connections, dependent: :destroy
validates :key, presence: true, uniqueness: true
validates :name, presence: true
validates :permissions, presence: true
store :permissions, coder: JSON, accessors: [:can_read, :can_stream, :can_metadata]
before_validation :generate_key, on: :create
def self.generate_key
SecureRandom.hex(32)
end
private
def generate_key
self.key ||= self.class.generate_key
end
end
```
### Federation Connections
```ruby
class FederationConnection < ApplicationRecord
belongs_to :api_key
belongs_to :storage_location
validates :remote_instance_url, presence: true, uniqueness: { scope: :api_key_id }
validates :status, inclusion: { in: %w[pending active suspended rejected] }
enum status: { pending: 0, active: 1, suspended: 2, rejected: 3 }
def active?
status == "active"
end
end
```
## Federation Client
### Velour Federation Client
```ruby
class VelourFederationClient
def initialize(instance_url:, api_key:)
@instance_url = instance_url.chomp('/')
@api_key = api_key
@http = Faraday.new(url: @instance_url) do |faraday|
faraday.headers['Authorization'] = "Bearer #{@api_key}"
faraday.headers['User-Agent'] = "Velour/#{Velour::VERSION}"
faraday.request :json
faraday.response :json
faraday.adapter Faraday.default_adapter
end
end
def ping?
response = @http.get('/api/v1/ping')
response.success?
rescue
false
end
def instance_info
response = @http.get('/api/v1/instance')
response.success? ? response.body : nil
end
def works(page: 1, per_page: 50)
response = @http.get('/api/v1/works', params: {
page: page,
per_page: per_page
})
response.success? ? response.body : []
end
def work_details(work_id)
response = @http.get("/api/v1/works/#{work_id}")
response.success? ? response.body : nil
end
def video_stream_url(video_id)
"#{@instance_url}/api/v1/videos/#{video_id}/stream"
end
def video_metadata(video_id)
response = @http.get("/api/v1/videos/#{video_id}")
response.success? ? response.body : nil
end
def search_videos(query:, page: 1, per_page: 20)
response = @http.get('/api/v1/videos/search', params: {
query: query,
page: page,
per_page: per_page
})
response.success? ? response.body : []
end
end
```
## Federation Scanner
### Velour Federation Scanner
```ruby
class VelourFederationScanner
def initialize(storage_location)
@storage_location = storage_location
@client = @storage_location.federation_client
end
def scan
return failure_result("Remote Velour instance not accessible") unless @storage_location.accessible?
instance_info = @client.instance_info
return failure_result("Failed to get instance info") unless instance_info
works = @client.works(per_page: 100) # Start with first 100 works
new_videos = process_remote_works(works)
success_result(new_videos, instance_info)
rescue => e
failure_result("Federation error: #{e.message}")
end
private
def process_remote_works(works)
new_videos = []
works.each do |work_data|
# Create or find local work
work = Work.find_or_create_by(
title: work_data['title'],
year: work_data['year']
) do |w|
w.description = work_data['description']
w.director = work_data['director']
w.rating = work_data['rating']
end
# Process videos for this work
work_data['videos'].each do |video_data|
video = Video.find_or_initialize_by(
filename: video_data['id'], # Use remote ID as filename
storage_location: @storage_location
)
if video.new_record?
video.update!(
work: work,
file_size: video_data['file_size'],
web_compatible: video_data['web_compatible'],
video_metadata: {
remote_video_id: video_data['id'],
remote_instance_url: @storage_location.remote_instance_url,
duration: video_data['duration'],
width: video_data['width'],
height: video_data['height'],
video_codec: video_data['video_codec'],
audio_codec: video_data['audio_codec']
},
fingerprints: video_data['fingerprints']
)
new_videos << video
# Note: We don't process remote videos locally
# We just catalog them for streaming
end
end
end
new_videos
end
def success_result(videos = [], instance_info = {})
{
success: true,
videos: videos,
message: "Found #{videos.length} videos from #{@storage_location.instance_name}",
instance_info: instance_info
}
end
def failure_result(message)
{ success: false, message: message }
end
end
```
## Federation Streamer
### Velour Federation Streamer
```ruby
class VelourFederationStreamer
def initialize(storage_location)
@storage_location = storage_location
@client = @storage_location.federation_client
end
def stream(video, range: nil)
remote_video_id = video.video_metadata['remote_video_id']
stream_url = @client.video_stream_url(remote_video_id)
if range
proxy_stream_with_range(stream_url, range)
else
proxy_stream(stream_url)
end
end
def thumbnail_url(video)
remote_video_id = video.video_metadata['remote_video_id']
"#{@storage_location.remote_instance_url}/api/v1/videos/#{remote_video_id}/thumbnail"
end
private
def proxy_stream(url)
response = Faraday.get(url) do |req|
req.headers['Authorization'] = "Bearer #{@storage_location.api_key}"
end
{
body: response.body,
status: response.status,
headers: response.headers
}
end
def proxy_stream_with_range(url, range)
response = Faraday.get(url) do |req|
req.headers['Authorization'] = "Bearer #{@storage_location.api_key}"
req.headers['Range'] = "bytes=#{range}"
end
{
body: response.body,
status: response.status,
headers: response.headers
}
end
end
```
## Federation API Endpoints
### API Controllers
```ruby
# app/controllers/api/v1/base_controller.rb
class Api::V1::BaseController < ActionController::Base
before_action :authenticate_api_request
private
def authenticate_api_request
token = request.headers['Authorization']&.gsub(/^Bearer\s+/, '')
api_key = ApiKey.find_by(key: token)
if api_key&.active?
@current_api_key = api_key
@current_user = api_key.user
else
render json: { error: 'Unauthorized' }, status: :unauthorized
end
end
def authorize_federation
head :forbidden unless @current_api_key&.can_read
end
def authorize_streaming
head :forbidden unless @current_api_key&.can_stream
end
end
# app/controllers/api/v1/instance_controller.rb
class Api::V1::InstanceController < Api::V1::BaseController
before_action :authorize_federation
def ping
render json: { status: 'ok', timestamp: Time.current.iso8601 }
end
def show
render json: {
name: 'Velour Instance',
version: Velour::VERSION,
description: 'Video library federation node',
total_works: Work.count,
total_videos: Video.count,
total_storage: Video.sum(:file_size)
}
end
end
# app/controllers/api/v1/works_controller.rb
class Api::V1::WorksController < Api::V1::BaseController
before_action :authorize_federation
def index
works = Work.includes(:videos)
.order(:title)
.page(params[:page])
.per(params[:per_page] || 50)
render json: {
works: works.map { |work| serialize_work(work) },
pagination: {
current_page: works.current_page,
total_pages: works.total_pages,
total_count: works.total_count
}
}
end
def show
work = Work.includes(videos: :storage_location).find(params[:id])
render json: serialize_work(work)
end
private
def serialize_work(work)
{
id: work.id,
title: work.title,
year: work.year,
description: work.description,
director: work.director,
rating: work.rating,
organized: work.organized,
created_at: work.created_at.iso8601,
videos: work.videos.map { |video| serialize_video(video) }
}
end
def serialize_video(video)
{
id: video.id,
filename: video.filename,
file_size: video.file_size,
web_compatible: video.web_compatible?,
duration: video.duration,
width: video.width,
height: video.height,
video_codec: video.video_codec,
audio_codec: video.audio_codec,
fingerprints: video.fingerprints,
storage_type: video.storage_location.storage_type,
created_at: video.created_at.iso8601
}
end
end
# app/controllers/api/v1/videos_controller.rb
class Api::V1::VideosController < Api::V1::BaseController
before_action :set_video, only: [:show, :stream, :thumbnail]
before_action :authorize_federation, except: [:stream, :thumbnail]
before_action :authorize_streaming, only: [:stream, :thumbnail]
def show
render json: serialize_video(@video)
end
def stream
streamer = @video.storage_location.streamer
result = streamer.stream(@video, range: request.headers['Range'])
send_data result[:body],
type: result[:headers]['Content-Type'] || 'video/mp4',
disposition: 'inline',
status: result[:status],
headers: result[:headers].slice('Content-Range', 'Content-Length', 'Accept-Ranges')
end
def thumbnail
if @video.video_assets.where(asset_type: 'thumbnail').exists?
asset = @video.video_assets.where(asset_type: 'thumbnail').first
redirect_to rails_blob_url(asset.file, disposition: 'inline')
else
head :not_found
end
end
def search
query = params[:query]
return render json: [] if query.blank?
videos = Video.joins(:work)
.where('works.title ILIKE ?', "%#{query}%")
.includes(:work, :storage_location)
.page(params[:page])
.per(params[:per_page] || 20)
render json: {
videos: videos.map { |video| serialize_video(video) },
pagination: {
current_page: videos.current_page,
total_pages: videos.total_pages,
total_count: videos.total_count
}
}
end
private
def set_video
@video = Video.find(params[:id])
end
def serialize_video(video)
{
id: video.id,
filename: video.filename,
work_id: video.work_id,
work_title: video.work.title,
file_size: video.file_size,
web_compatible: video.web_compatible?,
duration: video.duration,
width: video.width,
height: video.height,
video_codec: video.video_codec,
audio_codec: video.audio_codec,
fingerprints: video.fingerprints,
storage_type: video.storage_location.storage_type,
created_at: video.created_at.iso8601
}
end
end
```
## Federation Management UI
### Federation Connections Management
```erb
<!-- app/views/admin/federation_connections/index.html.erb -->
<div class="container mx-auto p-6">
<h1 class="text-3xl font-bold mb-6">Federation Connections</h1>
<% if @api_keys.any? %>
<div class="mb-6">
<%= link_to 'Create New Connection', new_admin_federation_connection_path, class: 'bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded' %>
</div>
<div class="bg-white shadow overflow-hidden sm:rounded-md">
<ul class="divide-y divide-gray-200">
<% @connections.each do |connection| %>
<li class="px-6 py-4">
<div class="flex items-center justify-between">
<div>
<p class="text-sm font-medium text-gray-900">
<%= connection.storage_location.name %>
</p>
<p class="text-sm text-gray-500">
<%= connection.remote_instance_url %>
</p>
<p class="text-sm text-gray-500">
API Key: <%= connection.api_key.name %>
</p>
</div>
<div class="flex items-center space-x-2">
<span class="inline-flex items-center px-2.5 py-0.5 rounded-full text-xs font-medium
<%= case connection.status
when 'active' then 'bg-green-100 text-green-800'
when 'pending' then 'bg-yellow-100 text-yellow-800'
when 'suspended' then 'bg-red-100 text-red-800'
when 'rejected' then 'bg-gray-100 text-gray-800'
end %>">
<%= connection.status.titleize %>
</span>
<%= link_to 'Edit', edit_admin_federation_connection_path(connection), class: 'text-blue-600 hover:text-blue-900' %>
<%= link_to 'Test', test_admin_federation_connection_path(connection), method: :post, class: 'text-green-600 hover:text-green-900' %>
<%= link_to 'Delete', admin_federation_connection_path(connection), method: :delete,
data: { confirm: 'Are you sure?' }, class: 'text-red-600 hover:text-red-900' %>
</div>
</div>
</li>
<% end %>
</ul>
</div>
<% else %>
<div class="bg-yellow-50 border-l-4 border-yellow-400 p-4">
<div class="flex">
<div class="flex-shrink-0">
<svg class="h-5 w-5 text-yellow-400" viewBox="0 0 20 20" fill="currentColor">
<path fill-rule="evenodd" d="M8.257 3.099c.765-1.36 2.722-1.36 3.486 0l5.58 9.92c.75 1.334-.213 2.98-1.742 2.98H4.42c-1.53 0-2.493-1.646-1.743-2.98l5.58-9.92zM11 13a1 1 0 11-2 0 1 1 0 012 0zm-1-8a1 1 0 00-1 1v3a1 1 0 002 0V6a1 1 0 00-1-1z" clip-rule="evenodd"/>
</svg>
</div>
<div class="ml-3">
<p class="text-sm text-yellow-700">
You need to create an API key before you can establish federation connections.
</p>
<div class="mt-2">
<%= link_to 'Create API Key', new_admin_api_key_path, class: 'text-yellow-700 underline hover:text-yellow-600' %>
</div>
</div>
</div>
</div>
<% end %>
</div>
```
## Security Considerations
### Federation Security Best Practices
1. **API Key Management**: Regular rotation and limited scope
2. **Access Control**: Minimum required permissions
3. **Rate Limiting**: Prevent abuse from federated instances
4. **Audit Logging**: Track all federated access
5. **Network Security**: HTTPS-only federation connections
### Rate Limiting
```ruby
# app/controllers/concerns/rate_limited.rb
module RateLimited
extend ActiveSupport::Concern
included do
before_action :check_rate_limit, only: [:index, :show, :stream]
end
private
def check_rate_limit
return unless @current_api_key
key = "federation_rate_limit:#{@current_api_key.id}"
count = Rails.cache.increment(key, 1, expires_in: 1.hour)
if count > rate_limit
Rails.logger.warn "Rate limit exceeded for API key #{@current_api_key.id}"
render json: { error: 'Rate limit exceeded' }, status: :too_many_requests
end
end
def rate_limit
case action_name
when 'stream'
1000 # requests per hour
else
5000 # requests per hour
end
end
end
```
## Environment Configuration
### Federation Configuration
```bash
# Federation settings
FEDERATION_ENABLED=true
MAX_FEDERATED_CONNECTIONS=10
FEDERATION_RATE_LIMIT_PER_HOUR=5000
# Federation security
FEDERATION_REQUIRE_HTTPS=true
FEDERATION_TOKEN_EXPIRY_HOURS=24
```
Phase 4 enables a distributed network of Velour instances that can share video libraries while maintaining security and access controls. This allows organizations to federate their video collections across multiple servers or locations.

324
docs/phases/phase_5.md Normal file
View File

@@ -0,0 +1,324 @@
# Velour Phase 5: Audio Support (Music & Audiobooks)
Phase 5 extends Velour from a video library to a comprehensive media library by adding support for music and audiobooks. This builds upon the extensible MediaFile architecture established in Phase 1.
## Technology Stack
### Audio Processing Components
- **FFmpeg** - Audio transcoding and metadata extraction (extends existing video processing)
- **Ruby Audio Gems** - ID3 tag parsing, waveform generation
- **Active Storage** - Album art and waveform visualization storage
- **MediaInfo** - Comprehensive audio metadata extraction
## Database Schema Extensions
### Audio Model (inherits from MediaFile)
```ruby
class Audio < MediaFile
# Audio-specific associations
has_many :audio_assets, dependent: :destroy # album art, waveforms
# Audio-specific metadata store
store :audio_metadata, accessors: [:sample_rate, :channels, :artist, :album, :track_number, :genre, :year]
# Audio-specific methods
def quality_label
return "Unknown" unless bit_rate
case bit_rate
when 0..128 then "128kbps"
when 129..192 then "192kbps"
when 193..256 then "256kbps"
when 257..320 then "320kbps"
else "Lossless"
end
end
def format_type
return "Unknown" unless format
case format&.downcase
when "mp3" then "MP3"
when "flac" then "FLAC"
when "wav" then "WAV"
when "aac", "m4a" then "AAC"
when "ogg" then "OGG Vorbis"
else format&.upcase
end
end
end
class AudioAsset < ApplicationRecord
belongs_to :audio
enum asset_type: { album_art: 0, waveform: 1, lyrics: 2 }
# Uses Active Storage for file storage
has_one_attached :file
end
```
### Extended Work Model
```ruby
class Work < ApplicationRecord
# Existing video associations
has_many :videos, dependent: :destroy
has_many :external_ids, dependent: :destroy
# New audio associations
has_many :audios, dependent: :destroy
# Enhanced primary media selection
def primary_media
(audios + videos).sort_by(&:created_at).last
end
def primary_video
videos.order(created_at: :desc).first
end
def primary_audio
audios.order(created_at: :desc).first
end
# Content type detection
def video_content?
videos.exists?
end
def audio_content?
audios.exists?
end
def mixed_content?
video_content? && audio_content?
end
end
```
## Audio Processing Pipeline
### AudioProcessorJob
```ruby
class AudioProcessorJob < ApplicationJob
queue_as :processing
def perform(audio_id)
audio = Audio.find(audio_id)
# Extract audio metadata
AudioMetadataExtractor.new(audio).extract!
# Generate album art if embedded
AlbumArtExtractor.new(audio).extract!
# Generate waveform visualization
WaveformGenerator.new(audio).generate!
# Check web compatibility and transcode if needed
unless AudioTranscoder.new(audio).web_compatible?
AudioTranscoderJob.perform_later(audio_id)
end
audio.update!(processed: true)
rescue => e
audio.update!(processing_error: e.message)
raise
end
end
```
### AudioTranscoderJob
```ruby
class AudioTranscoderJob < ApplicationJob
queue_as :transcoding
def perform(audio_id)
audio = Audio.find(audio_id)
AudioTranscoder.new(audio).transcode_for_web!
end
end
```
## File Discovery Extensions
### Enhanced FileScannerService
```ruby
class FileScannerService
AUDIO_EXTENSIONS = %w[mp3 flac wav aac m4a ogg wma].freeze
def scan_directory(storage_location)
# Existing video scanning logic
scan_videos(storage_location)
# New audio scanning logic
scan_audio(storage_location)
end
private
def scan_audio(storage_location)
AUDIO_EXTENSIONS.each do |ext|
Dir.glob(File.join(storage_location.path, "**", "*.#{ext}")).each do |file_path|
process_audio_file(file_path, storage_location)
end
end
end
def process_audio_file(file_path, storage_location)
filename = File.basename(file_path)
return if Audio.joins(:storage_location).exists?(filename: filename, storage_locations: { id: storage_location.id })
# Create Work based on filename parsing (album/track structure)
work = find_or_create_audio_work(filename, file_path)
# Create Audio record
Audio.create!(
work: work,
storage_location: storage_location,
filename: filename,
xxhash64: calculate_xxhash64(file_path)
)
AudioProcessorJob.perform_later(audio.id)
end
end
```
## User Interface Extensions
### Audio Player Integration
- **Video.js Audio Plugin** - Extend existing video player for audio
- **Waveform Visualization** - Interactive seeking with waveform display
- **Chapter Support** - Essential for audiobooks
- **Speed Control** - Variable playback speed for audiobooks
### Library Organization
- **Album View** - Grid layout with album art
- **Artist Pages** - Discography and album organization
- **Audiobook Progress** - Chapter tracking and resume functionality
- **Mixed Media Collections** - Works containing both video and audio content
### Audio-Specific Features
- **Playlist Creation** - Custom playlists for music
- **Shuffle Play** - Random playback for albums/artists
- **Gapless Playback** - Seamless track transitions
- **Lyrics Display** - Embedded or external lyrics support
## Implementation Timeline
### Phase 5A: Audio Foundation (Week 1-2)
- Create Audio model inheriting from MediaFile
- Implement AudioProcessorJob and audio metadata extraction
- Extend FileScannerService for audio formats
- Basic audio streaming endpoint
### Phase 5B: Audio Processing (Week 3)
- Album art extraction and storage
- Waveform generation
- Audio transcoding for web compatibility
- Quality optimization and format conversion
### Phase 5C: User Interface (Week 4)
- Audio player component (extends Video.js)
- Album and artist browsing interfaces
- Audio library management views
- Search and filtering for audio content
### Phase 5D: Advanced Features (Week 5)
- Chapter support for audiobooks
- Playlist creation and management
- Mixed media Works (video + audio)
- Audio-specific user preferences
## Migration Strategy
### Database Migrations
```ruby
# Extend videos table for STI (already done in Phase 1)
# Add audio-specific columns if needed
class AddAudioFeatures < ActiveRecord::Migration[8.1]
def change
create_table :audio_assets do |t|
t.references :audio, null: false, foreign_key: true
t.string :asset_type
t.timestamps
end
# Audio-specific indexes
add_index :audios, :artist if column_exists?(:audios, :artist)
add_index :audios, :album if column_exists?(:audios, :album)
end
end
```
### Backward Compatibility
- All existing video functionality remains unchanged
- Video URLs and routes continue to work identically
- Database migration is additive (type column only)
- No breaking changes to existing API
## Configuration
### Environment Variables
```bash
# Audio Processing (extends existing video processing)
FFMPEG_PATH=/usr/bin/ffmpeg
AUDIO_TRANSCODE_QUALITY=high
MAX_AUDIO_TRANSCODE_SIZE_GB=10
# Audio Features
ENABLE_AUDIO_SCANNING=true
ENABLE_WAVEFORM_GENERATION=true
AUDIO_THUMBNAIL_SIZE=300x300
```
### Storage Considerations
- Album art storage in Active Storage
- Waveform images (generated per track)
- Potential audio transcoding cache
- Audio-specific metadata storage
## Testing Strategy
### Model Tests
- Audio model validation and inheritance
- Work model mixed content handling
- Audio metadata extraction accuracy
### Integration Tests
- Audio processing pipeline end-to-end
- Audio streaming with seeking support
- File scanner audio discovery
### System Tests
- Audio player functionality
- Album/artist interface navigation
- Mixed media library browsing
## Performance Considerations
### Audio Processing
- Parallel audio metadata extraction
- Efficient album art extraction
- Optimized waveform generation
- Background transcoding queue management
### Storage Optimization
- Compressed waveform storage
- Album art caching and optimization
- Efficient audio streaming with range requests
### User Experience
- Fast audio library browsing
- Quick album art loading
- Responsive audio player controls
## Future Extensions
### Phase 5+ Possibilities
- **Podcast Support** - RSS feed integration and episode management
- **Radio Streaming** - Internet radio station integration
- **Music Discovery** - Similar artist recommendations
- **Audio Bookmarks** - Detailed note-taking for audiobooks
- **Social Features** - Sharing playlists and recommendations
This phase transforms Velour from a video library into a comprehensive personal media platform while maintaining the simplicity and robustness of the existing architecture.