Much base work started
CI / scan_ruby (push) Has been cancelled
CI / scan_js (push) Has been cancelled
CI / lint (push) Has been cancelled
CI / test (push) Has been cancelled
CI / system-test (push) Has been cancelled

This commit is contained in:
Dan Milne
2025-10-31 14:36:14 +11:00
parent 4a35bf6758
commit 88a906064f
97 changed files with 5333 additions and 2774 deletions
+218 -2762
View File
File diff suppressed because it is too large Load Diff
+548
View File
@@ -0,0 +1,548 @@
# Velour Phase 1: MVP (Local Filesystem)
Phase 1 delivers a complete video library application for local files with grouping, transcoding, and playback. This is the foundation that provides immediate value.
**Architecture Note**: This phase implements a extensible MediaFile architecture using Single Table Inheritance (STI). Video inherits from MediaFile, preparing the system for future audio support in Phase 5.
## Technology Stack
### Core Components
- **Ruby on Rails 8.x** with SQLite3
- **Hotwire** (Turbo + Stimulus) for frontend
- **Solid Queue** for background jobs
- **Video.js** for video playback
- **FFmpeg** for video processing
- **Active Storage** for thumbnails/assets
- **TailwindCSS** for styling
## Database Schema (Core Models)
### Work Model
```ruby
class Work < ApplicationRecord
validates :title, presence: true
has_many :videos, dependent: :destroy
has_many :external_ids, dependent: :destroy
scope :organized, -> { where(organized: true) }
scope :unorganized, -> { where(organized: false) }
def primary_video
videos.order(created_at: :desc).first
end
end
```
### MediaFile Model (Base Class)
```ruby
class MediaFile < ApplicationRecord
# Base class for all media files using STI
include Streamable, Processable
# Common associations
belongs_to :work
belongs_to :storage_location
has_many :playback_sessions, dependent: :destroy
# Common metadata stores
store :fingerprints, accessors: [:xxhash64, :md5, :oshash, :phash]
store :media_metadata, accessors: [:duration, :codec, :bit_rate, :format]
# Common validations and methods
validates :filename, presence: true
validates :xxhash64, presence: true, uniqueness: { scope: :storage_location_id }
scope :web_compatible, -> { where(web_compatible: true) }
scope :needs_transcoding, -> { where(web_compatible: false) }
def display_title
work&.display_title || filename
end
def full_file_path
File.join(storage_location.path, filename)
end
def format_duration
# Duration formatting logic
end
end
```
### Video Model (Inherits from MediaFile)
```ruby
class Video < MediaFile
# Video-specific associations
has_many :video_assets, dependent: :destroy
# Video-specific metadata
store :video_metadata, accessors: [:width, :height, :video_codec, :audio_codec, :frame_rate]
# Video-specific methods
def resolution_label
# Resolution-based quality labeling (SD, 720p, 1080p, 4K, etc.)
end
end
```
### StorageLocation Model
```ruby
class StorageLocation < ApplicationRecord
has_many :videos, dependent: :destroy
validates :name, presence: true
validates :path, presence: true, uniqueness: true
validates :storage_type, presence: true, inclusion: { in: %w[local] }
validate :path_must_exist_and_be_readable
def accessible?
File.exist?(path) && File.readable?(path)
end
private
def path_must_exist_and_be_readable
errors.add(:path, "must exist and be readable") unless accessible?
end
end
```
## Storage Architecture (Local Only)
### Directory Structure
```bash
# User's original media directories (read-only references)
/movies/action/Die Hard (1988).mkv
/movies/anime/Your Name (2016).webm
/movies/scifi/The Matrix (1999).avi
# Velour transcoded files (same directories, web-compatible)
/movies/action/Die Hard (1988).web.mp4
/movies/anime/Your Name (2016).web.mp4
/movies/scifi/The Matrix (1999).web.mp4
# Velour managed directories
./velour_data/
├── assets/ # Active Storage for generated content
│ └── thumbnails/ # Video screenshots
└── tmp/ # Temporary processing files
└── transcodes/ # Temporary transcode storage
```
### Docker Volume Mounting with Heuristic Discovery
Users mount their video directories under `/videos` and Velour automatically discovers and categorizes them:
```yaml
# docker-compose.yml
volumes:
- /path/to/user/movies:/videos/movies:ro
- /path/to/user/tv_shows:/videos/tv:ro
- /path/to/user/documentaries:/videos/docs:ro
- /path/to/user/anime:/videos/anime:ro
- ./velour_data:/app/velour_data
```
### Automatic Storage Location Discovery
```ruby
# app/services/storage_discovery_service.rb
class StorageDiscoveryService
CATEGORIES = {
'movies' => 'Movies',
'tv' => 'TV Shows',
'tv_shows' => 'TV Shows',
'series' => 'TV Shows',
'docs' => 'Documentaries',
'documentaries' => 'Documentaries',
'anime' => 'Anime',
'cartoons' => 'Animation',
'animation' => 'Animation',
'sports' => 'Sports',
'music' => 'Music Videos',
'music_videos' => 'Music Videos',
'kids' => 'Kids Content',
'family' => 'Family Content'
}.freeze
def self.discover_and_create
base_path = '/videos'
return [] unless Dir.exist?(base_path)
discovered = []
Dir.children(base_path).each do |subdir|
dir_path = File.join(base_path, subdir)
next unless Dir.exist?(dir_path)
category = categorize_directory(subdir)
storage = StorageLocation.find_or_create_by!(
name: "#{category}: #{subdir.titleize}",
path: dir_path,
storage_type: 'local'
)
discovered << storage
end
discovered
end
def self.categorize_directory(dirname)
downcase = dirname.downcase
CATEGORIES[downcase] || CATEGORIES.find { |key, _| downcase.include?(key) }&.last || 'Other'
end
end
```
## Video Processing Pipeline
### Background Job with ActiveJob 8.1 Continuations
```ruby
class VideoProcessorJob < ApplicationJob
include ActiveJob::Statuses
def perform(video_id)
video = Video.find(video_id)
progress.update(stage: "metadata", total: 100, current: 0)
metadata = VideoMetadataExtractor.new(video.full_file_path).extract
video.update!(video_metadata: metadata)
progress.update(stage: "metadata", total: 100, current: 100)
progress.update(stage: "thumbnail", total: 100, current: 0)
generate_thumbnail(video)
progress.update(stage: "thumbnail", total: 100, current: 100)
unless video.web_compatible?
progress.update(stage: "transcode", total: 100, current: 0)
transcode_video(video)
progress.update(stage: "transcode", total: 100, current: 100)
end
progress.update(stage: "complete", total: 100, current: 100)
end
private
def generate_thumbnail(video)
# Generate thumbnail at 10% of duration
thumbnail_path = VideoTranscoder.new.extract_frame(video.full_file_path, video.duration * 0.1)
video.video_assets.create!(asset_type: "thumbnail", file: File.open(thumbnail_path))
end
def transcode_video(video)
transcoder = VideoTranscoder.new
output_path = video.full_file_path.gsub(/\.[^.]+$/, '.web.mp4')
transcoder.transcode_for_web(
input_path: video.full_file_path,
output_path: output_path,
on_progress: ->(current, total) { progress.update(current: current) }
)
video.update!(
transcoded_path: File.basename(output_path),
transcoded_permanently: true,
web_compatible: true
)
end
end
```
## Frontend Architecture
### Uninterrupted Video Playback
```erb
<!-- videos/show.html.erb -->
<turbo-frame id="video-player-frame" data-turbo-permanent>
<div class="video-container">
<video
id="video-player"
data-controller="video-player"
data-video-player-video-id-value="<%= @video.id %>"
data-video-player-start-position-value="<%= @last_position %>"
class="w-full"
controls>
<source src="<%= stream_video_path(@video) %>" type="video/mp4">
</video>
</div>
</turbo-frame>
<turbo-frame id="video-info">
<h1><%= @work.title %></h1>
<p>Duration: <%= format_duration(@video.duration) %></p>
</turbo-frame>
```
### Video Player Stimulus Controller
```javascript
// app/javascript/controllers/video_player_controller.js
import { Controller } from "@hotwired/stimulus"
export default class extends Controller {
static values = { videoId: Number, startPosition: Number }
connect() {
this.player = videojs(this.element, {
controls: true,
responsive: true,
fluid: true,
playbackRates: [0.5, 1, 1.25, 1.5, 2]
})
this.player.ready(() => {
if (this.startPositionValue > 0) {
this.player.currentTime(this.startPositionValue)
}
})
// Save position every 10 seconds
this.interval = setInterval(() => {
this.savePosition()
}, 10000)
// Save on pause
this.player.on("pause", () => this.savePosition())
}
disconnect() {
clearInterval(this.interval)
this.player.dispose()
}
savePosition() {
const position = Math.floor(this.player.currentTime())
fetch(`/videos/${this.videoIdValue}/playback-position`, {
method: "PATCH",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ position })
})
}
}
```
## Service Objects
### File Scanner Service
```ruby
class FileScannerService
def initialize(storage_location)
@storage_location = storage_location
end
def scan
return failure_result("Storage location not accessible") unless @storage_location.accessible?
video_files = find_video_files
new_videos = process_files(video_files)
success_result(new_videos)
rescue => e
failure_result(e.message)
end
private
def find_video_files
Dir.glob(File.join(@storage_location.path, "**", "*.{mp4,avi,mkv,mov,wmv,flv,webm,m4v}"))
end
def process_files(file_paths)
new_videos = []
file_paths.each do |file_path|
filename = File.basename(file_path)
next if Video.exists?(filename: filename, storage_location: @storage_location)
video = Video.create!(
filename: filename,
storage_location: @storage_location,
work: Work.find_or_create_by(title: extract_title(filename))
)
new_videos << video
VideoProcessorJob.perform_later(video.id)
end
new_videos
end
def extract_title(filename)
# Simple title extraction - can be enhanced
File.basename(filename, ".*").gsub(/[\[\(].*?[\]\)]/, "").strip
end
def success_result(videos = [])
{ success: true, videos: videos, message: "Found #{videos.length} new videos" }
end
def failure_result(message)
{ success: false, message: message }
end
end
```
### Video Transcoder Service
```ruby
class VideoTranscoder
def transcode_for_web(input_path:, output_path:, on_progress: nil)
movie = FFMPEG::Movie.new(input_path)
# Calculate progress callback
progress_callback = ->(progress) {
on_progress&.call(progress, 100)
}
movie.transcode(output_path, {
video_codec: "libx264",
audio_codec: "aac",
custom: %w[
-pix_fmt yuv420p
-preset medium
-crf 23
-movflags +faststart
-tune fastdecode
]
}, &progress_callback)
end
def extract_frame(input_path, seconds)
movie = FFMPEG::Movie.new(input_path)
output_path = "#{Rails.root}/tmp/thumbnail_#{SecureRandom.hex(8)}.jpg"
movie.screenshot(output_path, seek_time: seconds, resolution: "320x240")
output_path
end
end
```
## Routes
```ruby
Rails.application.routes.draw do
root "storage_locations#index"
resources :storage_locations, only: [:index, :show, :create, :destroy] do
member do
post :scan
end
end
resources :works, only: [:index, :show] do
resources :videos, only: [:show]
end
resources :videos, only: [] do
member do
get :stream
patch :playback_position
post :retry_processing
end
resources :playback_sessions, only: [:create]
end
# Real-time job progress
resources :jobs, only: [:show] do
member do
get :progress
end
end
end
```
## Implementation Sub-Phases
### Phase 1A: Core Foundation (Week 1-2)
1. Generate models with migrations
2. Implement model validations and associations
3. Create storage adapter pattern
4. Create FileScannerService
5. Basic UI for storage locations and video listing
### Phase 1B: Video Playback (Week 3)
1. Video streaming controller with byte-range support
2. Video.js integration with Stimulus controller
3. Playback session tracking
4. Resume functionality
### Phase 1C: Processing Pipeline (Week 4)
1. Video metadata extraction with FFmpeg
2. Background job processing with progress tracking
3. Thumbnail generation and storage
4. Processing UI with status indicators
### Phase 1D: Works & Grouping (Week 5)
1. Duplicate detection by file hash
2. Manual grouping interface
3. Works display with version selection
4. Search, filtering, and pagination
## Configuration
### Environment Variables
```bash
# Required
RAILS_ENV=development
RAILS_MASTER_KEY=your_master_key
# Video processing
FFMPEG_PATH=/usr/bin/ffmpeg
FFPROBE_PATH=/usr/bin/ffprobe
# Storage
VIDEOS_PATH=./velour_data/videos
MAX_TRANSCODE_SIZE_GB=50
# Background jobs (SolidQueue runs with defaults)
SOLID_QUEUE_PROCESSES="*:2" # 2 workers for all queues
```
### Docker Configuration
```dockerfile
FROM ruby:3.3-alpine
# Install FFmpeg and other dependencies
RUN apk add --no-cache \
ffmpeg \
imagemagick \
sqlite-dev \
nodejs \
npm
# Install Rails and other gems
COPY Gemfile* /app/
RUN bundle install
# Copy application code
COPY . /app/
# Create directories
RUN mkdir -p /app/velour_data/{assets,tmp}
EXPOSE 3000
CMD ["rails", "server", "-b", "0.0.0.0"]
```
## Testing Strategy
### Model Tests
- Model validations and associations
- Scopes and class methods
- JSON store accessor functionality
### Service Tests
- FileScannerService with sample directory
- VideoTranscoder service methods
- Background job processing
### System Tests
- Video playback functionality
- File scanning workflow
- Work grouping interface
This Phase 1 delivers a complete, useful video library application that provides immediate value while establishing the foundation for future enhancements.
+486
View File
@@ -0,0 +1,486 @@
# Velour Phase 2: Authentication & Multi-User
Phase 2 adds user management, authentication, and multi-user support while maintaining the simplicity of the core application.
## Authentication Architecture
### Rails Authentication Generators + OIDC Extension
We use Rails' built-in authentication generators as the foundation, extended with OIDC support for enterprise environments.
### User Model
```ruby
class User < ApplicationRecord
# Include default devise modules or Rails authentication
# Devise modules: :database_authenticatable, :registerable,
# :recoverable, :rememberable, :validatable
has_many :playback_sessions, dependent: :destroy
has_many :user_preferences, dependent: :destroy
validates :email, presence: true, uniqueness: true
enum role: { user: 0, admin: 1 }
def admin?
role == "admin" || email == ENV.fetch("ADMIN_EMAIL", "").downcase
end
def can_manage_storage?
admin?
end
def can_manage_users?
admin?
end
end
```
### Authentication Setup
```bash
# Generate Rails authentication
rails generate authentication
# Add OIDC support
gem 'omniauth-openid-connect'
bundle install
```
### OIDC Configuration
```ruby
# config/initializers/omniauth.rb
Rails.application.config.middleware.use OmniAuth::Builder do
provider :openid_connect, {
name: :oidc,
issuer: ENV['OIDC_ISSUER'],
client_id: ENV['OIDC_CLIENT_ID'],
client_secret: ENV['OIDC_CLIENT_SECRET'],
scope: [:openid, :email, :profile],
response_type: :code,
client_options: {
identifier: ENV['OIDC_CLIENT_ID'],
secret: ENV['OIDC_CLIENT_SECRET'],
redirect_uri: "#{ENV['RAILS_HOST']}/auth/oidc/callback"
}
}
end
```
## First User Bootstrap Flow
### Initial Setup
```ruby
# db/seeds.rb
admin_email = ENV.fetch('ADMIN_EMAIL', 'admin@velour.local')
User.find_or_create_by!(email: admin_email) do |user|
user.password = SecureRandom.hex(16)
user.role = :admin
puts "Created admin user: #{admin_email}"
puts "Password: #{user.password}"
end
```
### First Login Controller
```ruby
class FirstSetupController < ApplicationController
before_action :ensure_no_users_exist
before_action :require_admin_setup, only: [:create_admin]
def show
@user = User.new
end
def create_admin
@user = User.new(user_params)
@user.role = :admin
if @user.save
session[:user_id] = @user.id
redirect_to root_path, notice: "Admin account created successfully!"
else
render :show, status: :unprocessable_entity
end
end
private
def ensure_no_users_exist
redirect_to root_path if User.exists?
end
def require_admin_setup
redirect_to first_setup_path unless ENV.key?('ADMIN_EMAIL')
end
def user_params
params.require(:user).permit(:email, :password, :password_confirmation)
end
end
```
## User Management
### User Preferences
```ruby
class UserPreference < ApplicationRecord
belongs_to :user
store :settings, coder: JSON, accessors: [
:default_video_quality,
:auto_play_next,
:subtitle_language,
:theme
]
validates :user_id, presence: true, uniqueness: true
end
```
### Per-User Playback Sessions
```ruby
class PlaybackSession < ApplicationRecord
belongs_to :user
belongs_to :video
scope :for_user, ->(user) { where(user: user) }
scope :recent, -> { order(updated_at: :desc) }
def self.resume_position_for(video, user)
for_user(user).where(video: video).last&.position || 0
end
end
```
## Authorization & Security
### Model-Level Authorization
```ruby
# app/models/concerns/authorizable.rb
module Authorizable
extend ActiveSupport::Concern
def viewable_by?(user)
true # All videos viewable by all users
end
def editable_by?(user)
user.admin?
end
end
# Include in Video and Work models
class Video < ApplicationRecord
include Authorizable
# ... rest of model
end
```
### Controller Authorization
```ruby
class ApplicationController < ActionController::Base
before_action :authenticate_user!
before_action :set_current_user
private
def set_current_user
Current.user = current_user
end
def require_admin
redirect_to root_path, alert: "Access denied" unless current_user&.admin?
end
end
class StorageLocationsController < ApplicationController
before_action :require_admin, except: [:index, :show]
# ... rest of controller
end
```
## Updated Controllers for Multi-User
### Videos Controller
```ruby
class VideosController < ApplicationController
before_action :set_video, only: [:show, :stream, :update_position]
before_action :authorize_video
def show
@work = @video.work
@last_position = PlaybackSession.resume_position_for(@video, current_user)
# Create playback session for tracking
@playback_session = current_user.playback_sessions.create!(
video: @video,
position: @last_position
)
end
def stream
unless @video.viewable_by?(current_user)
head :forbidden
return
end
send_file @video.web_stream_path,
type: "video/mp4",
disposition: "inline",
range: request.headers['Range']
end
def update_position
current_user.playback_sessions.where(video: @video).last&.update!(
position: params[:position],
completed: params[:completed] || false
)
head :ok
end
private
def set_video
@video = Video.find(params[:id])
end
def authorize_video
head :forbidden unless @video.viewable_by?(current_user)
end
end
```
### Storage Locations Controller (Admin Only)
```ruby
class StorageLocationsController < ApplicationController
before_action :require_admin, except: [:index, :show]
before_action :set_storage_location, only: [:show, :destroy, :scan]
def index
@storage_locations = StorageLocation.accessible.order(:name)
end
def show
@videos = @storage_location.videos.includes(:work).order(:filename)
end
def create
@storage_location = StorageLocation.new(storage_location_params)
if @storage_location.save
redirect_to @storage_location, notice: 'Storage location was successfully created.'
else
render :new, status: :unprocessable_entity
end
end
def destroy
if @storage_location.videos.exists?
redirect_to @storage_location, alert: 'Cannot delete storage location with videos.'
else
@storage_location.destroy
redirect_to storage_locations_path, notice: 'Storage location was successfully deleted.'
end
end
def scan
service = FileScannerService.new(@storage_location)
result = service.scan
if result[:success]
redirect_to @storage_location, notice: result[:message]
else
redirect_to @storage_location, alert: result[:message]
end
end
private
def set_storage_location
@storage_location = StorageLocation.find(params[:id])
end
def storage_location_params
params.require(:storage_location).permit(:name, :path, :storage_type)
end
end
```
## User Interface Updates
### User Navigation
```erb
<!-- app/views/layouts/application.html.erb -->
<nav class="bg-gray-800 text-white p-4">
<div class="container mx-auto flex justify-between items-center">
<%= link_to 'Velour', root_path, class: 'text-xl font-bold' %>
<div class="flex items-center space-x-4">
<%= link_to 'Library', works_path, class: 'hover:text-gray-300' %>
<%= link_to 'Storage', storage_locations_path, class: 'hover:text-gray-300' if current_user.admin? %>
<div class="relative">
<button data-action="click->dropdown#toggle" class="flex items-center">
<%= current_user.email %>
<svg class="w-4 h-4 ml-1" fill="currentColor" viewBox="0 0 20 20">
<path fill-rule="evenodd" d="M5.293 7.293a1 1 0 011.414 0L10 10.586l3.293-3.293a1 1 0 111.414 1.414l-4 4a1 1 0 01-1.414 0l-4-4a1 1 0 010-1.414z" clip-rule="evenodd"/>
</svg>
</button>
<div data-dropdown-target="menu" class="hidden absolute right-0 mt-2 w-48 bg-white rounded-md shadow-lg py-1 z-50">
<%= link_to 'Settings', edit_user_registration_path, class: 'block px-4 py-2 text-sm text-gray-700 hover:bg-gray-100' %>
<%= link_to 'Admin Panel', admin_path, class: 'block px-4 py-2 text-sm text-gray-700 hover:bg-gray-100' if current_user.admin? %>
<%= button_to 'Sign Out', destroy_user_session_path, method: :delete, class: 'block w-full text-left px-4 py-2 text-sm text-gray-700 hover:bg-gray-100' %>
</div>
</div>
</div>
</div>
</nav>
```
### Admin Panel
```erb
<!-- app/views/admin/dashboard.html.erb -->
<div class="container mx-auto p-6">
<h1 class="text-3xl font-bold mb-6">Admin Dashboard</h1>
<div class="grid grid-cols-1 md:grid-cols-3 gap-6">
<div class="bg-white p-6 rounded-lg shadow">
<h2 class="text-xl font-semibold mb-4">Users</h2>
<p class="text-3xl font-bold text-blue-600"><%= User.count %></p>
<p class="text-gray-600">Total users</p>
<%= link_to 'Manage Users', admin_users_path, class: 'mt-2 text-blue-500 hover:text-blue-700' %>
</div>
<div class="bg-white p-6 rounded-lg shadow">
<h2 class="text-xl font-semibold mb-4">Videos</h2>
<p class="text-3xl font-bold text-green-600"><%= Video.count %></p>
<p class="text-gray-600">Total videos</p>
<%= link_to 'View Library', works_path, class: 'mt-2 text-green-500 hover:text-green-700' %>
</div>
<div class="bg-white p-6 rounded-lg shadow">
<h2 class="text-xl font-semibold mb-4">Storage</h2>
<p class="text-3xl font-bold text-purple-600"><%= StorageLocation.count %></p>
<p class="text-gray-600">Storage locations</p>
<%= link_to 'Manage Storage', storage_locations_path, class: 'mt-2 text-purple-500 hover:text-purple-700' %>
</div>
</div>
<div class="mt-8 bg-white p-6 rounded-lg shadow">
<h2 class="text-xl font-semibold mb-4">System Status</h2>
<div class="space-y-2">
<div class="flex justify-between">
<span>Background Jobs:</span>
<span class="font-mono"><%= SolidQueue::Job.count %></span>
</div>
<div class="flex justify-between">
<span>Processing Jobs:</span>
<span class="font-mono"><%= SolidQueue::Job.where(finished_at: nil).count %></span>
</div>
<div class="flex justify-between">
<span>Failed Jobs:</span>
<span class="font-mono"><%= SolidQueue::FailedExecution.count %></span>
</div>
</div>
</div>
</div>
```
## Environment Configuration
### New Environment Variables
```bash
# Authentication
ADMIN_EMAIL=admin@yourdomain.com
RAILS_HOST=https://your-velour-domain.com
# OIDC (optional)
OIDC_ISSUER=https://your-oidc-provider.com
OIDC_CLIENT_ID=your_client_id
OIDC_CLIENT_SECRET=your_client_secret
# Session management
RAILS_SESSION_COOKIE_SECURE=true
RAILS_SESSION_COOKIE_SAME_SITE=lax
```
## Testing for Phase 2
### Authentication Tests
```ruby
# test/integration/authentication_test.rb
class AuthenticationTest < ActionDispatch::IntegrationTest
test "first user can create admin account" do
User.delete_all
get first_setup_path
assert_response :success
post first_setup_path, params: {
user: {
email: "admin@example.com",
password: "password123",
password_confirmation: "password123"
}
}
assert_redirected_to root_path
assert_equal "admin@example.com", User.last.email
assert User.last.admin?
end
test "regular users cannot access admin features" do
user = users(:regular)
sign_in user
get storage_locations_path
assert_redirected_to root_path
post storage_locations_path, params: {
storage_location: {
name: "Test",
path: "/test",
storage_type: "local"
}
}
assert_redirected_to root_path
end
end
```
## Migration from Phase 1
### Database Migration
```ruby
class AddAuthenticationToUsers < ActiveRecord::Migration[7.1]
def change
create_table :users do |t|
t.string :email, null: false, index: { unique: true }
t.string :encrypted_password, null: false
t.string :role, default: 'user', null: false
t.timestamps null: false
end
create_table :user_preferences do |t|
t.references :user, null: false, foreign_key: true
t.json :settings, default: {}
t.timestamps null: false
end
# Add user_id to existing playback_sessions
add_reference :playback_sessions, :user, null: false, foreign_key: true
# Create first admin if ADMIN_EMAIL is set
if ENV.key?('ADMIN_EMAIL')
User.create!(
email: ENV['ADMIN_EMAIL'],
password: SecureRandom.hex(16),
role: 'admin'
)
end
end
end
```
Phase 2 provides a complete multi-user system while maintaining the simplicity of Phase 1. Users can have personal playback history, and administrators can manage the system through a clean interface.
+773
View File
@@ -0,0 +1,773 @@
# Velour Phase 3: Remote Sources & Import
Phase 3 extends the video library to support remote storage sources like S3, JellyFin servers, and web directories. This allows users to access and import videos from multiple locations.
## Extended Storage Architecture
### New Storage Location Types
```ruby
class StorageLocation < ApplicationRecord
has_many :videos, dependent: :destroy
validates :name, presence: true
validates :storage_type, presence: true, inclusion: { in: %w[local s3 jellyfin web] }
store :configuration, accessors: [
# S3 configuration
:bucket, :region, :access_key_id, :secret_access_key, :endpoint,
# JellyFin configuration
:server_url, :api_key, :username,
# Web directory configuration
:base_url, :auth_type, :username, :password, :headers
], coder: JSON
# Storage-type specific validations
validate :validate_s3_configuration, if: -> { s3? }
validate :validate_jellyfin_configuration, if: -> { jellyfin? }
validate :validate_web_configuration, if: -> { web? }
enum storage_type: { local: 0, s3: 1, jellyfin: 2, web: 3 }
def accessible?
case storage_type
when 'local'
File.exist?(path) && File.readable?(path)
when 's3'
s3_client&.bucket(bucket)&.exists?
when 'jellyfin'
jellyfin_client&.ping?
when 'web'
web_accessible?
else
false
end
end
def scanner
case storage_type
when 'local'
LocalFileScanner.new(self)
when 's3'
S3Scanner.new(self)
when 'jellyfin'
JellyFinScanner.new(self)
when 'web'
WebDirectoryScanner.new(self)
end
end
def streamer
case storage_type
when 'local'
LocalStreamer.new(self)
when 's3'
S3Streamer.new(self)
when 'jellyfin'
JellyFinStreamer.new(self)
when 'web'
WebStreamer.new(self)
end
end
private
def validate_s3_configuration
%w[bucket region access_key_id secret_access_key].each do |field|
errors.add(:configuration, "#{field} is required for S3 storage") if send(field).blank?
end
end
def validate_jellyfin_configuration
%w[server_url api_key].each do |field|
errors.add(:configuration, "#{field} is required for JellyFin storage") if send(field).blank?
end
end
def validate_web_configuration
errors.add(:configuration, "base_url is required for web storage") if base_url.blank?
end
end
```
## S3 Storage Implementation
### S3 Scanner Service
```ruby
class S3Scanner
def initialize(storage_location)
@storage_location = storage_location
@client = s3_client
end
def scan
return failure_result("S3 bucket not accessible") unless @storage_location.accessible?
video_files = find_video_files_in_s3
new_videos = process_s3_files(video_files)
success_result(new_videos)
rescue Aws::Errors::ServiceError => e
failure_result("S3 error: #{e.message}")
end
private
def s3_client
@s3_client ||= Aws::S3::Client.new(
region: @storage_location.region,
access_key_id: @storage_location.access_key_id,
secret_access_key: @storage_location.secret_access_key,
endpoint: @storage_location.endpoint # Optional for S3-compatible services
)
end
def find_video_files_in_s3
bucket = Aws::S3::Bucket.new(@storage_location.bucket, client: s3_client)
video_extensions = %w[.mp4 .avi .mkv .mov .wmv .flv .webm .m4v]
bucket.objects(prefix: "")
.select { |obj| video_extensions.any? { |ext| obj.key.end_with?(ext) } }
.to_a
end
def process_s3_files(s3_objects)
new_videos = []
s3_objects.each do |s3_object|
filename = File.basename(s3_object.key)
next if Video.exists?(filename: filename, storage_location: @storage_location)
video = Video.create!(
filename: filename,
storage_location: @storage_location,
work: Work.find_or_create_by(title: extract_title(filename)),
file_size: s3_object.size,
video_metadata: {
remote_url: s3_object.key,
last_modified: s3_object.last_modified
}
)
new_videos << video
VideoProcessorJob.perform_later(video.id)
end
new_videos
end
def extract_title(filename)
File.basename(filename, ".*").gsub(/[\[\(].*?[\]\)]/, "").strip
end
def success_result(videos = [])
{ success: true, videos: videos, message: "Found #{videos.length} new videos in S3" }
end
def failure_result(message)
{ success: false, message: message }
end
end
```
### S3 Streamer
```ruby
class S3Streamer
def initialize(storage_location)
@storage_location = storage_location
@client = s3_client
end
def stream(video, range: nil)
s3_object = s3_object_for_video(video)
if range
# Handle byte-range requests for seeking
range_header = "bytes=#{range}"
resp = @client.get_object(
bucket: @storage_location.bucket,
key: video.video_metadata['remote_url'],
range: range_header
)
{
body: resp.body,
status: 206, # Partial content
headers: {
'Content-Range' => "bytes #{range}/#{s3_object.size}",
'Content-Length' => resp.content_length,
'Accept-Ranges' => 'bytes',
'Content-Type' => 'video/mp4'
}
}
else
resp = @client.get_object(
bucket: @storage_location.bucket,
key: video.video_metadata['remote_url']
)
{
body: resp.body,
status: 200,
headers: {
'Content-Length' => resp.content_length,
'Content-Type' => 'video/mp4'
}
}
end
end
def presigned_url(video, expires_in: 1.hour)
signer = Aws::S3::Presigner.new(client: @client)
signer.presigned_url(
:get_object,
bucket: @storage_location.bucket,
key: video.video_metadata['remote_url'],
expires_in: expires_in.to_i
)
end
private
def s3_client
@s3_client ||= Aws::S3::Client.new(
region: @storage_location.region,
access_key_id: @storage_location.access_key_id,
secret_access_key: @storage_location.secret_access_key,
endpoint: @storage_location.endpoint
)
end
def s3_object_for_video(video)
@client.get_object(
bucket: @storage_location.bucket,
key: video.video_metadata['remote_url']
)
end
end
```
## JellyFin Integration
### JellyFin Client
```ruby
class JellyFinClient
def initialize(server_url:, api_key:, username: nil)
@server_url = server_url.chomp('/')
@api_key = api_key
@username = username
@http = Faraday.new(url: @server_url) do |faraday|
faraday.headers['X-Emby-Token'] = @api_key
faraday.adapter Faraday.default_adapter
end
end
def ping?
response = @http.get('/System/Ping')
response.success?
rescue
false
end
def libraries
response = @http.get('/Library/VirtualFolders')
JSON.parse(response.body)
end
def movies(library_id = nil)
path = library_id ? "/Users/#{user_id}/Items?ParentId=#{library_id}&IncludeItemTypes=Movie&Recursive=true" : "/Users/#{user_id}/Items?IncludeItemTypes=Movie&Recursive=true"
response = @http.get(path)
JSON.parse(response.body)['Items']
end
def tv_shows(library_id = nil)
path = library_id ? "/Users/#{user_id}/Items?ParentId=#{library_id}&IncludeItemTypes=Series&Recursive=true" : "/Users/#{user_id}/Items?IncludeItemTypes=Series&Recursive=true"
response = @http.get(path)
JSON.parse(response.body)['Items']
end
def episodes(show_id)
response = @http.get("/Shows/#{show_id}/Episodes?UserId=#{user_id}")
JSON.parse(response.body)['Items']
end
def streaming_url(item_id)
"#{@server_url}/Videos/#{item_id}/stream?Static=true&MediaSourceId=#{item_id}&DeviceId=Velour&api_key=#{@api_key}"
end
def item_details(item_id)
response = @http.get("/Users/#{user_id}/Items/#{item_id}")
JSON.parse(response.body)
end
private
def user_id
@user_id ||= begin
response = @http.get('/Users')
users = JSON.parse(response.body)
if @username
user = users.find { |u| u['Name'] == @username }
user&.dig('Id') || users.first['Id']
else
users.first['Id']
end
end
end
end
```
### JellyFin Scanner
```ruby
class JellyFinScanner
def initialize(storage_location)
@storage_location = storage_location
@client = jellyfin_client
end
def scan
return failure_result("JellyFin server not accessible") unless @storage_location.accessible?
movies = @client.movies
shows = @client.tv_shows
episodes = []
shows.each do |show|
episodes.concat(@client.episodes(show['Id']))
end
all_items = movies + episodes
new_videos = process_jellyfin_items(all_items)
success_result(new_videos)
rescue => e
failure_result("JellyFin error: #{e.message}")
end
private
def jellyfin_client
@client ||= JellyFinClient.new(
server_url: @storage_location.server_url,
api_key: @storage_location.api_key,
username: @storage_location.username
)
end
def process_jellyfin_items(items)
new_videos = []
items.each do |item|
next unless item['MediaType'] == 'Video'
title = item['Name']
year = item['ProductionYear']
work_title = year ? "#{title} (#{year})" : title
work = Work.find_or_create_by(title: work_title) do |w|
w.year = year
w.description = item['Overview']
end
video = Video.find_or_initialize_by(
filename: item['Id'],
storage_location: @storage_location
)
if video.new_record?
video.update!(
work: work,
video_metadata: {
jellyfin_id: item['Id'],
media_type: item['Type'],
runtime: item['RunTimeTicks'] ? item['RunTimeTicks'] / 10_000_000 : nil,
premiere_date: item['PremiereDate'],
community_rating: item['CommunityRating'],
genres: item['Genres']
}
)
new_videos << video
VideoProcessorJob.perform_later(video.id)
end
end
new_videos
end
def success_result(videos = [])
{ success: true, videos: videos, message: "Found #{videos.length} new videos from JellyFin" }
end
def failure_result(message)
{ success: false, message: message }
end
end
```
### JellyFin Streamer
```ruby
class JellyFinStreamer
def initialize(storage_location)
@storage_location = storage_location
@client = jellyfin_client
end
def stream(video, range: nil)
jellyfin_id = video.video_metadata['jellyfin_id']
stream_url = @client.streaming_url(jellyfin_id)
# For JellyFin, we typically proxy the stream
if range
proxy_stream_with_range(stream_url, range)
else
proxy_stream(stream_url)
end
end
private
def jellyfin_client
@client ||= JellyFinClient.new(
server_url: @storage_location.server_url,
api_key: @storage_location.api_key,
username: @storage_location.username
)
end
def proxy_stream(url)
response = Faraday.get(url)
{
body: response.body,
status: response.status,
headers: response.headers
}
end
def proxy_stream_with_range(url, range)
response = Faraday.get(url, nil, { 'Range' => "bytes=#{range}" })
{
body: response.body,
status: response.status,
headers: response.headers
}
end
end
```
## Video Import System
### Import Job with Progress Tracking
```ruby
class VideoImportJob < ApplicationJob
include ActiveJob::Statuses
def perform(video_id, destination_storage_location_id)
video = Video.find(video_id)
destination = StorageLocation.find(destination_storage_location_id)
progress.update(stage: "download", total: 100, current: 0)
# Download file from source
downloaded_file = download_video(video, destination) do |current, total|
progress.update(current: (current.to_f / total * 50).to_i) # Download is 50% of progress
end
progress.update(stage: "process", total: 100, current: 50)
# Create new video record in destination
new_video = Video.create!(
filename: video.filename,
storage_location: destination,
work: video.work,
file_size: video.file_size
)
# Copy file to destination
destination_path = File.join(destination.path, video.filename)
FileUtils.cp(downloaded_file.path, destination_path)
# Process the new video
VideoProcessorJob.perform_later(new_video.id)
progress.update(stage: "complete", total: 100, current: 100)
# Clean up temp file
File.delete(downloaded_file.path)
end
private
def download_video(video, destination, &block)
case video.storage_location.storage_type
when 's3'
download_from_s3(video, &block)
when 'jellyfin'
download_from_jellyfin(video, &block)
when 'web'
download_from_web(video, &block)
else
raise "Unsupported import from #{video.storage_location.storage_type}"
end
end
def download_from_s3(video, &block)
temp_file = Tempfile.new(['video_import', File.extname(video.filename)])
s3_client = Aws::S3::Client.new(
region: video.storage_location.region,
access_key_id: video.storage_location.access_key_id,
secret_access_key: video.storage_location.secret_access_key
)
s3_client.get_object(
bucket: video.storage_location.bucket,
key: video.video_metadata['remote_url'],
response_target: temp_file.path
) do |chunk|
yield(chunk.bytes_written, chunk.size) if block_given?
end
temp_file
end
def download_from_jellyfin(video, &block)
temp_file = Tempfile.new(['video_import', File.extname(video.filename)])
jellyfin_id = video.video_metadata['jellyfin_id']
client = JellyFinClient.new(
server_url: video.storage_location.server_url,
api_key: video.storage_location.api_key
)
stream_url = client.streaming_url(jellyfin_id)
# Download with progress tracking
uri = URI(stream_url)
Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https') do |http|
request = Net::HTTP::Get.new(uri)
http.request(request) do |response|
total_size = response['Content-Length'].to_i
downloaded = 0
response.read_body do |chunk|
temp_file.write(chunk)
downloaded += chunk.bytesize
yield(downloaded, total_size) if block_given?
end
end
end
temp_file
end
end
```
### Import UI
```erb
<!-- app/views/videos/_import_button.html.erb -->
<% if video.storage_location.remote? && current_user.admin? %>
<div data-controller="import-dialog">
<button
data-action="click->import-dialog#show"
class="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded">
Import to Local Storage
</button>
<div
data-import-dialog-target="dialog"
class="hidden fixed inset-0 bg-gray-600 bg-opacity-50 overflow-y-auto h-full w-full z-50">
<div class="relative top-20 mx-auto p-5 border w-96 shadow-lg rounded-md bg-white">
<h3 class="text-lg font-bold text-gray-900 mb-4">Import Video</h3>
<p class="text-gray-600 mb-4">
Import "<%= video.filename %>" to a local storage location for offline access and transcoding.
</p>
<div class="mb-4">
<label class="block text-gray-700 text-sm font-bold mb-2">
Destination Storage:
</label>
<select
name="destination_storage_location_id"
data-import-dialog-target="destination"
class="shadow appearance-none border rounded w-full py-2 px-3 text-gray-700 leading-tight">
<% StorageLocation.local.accessible.each do |location| %>
<option value="<%= location.id %>"><%= location.name %></option>
<% end %>
</select>
</div>
<div class="flex justify-end space-x-2">
<button
data-action="click->import-dialog#hide"
class="bg-gray-500 hover:bg-gray-700 text-white font-bold py-2 px-4 rounded">
Cancel
</button>
<button
data-action="click->import-dialog#import"
data-video-id="<%= video.id %>"
class="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded">
Import
</button>
</div>
</div>
</div>
<!-- Progress display -->
<div
data-import-dialog-target="progress"
class="hidden mt-2">
<div class="bg-blue-100 border-l-4 border-blue-500 text-blue-700 p-4">
<p class="font-bold">Importing video...</p>
<div class="mt-2">
<div class="bg-blue-200 rounded-full h-2">
<div
data-import-dialog-target="progressBar"
class="bg-blue-600 h-2 rounded-full transition-all duration-300"
style="width: 0%"></div>
</div>
<p class="text-sm mt-1" data-import-dialog-target="progressText">Starting...</p>
</div>
</div>
</div>
</div>
<% end %>
```
### Import Stimulus Controller
```javascript
// app/javascript/controllers/import_dialog_controller.js
import { Controller } from "@hotwired/stimulus"
import { get } from "@rails/request.js"
export default class extends Controller {
static targets = ["dialog", "progress", "progressBar", "progressText", "destination"]
show() {
this.dialogTarget.classList.remove("hidden")
}
hide() {
this.dialogTarget.classList.add("hidden")
}
async import(event) {
const videoId = event.target.dataset.videoId
const destinationId = this.destinationTarget.value
this.hide()
this.progressTarget.classList.remove("hidden")
try {
// Start import job
const response = await post("/videos/import", {
body: JSON.stringify({
video_id: videoId,
destination_storage_location_id: destinationId
})
})
const { jobId } = await response.json
// Poll for progress
this.pollProgress(jobId)
} catch (error) {
console.error("Import failed:", error)
this.progressTarget.innerHTML = `
<div class="bg-red-100 border-l-4 border-red-500 text-red-700 p-4">
<p class="font-bold">Import failed</p>
<p class="text-sm">${error.message}</p>
</div>
`
}
}
async pollProgress(jobId) {
const updateProgress = async () => {
try {
const response = await get(`/jobs/${jobId}/progress`)
const progress = await response.json
this.progressBarTarget.style.width = `${progress.current}%`
this.progressTextTarget.textContent = `${progress.stage}: ${progress.current}%`
if (progress.stage === "complete") {
this.progressTarget.innerHTML = `
<div class="bg-green-100 border-l-4 border-green-500 text-green-700 p-4">
<p class="font-bold">Import complete!</p>
</div>
`
setTimeout(() => {
window.location.reload()
}, 2000)
} else if (progress.stage === "failed") {
this.progressTarget.innerHTML = `
<div class="bg-red-100 border-l-4 border-red-500 text-red-700 p-4">
<p class="font-bold">Import failed</p>
<p class="text-sm">${progress.error || "Unknown error"}</p>
</div>
`
} else {
setTimeout(updateProgress, 1000)
}
} catch (error) {
console.error("Failed to get progress:", error)
setTimeout(updateProgress, 1000)
}
}
updateProgress()
}
}
```
## Environment Configuration
### New Environment Variables
```bash
# S3 Configuration
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_DEFAULT_REGION=us-east-1
# Import Settings
MAX_IMPORT_SIZE_GB=10
IMPORT_TIMEOUT_MINUTES=60
# Rate limiting
JELLYFIN_RATE_LIMIT_DELAY=1 # seconds between requests
```
### New Gems
```ruby
# Gemfile
gem 'aws-sdk-s3', '~> 1'
gem 'faraday', '~> 2.0'
gem 'httparty', '~> 0.21'
```
## Routes for Phase 3
```ruby
# Add to existing routes
resources :videos, only: [] do
member do
post :import
end
end
namespace :admin do
resources :storage_locations do
member do
post :test_connection
end
end
end
```
Phase 3 enables users to access video libraries from multiple remote sources while maintaining a unified interface. The import system allows bringing remote videos into local storage for offline access and transcoding.
+636
View File
@@ -0,0 +1,636 @@
# Velour Phase 4: Federation
Phase 4 enables federation between multiple Velour instances, allowing users to share and access video libraries across different servers while maintaining security and access controls.
## Federation Architecture
### Overview
Velour federation allows instances to:
- Share video libraries with other trusted instances
- Stream videos from remote servers
- Sync metadata and work information
- Maintain access control and authentication
### Federation Storage Location Type
```ruby
class StorageLocation < ApplicationRecord
# ... existing code ...
validates :storage_type, inclusion: { in: %w[local s3 jellyfin web velour] }
store :configuration, accessors: [
# Existing configurations...
# Velour federation configuration
:remote_instance_url, :api_key, :instance_name, :trusted_instances
], coder: JSON
enum storage_type: { local: 0, s3: 1, jellyfin: 2, web: 3, velour: 4 }
def federation_client
return nil unless velour?
@federation_client ||= VelourFederationClient.new(
instance_url: remote_instance_url,
api_key: api_key
)
end
def accessible?
case storage_type
# ... existing cases ...
when 'velour'
federation_client&.ping?
else
false
end
end
def scanner
case storage_type
# ... existing cases ...
when 'velour'
VelourFederationScanner.new(self)
else
super
end
end
def streamer
case storage_type
# ... existing cases ...
when 'velour'
VelourFederationStreamer.new(self)
else
super
end
end
end
```
## Federation API Authentication
### API Key Management
```ruby
class ApiKey < ApplicationRecord
belongs_to :user
has_many :federation_connections, dependent: :destroy
validates :key, presence: true, uniqueness: true
validates :name, presence: true
validates :permissions, presence: true
store :permissions, coder: JSON, accessors: [:can_read, :can_stream, :can_metadata]
before_validation :generate_key, on: :create
def self.generate_key
SecureRandom.hex(32)
end
private
def generate_key
self.key ||= self.class.generate_key
end
end
```
### Federation Connections
```ruby
class FederationConnection < ApplicationRecord
belongs_to :api_key
belongs_to :storage_location
validates :remote_instance_url, presence: true, uniqueness: { scope: :api_key_id }
validates :status, inclusion: { in: %w[pending active suspended rejected] }
enum status: { pending: 0, active: 1, suspended: 2, rejected: 3 }
def active?
status == "active"
end
end
```
## Federation Client
### Velour Federation Client
```ruby
class VelourFederationClient
def initialize(instance_url:, api_key:)
@instance_url = instance_url.chomp('/')
@api_key = api_key
@http = Faraday.new(url: @instance_url) do |faraday|
faraday.headers['Authorization'] = "Bearer #{@api_key}"
faraday.headers['User-Agent'] = "Velour/#{Velour::VERSION}"
faraday.request :json
faraday.response :json
faraday.adapter Faraday.default_adapter
end
end
def ping?
response = @http.get('/api/v1/ping')
response.success?
rescue
false
end
def instance_info
response = @http.get('/api/v1/instance')
response.success? ? response.body : nil
end
def works(page: 1, per_page: 50)
response = @http.get('/api/v1/works', params: {
page: page,
per_page: per_page
})
response.success? ? response.body : []
end
def work_details(work_id)
response = @http.get("/api/v1/works/#{work_id}")
response.success? ? response.body : nil
end
def video_stream_url(video_id)
"#{@instance_url}/api/v1/videos/#{video_id}/stream"
end
def video_metadata(video_id)
response = @http.get("/api/v1/videos/#{video_id}")
response.success? ? response.body : nil
end
def search_videos(query:, page: 1, per_page: 20)
response = @http.get('/api/v1/videos/search', params: {
query: query,
page: page,
per_page: per_page
})
response.success? ? response.body : []
end
end
```
## Federation Scanner
### Velour Federation Scanner
```ruby
class VelourFederationScanner
def initialize(storage_location)
@storage_location = storage_location
@client = @storage_location.federation_client
end
def scan
return failure_result("Remote Velour instance not accessible") unless @storage_location.accessible?
instance_info = @client.instance_info
return failure_result("Failed to get instance info") unless instance_info
works = @client.works(per_page: 100) # Start with first 100 works
new_videos = process_remote_works(works)
success_result(new_videos, instance_info)
rescue => e
failure_result("Federation error: #{e.message}")
end
private
def process_remote_works(works)
new_videos = []
works.each do |work_data|
# Create or find local work
work = Work.find_or_create_by(
title: work_data['title'],
year: work_data['year']
) do |w|
w.description = work_data['description']
w.director = work_data['director']
w.rating = work_data['rating']
end
# Process videos for this work
work_data['videos'].each do |video_data|
video = Video.find_or_initialize_by(
filename: video_data['id'], # Use remote ID as filename
storage_location: @storage_location
)
if video.new_record?
video.update!(
work: work,
file_size: video_data['file_size'],
web_compatible: video_data['web_compatible'],
video_metadata: {
remote_video_id: video_data['id'],
remote_instance_url: @storage_location.remote_instance_url,
duration: video_data['duration'],
width: video_data['width'],
height: video_data['height'],
video_codec: video_data['video_codec'],
audio_codec: video_data['audio_codec']
},
fingerprints: video_data['fingerprints']
)
new_videos << video
# Note: We don't process remote videos locally
# We just catalog them for streaming
end
end
end
new_videos
end
def success_result(videos = [], instance_info = {})
{
success: true,
videos: videos,
message: "Found #{videos.length} videos from #{@storage_location.instance_name}",
instance_info: instance_info
}
end
def failure_result(message)
{ success: false, message: message }
end
end
```
## Federation Streamer
### Velour Federation Streamer
```ruby
class VelourFederationStreamer
def initialize(storage_location)
@storage_location = storage_location
@client = @storage_location.federation_client
end
def stream(video, range: nil)
remote_video_id = video.video_metadata['remote_video_id']
stream_url = @client.video_stream_url(remote_video_id)
if range
proxy_stream_with_range(stream_url, range)
else
proxy_stream(stream_url)
end
end
def thumbnail_url(video)
remote_video_id = video.video_metadata['remote_video_id']
"#{@storage_location.remote_instance_url}/api/v1/videos/#{remote_video_id}/thumbnail"
end
private
def proxy_stream(url)
response = Faraday.get(url) do |req|
req.headers['Authorization'] = "Bearer #{@storage_location.api_key}"
end
{
body: response.body,
status: response.status,
headers: response.headers
}
end
def proxy_stream_with_range(url, range)
response = Faraday.get(url) do |req|
req.headers['Authorization'] = "Bearer #{@storage_location.api_key}"
req.headers['Range'] = "bytes=#{range}"
end
{
body: response.body,
status: response.status,
headers: response.headers
}
end
end
```
## Federation API Endpoints
### API Controllers
```ruby
# app/controllers/api/v1/base_controller.rb
class Api::V1::BaseController < ActionController::Base
before_action :authenticate_api_request
private
def authenticate_api_request
token = request.headers['Authorization']&.gsub(/^Bearer\s+/, '')
api_key = ApiKey.find_by(key: token)
if api_key&.active?
@current_api_key = api_key
@current_user = api_key.user
else
render json: { error: 'Unauthorized' }, status: :unauthorized
end
end
def authorize_federation
head :forbidden unless @current_api_key&.can_read
end
def authorize_streaming
head :forbidden unless @current_api_key&.can_stream
end
end
# app/controllers/api/v1/instance_controller.rb
class Api::V1::InstanceController < Api::V1::BaseController
before_action :authorize_federation
def ping
render json: { status: 'ok', timestamp: Time.current.iso8601 }
end
def show
render json: {
name: 'Velour Instance',
version: Velour::VERSION,
description: 'Video library federation node',
total_works: Work.count,
total_videos: Video.count,
total_storage: Video.sum(:file_size)
}
end
end
# app/controllers/api/v1/works_controller.rb
class Api::V1::WorksController < Api::V1::BaseController
before_action :authorize_federation
def index
works = Work.includes(:videos)
.order(:title)
.page(params[:page])
.per(params[:per_page] || 50)
render json: {
works: works.map { |work| serialize_work(work) },
pagination: {
current_page: works.current_page,
total_pages: works.total_pages,
total_count: works.total_count
}
}
end
def show
work = Work.includes(videos: :storage_location).find(params[:id])
render json: serialize_work(work)
end
private
def serialize_work(work)
{
id: work.id,
title: work.title,
year: work.year,
description: work.description,
director: work.director,
rating: work.rating,
organized: work.organized,
created_at: work.created_at.iso8601,
videos: work.videos.map { |video| serialize_video(video) }
}
end
def serialize_video(video)
{
id: video.id,
filename: video.filename,
file_size: video.file_size,
web_compatible: video.web_compatible?,
duration: video.duration,
width: video.width,
height: video.height,
video_codec: video.video_codec,
audio_codec: video.audio_codec,
fingerprints: video.fingerprints,
storage_type: video.storage_location.storage_type,
created_at: video.created_at.iso8601
}
end
end
# app/controllers/api/v1/videos_controller.rb
class Api::V1::VideosController < Api::V1::BaseController
before_action :set_video, only: [:show, :stream, :thumbnail]
before_action :authorize_federation, except: [:stream, :thumbnail]
before_action :authorize_streaming, only: [:stream, :thumbnail]
def show
render json: serialize_video(@video)
end
def stream
streamer = @video.storage_location.streamer
result = streamer.stream(@video, range: request.headers['Range'])
send_data result[:body],
type: result[:headers]['Content-Type'] || 'video/mp4',
disposition: 'inline',
status: result[:status],
headers: result[:headers].slice('Content-Range', 'Content-Length', 'Accept-Ranges')
end
def thumbnail
if @video.video_assets.where(asset_type: 'thumbnail').exists?
asset = @video.video_assets.where(asset_type: 'thumbnail').first
redirect_to rails_blob_url(asset.file, disposition: 'inline')
else
head :not_found
end
end
def search
query = params[:query]
return render json: [] if query.blank?
videos = Video.joins(:work)
.where('works.title ILIKE ?', "%#{query}%")
.includes(:work, :storage_location)
.page(params[:page])
.per(params[:per_page] || 20)
render json: {
videos: videos.map { |video| serialize_video(video) },
pagination: {
current_page: videos.current_page,
total_pages: videos.total_pages,
total_count: videos.total_count
}
}
end
private
def set_video
@video = Video.find(params[:id])
end
def serialize_video(video)
{
id: video.id,
filename: video.filename,
work_id: video.work_id,
work_title: video.work.title,
file_size: video.file_size,
web_compatible: video.web_compatible?,
duration: video.duration,
width: video.width,
height: video.height,
video_codec: video.video_codec,
audio_codec: video.audio_codec,
fingerprints: video.fingerprints,
storage_type: video.storage_location.storage_type,
created_at: video.created_at.iso8601
}
end
end
```
## Federation Management UI
### Federation Connections Management
```erb
<!-- app/views/admin/federation_connections/index.html.erb -->
<div class="container mx-auto p-6">
<h1 class="text-3xl font-bold mb-6">Federation Connections</h1>
<% if @api_keys.any? %>
<div class="mb-6">
<%= link_to 'Create New Connection', new_admin_federation_connection_path, class: 'bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded' %>
</div>
<div class="bg-white shadow overflow-hidden sm:rounded-md">
<ul class="divide-y divide-gray-200">
<% @connections.each do |connection| %>
<li class="px-6 py-4">
<div class="flex items-center justify-between">
<div>
<p class="text-sm font-medium text-gray-900">
<%= connection.storage_location.name %>
</p>
<p class="text-sm text-gray-500">
<%= connection.remote_instance_url %>
</p>
<p class="text-sm text-gray-500">
API Key: <%= connection.api_key.name %>
</p>
</div>
<div class="flex items-center space-x-2">
<span class="inline-flex items-center px-2.5 py-0.5 rounded-full text-xs font-medium
<%= case connection.status
when 'active' then 'bg-green-100 text-green-800'
when 'pending' then 'bg-yellow-100 text-yellow-800'
when 'suspended' then 'bg-red-100 text-red-800'
when 'rejected' then 'bg-gray-100 text-gray-800'
end %>">
<%= connection.status.titleize %>
</span>
<%= link_to 'Edit', edit_admin_federation_connection_path(connection), class: 'text-blue-600 hover:text-blue-900' %>
<%= link_to 'Test', test_admin_federation_connection_path(connection), method: :post, class: 'text-green-600 hover:text-green-900' %>
<%= link_to 'Delete', admin_federation_connection_path(connection), method: :delete,
data: { confirm: 'Are you sure?' }, class: 'text-red-600 hover:text-red-900' %>
</div>
</div>
</li>
<% end %>
</ul>
</div>
<% else %>
<div class="bg-yellow-50 border-l-4 border-yellow-400 p-4">
<div class="flex">
<div class="flex-shrink-0">
<svg class="h-5 w-5 text-yellow-400" viewBox="0 0 20 20" fill="currentColor">
<path fill-rule="evenodd" d="M8.257 3.099c.765-1.36 2.722-1.36 3.486 0l5.58 9.92c.75 1.334-.213 2.98-1.742 2.98H4.42c-1.53 0-2.493-1.646-1.743-2.98l5.58-9.92zM11 13a1 1 0 11-2 0 1 1 0 012 0zm-1-8a1 1 0 00-1 1v3a1 1 0 002 0V6a1 1 0 00-1-1z" clip-rule="evenodd"/>
</svg>
</div>
<div class="ml-3">
<p class="text-sm text-yellow-700">
You need to create an API key before you can establish federation connections.
</p>
<div class="mt-2">
<%= link_to 'Create API Key', new_admin_api_key_path, class: 'text-yellow-700 underline hover:text-yellow-600' %>
</div>
</div>
</div>
</div>
<% end %>
</div>
```
## Security Considerations
### Federation Security Best Practices
1. **API Key Management**: Regular rotation and limited scope
2. **Access Control**: Minimum required permissions
3. **Rate Limiting**: Prevent abuse from federated instances
4. **Audit Logging**: Track all federated access
5. **Network Security**: HTTPS-only federation connections
### Rate Limiting
```ruby
# app/controllers/concerns/rate_limited.rb
module RateLimited
extend ActiveSupport::Concern
included do
before_action :check_rate_limit, only: [:index, :show, :stream]
end
private
def check_rate_limit
return unless @current_api_key
key = "federation_rate_limit:#{@current_api_key.id}"
count = Rails.cache.increment(key, 1, expires_in: 1.hour)
if count > rate_limit
Rails.logger.warn "Rate limit exceeded for API key #{@current_api_key.id}"
render json: { error: 'Rate limit exceeded' }, status: :too_many_requests
end
end
def rate_limit
case action_name
when 'stream'
1000 # requests per hour
else
5000 # requests per hour
end
end
end
```
## Environment Configuration
### Federation Configuration
```bash
# Federation settings
FEDERATION_ENABLED=true
MAX_FEDERATED_CONNECTIONS=10
FEDERATION_RATE_LIMIT_PER_HOUR=5000
# Federation security
FEDERATION_REQUIRE_HTTPS=true
FEDERATION_TOKEN_EXPIRY_HOURS=24
```
Phase 4 enables a distributed network of Velour instances that can share video libraries while maintaining security and access controls. This allows organizations to federate their video collections across multiple servers or locations.
+324
View File
@@ -0,0 +1,324 @@
# Velour Phase 5: Audio Support (Music & Audiobooks)
Phase 5 extends Velour from a video library to a comprehensive media library by adding support for music and audiobooks. This builds upon the extensible MediaFile architecture established in Phase 1.
## Technology Stack
### Audio Processing Components
- **FFmpeg** - Audio transcoding and metadata extraction (extends existing video processing)
- **Ruby Audio Gems** - ID3 tag parsing, waveform generation
- **Active Storage** - Album art and waveform visualization storage
- **MediaInfo** - Comprehensive audio metadata extraction
## Database Schema Extensions
### Audio Model (inherits from MediaFile)
```ruby
class Audio < MediaFile
# Audio-specific associations
has_many :audio_assets, dependent: :destroy # album art, waveforms
# Audio-specific metadata store
store :audio_metadata, accessors: [:sample_rate, :channels, :artist, :album, :track_number, :genre, :year]
# Audio-specific methods
def quality_label
return "Unknown" unless bit_rate
case bit_rate
when 0..128 then "128kbps"
when 129..192 then "192kbps"
when 193..256 then "256kbps"
when 257..320 then "320kbps"
else "Lossless"
end
end
def format_type
return "Unknown" unless format
case format&.downcase
when "mp3" then "MP3"
when "flac" then "FLAC"
when "wav" then "WAV"
when "aac", "m4a" then "AAC"
when "ogg" then "OGG Vorbis"
else format&.upcase
end
end
end
class AudioAsset < ApplicationRecord
belongs_to :audio
enum asset_type: { album_art: 0, waveform: 1, lyrics: 2 }
# Uses Active Storage for file storage
has_one_attached :file
end
```
### Extended Work Model
```ruby
class Work < ApplicationRecord
# Existing video associations
has_many :videos, dependent: :destroy
has_many :external_ids, dependent: :destroy
# New audio associations
has_many :audios, dependent: :destroy
# Enhanced primary media selection
def primary_media
(audios + videos).sort_by(&:created_at).last
end
def primary_video
videos.order(created_at: :desc).first
end
def primary_audio
audios.order(created_at: :desc).first
end
# Content type detection
def video_content?
videos.exists?
end
def audio_content?
audios.exists?
end
def mixed_content?
video_content? && audio_content?
end
end
```
## Audio Processing Pipeline
### AudioProcessorJob
```ruby
class AudioProcessorJob < ApplicationJob
queue_as :processing
def perform(audio_id)
audio = Audio.find(audio_id)
# Extract audio metadata
AudioMetadataExtractor.new(audio).extract!
# Generate album art if embedded
AlbumArtExtractor.new(audio).extract!
# Generate waveform visualization
WaveformGenerator.new(audio).generate!
# Check web compatibility and transcode if needed
unless AudioTranscoder.new(audio).web_compatible?
AudioTranscoderJob.perform_later(audio_id)
end
audio.update!(processed: true)
rescue => e
audio.update!(processing_error: e.message)
raise
end
end
```
### AudioTranscoderJob
```ruby
class AudioTranscoderJob < ApplicationJob
queue_as :transcoding
def perform(audio_id)
audio = Audio.find(audio_id)
AudioTranscoder.new(audio).transcode_for_web!
end
end
```
## File Discovery Extensions
### Enhanced FileScannerService
```ruby
class FileScannerService
AUDIO_EXTENSIONS = %w[mp3 flac wav aac m4a ogg wma].freeze
def scan_directory(storage_location)
# Existing video scanning logic
scan_videos(storage_location)
# New audio scanning logic
scan_audio(storage_location)
end
private
def scan_audio(storage_location)
AUDIO_EXTENSIONS.each do |ext|
Dir.glob(File.join(storage_location.path, "**", "*.#{ext}")).each do |file_path|
process_audio_file(file_path, storage_location)
end
end
end
def process_audio_file(file_path, storage_location)
filename = File.basename(file_path)
return if Audio.joins(:storage_location).exists?(filename: filename, storage_locations: { id: storage_location.id })
# Create Work based on filename parsing (album/track structure)
work = find_or_create_audio_work(filename, file_path)
# Create Audio record
Audio.create!(
work: work,
storage_location: storage_location,
filename: filename,
xxhash64: calculate_xxhash64(file_path)
)
AudioProcessorJob.perform_later(audio.id)
end
end
```
## User Interface Extensions
### Audio Player Integration
- **Video.js Audio Plugin** - Extend existing video player for audio
- **Waveform Visualization** - Interactive seeking with waveform display
- **Chapter Support** - Essential for audiobooks
- **Speed Control** - Variable playback speed for audiobooks
### Library Organization
- **Album View** - Grid layout with album art
- **Artist Pages** - Discography and album organization
- **Audiobook Progress** - Chapter tracking and resume functionality
- **Mixed Media Collections** - Works containing both video and audio content
### Audio-Specific Features
- **Playlist Creation** - Custom playlists for music
- **Shuffle Play** - Random playback for albums/artists
- **Gapless Playback** - Seamless track transitions
- **Lyrics Display** - Embedded or external lyrics support
## Implementation Timeline
### Phase 5A: Audio Foundation (Week 1-2)
- Create Audio model inheriting from MediaFile
- Implement AudioProcessorJob and audio metadata extraction
- Extend FileScannerService for audio formats
- Basic audio streaming endpoint
### Phase 5B: Audio Processing (Week 3)
- Album art extraction and storage
- Waveform generation
- Audio transcoding for web compatibility
- Quality optimization and format conversion
### Phase 5C: User Interface (Week 4)
- Audio player component (extends Video.js)
- Album and artist browsing interfaces
- Audio library management views
- Search and filtering for audio content
### Phase 5D: Advanced Features (Week 5)
- Chapter support for audiobooks
- Playlist creation and management
- Mixed media Works (video + audio)
- Audio-specific user preferences
## Migration Strategy
### Database Migrations
```ruby
# Extend videos table for STI (already done in Phase 1)
# Add audio-specific columns if needed
class AddAudioFeatures < ActiveRecord::Migration[8.1]
def change
create_table :audio_assets do |t|
t.references :audio, null: false, foreign_key: true
t.string :asset_type
t.timestamps
end
# Audio-specific indexes
add_index :audios, :artist if column_exists?(:audios, :artist)
add_index :audios, :album if column_exists?(:audios, :album)
end
end
```
### Backward Compatibility
- All existing video functionality remains unchanged
- Video URLs and routes continue to work identically
- Database migration is additive (type column only)
- No breaking changes to existing API
## Configuration
### Environment Variables
```bash
# Audio Processing (extends existing video processing)
FFMPEG_PATH=/usr/bin/ffmpeg
AUDIO_TRANSCODE_QUALITY=high
MAX_AUDIO_TRANSCODE_SIZE_GB=10
# Audio Features
ENABLE_AUDIO_SCANNING=true
ENABLE_WAVEFORM_GENERATION=true
AUDIO_THUMBNAIL_SIZE=300x300
```
### Storage Considerations
- Album art storage in Active Storage
- Waveform images (generated per track)
- Potential audio transcoding cache
- Audio-specific metadata storage
## Testing Strategy
### Model Tests
- Audio model validation and inheritance
- Work model mixed content handling
- Audio metadata extraction accuracy
### Integration Tests
- Audio processing pipeline end-to-end
- Audio streaming with seeking support
- File scanner audio discovery
### System Tests
- Audio player functionality
- Album/artist interface navigation
- Mixed media library browsing
## Performance Considerations
### Audio Processing
- Parallel audio metadata extraction
- Efficient album art extraction
- Optimized waveform generation
- Background transcoding queue management
### Storage Optimization
- Compressed waveform storage
- Album art caching and optimization
- Efficient audio streaming with range requests
### User Experience
- Fast audio library browsing
- Quick album art loading
- Responsive audio player controls
## Future Extensions
### Phase 5+ Possibilities
- **Podcast Support** - RSS feed integration and episode management
- **Radio Streaming** - Internet radio station integration
- **Music Discovery** - Similar artist recommendations
- **Audio Bookmarks** - Detailed note-taking for audiobooks
- **Social Features** - Sharing playlists and recommendations
This phase transforms Velour from a video library into a comprehensive personal media platform while maintaining the simplicity and robustness of the existing architecture.