pgsql-jellyfin/docs/MULTI_INSTANCE_SUPPORT_PLAN.md

Multi-Instance Jellyfin Support - Implementation Plan

Overview

Enable multiple Jellyfin instances to share the same PostgreSQL database safely, assuming they run on the same OS with identical path structures.

---

Architecture Requirements

Key Challenges

1. Concurrent Database Access - Multiple instances reading/writing simultaneously
2. Session Isolation - Each instance manages its own user sessions (streaming, transcoding)
3. Cache Coordination - In-memory caches need invalidation across instances
4. Library Scanning - Prevent concurrent scans of the same library
5. File System Monitoring - Multiple instances watching same paths
6. Configuration Isolation - Instance-specific settings vs shared data
7. Database Migrations - Ensure only one instance migrates schema

---

Solution Architecture

1. Instance Registration System

Add an Instances table to track active Jellyfin instances:

CREATE TABLE IF NOT EXISTS library."Instances" (
    "InstanceId" UUID PRIMARY KEY,
    "Hostname" VARCHAR(255) NOT NULL,
    "ProcessId" INTEGER NOT NULL,
    "HttpPort" INTEGER NOT NULL,
    "HttpsPort" INTEGER,
    "Version" VARCHAR(50) NOT NULL,
    "StartedAt" TIMESTAMP NOT NULL,
    "LastHeartbeat" TIMESTAMP NOT NULL,
    "Status" VARCHAR(50) NOT NULL, -- Active, Shutdown, Failed
    "IsPrimary" BOOLEAN NOT NULL DEFAULT FALSE,
    "Capabilities" JSONB -- {canScan: true, canTranscode: true, etc}
);

CREATE INDEX idx_instances_lastheartbeat ON library."Instances"("LastHeartbeat");
CREATE INDEX idx_instances_status ON library."Instances"("Status");

2. Session Isolation

Sessions (streaming, transcoding) are instance-specific and should NOT be shared:

-- Add InstanceId to Sessions table
ALTER TABLE library."Sessions" ADD COLUMN "InstanceId" UUID;
ALTER TABLE library."Sessions" ADD CONSTRAINT fk_session_instance 
    FOREIGN KEY ("InstanceId") REFERENCES library."Instances"("InstanceId") ON DELETE CASCADE;

CREATE INDEX idx_sessions_instance ON library."Sessions"("InstanceId");

Why: Each instance has its own:

• Transcoding processes
• Network connections
• Resource limits
• WebSocket connections

3. Distributed Locking for Library Operations

Use PostgreSQL advisory locks for critical operations:

public interface IDistributedLockManager
{
    Task<IAsyncDisposable> AcquireLockAsync(string lockName, TimeSpan timeout, CancellationToken cancellationToken);
    Task<bool> TryAcquireLockAsync(string lockName, TimeSpan timeout, CancellationToken cancellationToken);
}

// Lock Types:
// - LibraryScan_{libraryId}
// - MetadataRefresh_{itemId}
// - DatabaseMigration
// - ConfigurationUpdate_{configType}

4. Cache Invalidation Strategy

Option A: Database Notifications (PostgreSQL LISTEN/NOTIFY)

-- Notification channel for cache invalidation
NOTIFY cache_invalidation, '{"type": "item", "id": "123-456-789", "operation": "update"}';

public interface ICacheCoordinator
{
    Task InvalidateItemAsync(Guid itemId);
    Task InvalidateUserDataAsync(Guid userId);
    Task InvalidateAllAsync(string cacheType);
}

Option B: Polling-Based Invalidation

Add a CacheInvalidations table:

CREATE TABLE library."CacheInvalidations" (
    "Id" BIGSERIAL PRIMARY KEY,
    "Timestamp" TIMESTAMP NOT NULL DEFAULT NOW(),
    "CacheType" VARCHAR(100) NOT NULL, -- Item, UserData, Configuration
    "EntityId" UUID,
    "Operation" VARCHAR(50) NOT NULL -- Update, Delete, Refresh
);

CREATE INDEX idx_cacheinvalidations_timestamp ON library."CacheInvalidations"("Timestamp");

5. Primary Instance Election

Use database for primary instance election (for administrative tasks):

public interface IPrimaryInstanceManager
{
    Task<bool> TryBecomePrimaryAsync(CancellationToken cancellationToken);
    Task<bool> IsPrimaryAsync(CancellationToken cancellationToken);
    Task ReleasePrimaryAsync();
}

Primary Instance Responsibilities:

• Database migrations
• Scheduled tasks (cleanup, backups)
• Library scanning coordination
• Plugin updates

Secondary Instance Responsibilities:

• Serve API requests
• Stream media
• Transcode
• User authentication

6. File System Monitor Coordination

Problem: Multiple instances watching same paths creates redundant work.

Solution: Coordinate through database:

CREATE TABLE library."FileSystemChanges" (
    "Id" BIGSERIAL PRIMARY KEY,
    "Path" TEXT NOT NULL,
    "ChangeType" VARCHAR(50) NOT NULL, -- Created, Modified, Deleted
    "DetectedAt" TIMESTAMP NOT NULL DEFAULT NOW(),
    "DetectedBy" UUID NOT NULL, -- InstanceId
    "ProcessedAt" TIMESTAMP,
    "ProcessedBy" UUID -- InstanceId
);

CREATE INDEX idx_filesystemchanges_processed ON library."FileSystemChanges"("ProcessedAt") 
    WHERE "ProcessedAt" IS NULL;

Workflow:

1. Instance A detects file change → writes to FileSystemChanges
2. Primary instance polls for unprocessed changes
3. Primary instance processes and marks as processed
4. All instances invalidate related caches

---

Configuration Strategy

Instance-Specific Configuration

Stored in: Local config/ directory per instance

• HTTP/HTTPS ports
• Transcoding paths (must be unique per instance)
• Cache directory
• Log directory
• PID file
• WebSocket settings

Shared Configuration

Stored in: Database

• Library paths (must be same across instances)
• User accounts and permissions
• Metadata providers
• Playback settings
• DLNA settings (disabled or coordinated)

---

Implementation Phases

Phase 1: Instance Registration & Heartbeat

Goal: Track which instances are active

Tasks:

1. Create Instances table migration
2. Implement InstanceRegistry service
3. Add startup registration
4. Add heartbeat mechanism (every 30 seconds)
5. Add cleanup for stale instances (no heartbeat > 2 minutes)

Files to Create:

• src/Jellyfin.Database/Entities/Instance.cs
• src/Jellyfin.Server.Implementations/Clustering/InstanceRegistry.cs
• src/Jellyfin.Server.Implementations/Clustering/IInstanceRegistry.cs
• Migration: YYYYMMDDHHMMSS_AddInstancesTable.cs

Phase 2: Distributed Locking

Goal: Prevent concurrent operations on same resources

Tasks:

1. Implement PostgreSQL advisory lock wrapper
2. Add lock management service
3. Wrap library scan operations with locks
4. Wrap metadata refresh with locks
5. Add migration lock

Files to Create:

• src/Jellyfin.Server.Implementations/Clustering/DistributedLockManager.cs
• src/Jellyfin.Server.Implementations/Clustering/IDistributedLockManager.cs
• Update: LibraryManager.cs to use locks

Phase 3: Session Isolation

Goal: Ensure sessions belong to specific instance

Tasks:

1. Add InstanceId column to sessions
2. Update SessionManager to filter by instance
3. Clean up sessions on instance shutdown
4. Add session migration for existing sessions

Files to Modify:

• src/Jellyfin.Database/Entities/Session.cs (if exists)
• Emby.Server.Implementations/Session/SessionManager.cs
• Migration: YYYYMMDDHHMMSS_AddInstanceIdToSessions.cs

Phase 4: Cache Coordination

Goal: Invalidate caches across all instances

Tasks:

1. Implement LISTEN/NOTIFY for PostgreSQL
2. Create CacheCoordinator service
3. Hook into item update events
4. Hook into user data update events
5. Add subscription management

Files to Create:

• src/Jellyfin.Server.Implementations/Clustering/CacheCoordinator.cs
• src/Jellyfin.Server.Implementations/Clustering/PostgresNotificationListener.cs

Phase 5: Primary Instance Election

Goal: Designate one instance for administrative tasks

Tasks:

1. Implement primary election algorithm
2. Add scheduled task coordination
3. Add migration coordination
4. Add backup coordination

Files to Create:

• src/Jellyfin.Server.Implementations/Clustering/PrimaryInstanceManager.cs
• Update: ApplicationHost.cs for primary election

Phase 6: File System Monitor Coordination

Goal: Reduce duplicate file scanning

Tasks:

1. Create FileSystemChanges table
2. Update LibraryMonitor to write to database
3. Add change processor on primary instance
4. Add polling mechanism

Files to Modify:

• Emby.Server.Implementations/IO/LibraryMonitor.cs
• Migration: YYYYMMDDHHMMSS_AddFileSystemChangesTable.cs

---

Database Schema Changes

Complete Migration Script

-- ============================================
-- Multi-Instance Support Migration
-- ============================================

-- 1. Instances Table
CREATE TABLE IF NOT EXISTS library."Instances" (
    "InstanceId" UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    "Hostname" VARCHAR(255) NOT NULL,
    "ProcessId" INTEGER NOT NULL,
    "HttpPort" INTEGER NOT NULL,
    "HttpsPort" INTEGER,
    "Version" VARCHAR(50) NOT NULL,
    "StartedAt" TIMESTAMP NOT NULL DEFAULT NOW(),
    "LastHeartbeat" TIMESTAMP NOT NULL DEFAULT NOW(),
    "Status" VARCHAR(50) NOT NULL DEFAULT 'Active',
    "IsPrimary" BOOLEAN NOT NULL DEFAULT FALSE,
    "Capabilities" JSONB DEFAULT '{}'::JSONB,
    "Configuration" JSONB DEFAULT '{}'::JSONB,
    CONSTRAINT chk_instance_status CHECK ("Status" IN ('Active', 'Shutdown', 'Failed', 'Maintenance'))
);

CREATE INDEX idx_instances_lastheartbeat ON library."Instances"("LastHeartbeat");
CREATE INDEX idx_instances_status ON library."Instances"("Status");
CREATE INDEX idx_instances_isprimary ON library."Instances"("IsPrimary") WHERE "IsPrimary" = TRUE;

-- 2. Distributed Locks Table
CREATE TABLE IF NOT EXISTS library."DistributedLocks" (
    "LockName" VARCHAR(255) PRIMARY KEY,
    "InstanceId" UUID NOT NULL,
    "AcquiredAt" TIMESTAMP NOT NULL DEFAULT NOW(),
    "ExpiresAt" TIMESTAMP NOT NULL,
    "RenewedAt" TIMESTAMP NOT NULL DEFAULT NOW(),
    CONSTRAINT fk_lock_instance FOREIGN KEY ("InstanceId") 
        REFERENCES library."Instances"("InstanceId") ON DELETE CASCADE
);

CREATE INDEX idx_locks_expiration ON library."DistributedLocks"("ExpiresAt");

-- 3. Cache Invalidations Table
CREATE TABLE IF NOT EXISTS library."CacheInvalidations" (
    "Id" BIGSERIAL PRIMARY KEY,
    "Timestamp" TIMESTAMP NOT NULL DEFAULT NOW(),
    "InstanceId" UUID NOT NULL,
    "CacheType" VARCHAR(100) NOT NULL,
    "EntityId" UUID,
    "EntityType" VARCHAR(100),
    "Operation" VARCHAR(50) NOT NULL,
    "Metadata" JSONB,
    CONSTRAINT fk_invalidation_instance FOREIGN KEY ("InstanceId") 
        REFERENCES library."Instances"("InstanceId") ON DELETE CASCADE,
    CONSTRAINT chk_operation CHECK ("Operation" IN ('Update', 'Delete', 'Refresh', 'Clear'))
);

CREATE INDEX idx_cacheinvalidations_timestamp ON library."CacheInvalidations"("Timestamp");
CREATE INDEX idx_cacheinvalidations_entityid ON library."CacheInvalidations"("EntityId") WHERE "EntityId" IS NOT NULL;

-- 4. File System Changes Table
CREATE TABLE IF NOT EXISTS library."FileSystemChanges" (
    "Id" BIGSERIAL PRIMARY KEY,
    "Path" TEXT NOT NULL,
    "ChangeType" VARCHAR(50) NOT NULL,
    "DetectedAt" TIMESTAMP NOT NULL DEFAULT NOW(),
    "DetectedBy" UUID NOT NULL,
    "ProcessedAt" TIMESTAMP,
    "ProcessedBy" UUID,
    "LibraryId" UUID,
    "Error" TEXT,
    CONSTRAINT fk_fschange_detectedby FOREIGN KEY ("DetectedBy") 
        REFERENCES library."Instances"("InstanceId") ON DELETE CASCADE,
    CONSTRAINT fk_fschange_processedby FOREIGN KEY ("ProcessedBy") 
        REFERENCES library."Instances"("InstanceId") ON DELETE SET NULL,
    CONSTRAINT chk_changetype CHECK ("ChangeType" IN ('Created', 'Modified', 'Deleted', 'Renamed'))
);

CREATE INDEX idx_filesystemchanges_processed ON library."FileSystemChanges"("ProcessedAt") 
    WHERE "ProcessedAt" IS NULL;
CREATE INDEX idx_filesystemchanges_detectedat ON library."FileSystemChanges"("DetectedAt");
CREATE INDEX idx_filesystemchanges_path ON library."FileSystemChanges"("Path");

-- 5. Activity Log - Add InstanceId
ALTER TABLE activitylog."ActivityLog" ADD COLUMN IF NOT EXISTS "InstanceId" UUID;
CREATE INDEX IF NOT EXISTS idx_activitylog_instance ON activitylog."ActivityLog"("InstanceId");

-- 6. Cleanup Function for Stale Instances
CREATE OR REPLACE FUNCTION library.cleanup_stale_instances()
RETURNS void AS $$
BEGIN
    UPDATE library."Instances"
    SET "Status" = 'Failed'
    WHERE "Status" = 'Active'
      AND "LastHeartbeat" < NOW() - INTERVAL '2 minutes';
END;
$$ LANGUAGE plpgsql;

-- 7. Function to Get Primary Instance
CREATE OR REPLACE FUNCTION library.get_primary_instance()
RETURNS UUID AS $$
DECLARE
    primary_id UUID;
BEGIN
    SELECT "InstanceId" INTO primary_id
    FROM library."Instances"
    WHERE "Status" = 'Active'
      AND "IsPrimary" = TRUE
      AND "LastHeartbeat" > NOW() - INTERVAL '1 minute'
    LIMIT 1;
    
    RETURN primary_id;
END;
$$ LANGUAGE plpgsql;

-- 8. Function to Elect Primary Instance
CREATE OR REPLACE FUNCTION library.elect_primary_instance()
RETURNS UUID AS $$
DECLARE
    elected_id UUID;
BEGIN
    -- Clear any existing primary that's not active
    UPDATE library."Instances"
    SET "IsPrimary" = FALSE
    WHERE "IsPrimary" = TRUE
      AND ("Status" != 'Active' OR "LastHeartbeat" < NOW() - INTERVAL '1 minute');
    
    -- Check if we have an active primary
    SELECT "InstanceId" INTO elected_id
    FROM library."Instances"
    WHERE "Status" = 'Active'
      AND "IsPrimary" = TRUE
      AND "LastHeartbeat" > NOW() - INTERVAL '1 minute'
    LIMIT 1;
    
    -- If no primary, elect the oldest active instance
    IF elected_id IS NULL THEN
        SELECT "InstanceId" INTO elected_id
        FROM library."Instances"
        WHERE "Status" = 'Active'
          AND "LastHeartbeat" > NOW() - INTERVAL '1 minute'
        ORDER BY "StartedAt" ASC
        LIMIT 1;
        
        IF elected_id IS NOT NULL THEN
            UPDATE library."Instances"
            SET "IsPrimary" = TRUE
            WHERE "InstanceId" = elected_id;
        END IF;
    END IF;
    
    RETURN elected_id;
END;
$$ LANGUAGE plpgsql;

-- 9. Notification Function for Cache Invalidation
CREATE OR REPLACE FUNCTION library.notify_cache_invalidation()
RETURNS TRIGGER AS $$
BEGIN
    PERFORM pg_notify(
        'cache_invalidation',
        json_build_object(
            'id', NEW."Id",
            'cacheType', NEW."CacheType",
            'entityId', NEW."EntityId",
            'operation', NEW."Operation"
        )::text
    );
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trigger_cache_invalidation_notify
AFTER INSERT ON library."CacheInvalidations"
FOR EACH ROW
EXECUTE FUNCTION library.notify_cache_invalidation();

-- 10. Grant Permissions
GRANT ALL ON TABLE library."Instances" TO jellyfin;
GRANT ALL ON TABLE library."DistributedLocks" TO jellyfin;
GRANT ALL ON TABLE library."CacheInvalidations" TO jellyfin;
GRANT ALL ON TABLE library."FileSystemChanges" TO jellyfin;
GRANT ALL ON SEQUENCE library."CacheInvalidations_Id_seq" TO jellyfin;
GRANT ALL ON SEQUENCE library."FileSystemChanges_Id_seq" TO jellyfin;
GRANT EXECUTE ON FUNCTION library.cleanup_stale_instances() TO jellyfin;
GRANT EXECUTE ON FUNCTION library.get_primary_instance() TO jellyfin;
GRANT EXECUTE ON FUNCTION library.elect_primary_instance() TO jellyfin;

-- 11. Create Cleanup Job (Optional - can be done via scheduled task)
-- This would require pg_cron extension:
-- SELECT cron.schedule('cleanup-stale-instances', '*/1 * * * *', 
--     'SELECT library.cleanup_stale_instances()');

---

Configuration File Changes

startup.json (Instance-Specific)

{
  "InstanceId": "generated-on-first-run-or-specified",
  "InstanceName": "Jellyfin-Server1",
  "DatabaseProvider": "Postgres",
  "ConnectionStrings": {
    "DefaultConnection": "Host=localhost;Database=jellyfin;Username=jellyfin;Password=***"
  },
  "EnableMultiInstance": true,
  "InstanceConfiguration": {
    "HttpPort": 8096,
    "HttpsPort": 8920,
    "CanBecomePrimary": true,
    "Capabilities": {
      "CanScan": true,
      "CanTranscode": true,
      "CanServeApi": true,
      "CanStream": true
    },
    "HeartbeatInterval": 30,
    "LockTimeout": 300
  }
}

---

Usage Scenarios

Scenario 1: Load Balanced Web Tier

┌─────────────┐
│ Load        │
│ Balancer    │
└──────┬──────┘
       │
  ┌────┴────┐
  │         │
┌─▼──┐   ┌─▼──┐
│ J1 │   │ J2 │  ← API Instances (Primary=False)
└─┬──┘   └─┬──┘
  │         │
  └────┬────┘
       │
  ┌────▼────┐
  │   DB    │  ← Shared PostgreSQL
  └─────────┘

• J1 & J2: Serve API requests, streaming
• Primary Instance: J1 (elected automatically)
• Shared: Database, metadata, user data
• Isolated: Sessions, transcoding, caches (with invalidation)

Scenario 2: Separated Scan & Serve

┌──────────────┐
│ Jellyfin-Scan│ ← Primary, scans libraries
│ (Background) │
└──────┬───────┘
       │
  ┌────▼────────────┐
  │ PostgreSQL DB   │
  └────┬────────────┘
       │
  ┌────┴──────┐
  │           │
┌─▼──┐     ┌─▼──┐
│ J1 │     │ J2 │  ← Secondary, serve only
└────┘     └────┘

• Scan Instance: Primary, CanScan=true, CanTranscode=false, CanServeApi=false
• Serve Instances: Secondary, CanServeApi=true, CanStream=true, CanTranscode=true

---

Performance Considerations

Pros

• ✅ Horizontal Scaling: Add more instances for more concurrent users
• ✅ High Availability: If one instance fails, others continue
• ✅ Specialized Instances: Dedicate instances to specific tasks
• ✅ Load Distribution: Spread transcoding/streaming across instances

Cons

• ⚠️ Network Latency: Database calls over network (vs local SQLite)
• ⚠️ Cache Complexity: Invalidation adds overhead
• ⚠️ Locking Overhead: Distributed locks slower than local
• ⚠️ Configuration Complexity: More moving parts to manage

Mitigation

• Use aggressive caching with proper invalidation
• Minimize database round-trips (batch operations)
• Use connection pooling effectively
• Monitor instance health closely

---

Testing Strategy

Test Cases

1. Instance Registration

   • Instance registers on startup
   • Heartbeat updates every 30 seconds
   • Stale instances marked as Failed
   • Instance unregisters on clean shutdown

2. Primary Election

   • Primary elected on first instance start
   • Primary re-elected when primary fails
   • Only one primary at a time

3. Distributed Locking

   • Lock acquired successfully
   • Lock prevents concurrent access
   • Lock released on completion
   • Lock expires if holder crashes

4. Cache Invalidation

   • Update on instance A invalidates cache on instance B
   • Delete operation propagates
   • Notifications delivered within 1 second

5. Session Isolation

   • Sessions belong to specific instance
   • Sessions cleaned up on instance shutdown
   • Sessions not visible to other instances

6. Library Scanning

   • Only one instance scans at a time
   • Scan lock prevents conflicts
   • Other instances see scan results

---

Rollout Plan

Development

1. Implement Phase 1 (Instance Registration) on branch multi-instance-testing
2. Test with 2 instances locally
3. Implement Phase 2 (Locking)
4. Test concurrent library scans

Staging

1. Deploy 3 instances behind load balancer
2. Run load tests
3. Test failover scenarios
4. Monitor cache invalidation performance

Production

1. Start with 2 instances
2. Monitor for 1 week
3. Gradually add more instances
4. Document operational procedures

---

Migration Guide for Existing Installations

For Current Single-Instance Users

Step 1: Backup database

pg_dump jellyfin > jellyfin_backup.sql

Step 2: Run migration

psql -U jellyfin -d jellyfin -f multi_instance_migration.sql

Step 3: Update startup.json

{
  "EnableMultiInstance": true
}

Step 4: Restart Jellyfin

For New Multi-Instance Deployment

Step 1: Set up shared PostgreSQL database

Step 2: Configure first instance

./jellyfin --datadir /opt/jellyfin/instance1 \
          --port 8096 \
          --instance-name "Jellyfin-Primary"

Step 3: Configure second instance

./jellyfin --datadir /opt/jellyfin/instance2 \
          --port 8097 \
          --instance-name "Jellyfin-Secondary" \
          --can-become-primary false

---

Next Steps

Would you like me to:

1. Implement Phase 1 (Instance Registration) with full code?
2. Create the complete EF Core migration for multi-instance support?
3. Implement the DistributedLockManager service?
4. Set up cache invalidation with PostgreSQL LISTEN/NOTIFY?

This is a significant architectural change but achievable with the plan above. The key is implementing it in phases and testing thoroughly at each step.