pgsql-jellyfin/docs/MULTI_INSTANCE_SUPPORT_PLAN.md

# Multi-Instance Jellyfin Support - Implementation Plan

## Overview

Enable multiple Jellyfin instances to share the same PostgreSQL database safely, assuming they run on the same OS with identical path structures.

---

## Architecture Requirements

### Key Challenges

1. **Concurrent Database Access** - Multiple instances reading/writing simultaneously
2. **Session Isolation** - Each instance manages its own user sessions (streaming, transcoding)
3. **Cache Coordination** - In-memory caches need invalidation across instances
4. **Library Scanning** - Prevent concurrent scans of the same library
5. **File System Monitoring** - Multiple instances watching same paths
6. **Configuration Isolation** - Instance-specific settings vs shared data
7. **Database Migrations** - Ensure only one instance migrates schema

---

## Solution Architecture

### 1. Instance Registration System

Add an `Instances` table to track active Jellyfin instances:

```sql
CREATE TABLE IF NOT EXISTS library."Instances" (
    "InstanceId" UUID PRIMARY KEY,
    "Hostname" VARCHAR(255) NOT NULL,
    "ProcessId" INTEGER NOT NULL,
    "HttpPort" INTEGER NOT NULL,
    "HttpsPort" INTEGER,
    "Version" VARCHAR(50) NOT NULL,
    "StartedAt" TIMESTAMP NOT NULL,
    "LastHeartbeat" TIMESTAMP NOT NULL,
    "Status" VARCHAR(50) NOT NULL, -- Active, Shutdown, Failed
    "IsPrimary" BOOLEAN NOT NULL DEFAULT FALSE,
    "Capabilities" JSONB -- {canScan: true, canTranscode: true, etc}
);

CREATE INDEX idx_instances_lastheartbeat ON library."Instances"("LastHeartbeat");
CREATE INDEX idx_instances_status ON library."Instances"("Status");
```

### 2. Session Isolation

Sessions (streaming, transcoding) are **instance-specific** and should NOT be shared:

```sql
-- Add InstanceId to Sessions table
ALTER TABLE library."Sessions" ADD COLUMN "InstanceId" UUID;
ALTER TABLE library."Sessions" ADD CONSTRAINT fk_session_instance 
    FOREIGN KEY ("InstanceId") REFERENCES library."Instances"("InstanceId") ON DELETE CASCADE;

CREATE INDEX idx_sessions_instance ON library."Sessions"("InstanceId");
```

**Why:** Each instance has its own:
- Transcoding processes
- Network connections
- Resource limits
- WebSocket connections

### 3. Distributed Locking for Library Operations

Use PostgreSQL advisory locks for critical operations:

```csharp
public interface IDistributedLockManager
{
    Task<IAsyncDisposable> AcquireLockAsync(string lockName, TimeSpan timeout, CancellationToken cancellationToken);
    Task<bool> TryAcquireLockAsync(string lockName, TimeSpan timeout, CancellationToken cancellationToken);
}

// Lock Types:
// - LibraryScan_{libraryId}
// - MetadataRefresh_{itemId}
// - DatabaseMigration
// - ConfigurationUpdate_{configType}
```

### 4. Cache Invalidation Strategy

**Option A: Database Notifications (PostgreSQL LISTEN/NOTIFY)**

```sql
-- Notification channel for cache invalidation
NOTIFY cache_invalidation, '{"type": "item", "id": "123-456-789", "operation": "update"}';
```

```csharp
public interface ICacheCoordinator
{
    Task InvalidateItemAsync(Guid itemId);
    Task InvalidateUserDataAsync(Guid userId);
    Task InvalidateAllAsync(string cacheType);
}
```

**Option B: Polling-Based Invalidation**

Add a `CacheInvalidations` table:

```sql
CREATE TABLE library."CacheInvalidations" (
    "Id" BIGSERIAL PRIMARY KEY,
    "Timestamp" TIMESTAMP NOT NULL DEFAULT NOW(),
    "CacheType" VARCHAR(100) NOT NULL, -- Item, UserData, Configuration
    "EntityId" UUID,
    "Operation" VARCHAR(50) NOT NULL -- Update, Delete, Refresh
);

CREATE INDEX idx_cacheinvalidations_timestamp ON library."CacheInvalidations"("Timestamp");
```

### 5. Primary Instance Election

Use database for primary instance election (for administrative tasks):

```csharp
public interface IPrimaryInstanceManager
{
    Task<bool> TryBecomePrimaryAsync(CancellationToken cancellationToken);
    Task<bool> IsPrimaryAsync(CancellationToken cancellationToken);
    Task ReleasePrimaryAsync();
}
```

**Primary Instance Responsibilities:**
- Database migrations
- Scheduled tasks (cleanup, backups)
- Library scanning coordination
- Plugin updates

**Secondary Instance Responsibilities:**
- Serve API requests
- Stream media
- Transcode
- User authentication

### 6. File System Monitor Coordination

**Problem:** Multiple instances watching same paths creates redundant work.

**Solution:** Coordinate through database:

```sql
CREATE TABLE library."FileSystemChanges" (
    "Id" BIGSERIAL PRIMARY KEY,
    "Path" TEXT NOT NULL,
    "ChangeType" VARCHAR(50) NOT NULL, -- Created, Modified, Deleted
    "DetectedAt" TIMESTAMP NOT NULL DEFAULT NOW(),
    "DetectedBy" UUID NOT NULL, -- InstanceId
    "ProcessedAt" TIMESTAMP,
    "ProcessedBy" UUID -- InstanceId
);

CREATE INDEX idx_filesystemchanges_processed ON library."FileSystemChanges"("ProcessedAt") 
    WHERE "ProcessedAt" IS NULL;
```

**Workflow:**
1. Instance A detects file change → writes to `FileSystemChanges`
2. Primary instance polls for unprocessed changes
3. Primary instance processes and marks as processed
4. All instances invalidate related caches

---

## Configuration Strategy

### Instance-Specific Configuration

**Stored in:** Local `config/` directory per instance

- HTTP/HTTPS ports
- Transcoding paths (must be unique per instance)
- Cache directory
- Log directory
- PID file
- WebSocket settings

### Shared Configuration

**Stored in:** Database

- Library paths (must be same across instances)
- User accounts and permissions
- Metadata providers
- Playback settings
- DLNA settings (disabled or coordinated)

---

## Implementation Phases

### Phase 1: Instance Registration & Heartbeat

**Goal:** Track which instances are active

**Tasks:**
1. Create `Instances` table migration
2. Implement `InstanceRegistry` service
3. Add startup registration
4. Add heartbeat mechanism (every 30 seconds)
5. Add cleanup for stale instances (no heartbeat > 2 minutes)

**Files to Create:**
- `src/Jellyfin.Database/Entities/Instance.cs`
- `src/Jellyfin.Server.Implementations/Clustering/InstanceRegistry.cs`
- `src/Jellyfin.Server.Implementations/Clustering/IInstanceRegistry.cs`
- Migration: `YYYYMMDDHHMMSS_AddInstancesTable.cs`

### Phase 2: Distributed Locking

**Goal:** Prevent concurrent operations on same resources

**Tasks:**
1. Implement PostgreSQL advisory lock wrapper
2. Add lock management service
3. Wrap library scan operations with locks
4. Wrap metadata refresh with locks
5. Add migration lock

**Files to Create:**
- `src/Jellyfin.Server.Implementations/Clustering/DistributedLockManager.cs`
- `src/Jellyfin.Server.Implementations/Clustering/IDistributedLockManager.cs`
- Update: `LibraryManager.cs` to use locks

### Phase 3: Session Isolation

**Goal:** Ensure sessions belong to specific instance

**Tasks:**
1. Add `InstanceId` column to sessions
2. Update `SessionManager` to filter by instance
3. Clean up sessions on instance shutdown
4. Add session migration for existing sessions

**Files to Modify:**
- `src/Jellyfin.Database/Entities/Session.cs` (if exists)
- `Emby.Server.Implementations/Session/SessionManager.cs`
- Migration: `YYYYMMDDHHMMSS_AddInstanceIdToSessions.cs`

### Phase 4: Cache Coordination

**Goal:** Invalidate caches across all instances

**Tasks:**
1. Implement LISTEN/NOTIFY for PostgreSQL
2. Create `CacheCoordinator` service
3. Hook into item update events
4. Hook into user data update events
5. Add subscription management

**Files to Create:**
- `src/Jellyfin.Server.Implementations/Clustering/CacheCoordinator.cs`
- `src/Jellyfin.Server.Implementations/Clustering/PostgresNotificationListener.cs`

### Phase 5: Primary Instance Election

**Goal:** Designate one instance for administrative tasks

**Tasks:**
1. Implement primary election algorithm
2. Add scheduled task coordination
3. Add migration coordination
4. Add backup coordination

**Files to Create:**
- `src/Jellyfin.Server.Implementations/Clustering/PrimaryInstanceManager.cs`
- Update: `ApplicationHost.cs` for primary election

### Phase 6: File System Monitor Coordination

**Goal:** Reduce duplicate file scanning

**Tasks:**
1. Create `FileSystemChanges` table
2. Update `LibraryMonitor` to write to database
3. Add change processor on primary instance
4. Add polling mechanism

**Files to Modify:**
- `Emby.Server.Implementations/IO/LibraryMonitor.cs`
- Migration: `YYYYMMDDHHMMSS_AddFileSystemChangesTable.cs`

---

## Database Schema Changes

### Complete Migration Script

```sql
-- ============================================
-- Multi-Instance Support Migration
-- ============================================

-- 1. Instances Table
CREATE TABLE IF NOT EXISTS library."Instances" (
    "InstanceId" UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    "Hostname" VARCHAR(255) NOT NULL,
    "ProcessId" INTEGER NOT NULL,
    "HttpPort" INTEGER NOT NULL,
    "HttpsPort" INTEGER,
    "Version" VARCHAR(50) NOT NULL,
    "StartedAt" TIMESTAMP NOT NULL DEFAULT NOW(),
    "LastHeartbeat" TIMESTAMP NOT NULL DEFAULT NOW(),
    "Status" VARCHAR(50) NOT NULL DEFAULT 'Active',
    "IsPrimary" BOOLEAN NOT NULL DEFAULT FALSE,
    "Capabilities" JSONB DEFAULT '{}'::JSONB,
    "Configuration" JSONB DEFAULT '{}'::JSONB,
    CONSTRAINT chk_instance_status CHECK ("Status" IN ('Active', 'Shutdown', 'Failed', 'Maintenance'))
);

CREATE INDEX idx_instances_lastheartbeat ON library."Instances"("LastHeartbeat");
CREATE INDEX idx_instances_status ON library."Instances"("Status");
CREATE INDEX idx_instances_isprimary ON library."Instances"("IsPrimary") WHERE "IsPrimary" = TRUE;

-- 2. Distributed Locks Table
CREATE TABLE IF NOT EXISTS library."DistributedLocks" (
    "LockName" VARCHAR(255) PRIMARY KEY,
    "InstanceId" UUID NOT NULL,
    "AcquiredAt" TIMESTAMP NOT NULL DEFAULT NOW(),
    "ExpiresAt" TIMESTAMP NOT NULL,
    "RenewedAt" TIMESTAMP NOT NULL DEFAULT NOW(),
    CONSTRAINT fk_lock_instance FOREIGN KEY ("InstanceId") 
        REFERENCES library."Instances"("InstanceId") ON DELETE CASCADE
);

CREATE INDEX idx_locks_expiration ON library."DistributedLocks"("ExpiresAt");

-- 3. Cache Invalidations Table
CREATE TABLE IF NOT EXISTS library."CacheInvalidations" (
    "Id" BIGSERIAL PRIMARY KEY,
    "Timestamp" TIMESTAMP NOT NULL DEFAULT NOW(),
    "InstanceId" UUID NOT NULL,
    "CacheType" VARCHAR(100) NOT NULL,
    "EntityId" UUID,
    "EntityType" VARCHAR(100),
    "Operation" VARCHAR(50) NOT NULL,
    "Metadata" JSONB,
    CONSTRAINT fk_invalidation_instance FOREIGN KEY ("InstanceId") 
        REFERENCES library."Instances"("InstanceId") ON DELETE CASCADE,
    CONSTRAINT chk_operation CHECK ("Operation" IN ('Update', 'Delete', 'Refresh', 'Clear'))
);

CREATE INDEX idx_cacheinvalidations_timestamp ON library."CacheInvalidations"("Timestamp");
CREATE INDEX idx_cacheinvalidations_entityid ON library."CacheInvalidations"("EntityId") WHERE "EntityId" IS NOT NULL;

-- 4. File System Changes Table
CREATE TABLE IF NOT EXISTS library."FileSystemChanges" (
    "Id" BIGSERIAL PRIMARY KEY,
    "Path" TEXT NOT NULL,
    "ChangeType" VARCHAR(50) NOT NULL,
    "DetectedAt" TIMESTAMP NOT NULL DEFAULT NOW(),
    "DetectedBy" UUID NOT NULL,
    "ProcessedAt" TIMESTAMP,
    "ProcessedBy" UUID,
    "LibraryId" UUID,
    "Error" TEXT,
    CONSTRAINT fk_fschange_detectedby FOREIGN KEY ("DetectedBy") 
        REFERENCES library."Instances"("InstanceId") ON DELETE CASCADE,
    CONSTRAINT fk_fschange_processedby FOREIGN KEY ("ProcessedBy") 
        REFERENCES library."Instances"("InstanceId") ON DELETE SET NULL,
    CONSTRAINT chk_changetype CHECK ("ChangeType" IN ('Created', 'Modified', 'Deleted', 'Renamed'))
);

CREATE INDEX idx_filesystemchanges_processed ON library."FileSystemChanges"("ProcessedAt") 
    WHERE "ProcessedAt" IS NULL;
CREATE INDEX idx_filesystemchanges_detectedat ON library."FileSystemChanges"("DetectedAt");
CREATE INDEX idx_filesystemchanges_path ON library."FileSystemChanges"("Path");

-- 5. Activity Log - Add InstanceId
ALTER TABLE activitylog."ActivityLog" ADD COLUMN IF NOT EXISTS "InstanceId" UUID;
CREATE INDEX IF NOT EXISTS idx_activitylog_instance ON activitylog."ActivityLog"("InstanceId");

-- 6. Cleanup Function for Stale Instances
CREATE OR REPLACE FUNCTION library.cleanup_stale_instances()
RETURNS void AS $$
BEGIN
    UPDATE library."Instances"
    SET "Status" = 'Failed'
    WHERE "Status" = 'Active'
      AND "LastHeartbeat" < NOW() - INTERVAL '2 minutes';
END;
$$ LANGUAGE plpgsql;

-- 7. Function to Get Primary Instance
CREATE OR REPLACE FUNCTION library.get_primary_instance()
RETURNS UUID AS $$
DECLARE
    primary_id UUID;
BEGIN
    SELECT "InstanceId" INTO primary_id
    FROM library."Instances"
    WHERE "Status" = 'Active'
      AND "IsPrimary" = TRUE
      AND "LastHeartbeat" > NOW() - INTERVAL '1 minute'
    LIMIT 1;
    
    RETURN primary_id;
END;
$$ LANGUAGE plpgsql;

-- 8. Function to Elect Primary Instance
CREATE OR REPLACE FUNCTION library.elect_primary_instance()
RETURNS UUID AS $$
DECLARE
    elected_id UUID;
BEGIN
    -- Clear any existing primary that's not active
    UPDATE library."Instances"
    SET "IsPrimary" = FALSE
    WHERE "IsPrimary" = TRUE
      AND ("Status" != 'Active' OR "LastHeartbeat" < NOW() - INTERVAL '1 minute');
    
    -- Check if we have an active primary
    SELECT "InstanceId" INTO elected_id
    FROM library."Instances"
    WHERE "Status" = 'Active'
      AND "IsPrimary" = TRUE
      AND "LastHeartbeat" > NOW() - INTERVAL '1 minute'
    LIMIT 1;
    
    -- If no primary, elect the oldest active instance
    IF elected_id IS NULL THEN
        SELECT "InstanceId" INTO elected_id
        FROM library."Instances"
        WHERE "Status" = 'Active'
          AND "LastHeartbeat" > NOW() - INTERVAL '1 minute'
        ORDER BY "StartedAt" ASC
        LIMIT 1;
        
        IF elected_id IS NOT NULL THEN
            UPDATE library."Instances"
            SET "IsPrimary" = TRUE
            WHERE "InstanceId" = elected_id;
        END IF;
    END IF;
    
    RETURN elected_id;
END;
$$ LANGUAGE plpgsql;

-- 9. Notification Function for Cache Invalidation
CREATE OR REPLACE FUNCTION library.notify_cache_invalidation()
RETURNS TRIGGER AS $$
BEGIN
    PERFORM pg_notify(
        'cache_invalidation',
        json_build_object(
            'id', NEW."Id",
            'cacheType', NEW."CacheType",
            'entityId', NEW."EntityId",
            'operation', NEW."Operation"
        )::text
    );
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trigger_cache_invalidation_notify
AFTER INSERT ON library."CacheInvalidations"
FOR EACH ROW
EXECUTE FUNCTION library.notify_cache_invalidation();

-- 10. Grant Permissions
GRANT ALL ON TABLE library."Instances" TO jellyfin;
GRANT ALL ON TABLE library."DistributedLocks" TO jellyfin;
GRANT ALL ON TABLE library."CacheInvalidations" TO jellyfin;
GRANT ALL ON TABLE library."FileSystemChanges" TO jellyfin;
GRANT ALL ON SEQUENCE library."CacheInvalidations_Id_seq" TO jellyfin;
GRANT ALL ON SEQUENCE library."FileSystemChanges_Id_seq" TO jellyfin;
GRANT EXECUTE ON FUNCTION library.cleanup_stale_instances() TO jellyfin;
GRANT EXECUTE ON FUNCTION library.get_primary_instance() TO jellyfin;
GRANT EXECUTE ON FUNCTION library.elect_primary_instance() TO jellyfin;

-- 11. Create Cleanup Job (Optional - can be done via scheduled task)
-- This would require pg_cron extension:
-- SELECT cron.schedule('cleanup-stale-instances', '*/1 * * * *', 
--     'SELECT library.cleanup_stale_instances()');
```

---

## Configuration File Changes

### startup.json (Instance-Specific)

```json
{
  "InstanceId": "generated-on-first-run-or-specified",
  "InstanceName": "Jellyfin-Server1",
  "DatabaseProvider": "Postgres",
  "ConnectionStrings": {
    "DefaultConnection": "Host=localhost;Database=jellyfin;Username=jellyfin;Password=***"
  },
  "EnableMultiInstance": true,
  "InstanceConfiguration": {
    "HttpPort": 8096,
    "HttpsPort": 8920,
    "CanBecomePrimary": true,
    "Capabilities": {
      "CanScan": true,
      "CanTranscode": true,
      "CanServeApi": true,
      "CanStream": true
    },
    "HeartbeatInterval": 30,
    "LockTimeout": 300
  }
}
```

---

## Usage Scenarios

### Scenario 1: Load Balanced Web Tier

```
┌─────────────┐
│ Load        │
│ Balancer    │
└──────┬──────┘
       │
  ┌────┴────┐
  │         │
┌─▼──┐   ┌─▼──┐
│ J1 │   │ J2 │  ← API Instances (Primary=False)
└─┬──┘   └─┬──┘
  │         │
  └────┬────┘
       │
  ┌────▼────┐
  │   DB    │  ← Shared PostgreSQL
  └─────────┘
```

- **J1 & J2:** Serve API requests, streaming
- **Primary Instance:** J1 (elected automatically)
- **Shared:** Database, metadata, user data
- **Isolated:** Sessions, transcoding, caches (with invalidation)

### Scenario 2: Separated Scan & Serve

```
┌──────────────┐
│ Jellyfin-Scan│ ← Primary, scans libraries
│ (Background) │
└──────┬───────┘
       │
  ┌────▼────────────┐
  │ PostgreSQL DB   │
  └────┬────────────┘
       │
  ┌────┴──────┐
  │           │
┌─▼──┐     ┌─▼──┐
│ J1 │     │ J2 │  ← Secondary, serve only
└────┘     └────┘
```

- **Scan Instance:** Primary, `CanScan=true, CanTranscode=false, CanServeApi=false`
- **Serve Instances:** Secondary, `CanServeApi=true, CanStream=true, CanTranscode=true`

---

## Performance Considerations

### Pros
- ✅ **Horizontal Scaling:** Add more instances for more concurrent users
- ✅ **High Availability:** If one instance fails, others continue
- ✅ **Specialized Instances:** Dedicate instances to specific tasks
- ✅ **Load Distribution:** Spread transcoding/streaming across instances

### Cons
- ⚠️ **Network Latency:** Database calls over network (vs local SQLite)
- ⚠️ **Cache Complexity:** Invalidation adds overhead
- ⚠️ **Locking Overhead:** Distributed locks slower than local
- ⚠️ **Configuration Complexity:** More moving parts to manage

### Mitigation
- Use aggressive caching with proper invalidation
- Minimize database round-trips (batch operations)
- Use connection pooling effectively
- Monitor instance health closely

---

## Testing Strategy

### Test Cases

1. **Instance Registration**
   - [ ] Instance registers on startup
   - [ ] Heartbeat updates every 30 seconds
   - [ ] Stale instances marked as Failed
   - [ ] Instance unregisters on clean shutdown

2. **Primary Election**
   - [ ] Primary elected on first instance start
   - [ ] Primary re-elected when primary fails
   - [ ] Only one primary at a time

3. **Distributed Locking**
   - [ ] Lock acquired successfully
   - [ ] Lock prevents concurrent access
   - [ ] Lock released on completion
   - [ ] Lock expires if holder crashes

4. **Cache Invalidation**
   - [ ] Update on instance A invalidates cache on instance B
   - [ ] Delete operation propagates
   - [ ] Notifications delivered within 1 second

5. **Session Isolation**
   - [ ] Sessions belong to specific instance
   - [ ] Sessions cleaned up on instance shutdown
   - [ ] Sessions not visible to other instances

6. **Library Scanning**
   - [ ] Only one instance scans at a time
   - [ ] Scan lock prevents conflicts
   - [ ] Other instances see scan results

---

## Rollout Plan

### Development
1. Implement Phase 1 (Instance Registration) on branch `multi-instance-testing`
2. Test with 2 instances locally
3. Implement Phase 2 (Locking)
4. Test concurrent library scans

### Staging
1. Deploy 3 instances behind load balancer
2. Run load tests
3. Test failover scenarios
4. Monitor cache invalidation performance

### Production
1. Start with 2 instances
2. Monitor for 1 week
3. Gradually add more instances
4. Document operational procedures

---

## Migration Guide for Existing Installations

### For Current Single-Instance Users

**Step 1:** Backup database
```bash
pg_dump jellyfin > jellyfin_backup.sql
```

**Step 2:** Run migration
```bash
psql -U jellyfin -d jellyfin -f multi_instance_migration.sql
```

**Step 3:** Update `startup.json`
```json
{
  "EnableMultiInstance": true
}
```

**Step 4:** Restart Jellyfin

### For New Multi-Instance Deployment

**Step 1:** Set up shared PostgreSQL database

**Step 2:** Configure first instance
```bash
./jellyfin --datadir /opt/jellyfin/instance1 \
          --port 8096 \
          --instance-name "Jellyfin-Primary"
```

**Step 3:** Configure second instance
```bash
./jellyfin --datadir /opt/jellyfin/instance2 \
          --port 8097 \
          --instance-name "Jellyfin-Secondary" \
          --can-become-primary false
```

---

## Next Steps

Would you like me to:

1. **Implement Phase 1** (Instance Registration) with full code?
2. **Create the complete EF Core migration** for multi-instance support?
3. **Implement the DistributedLockManager** service?
4. **Set up cache invalidation** with PostgreSQL LISTEN/NOTIFY?

This is a significant architectural change but achievable with the plan above. The key is implementing it in phases and testing thoroughly at each step.