wjones/pgsql-jellyfin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

PostgreSQL: Production fixes—UPSERT, remote backup, auth logs

Browse Source 
- Fix: Atomic UPSERT for BaseItemProviders (resolves duplicate key errors during concurrent metadata refresh; clears navigation property to prevent EF Core tracking conflicts)
- Add: Remote PostgreSQL backup support (removes localhost-only restriction; works with pg_dump/pg_restore for both local and remote DBs)
- Add: Configurable backup disable option (`disable-backups`)
- Fix: Query timeout and performance (documented `command-timeout` config, added performance index scripts)
- Fix: Authentication errors now log as warnings with clear messages (ExceptionMiddleware), reducing log noise
- Fix: SyncPlay authorization handler validates user before lookup, logs warnings for unauthenticated/unknown users (returns 403/404)
- Fix: Database deadlock detection logs warnings and allows EF Core auto-retry
- Add: Configurable LibraryMonitorDelay (min 30s, default 60s)
- Fix: SQLite migration filtering—skip SQLite-only migrations on PostgreSQL
- Chore: Suppress StyleCop warnings (SA1137, etc.) for project consistency
- Docs: 21 documentation files added/updated (config, backup, performance, troubleshooting, session summary)
- All changes are backward compatible and production-ready
This commit is contained in:
Wendell Jones
2026-03-03 16:35:27 -05:00
parent 163a037642
commit c76853a442
27 changed files with 3320 additions and 260 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Jellyfin.Database/Jellyfin.Database.Providers.Postgres/BACKUP_CONFIGURATION.md
-258
View File
@@ -1,258 +0,0 @@
# PostgreSQL Backup Configuration
This document explains how to configure PostgreSQL backup options in Jellyfin using the `database.xml` configuration file.
## **IMPORTANT: Local Database Requirement**
**pg_dump and pg_restore backup features are ONLY available when the PostgreSQL database host is `localhost` or `127.0.0.1` (or `::1` for IPv6).**
- ✅ **Local database** (localhost/127.0.0.1): Automated backups via pg_dump/pg_restore are enabled
- ❌ **Remote database**: Backup settings are ignored; you must manage backups externally
When using a remote PostgreSQL server, Jellyfin will log:
> "PostgreSQL database is on remote server ({Host}). Backup operations via pg_dump/pg_restore are disabled"
And backup operations will show:
> "Backup deletion is not implemented for PostgreSQL on a remote server. Manage backups externally."
## Configuration File Location
The `database.xml` file should be placed in your Jellyfin configuration directory:
- **Windows**: `%AppData%\Jellyfin\Server\config\database.xml`
- **Linux**: `~/.config/jellyfin/config/database.xml` or `/etc/jellyfin/config/database.xml`
- **Docker**: Mount to `/config/database.xml`
## Basic Configuration
Here's a minimal configuration for PostgreSQL with default backup settings:
```xml
<DatabaseConfigurationOptions>
<DatabaseType>Jellyfin-PostgreSQL</DatabaseType>
<LockingBehavior>NoLock</LockingBehavior>
<CustomProviderOptions>
<PluginName>Jellyfin-PostgreSQL</PluginName>
<PluginAssembly>Jellyfin.Database.Providers.Postgres</PluginAssembly>
<ConnectionString>Host=localhost;Port=5432;Database=jellyfin;Username=jellyfin;Password=your_password</ConnectionString>
</CustomProviderOptions>
<!-- Backup configuration (all settings are optional) -->
<BackupOptions>
<PgDumpPath>pg_dump</PgDumpPath>
<BackupFormat>custom</BackupFormat>
</BackupOptions>
</DatabaseConfigurationOptions>
```
## Backup Options Reference
### PgDumpPath
**Type**: `string` (optional)
**Default**: Searches in system PATH
The full path to the `pg_dump` executable. If not specified, Jellyfin will search common locations:
- Windows: `C:\Program Files\PostgreSQL\{version}\bin\pg_dump.exe`
- Linux: `/usr/bin/pg_dump`, `/usr/local/bin/pg_dump`
**Examples**:
```xml
<!-- Windows -->
<PgDumpPath>C:\Program Files\PostgreSQL\16\bin\pg_dump.exe</PgDumpPath>
<!-- Linux -->
<PgDumpPath>/usr/bin/pg_dump</PgDumpPath>
<!-- Search in PATH -->
<PgDumpPath>pg_dump</PgDumpPath>
```
### PgRestorePath
**Type**: `string` (optional)
**Default**: Searches in system PATH
Path to the `pg_restore` executable. Same search behavior as `PgDumpPath`.
### BackupFormat
**Type**: `string`
**Default**: `custom`
**Valid values**: `custom`, `plain`, `directory`, `tar`
The backup file format:
- **custom**: Compressed custom format (recommended) - can be restored with `pg_restore`
- **plain**: Plain SQL script - can be restored with `psql`
- **directory**: Directory format with one file per table
- **tar**: Tar archive format
```xml
<BackupFormat>custom</BackupFormat>
```
### IncludeBlobs
**Type**: `boolean`
**Default**: `true`
Whether to include large objects (BLOBs) in the backup.
```xml
<IncludeBlobs>true</IncludeBlobs>
```
### CompressionLevel
**Type**: `integer` (0-9)
**Default**: `6`
Compression level for custom and directory formats. Higher values mean better compression but slower speed.
- `0` = No compression
- `9` = Maximum compression
- `6` = Good balance (recommended)
```xml
<CompressionLevel>6</CompressionLevel>
```
### TimeoutSeconds
**Type**: `integer`
**Default**: `1800` (30 minutes)
Maximum time in seconds to wait for backup/restore operations to complete.
```xml
<TimeoutSeconds>1800</TimeoutSeconds>
```
### VerboseOutput
**Type**: `boolean`
**Default**: `true`
Enable verbose logging output from pg_dump/pg_restore operations.
```xml
<VerboseOutput>true</VerboseOutput>
```
### ParallelJobs
**Type**: `integer` (optional)
**Default**: `null`
Number of parallel jobs for backup operations. Only works with `directory` format.
```xml
<BackupFormat>directory</BackupFormat>
<ParallelJobs>4</ParallelJobs>
```
### AdditionalArguments
**Type**: `string` (optional)
**Default**: `null`
Additional command-line arguments to pass to pg_dump.
**Examples**:
```xml
<!-- Exclude specific table data -->
<AdditionalArguments>--exclude-table-data=public.temp_logs</AdditionalArguments>
<!-- Exclude schema -->
<AdditionalArguments>--exclude-schema=archive</AdditionalArguments>
<!-- Multiple arguments -->
<AdditionalArguments>--exclude-table-data=public.logs --exclude-schema=test</AdditionalArguments>
```
## Complete Example Configuration
```xml
<DatabaseConfigurationOptions>
<DatabaseType>Jellyfin-PostgreSQL</DatabaseType>
<LockingBehavior>NoLock</LockingBehavior>
<CustomProviderOptions>
<PluginName>Jellyfin-PostgreSQL</PluginName>
<PluginAssembly>Jellyfin.Database.Providers.Postgres</PluginAssembly>
<ConnectionString>Host=localhost;Port=5432;Database=jellyfin;Username=jellyfin;Password=your_password</ConnectionString>
</CustomProviderOptions>
<BackupOptions>
<!-- Paths to executables -->
<PgDumpPath>C:\Program Files\PostgreSQL\16\bin\pg_dump.exe</PgDumpPath>
<PgRestorePath>C:\Program Files\PostgreSQL\16\bin\pg_restore.exe</PgRestorePath>
<!-- Backup format and compression -->
<BackupFormat>custom</BackupFormat>
<CompressionLevel>6</CompressionLevel>
<IncludeBlobs>true</IncludeBlobs>
<!-- Performance and timeout -->
<TimeoutSeconds>3600</TimeoutSeconds>
<VerboseOutput>true</VerboseOutput>
<!-- Optional: Parallel processing for directory format -->
<!-- <ParallelJobs>4</ParallelJobs> -->
<!-- Optional: Additional pg_dump arguments -->
<!-- <AdditionalArguments>--exclude-table-data=public.temp_data</AdditionalArguments> -->
</BackupOptions>
</DatabaseConfigurationOptions>
```
## Usage in Code
The backup service will automatically read these settings from `database.xml`:
```csharp
// Inject the service
public class MyClass
{
private readonly PostgresBackupService _backupService;
public MyClass(PostgresBackupService backupService)
{
_backupService = backupService;
}
// Create a backup
public async Task CreateBackup()
{
var backupPath = await _backupService.CreateBackupAsync("/path/to/backups");
Console.WriteLine($"Backup created: {backupPath}");
}
// Restore a backup
public async Task RestoreBackup(string backupFile)
{
await _backupService.RestoreBackupAsync(backupFile);
Console.WriteLine("Backup restored successfully");
}
}
```
## Troubleshooting
### "pg_dump executable not found"
1. Verify PostgreSQL is installed
2. Check that `pg_dump` is in your system PATH, or
3. Specify the full path in `<PgDumpPath>` configuration
### Permission Issues
- Ensure the user running Jellyfin has execute permissions for `pg_dump`
- Verify database credentials have backup privileges
### Backup Takes Too Long
- Increase `<TimeoutSeconds>` value
- Consider using `<ParallelJobs>` with directory format for large databases
- Reduce `<CompressionLevel>` for faster backups
### Restore Fails
- Verify the backup file format matches what pg_restore expects
- Check that the target database exists and is accessible
- Review logs for specific error messages
## Security Notes
1. **Password Storage**: The connection string in `database.xml` contains the database password. Ensure this file has appropriate permissions (readable only by the Jellyfin user).
2. **PGPASSWORD**: The service uses the `PGPASSWORD` environment variable to pass credentials to pg_dump, which is more secure than command-line arguments.
3. **Backup Files**: Backup files contain your entire database. Store them securely and restrict access appropriately.
src/Jellyfin.Database/Jellyfin.Database.Providers.Postgres/MIGRATION_GUIDE.md
-215
View File
@@ -1,215 +0,0 @@
# PostgreSQL Database Provider - Migration Guide
## Overview
The `Jellyfin.Database.Providers.Postgres` project has been successfully created as a PostgreSQL alternative to the SQLite provider. This document outlines the key changes and migration path.
## Project Structure
```
src/Jellyfin.Database/Jellyfin.Database.Providers.Postgres/
├── Jellyfin.Database.Providers.Postgres.csproj
├── PostgresDatabaseProvider.cs
├── ModelBuilderExtensions.cs
├── GlobalSuppressions.cs
├── Properties/
│ └── AssemblyInfo.cs
├── ValueConverters/
│ └── DateTimeKindValueConverter.cs
├── Migrations/
│ └── PostgresDesignTimeJellyfinDbFactory.cs
└── README.md
```
## Key Differences from SQLite Provider
### 1. Database Provider Attribute
```csharp
[JellyfinDatabaseProviderKey("Jellyfin-PostgreSQL")]
```
### 2. Connection String Builder
- SQLite: `SqliteConnectionStringBuilder`
- PostgreSQL: `NpgsqlConnectionStringBuilder`
### 3. Connection Options
**SQLite:**
- DataSource (file path)
- Cache mode
- Journal mode
- Pragma settings
**PostgreSQL:**
- Host, Port
- Database name
- Username, Password
- Connection pooling
- Timeout settings
### 4. Optimization Commands
**SQLite:**
```csharp
PRAGMA wal_checkpoint(TRUNCATE)
PRAGMA optimize
VACUUM
```
**PostgreSQL:**
```csharp
VACUUM ANALYZE
```
### 5. Data Purge
**SQLite:**
```sql
DELETE FROM "table";
```
**PostgreSQL:**
```sql
TRUNCATE TABLE "table" CASCADE;
```
### 6. Backup/Restore
- **SQLite**: File-based backup (Copy jellyfin.db)
- **PostgreSQL**: Use `pg_dump` and `pg_restore` tools
## Configuration Example
### appsettings.json
```json
{
"Database": {
"Provider": "Jellyfin-PostgreSQL",
"CustomProviderOptions": {
"Options": [
{ "Key": "host", "Value": "localhost" },
{ "Key": "port", "Value": "5432" },
{ "Key": "database", "Value": "jellyfin" },
{ "Key": "username", "Value": "jellyfin" },
{ "Key": "password", "Value": "your_password" }
]
}
}
}
```
## Migration from SQLite to PostgreSQL
### 1. Export Data from SQLite
```bash
# Use a tool like pgloader or write custom migration scripts
sqlite3 jellyfin.db .dump > jellyfin_dump.sql
```
### 2. Transform SQL (if needed)
SQLite-specific SQL may need transformation for PostgreSQL compatibility:
- INTEGER PRIMARY KEY → SERIAL PRIMARY KEY
- DATETIME → TIMESTAMP
- Boolean values (0/1 → false/true)
### 3. Import to PostgreSQL
```bash
# Create database
createdb -U jellyfin jellyfin
# Import (after transformation)
psql -U jellyfin -d jellyfin -f transformed_dump.sql
```
### 4. Update Configuration
Point Jellyfin to PostgreSQL using the configuration above.
## Creating Migrations
```bash
cd src/Jellyfin.Database/Jellyfin.Database.Providers.Postgres
# Add migration
dotnet ef migrations add MigrationName --context JellyfinDbContext
# Apply migrations
dotnet ef database update --context JellyfinDbContext
```
## Package Dependencies
- `Npgsql.EntityFrameworkCore.PostgreSQL` 9.0.2 (with EF Core 10 override)
- `Microsoft.EntityFrameworkCore` 10.0.3
- `Microsoft.EntityFrameworkCore.Relational` 10.0.3
- `Microsoft.EntityFrameworkCore.Design` 10.0.3
- `Microsoft.EntityFrameworkCore.Tools` 10.0.3
**Note**: When Npgsql 10.x is released, update `Directory.Packages.props`.
## Testing
### Unit Tests
Tests should be added to verify:
1. Connection string building
2. Migration application
3. CRUD operations
4. Transaction handling
### Integration Tests
1. Test against real PostgreSQL instance
2. Verify data type mappings
3. Test concurrent access
4. Performance benchmarks
## Known Limitations
1. **Version Compatibility**: Using Npgsql 9.x with EF Core 10.x (temporary)
2. **Backup/Restore**: Not implemented within provider (use PostgreSQL tools)
3. **Migrations**: Will need to be generated fresh (not ported from SQLite)
## Advantages of PostgreSQL
1. **Multi-user Support**: Better concurrent access handling
2. **ACID Compliance**: Full transaction support
3. **Scalability**: Better performance with large datasets
4. **JSON Support**: Native JSONB type for complex data
5. **Replication**: Built-in master-slave replication
6. **Extensions**: PostGIS, full-text search, etc.
## Next Steps
1. **Generate Initial Migration**
```bash
dotnet ef migrations add InitialCreate --context JellyfinDbContext
```
2. **Test with Development Database**
- Set up local PostgreSQL
- Apply migrations
- Test Jellyfin functionality
3. **Performance Testing**
- Compare query performance vs SQLite
- Optimize indexes
- Tune PostgreSQL configuration
4. **Documentation**
- Update user documentation
- Add deployment guides
- Create troubleshooting guide
5. **CI/CD Integration**
- Add PostgreSQL to test pipeline
- Test migrations in CI
- Performance regression tests
## Support
For issues specific to the PostgreSQL provider:
1. Check PostgreSQL logs: `/var/log/postgresql/`
2. Enable EF Core logging in Jellyfin
3. Verify PostgreSQL version compatibility (12+)
4. Check connection string in configuration
## References
- [Npgsql Documentation](https://www.npgsql.org/efcore/)
- [PostgreSQL Documentation](https://www.postgresql.org/docs/)
- [Entity Framework Core](https://docs.microsoft.com/en-us/ef/core/)
- [Jellyfin Database Architecture](https://jellyfin.org/docs/)
src/Jellyfin.Database/Jellyfin.Database.Providers.Postgres/PostgresDatabaseProvider.cs
+31 -8
View File
@@ -146,17 +146,40 @@ public sealed class PostgresDatabaseProvider : IJellyfinDatabaseProvider
connectionBuilder.MaxPoolSize,
connectionBuilder.Multiplexing);
// Initialize backup service if host is local and configuration manager is available
if (IsLocalHost(currentHost) && configurationManager is not null)
// Initialize backup service if configuration manager is available
// Backup works for both local and remote servers via pg_dump/pg_restore client tools
if (configurationManager is not null)
{
var loggerFactory = Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance;
var backupLogger = loggerFactory.CreateLogger<PostgresBackupService>();
backupService = new PostgresBackupService(backupLogger, configurationManager);
logger.LogInformation("PostgreSQL backup service initialized for local database (will use external pg_dump/pg_restore tools)");
// Check if backups are explicitly disabled
var disableBackups = GetOption(customOptions, "disable-backups", e => e.Equals(bool.TrueString, StringComparison.OrdinalIgnoreCase), () => false);
if (disableBackups)
{
logger.LogInformation("PostgreSQL backup service is disabled (disable-backups=true in configuration)");
backupService = null;
}
else
{
var loggerFactory = Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance;
var backupLogger = loggerFactory.CreateLogger<PostgresBackupService>();
backupService = new PostgresBackupService(backupLogger, configurationManager);
if (IsLocalHost(currentHost))
{
logger.LogInformation("PostgreSQL backup service initialized for local database at {Host} (using pg_dump/pg_restore tools)", currentHost);
}
else
{
logger.LogInformation("PostgreSQL backup service initialized for remote database at {Host} (using pg_dump/pg_restore tools)", currentHost);
logger.LogWarning("Note: Ensure pg_dump and pg_restore client tools are installed locally and network connectivity to {Host} is available", currentHost);
}
logger.LogInformation("PostgreSQL client binaries (pg_dump/pg_restore) must be in system PATH or specify full paths in BackupOptions");
}
}
else if (!IsLocalHost(currentHost))
else
{
logger.LogInformation("PostgreSQL database is on remote server ({Host}). Backup operations via external pg_dump/pg_restore tools are disabled", currentHost);
logger.LogWarning("Configuration manager not available - PostgreSQL backup service disabled");
}
options
src/Jellyfin.Database/Jellyfin.Database.Providers.Postgres/README.md
-132
View File
@@ -1,132 +0,0 @@
# Jellyfin.Database.Providers.Postgres
PostgreSQL database provider for Jellyfin.
## ⚠️ Important Note
This provider currently uses `Npgsql.EntityFrameworkCore.PostgreSQL 9.0.2` with EF Core 10.0.3, which requires overriding package version constraints. This is a temporary workaround until Npgsql releases a version compatible with EF Core 10.
**Compatibility Warning**: This may cause runtime issues. Monitor for:
- Npgsql.EntityFrameworkCore.PostgreSQL 10.x release
- Update `Directory.Packages.props` when available
## Configuration
To use PostgreSQL as the database backend, configure the following options in your Jellyfin configuration:
```json
{
"Database": {
"Provider": "Jellyfin-PostgreSQL",
"CustomProviderOptions": {
"Options": [
{ "Key": "host", "Value": "localhost" },
{ "Key": "port", "Value": "5432" },
{ "Key": "database", "Value": "jellyfin" },
{ "Key": "username", "Value": "jellyfin" },
{ "Key": "password", "Value": "your_secure_password" },
{ "Key": "pooling", "Value": "true" },
{ "Key": "max-pool-size", "Value": "100" },
{ "Key": "min-pool-size", "Value": "0" },
{ "Key": "command-timeout", "Value": "30" },
{ "Key": "connection-timeout", "Value": "15" }
]
}
}
}
```
## Database Setup
Before using this provider, ensure PostgreSQL is installed and create the database:
```sql
CREATE DATABASE jellyfin;
CREATE USER jellyfin WITH PASSWORD 'your_secure_password';
GRANT ALL PRIVILEGES ON DATABASE jellyfin TO jellyfin;
```
## Creating Migrations
To create a new migration:
```bash
cd src/Jellyfin.Database/Jellyfin.Database.Providers.Postgres
dotnet ef migrations add YourMigrationName --context JellyfinDbContext
```
## Applying Migrations
Migrations will be applied automatically when Jellyfin starts. You can also apply them manually:
```bash
dotnet ef database update --context JellyfinDbContext
```
## Backup and Restore
Unlike SQLite, PostgreSQL backups should be managed externally using PostgreSQL tools:
```bash
# Backup
pg_dump -U jellyfin -h localhost jellyfin > jellyfin_backup.sql
# Restore
psql -U jellyfin -h localhost jellyfin < jellyfin_backup.sql
```
## Configuration Options
| Key | Default | Description |
|-----|---------|-------------|
| host | localhost | PostgreSQL server hostname |
| port | 5432 | PostgreSQL server port |
| database | jellyfin | Database name |
| username | jellyfin | Database username |
| password | (empty) | Database password |
| pooling | true | Enable connection pooling |
| max-pool-size | 100 | Maximum number of connections in the pool |
| min-pool-size | 0 | Minimum number of connections in the pool |
| command-timeout | 30 | Command timeout in seconds |
| connection-timeout | 15 | Connection timeout in seconds |
| multiplexing | false | ⚠️ **Advanced**: Enable command multiplexing (requires all async operations) |
| EnableSensitiveDataLogging | false | Enable sensitive data logging (for debugging) |
### ⚠️ Multiplexing Warning
**Multiplexing is disabled by default** because it requires all database operations to be asynchronous. Enabling multiplexing will cause errors like:
```
System.NotSupportedException: Synchronous command execution is not supported when multiplexing is on
```
Only enable multiplexing if you have modified Jellyfin code to use fully async database operations.
## Performance Tuning
For better performance, consider:
1. **Indexes**: The provider will create necessary indexes through migrations
2. **Connection Pooling**: Enabled by default with max 100 connections
- Adjust `max-pool-size` based on your concurrent user count
- Higher values allow more simultaneous database operations but use more resources
3. **Vacuum**: Scheduled optimization runs `VACUUM ANALYZE` periodically
4. **PostgreSQL Configuration**: Adjust `shared_buffers`, `effective_cache_size`, etc. in postgresql.conf
### Connection Pool Sizing
A good rule of thumb for `max-pool-size`:
- **Small deployments** (1-10 users): 20-50 connections
- **Medium deployments** (10-50 users): 50-100 connections
- **Large deployments** (50+ users): 100-200 connections
Monitor your PostgreSQL server's active connections with:
```sql
SELECT count(*) FROM pg_stat_activity WHERE datname = 'jellyfin';
```
## Notes
- Migrations from SQLite are not automatically handled. You'll need to export data from SQLite and import into PostgreSQL manually.
- The provider uses UTC timestamps throughout for consistency.
- Transaction isolation level and other PostgreSQL-specific features can be configured through Npgsql connection string parameters.