pgsql-jellyfin/docs/database-query-optimization.md

Database Query Optimization Guide

Query Performance Issues and Solutions

Issue: Correlated Subquery Performance Problem

Problem Description

Prior to optimization, queries using GroupBy().Select(e => e.FirstOrDefault()).Select(e => e.Id) pattern were generating inefficient correlated subqueries in PostgreSQL, causing timeout errors (30+ seconds).

Example of problematic pattern:

var tempQuery = dbQuery
    .GroupBy(e => e.PresentationUniqueKey)
    .Select(e => e.FirstOrDefault())
    .Select(e => e!.Id);
dbQuery = context.BaseItems.Where(e => tempQuery.Contains(e.Id));

Generated SQL (PROBLEMATIC):

WHERE b.Id IN (
    SELECT (
        SELECT b1.Id
        FROM library.BaseItems AS b1
        WHERE b0.PresentationUniqueKey = b1.PresentationUniqueKey
        LIMIT 1
    )
    FROM library.BaseItems AS b0
    GROUP BY b0.PresentationUniqueKey
)

This creates a correlated subquery that executes once for each group, resulting in exponential performance degradation.

Solution

Replace FirstOrDefault() with Min(x => x.Id) to generate an efficient aggregate query:

Optimized pattern:

var tempQuery = dbQuery
    .GroupBy(e => e.PresentationUniqueKey)
    .Select(e => e.Min(x => x.Id));
dbQuery = context.BaseItems.Where(e => tempQuery.Contains(e.Id));

Generated SQL (OPTIMIZED):

WHERE b.Id IN (
    SELECT MIN(b0.Id)
    FROM library.BaseItems AS b0
    GROUP BY b0.PresentationUniqueKey
)

This generates a simple GROUP BY with MIN aggregate, which PostgreSQL can optimize efficiently.

Performance Impact

• Before: 30,000+ ms (timeout)
• After: ~10-50 ms (estimated based on query complexity)
• Improvement: ~600-3000x faster

Affected Queries

This optimization was applied to:

1. ApplyGroupingFilter with GroupBySeriesPresentationUniqueKey and PresentationUniqueKey
2. All three grouping scenarios in the BaseItemRepository

Testing

After applying this optimization:

1. Monitor query logs to verify improved SQL generation
2. Test library browsing performance, especially for:
   • TV show episode lists
   • Duplicate media detection
   • Collection grouping
3. Verify that the "first" item from each group is consistently selected (by Id ordering)

Notes

• Using Min(Id) instead of FirstOrDefault() ensures deterministic selection
• The selected item will be the one with the lowest GUID value in each group
• This change maintains functional equivalence while dramatically improving performance
• If a different selection criterion is needed (e.g., by date), use Min(x => x.DateCreated) and join back to get the Id

Related Files

• Jellyfin.Server.Implementations/Item/BaseItemRepository.cs - Line 572-603 (ApplyGroupingFilter method)

Additional Recommendations

1. Consider Using AsSplitQuery() for Related Data

When loading items with multiple relationships (providers, images, user data), consider using split queries:

dbQuery = dbQuery
    .Include(e => e.Provider)
    .Include(e => e.UserData)
    .Include(e => e.Images)
    .AsSplitQuery();  // Prevents cartesian explosion

2. Increase Command Timeout for Complex Queries

If queries legitimately need more time, increase the command timeout in DbContext configuration:

opt.CommandTimeout(60);  // 60 seconds

3. Database Indexing

Ensure proper indexes exist on:

• BaseItems.PresentationUniqueKey
• BaseItems.SeriesPresentationUniqueKey
• BaseItems.IsVirtualItem
• BaseItems.TopParentId

Check with:

SELECT * FROM pg_indexes WHERE tablename = 'BaseItems';

4. Query Logging Configuration

To debug slow queries, enable Entity Framework Core query logging in logging.json:

{
    "Serilog": {
        "MinimumLevel": {
            "Override": {
                "Microsoft.EntityFrameworkCore.Database.Command": "Debug"
            }
        }
    }
}

See src/Jellyfin.Database/readme.md for more details on query logging.