# TV Shows/Series Query Performance Analysis

## Executive Summary

TV Shows queries are **inherently slower** than other item types due to:
1. **Extra table joins** - TvExtras table required for Series grouping/deduplication
2. **Complex filtering logic** - Series-specific handling in IsPlayed filter and tag inheritance
3. **In-memory deduplication** - All matching episodes must be loaded before grouping (N+1 risk)
4. **Missing composite indexes** - No index on `(Type, IsFolder)` for efficient Series filtering
5. **SeriesPresentationUniqueKey deduplication** - Requires loading all results before grouping

---

## 1. TV Shows Query Logic in BaseItemRepository

### Location
[BaseItemRepository.cs](Jellyfin.Server.Implementations/Item/BaseItemRepository.cs#L421)

### Query Flow for TV Shows

**Step 1: PrepareItemQuery** (Line 911)
```csharp
private IQueryable<BaseItemEntity> PrepareItemQuery(JellyfinDbContext context, InternalItemsQuery filter)
{
    IQueryable<BaseItemEntity> dbQuery = context.BaseItems.AsNoTracking();
    dbQuery = dbQuery.AsSingleQuery();
    return dbQuery;
}
```
- Creates a base query from BaseItems table
- `AsSingleQuery()` - Prevents EF Core query splitting (important for TV Shows to avoid N+1)

**Step 2: TranslateQuery** (Line 2577)
- Applies filtering including:
  - Type filtering: `e.Type != "Series"` (for excluding Series)
  - IsSeries flag: `e.IsSeries == filter.IsSeries`
  - IsPlayed special logic (line 2995-3000)

**Step 3: ApplyOrder** - Orders results

**Step 4: ApplyGroupingInMemory** (Line 814)
- **CRITICAL**: Groups results IN MEMORY (not in database)
- Uses `DistinctBy()` with reflection on anonymous types
- For Series: `DistinctBy(e => e.TvExtras!.SeriesPresentationUniqueKey)`

**Step 5: ApplyNavigations** (Line 780)
- Eagerly loads related data via `.Include()`:
  ```csharp
  .Include(e => e.TvExtras)          // TV Series metadata
  .Include(e => e.LiveTvExtras)      // Live TV metadata  
  .Include(e => e.AudioExtras)       // Audio metadata
  .Include(e => e.Provider)          // Provider IDs
  .Include(e => e.LockedFields)      // Locked metadata fields
  .Include(e => e.UserData)          // Watch history (if EnableUserData=true)
  .Include(e => e.Images)            // Item images
  .Include(e => e.TrailerTypes)      // Trailer types (if requested)
  ```

---

## 2. Type Filtering for Series

### Location: Lines 2719-2748

**How Type Filtering Works:**

```csharp
var includeTypes = filter.IncludeItemTypes;

if (filter.IncludeItemTypes.Length == 0)
{
    var excludeTypes = filter.ExcludeItemTypes;
    if (excludeTypes.Length == 1)
    {
        baseQuery = baseQuery.Where(e => e.Type != excludeTypeName);
    }
    else if (excludeTypes.Length > 1)
    {
        var excludeTypeName = new List<string>();
        foreach (var excludeType in excludeTypes)
        {
            if (_itemTypeLookup.BaseItemKindNames.TryGetValue(excludeType, out var baseItemKindName))
            {
                excludeTypeName.Add(baseItemKindName!);
            }
        }
        baseQuery = baseQuery.Where(e => !excludeTypeName.Contains(e.Type));
    }
}
else
{
    string[] types = includeTypes.Select(f => _itemTypeLookup.BaseItemKindNames.GetValueOrDefault(f))
        .Where(e => e != null).ToArray()!;
    baseQuery = baseQuery.WhereOneOrMany(types, f => f.Type);
}
```

**Problem**: No index on `Type` column alone
- When filtering for Series: `e.Type == "Series"`
- Requires table scan or composite index
- **Index Status**: Only composite indexes exist:
  - `baseitems_parentid_idx` → `(ParentId, Type)`
  - `baseitems_topparentid_idx` → `(TopParentId, Type)`
  - **Missing**: Standalone `Type` index or `(IsFolder, Type)` composite

### Workaround Optimization
The code uses `IsFolder` to optimize Series queries since Series are folders:
```csharp
if (filter.IsFolder.HasValue)
{
    baseQuery = baseQuery.Where(e => e.IsFolder == filter.IsFolder);
}
```
**But**: This is manual and requires caller to specify `IsFolder=true` when filtering for Series.

---

## 3. Series-Specific Filtering: Child Count & Parent Relationships

### Location: Lines 3416-3438

**SeriesPresentationUniqueKey Filtering** (Line 3416-3419)
```csharp
if (!string.IsNullOrWhiteSpace(filter.SeriesPresentationUniqueKey))
{
    baseQuery = baseQuery
        .Where(e => e.TvExtras!.SeriesPresentationUniqueKey == filter.SeriesPresentationUniqueKey);
}
```
- Filters episodes by their parent Series
- Requires **join to TvExtras table**

**Tag Inheritance for Episodes** (Line 3424-3438)
```csharp
if (filter.ExcludeInheritedTags.Length > 0)
{
    var excludedTags = filter.ExcludeInheritedTags;
    baseQuery = baseQuery.Where(e =>
        !e.ItemValues!.Any(f => f.ItemValue.Type == ItemValueType.Tags && excludedTags.Contains(f.ItemValue.CleanValue))
        && (!e.TvExtras!.SeriesId.HasValue || !context.ItemValuesMap.Any(f => 
            f.ItemId == e.TvExtras!.SeriesId.Value && 
            f.ItemValue.Type == ItemValueType.Tags && 
            excludedTags.Contains(f.ItemValue.CleanValue))));
}

if (filter.IncludeInheritedTags.Length > 0)
{
    var includeTags = filter.IncludeInheritedTags;
    var isPlaylistOnlyQuery = includeTypes.Length == 1 && includeTypes.FirstOrDefault() == BaseItemKind.Playlist;
    baseQuery = baseQuery.Where(e =>
        e.ItemValues!.Any(f => f.ItemValue.Type == ItemValueType.Tags && includeTags.Contains(f.ItemValue.CleanValue))
        
        // For seasons and episodes, we also need to check the parent series' tags.
        || (e.TvExtras!.SeriesId.HasValue && context.ItemValuesMap.Any(f => 
            f.ItemId == e.TvExtras!.SeriesId.Value && 
            f.ItemValue.Type == ItemValueType.Tags && 
            includeTags.Contains(f.ItemValue.CleanValue)))
        
        || (isPlaylistOnlyQuery && e.Data!.Contains($"OwnerUserId\":\"{filter.User!.Id:N}\"")));
}
```

**Performance Issue**: 
- **For each episode**, the query checks the parent Series' tags
- Requires correlated subqueries in tag filtering
- Forces joins to TvExtras, ItemValuesMap, and ItemValues tables

### IsPlayed Special Handling (Line 2995-3000)
```csharp
if (filter.IsPlayed.HasValue)
{
    if (filter.IncludeItemTypes.Length == 1 && filter.IncludeItemTypes[0] == BaseItemKind.Series)
    {
        // SPECIAL CASE: For Series queries, check if ANY episode was played
        baseQuery = baseQuery.Where(e => 
            context.BaseItems.Where(e => e.Id != EF.Constant(PlaceholderId))
                .Where(e => e.IsFolder == false && e.IsVirtualItem == false)
                .Where(f => f.UserData!.FirstOrDefault(e => e.UserId == filter.User!.Id && e.Played)!.Played)
                .Any(f => f.TvExtras!.SeriesPresentationUniqueKey == e.PresentationUniqueKey) == filter.IsPlayed);
    }
    else
    {
        // Standard: Check if THIS item was played
        baseQuery = baseQuery
            .Select(e => new
            {
                IsPlayed = e.UserData!.Where(f => f.UserId == filter.User!.Id).Select(f => (bool?)f.Played).FirstOrDefault() ?? false,
                Item = e
            })
            .Where(e => e.IsPlayed == filter.IsPlayed)
            .Select(f => f.Item);
    }
}
```

**The Problem**: 
- **Correlated subquery inside WHERE clause**
- For each Series, database checks ALL episodes to see if ANY were played
- This is a **massive performance killer** for Series queries
- Example: 100 Series × (each checks all episodes in library) = thousands of subqueries

---

## 4. N+1 Loading Patterns Specific to TV Shows

### Location: Lines 450-530 (GetItemsAsync)

**The Two-Query Pattern:**

**Query 1: Get IDs with Keys** (Line 455-460)
```csharp
var allIds = await dbQuery
    .Select(e => new { e.Id, e.PresentationUniqueKey, SeriesPresentationUniqueKey = e.TvExtras!.SeriesPresentationUniqueKey })
    .ToListAsync(cancellationToken)
    .ConfigureAwait(false);
```
- Gets **all matching rows** (including duplicates)
- Must load TvExtras for grouping

**Query 2: Load Full Entities** (Line 475-481)
```csharp
var dbQueryWithNavs = ApplyNavigations(context.BaseItems.Where(e => pagedIds.Contains(e.Id)), filter);
var items = await dbQueryWithNavs
    .ToListAsync(cancellationToken)
    .ConfigureAwait(false);
```
- **Second query**: Loads full entities including all navigations

**The N+1 Risk for TV Shows:**
1. User has 500 Series with 10,000 Episodes total
2. When browsing "TV Shows" page:
   - User clicks "TV Shows" → wants to see list of Series (not episodes)
   - Query 1 loads ALL 10,000 episodes (to deduplicate by SeriesPresentationUniqueKey)
   - Only then groups to 500 Series
   - Query 2 loads 500 Series entities with all their data
3. If subsequent calls ask for "child count" or "episode count" per Series → additional queries per Series

### Why This is Different Than Movies/Music
- **Movies**: No grouping needed, 1:1 with items displayed
- **Music Albums**: Grouped by Album, but typically fewer duplicates
- **TV Shows**: 1 Series + N Episodes = N+1 rows to load and deduplicate

---

## 5. In-Memory Deduplication (ApplyGroupingInMemory)

### Location: Lines 814-850

```csharp
private List<Guid> ApplyGroupingInMemory<T>(List<T> items, InternalItemsQuery filter)
    where T : class
{
    var enableGroupByPresentationUniqueKey = EnableGroupByPresentationUniqueKey(filter);

    var idProp = typeof(T).GetProperty("Id");
    var presentationKeyProp = typeof(T).GetProperty("PresentationUniqueKey");
    var seriesKeyProp = typeof(T).GetProperty("SeriesPresentationUniqueKey");

    IEnumerable<T> filtered = items;

    if (enableGroupByPresentationUniqueKey && filter.GroupBySeriesPresentationUniqueKey)
    {
        filtered = items.DistinctBy(e => new
        {
            PresentationKey = presentationKeyProp.GetValue(e),
            SeriesKey = seriesKeyProp.GetValue(e)
        });
    }
    else if (enableGroupByPresentationUniqueKey)
    {
        filtered = items.DistinctBy(e => presentationKeyProp.GetValue(e));
    }
    else if (filter.GroupBySeriesPresentationUniqueKey)
    {
        filtered = items.DistinctBy(e => seriesKeyProp.GetValue(e));
    }
    else
    {
        filtered = items.DistinctBy(e => idProp.GetValue(e));
    }

    return filtered.Select(e => (Guid)idProp.GetValue(e)!).ToList();
}
```

**Why Not Database Grouping?**
- Comment in code: "See docs/query-optimization-complete-story.md"
- **EF Core limitation**: `DistinctBy()` on complex types cannot be translated to SQL
- **PostgreSQL issue**: UUID constraints prevent some grouping patterns
- **Workaround**: Use `DISTINCT ON` via `DistinctBy()` in EF Core (translates correctly)
- But `DistinctBy()` within complex SELECT is not always translatable

**Performance Impact for TV Shows:**
- Must load **ALL matching episodes** from database
- Then deduplicate in C# code using reflection
- For large libraries: 10,000+ rows loaded just to get 500 unique Series
- **Memory spike**: Temporary list of 10,000 items before deduplication

---

## 6. Database Indexes: What Exists vs. What's Missing

### Current Base Indexes on BaseItems

[Migration: 20260226170000_AddBasePerformanceIndexes.cs](src/Jellyfin.Database/Jellyfin.Database.Providers.Postgres/Migrations/20260226170000_AddBasePerformanceIndexes.cs)

**Indexes That Support Series Queries:**
| Index | Columns | Supports |
|-------|---------|----------|
| `baseitems_parentid_idx` | `(ParentId, Type)` | Series grouped by parent folder |
| `baseitems_topparentid_idx` | `(TopParentId, Type)` | Series filtered by library |
| `baseitems_seriespresentationuniquekey_idx` | `(SeriesPresentationUniqueKey, IndexNumber, ParentIndexNumber)` | ✅ Episodes by Series |
| `baseitems_sortname_idx` | `(SortName)` | Series sorted by name |
| `baseitems_premieredate_idx` | `(PremiereDate DESC)` | Series sorted by date |

**Critical Indexes Missing:**
| Index | Impact | Why Missing |
|-------|--------|------------|
| `Type` (standalone) | Full table scan when filtering by Type alone | Not included in migration |
| `(IsFolder, Type)` | Fast Series queries without ParentId filter | Not included in migration |
| `(Type, IsFolder)` | Fast queries: "all Series" without folder hierarchy | Not included in migration |
| `Type` with filter on `IsSeries=true` | IsSeries column not indexed | Not included in migration |

**Why This Matters:**
```sql
-- Current: Uses composite index (ParentId, Type) if available
SELECT * FROM BaseItems WHERE ParentId = $1 AND Type = 'Series';

-- Missing: NO index for this common query pattern
SELECT * FROM BaseItems WHERE Type = 'Series';

-- Missing: NO index for this optimization
SELECT * FROM BaseItems WHERE IsFolder = true AND Type = 'Series';
```

### TvExtras Table Indexes
- **Missing**: Index on `SeriesId` (needed for tag inheritance queries)
- **Missing**: Composite `(ItemId, SeriesPresentationUniqueKey)`

---

## 7. ApplyGroupingFilter Logic

### Location: Lines 740-778

```csharp
private IQueryable<BaseItemEntity> ApplyGroupingFilter(JellyfinDbContext context, IQueryable<BaseItemEntity> dbQuery, InternalItemsQuery filter)
{
    var enableGroupByPresentationUniqueKey = EnableGroupByPresentationUniqueKey(filter);
    
    if (enableGroupByPresentationUniqueKey && filter.GroupBySeriesPresentationUniqueKey)
    {
        // PERFORMANCE FIX: Use DistinctBy() instead of GroupBy().Select().FirstOrDefault()
        dbQuery = dbQuery.DistinctBy(e => new { e.PresentationUniqueKey, SeriesPresentationUniqueKey = e.TvExtras!.SeriesPresentationUniqueKey });
    }
    else if (enableGroupByPresentationUniqueKey)
    {
        // PERFORMANCE FIX: Use DistinctBy() to generate DISTINCT ON instead of correlated subquery
        dbQuery = dbQuery.DistinctBy(e => e.PresentationUniqueKey);
    }
    else if (filter.GroupBySeriesPresentationUniqueKey)
    {
        // PERFORMANCE FIX: Use DistinctBy() instead of GroupBy().Select().FirstOrDefault()
        dbQuery = dbQuery.DistinctBy(e => e.TvExtras!.SeriesPresentationUniqueKey);
    }
    else
    {
        dbQuery = dbQuery.Distinct();
    }

    dbQuery = ApplyOrder(dbQuery, filter, context);
    return dbQuery;
}
```

**Key Insight**: 
- `DistinctBy()` translates to PostgreSQL's `DISTINCT ON` clause
- This is **much faster** than correlated subqueries
- But still must fetch all matching rows first

**For Series Queries:**
- `GroupBySeriesPresentationUniqueKey=true`
- Translates to: `SELECT DISTINCT ON (TvExtras.SeriesPresentationUniqueKey) ...`
- This means: "Give me one row per unique Series"

**The Problem:**
- Must ORDER BY before using DISTINCT ON (PostgreSQL requirement)
- With many Episodes, sorting 10,000 rows then deduplicating is slower than database-level grouping

---

## 8. Complete Series Query Execution Example

### Scenario: User Clicks "TV Shows" in Library

**URL**: `/Items?IncludeItemTypes=Series&ParentId=X`

**What Happens:**

```
1. GetItemsAsync() called with:
   - IncludeItemTypes: [Series]
   - ParentId: X (TV Shows folder)
   - EnableTotalRecordCount: true
   
2. PrepareItemQuery()
   └─ IQueryable<BaseItemEntity> dbQuery = context.BaseItems.AsNoTracking().AsSingleQuery()
   
3. TranslateQuery()
   └─ WHERE Type = 'Series'
   └─ WHERE ParentId = X
   └─ (If indexed: uses baseitems_parentid_idx)
   
4. ApplyOrder()
   └─ ORDER BY SortName (or other sort)
   
5. QUERY 1 (Lines 455-460):
   SELECT Id, PresentationUniqueKey, TvExtras.SeriesPresentationUniqueKey
   FROM BaseItems
   LEFT JOIN TvExtras ON BaseItems.Id = TvExtras.ItemId
   WHERE ParentId = X AND Type = 'Series'
   ORDER BY SortName
   └─ Result: 500 Series rows (1 row per Series, TvExtras joined)
   
6. ApplyGroupingInMemory()
   └─ Already unique, no deduplication needed for Series-only queries
   
7. ApplyPagingToIds()
   └─ Skip 0, Take 100 (if paginated)
   
8. QUERY 2 (Lines 475-481):
   SELECT * FROM BaseItems
   LEFT JOIN TvExtras ON BaseItems.Id = TvExtras.ItemId
   LEFT JOIN Provider ON BaseItems.Id = Provider.ItemId
   LEFT JOIN Images ON BaseItems.Id = Images.ItemId
   LEFT JOIN UserData ON BaseItems.Id = UserData.ItemId
   WHERE Id IN (500 Series IDs from paged list)
   └─ Result: Full Series entities with all related data
   
9. Deserialize to DTOs
   └─ Convert BaseItemEntity to BaseItemDto
   └─ Map UserData if available
   
10. Return to client
```

**Total Queries: 2**
- **Query 1**: Get Series IDs (fast, uses index)
- **Query 2**: Get full entities (includes multiple JOINs)

**Bottleneck**: 
- If there are 10,000 Episodes with same Parent, Query 1 returns 10,000 rows (one per episode), then deduplicates
- This is rare for direct Series queries, but happens when:
  - Querying Episodes: `IncludeItemTypes=Episode&ParentId=X`
  - Querying all content: `IncludeItemTypes=[Series,Episode,Movie]`

---

## 9. Performance Comparison: Series vs. Movies vs. Music

### Query Pattern Differences

**Movies** (Fast):
```csharp
// No special handling, 1:1 with display
WHERE Type = 'Movie'
// Result: 100 rows = 100 Movies shown
```

**Music Albums** (Medium):
```csharp
// Grouped by Album name
WHERE Type = 'MusicAlbum'
SELECT DISTINCT ON (Album) ...
// Result: 100 rows = 100 Albums
```

**TV Shows** (Slow):
```csharp
// Two cases:
// Case 1: Query for Series themselves
WHERE Type = 'Series'
// Result: 500 rows = 500 Series (GOOD)

// Case 2: Query for all Episodes (common in "latest" or "browse all")
WHERE Type = 'Episode'
SELECT DISTINCT ON (TvExtras.SeriesPresentationUniqueKey) ...
// Result: 10,000 rows → deduplicate to 500 → return 100
// PROBLEM: Loaded 10,000, returned 100
```

### Why Series Are Slower

| Aspect | Movies | TV Shows | Impact |
|--------|--------|----------|--------|
| Deduplication | None | Required (Series ID) | 10-100x more rows loaded |
| Table joins | 1 (BaseItems) | 3+ (BaseItems + TvExtras + Tags checks) | Multiple JOINs per query |
| Filter complexity | Simple | Complex (tag inheritance, IsPlayed special case) | More CPU, more conditions |
| Index usage | Direct on Type | Composite (ParentId, Type) usually needed | Fewer index options |
| Related data | Simple | Children (episodes), parent (series for episodes) | More eager-load navigations |
| In-memory ops | Minimal | Full list loaded for deduplication | Memory spike |

---

## 10. Special Handling That Slows TV Shows

### 1. IsPlayed Special Case (Line 2995)
```csharp
if (filter.IncludeItemTypes.Length == 1 && filter.IncludeItemTypes[0] == BaseItemKind.Series)
{
    // Correlated subquery: For each Series, check if ANY episode was played
    baseQuery = baseQuery.Where(e => 
        context.BaseItems
            .Where(f => f.TvExtras!.SeriesPresentationUniqueKey == e.PresentationUniqueKey)
            .Any(f => f.UserData!.FirstOrDefault()!.Played));
}
```
**Impact**: Extra subquery per Series

### 2. Tag Inheritance for Episodes (Line 3427-3438)
```csharp
// Check episode's tags AND parent series' tags
&& (!e.TvExtras!.SeriesId.HasValue || 
    !context.ItemValuesMap.Any(f => 
        f.ItemId == e.TvExtras!.SeriesId.Value && 
        f.ItemValue.Type == ItemValueType.Tags))
```
**Impact**: Joins to ItemValuesMap for every tag filter on episodes

### 3. AsSingleQuery() (Line 914)
```csharp
dbQuery = dbQuery.AsSingleQuery();
```
**Impact**: Prevents EF Core from splitting queries, but means all JOINs happen in one huge query
- Better: Avoids N+1
- Worse: One massive JOIN chain for all navigations

---

## 11. Recommendations for Optimization

### Immediate (No Schema Changes)
1. **Add Index on Type column**
   ```sql
   CREATE INDEX CONCURRENTLY idx_baseitems_type 
   ON library."BaseItems" ("Type");
   ```

2. **Add Composite Index for IsFolder + Type**
   ```sql
   CREATE INDEX CONCURRENTLY idx_baseitems_isfolder_type 
   ON library."BaseItems" ("IsFolder", "Type");
   ```

3. **Add Index on TvExtras.SeriesId**
   ```sql
   CREATE INDEX CONCURRENTLY idx_baseitem_tv_extras_seriesid 
   ON library."BaseItemTvExtras" ("SeriesId");
   ```

### Short-term (Application Changes)
1. **Cache Series child counts** to avoid repeated queries
2. **Lazy-load TvExtras** only when needed (not always via Include)
3. **Optimize IsPlayed query** for Series - precompute in separate query

### Long-term (Architecture Changes)
1. **Denormalize Series child count** into BaseItems table
2. **Create materialized view** for Series with episode count, last played date
3. **Use read-only replica** for expensive Series queries
4. **Implement Series-specific cache layer** in BaseItemRepository

---

## Summary: Why TV Shows Are Slower

| Factor | Impact Level | Reason |
|--------|--------------|--------|
| **Extra Table Joins** | ⭐⭐⭐⭐⭐ | TvExtras, tag checks, user data |
| **In-Memory Deduplication** | ⭐⭐⭐⭐ | All episodes loaded, deduplicated in C# |
| **Missing Indexes** | ⭐⭐⭐⭐ | No Type or (IsFolder, Type) indexes |
| **Correlated Subqueries** | ⭐⭐⭐⭐ | IsPlayed, tag inheritance, child counts |
| **Complex Filtering** | ⭐⭐⭐ | Tag inheritance from parent Series |
| **AsSingleQuery()** | ⭐⭐ | One huge JOIN chain, not optimizable |

**Total Impact**: TV Shows queries can be **10-100x slower** than Movies, depending on library size and query type.