Add SQL query patterns documentation and Linux package build scripts
- Created `TV_SHOWS_SQL_QUERY_PATTERNS.md` to document SQL query patterns for TV shows, including performance issues and missing indexes. - Added `README.md` for Linux package building, detailing steps for creating Debian and Red Hat packages. - Implemented build scripts for Debian and Red Hat, including service files and post-installation hooks. - Added necessary scripts for managing Jellyfin service lifecycle on both Debian and Red Hat systems. - Included package specifications and installation instructions for both distributions.
This commit is contained in:
@@ -0,0 +1,157 @@
|
||||
# TV Shows Query Performance - Quick Summary
|
||||
|
||||
## Problem Statement
|
||||
TV Shows queries are **10-100x slower** than other item types due to architectural limitations in how Series are queried and deduplicated.
|
||||
|
||||
## Critical Findings
|
||||
|
||||
### 1. Query Architecture
|
||||
**Current Flow**:
|
||||
1. Query database for matching Series (with TvExtras JOIN)
|
||||
2. Load ALL results into memory (including episodes)
|
||||
3. Deduplicate by `SeriesPresentationUniqueKey` in C#
|
||||
4. Apply paging
|
||||
5. Load full entities with all navigations (second query)
|
||||
|
||||
**Example Problem**:
|
||||
- User has 500 TV Series with 10,000 Episodes total
|
||||
- Query to display Series list:
|
||||
- Query 1: Returns 10,000 Episode rows (all must be loaded to deduplicate)
|
||||
- In-memory: Deduplicates to 500 unique Series
|
||||
- Query 2: Loads 500 Series entities
|
||||
|
||||
### 2. Missing Database Indexes
|
||||
From migration `20260226170000_AddBasePerformanceIndexes.cs`:
|
||||
|
||||
**What EXISTS:**
|
||||
- `baseitems_parentid_idx` → `(ParentId, Type)`
|
||||
- `baseitems_topparentid_idx` → `(TopParentId, Type)`
|
||||
- `baseitems_seriespresentationuniquekey_idx` → `(SeriesPresentationUniqueKey, ...)`
|
||||
|
||||
**What's MISSING:**
|
||||
- Standalone `Type` index
|
||||
- Composite `(IsFolder, Type)` index
|
||||
- Index on `TvExtras.SeriesId`
|
||||
- No filtering on `IsSeries` column
|
||||
|
||||
### 3. Inefficient Queries for Series
|
||||
|
||||
**Line 2995-3000**: IsPlayed special case
|
||||
```csharp
|
||||
// For each Series, runs a correlated subquery to check if ANY episode was played
|
||||
WHERE ... (
|
||||
SELECT 1 FROM BaseItems
|
||||
WHERE TvExtras.SeriesPresentationUniqueKey = e.PresentationUniqueKey
|
||||
AND UserData.Played = true
|
||||
)
|
||||
```
|
||||
**Impact**: 1 Series × (number of episodes to check) subqueries
|
||||
|
||||
**Line 3427-3438**: Tag inheritance for episodes
|
||||
```csharp
|
||||
// For each episode, must check both episode tags AND parent series tags
|
||||
&& (!e.TvExtras.SeriesId.HasValue ||
|
||||
!context.ItemValuesMap.Any(f =>
|
||||
f.ItemId == e.TvExtras.SeriesId.Value
|
||||
AND f.ItemValue.Type == ItemValueType.Tags))
|
||||
```
|
||||
**Impact**: Additional ItemValuesMap joins per episode with tag filters
|
||||
|
||||
### 4. In-Memory Deduplication
|
||||
**Location**: Line 814-850 in BaseItemRepository.cs
|
||||
|
||||
Uses reflection on anonymous types to deduplicate results in C#:
|
||||
```csharp
|
||||
private List<Guid> ApplyGroupingInMemory<T>(List<T> items, InternalItemsQuery filter)
|
||||
{
|
||||
// Reflects properties, uses DistinctBy()
|
||||
filtered = items.DistinctBy(e => seriesKeyProp.GetValue(e));
|
||||
return filtered.Select(...).ToList();
|
||||
}
|
||||
```
|
||||
**Why not database?** EF Core limitations - `DistinctBy()` on complex types can't be translated to SQL consistently
|
||||
|
||||
**Impact**:
|
||||
- All matching rows must be loaded from database
|
||||
- Temporary memory spike with 10,000+ item objects
|
||||
- Slow reflection-based deduplication
|
||||
|
||||
### 5. Table Joins Required
|
||||
For Series queries, ApplyNavigations (line 780-809) eagerly loads:
|
||||
- `TvExtras` - Series/Season/Episode metadata
|
||||
- `LiveTvExtras` - Live TV metadata
|
||||
- `AudioExtras` - Audio metadata
|
||||
- `Provider` - Provider IDs
|
||||
- `LockedFields` - Locked metadata
|
||||
- `UserData` - Watch history
|
||||
- `Images` - Item images
|
||||
- `TrailerTypes` - Trailer types (if requested)
|
||||
|
||||
**This creates a massive JOIN chain** instead of separate queries
|
||||
|
||||
### 6. Code Architecture Decisions
|
||||
|
||||
**Line 914**: `AsSingleQuery()`
|
||||
- Forces EF Core to use one giant query instead of splitting
|
||||
- **Good**: Avoids N+1 on navigations
|
||||
- **Bad**: One huge JOIN with poor optimization potential
|
||||
|
||||
**Line 455-460**: Select only IDs and keys, then load full entities
|
||||
- Meant to optimize but actually forces:
|
||||
1. Full Series table scan with TvExtras JOIN
|
||||
2. Select into memory
|
||||
3. Deduplicate
|
||||
4. Second query for full entities
|
||||
|
||||
## Performance Comparison
|
||||
|
||||
| Type | Bottleneck | Speed Relative to Movies |
|
||||
|------|-----------|------------------------|
|
||||
| Movies | Minimal dedup, direct Type filter | 1x (baseline) |
|
||||
| Music Albums | Moderate dedup by Album | 2-5x slower |
|
||||
| TV Shows | Massive dedup, correlated subqueries, tag inheritance | 10-100x slower |
|
||||
|
||||
## When TV Shows Are Slower
|
||||
|
||||
**❌ SLOW**:
|
||||
1. Query for Episodes: `IncludeItemTypes=Episode` (loads all 10,000 to deduplicate)
|
||||
2. Filter by tags: `Tags=Action` (must join parent Series tags)
|
||||
3. Filter by played status: `IsPlayed=true` (correlated subquery per Series)
|
||||
4. Browse large library with series+episodes mixed query
|
||||
5. Any query requesting both Series AND Episodes
|
||||
|
||||
**✅ FASTER**:
|
||||
1. Query for just Series: `IncludeItemTypes=Series` (no Episodes to load)
|
||||
2. Query with TopParentId (uses composite index)
|
||||
3. Direct queries with no deduplication needed
|
||||
4. Small libraries with few episodes per series
|
||||
|
||||
## Why This Matters
|
||||
|
||||
**Scale Impact**:
|
||||
- **100 Series × 10 Episodes each**: ~1000 rows loaded, deduplicated to 100 (10x overhead)
|
||||
- **500 Series × 50 Episodes each**: ~25,000 rows loaded, deduplicated to 500 (50x overhead)
|
||||
- **2000 Series × 100 Episodes each**: ~200,000 rows loaded, deduplicated to 2000 (100x overhead)
|
||||
|
||||
**For remote databases**: Even worse due to network latency × 50-100 overhead
|
||||
|
||||
## Recommended Fixes (Priority Order)
|
||||
|
||||
### High Priority (Quick Wins)
|
||||
1. Add index on `Type` column
|
||||
2. Add index on `(IsFolder, Type)`
|
||||
3. Add index on `TvExtras.SeriesId`
|
||||
4. Optimize IsPlayed query to avoid correlated subquery
|
||||
|
||||
### Medium Priority (Architecture)
|
||||
1. Implement caching for Series child counts
|
||||
2. Lazy-load TvExtras only when needed
|
||||
3. Create view for Series with precomputed episode counts
|
||||
|
||||
### Low Priority (Long-term)
|
||||
1. Denormalize episode count into Series row
|
||||
2. Separate read replica for expensive queries
|
||||
3. Move deduplication fully to database with DISTINCT ON
|
||||
|
||||
## Documentation
|
||||
See **TV_SHOWS_QUERY_PERFORMANCE_ANALYSIS.md** for complete details with code references and SQL examples.
|
||||
Reference in New Issue
Block a user