Implement N+1 query optimization and response caching strategies
- Added a comprehensive quick start guide for N+1 optimization in QUICK_START.md, detailing the problem, fixes, and deployment steps. - Created RESPONSE_CACHING_STRATEGY.md to outline caching strategies for Jellyfin API endpoints, including implementation details and performance projections. - Developed TECHNICAL_REFERENCE.md to document changes made in DtoService.cs, including method modifications and performance characteristics. - Introduced a PowerShell script (convert_sql_identifiers.ps1) to convert SQL identifiers from PascalCase to lowercase/snake_case for consistency in database schema.
This commit is contained in:
@@ -0,0 +1,417 @@
|
||||
# Query Path Map - Specific SQL & Code Locations
|
||||
|
||||
## Query Execution Paths
|
||||
|
||||
### Path 1: Primary Item Query (Always Executed)
|
||||
```
|
||||
Queries: 2-3 depending on ApplyGroupingInMemory logic
|
||||
|
||||
1. QUERY: Get filtered item IDs
|
||||
SELECT e.Id, e.PresentationUniqueKey,
|
||||
e.TvExtras!.SeriesPresentationUniqueKey
|
||||
FROM BaseItems e
|
||||
WHERE [filter conditions]
|
||||
ORDER BY [sort fields]
|
||||
|
||||
File: BaseItemRepository.cs:453-455
|
||||
Code: dbQuery.Select(e => new { e.Id, e.PresentationUniqueKey, ... })
|
||||
.ToListAsync()
|
||||
|
||||
2. QUERY: Load full items with navigations (INCLUDES UserData)
|
||||
SELECT [ALL COLUMNS] FROM BaseItems e
|
||||
LEFT JOIN UserData ON e.Id = UserData.ItemId
|
||||
LEFT JOIN Provider ON e.Id = Provider.ItemId
|
||||
LEFT JOIN Images ON e.Id = Images.ItemId
|
||||
WHERE e.Id IN (SELECT [paged IDs])
|
||||
|
||||
File: BaseItemRepository.cs:468-473
|
||||
Code: ApplyNavigations(context.BaseItems.Where(...), filter)
|
||||
.Include(e => e.UserData)
|
||||
.Include(e => e.Provider)
|
||||
.Include(e => e.Images)
|
||||
.ToListAsync()
|
||||
|
||||
3. QUERY: (Optional) Get TV/Audio extras if needed
|
||||
Already included via:
|
||||
.Include(e => e.TvExtras)
|
||||
.Include(e => e.LiveTvExtras)
|
||||
.Include(e => e.AudioExtras)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Path 2: ItemCounts N+1 Pattern (IF ItemCounts field requested)
|
||||
```
|
||||
Queries: +1 per item in result set
|
||||
|
||||
For each item:
|
||||
IF options.ContainsField(ItemFields.ItemCounts):
|
||||
QUERY: Get item counts for THIS ITEM
|
||||
(Query details depend on SetItemByNameInfo implementation)
|
||||
|
||||
File: DtoService.cs:182-183
|
||||
Code: for (int index = 0; index < accessibleItems.Count; index++)
|
||||
{
|
||||
if (options.ContainsField(ItemFields.ItemCounts))
|
||||
{
|
||||
SetItemByNameInfo(dto, user); // ← CALLED PER ITEM
|
||||
}
|
||||
}
|
||||
|
||||
Example: If 20 items returned:
|
||||
Base query: 2 queries
|
||||
ItemCounts: +20 queries
|
||||
TOTAL: 22 queries
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Path 3: ChildCount Potential N+1 (IF ChildCount field requested)
|
||||
```
|
||||
Queries: +1 per folder item (not for non-folder items)
|
||||
|
||||
For each folder item:
|
||||
IF options.ContainsField(ItemFields.ChildCount):
|
||||
IF folder is ICollectionFolder OR UserView:
|
||||
NO QUERY: return Random.Shared.Next(1, 10)
|
||||
ELSE:
|
||||
QUERY: Get child count for THIS folder
|
||||
File: DtoService.cs:530-533
|
||||
Code: GetChildCount(folder, user)
|
||||
→ folder.GetChildCount(user)
|
||||
|
||||
Likely query pattern:
|
||||
SELECT COUNT(*) FROM BaseItems
|
||||
WHERE ParentId = @folderId
|
||||
AND [visibility conditions for user]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Path 4: UserData Attachment (NO N+1 - Eagerly Loaded)
|
||||
```
|
||||
NO ADDITIONAL QUERIES - UserData is already in memory from Path 1
|
||||
|
||||
For each item:
|
||||
AttachUserSpecificInfo(dto, item, user, options)
|
||||
└─ GetUserData(user, item)
|
||||
└─ item.UserData?.Where(e => e.UserId.Equals(user.Id))
|
||||
|
||||
This is an in-memory LINQ query on the eagerly-loaded collection.
|
||||
|
||||
File: UserDataManager.cs:247-253
|
||||
Code: public UserItemData? GetUserData(User user, BaseItem item)
|
||||
{
|
||||
return item.UserData?.Where(e => e.UserId.Equals(user.Id))
|
||||
.Select(Map)
|
||||
.FirstOrDefault() ?? new UserItemData()
|
||||
{
|
||||
Key = item.GetUserDataKeys()[0],
|
||||
};
|
||||
}
|
||||
|
||||
NO DATABASE ACCESS - all data already loaded.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Detailed Query Analysis by Request Type
|
||||
|
||||
### Request Type A: Basic Items Listing
|
||||
```
|
||||
GET /Items?userId={id}&startIndex=0&limit=20
|
||||
```
|
||||
|
||||
**Queries Executed**:
|
||||
```
|
||||
1. Get IDs (filtering/sorting/paging)
|
||||
2. Load full items with navigations
|
||||
TOTAL: 2 queries
|
||||
```
|
||||
|
||||
**SQL Pattern**:
|
||||
```sql
|
||||
-- Query 1: Get filtered+sorted IDs
|
||||
SELECT e.Id, e.PresentationUniqueKey, e.TvExtras.SeriesPresentationUniqueKey
|
||||
FROM BaseItems e
|
||||
WHERE [filter conditions]
|
||||
ORDER BY [sort fields]
|
||||
OFFSET 0 ROWS FETCH NEXT 20 ROWS ONLY;
|
||||
|
||||
-- Query 2: Load full entities with navigations
|
||||
SELECT b.*,
|
||||
ud.Id, ud.UserId, ud.PlayCount, ...,
|
||||
p.Id, p.ProviderName, ...,
|
||||
i.Id, i.ImageType, ...,
|
||||
tv.Id, ...,
|
||||
la.Id, ...,
|
||||
ae.Id, ...
|
||||
FROM BaseItems b
|
||||
LEFT JOIN UserData ud ON b.Id = ud.ItemId
|
||||
LEFT JOIN Provider p ON b.Id = p.ItemId
|
||||
LEFT JOIN Images i ON b.Id = i.ItemId
|
||||
LEFT JOIN TvExtras tv ON b.Id = tv.ItemId
|
||||
LEFT JOIN LiveTvExtras la ON b.Id = la.ItemId
|
||||
LEFT JOIN AudioExtras ae ON b.Id = ae.ItemId
|
||||
WHERE b.Id IN (...20 IDs...);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Request Type B: With ItemCounts Field
|
||||
```
|
||||
GET /Items?userId={id}&startIndex=0&limit=20&fields=ItemCounts
|
||||
```
|
||||
|
||||
**Queries Executed**:
|
||||
```
|
||||
1. Get IDs (filtering/sorting/paging)
|
||||
2. Load full items with navigations
|
||||
3-22. SetItemByNameInfo for each of 20 items
|
||||
TOTAL: 22 queries
|
||||
```
|
||||
|
||||
**Performance Impact**: 10x slower than basic listing
|
||||
|
||||
---
|
||||
|
||||
### Request Type C: With ChildCount Field
|
||||
```
|
||||
GET /Items?userId={id}&startIndex=0&limit=20&fields=ChildCount&includeItemTypes=Folder
|
||||
```
|
||||
|
||||
**Queries Executed**:
|
||||
```
|
||||
1. Get IDs (filtering/sorting/paging)
|
||||
2. Load full items with navigations
|
||||
3-N. GetChildCount for non-collection folders
|
||||
TOTAL: 2 + (count of non-collection folders) queries
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Request Type D: Multiple Concurrent Calls (Real Page Load)
|
||||
```
|
||||
GET /Items?includeItemTypes=Movie&limit=16
|
||||
GET /Items?includeItemTypes=Series&limit=16
|
||||
GET /Items?filters=IsResumable&limit=16
|
||||
GET /Users/{userId}/Items/Resume
|
||||
GET /Playlists
|
||||
```
|
||||
|
||||
**Queries Executed** (assuming no ItemCounts/ChildCount):
|
||||
```
|
||||
Call 1 (Movies): 2 queries
|
||||
Call 2 (Series): 2 queries
|
||||
Call 3 (Resumable): 2 queries
|
||||
Call 4 (Resume items): 2 queries
|
||||
Call 5 (Playlists): 2 queries
|
||||
TOTAL: 10 queries
|
||||
```
|
||||
|
||||
**Plus if ItemCounts requested**:
|
||||
```
|
||||
Call 1: 2 + 16 = 18 queries
|
||||
Call 2: 2 + 16 = 18 queries
|
||||
Call 3: 2 + 16 = 18 queries
|
||||
Call 4: 2 + 16 = 18 queries
|
||||
Call 5: 2 + 16 = 18 queries
|
||||
TOTAL: 90 queries
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How to Trace Specific Queries
|
||||
|
||||
### Enable EF Core Command Logging
|
||||
**File**: [Jellyfin.Server/Resources/Configuration/logging.json](Jellyfin.Server/Resources/Configuration/logging.json)
|
||||
|
||||
```json
|
||||
{
|
||||
"Serilog": {
|
||||
"MinimumLevel": {
|
||||
"Default": "Information",
|
||||
"Override": {
|
||||
"Microsoft.EntityFrameworkCore.Database.Command": "Debug"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Restart and Capture Logs
|
||||
```bash
|
||||
# Clear existing logs
|
||||
sudo rm /var/log/jellyfin/log_*.log
|
||||
|
||||
# Restart service
|
||||
sudo systemctl restart jellyfin
|
||||
|
||||
# Load web page
|
||||
# (navigate to Jellyfin in browser)
|
||||
|
||||
# Examine logs
|
||||
tail -1000 /var/log/jellyfin/log_*.log | grep "Database.Command"
|
||||
|
||||
# Or search for specific query patterns
|
||||
grep "FROM BaseItems" /var/log/jellyfin/log_*.log | wc -l
|
||||
grep "FROM UserData" /var/log/jellyfin/log_*.log | wc -l
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Code Sections Responsible for Each Query
|
||||
|
||||
| Query | Triggered By | Code Location | Frequency |
|
||||
|-------|--------------|---------------|-----------|
|
||||
| **Primary ID Query** | BaseItemRepository.GetItemsAsync() | BaseItemRepository.cs:453-455 | Always (1 per request) |
|
||||
| **Load Full Items** | ApplyNavigations() | BaseItemRepository.cs:468-473 | Always (1 per request) |
|
||||
| **SetItemByNameInfo** | ItemCounts field | DtoService.cs:182-183 | Per item if requested |
|
||||
| **GetChildCount** | ChildCount field | DtoService.cs:497-506 | Per folder if requested |
|
||||
| **AttachPeople** | People field | DtoService.cs:241-244 | Per item if requested |
|
||||
| **GetStaticMediaSources** | MediaSources field | DtoService.cs:264-268 | Per item if requested |
|
||||
|
||||
---
|
||||
|
||||
## Memory vs. Database Access
|
||||
|
||||
```
|
||||
┌─ BaseItemRepository.GetItemsAsync()
|
||||
│
|
||||
├─ Query 1: Get ID list
|
||||
│ └─ Database access
|
||||
│
|
||||
├─ Query 2: Load full items + navigations
|
||||
│ ├─ Database access
|
||||
│ └─ Result: BaseItem objects in memory with:
|
||||
│ ├─ UserData collection (loaded)
|
||||
│ ├─ Provider data (loaded)
|
||||
│ ├─ Images collection (loaded)
|
||||
│ └─ Etc.
|
||||
│
|
||||
└─ DtoService.GetBaseItemDtos()
|
||||
│
|
||||
└─ For each item in memory:
|
||||
├─ GetBaseItemDtoInternal()
|
||||
│ ├─ AttachUserSpecificInfo()
|
||||
│ │ └─ GetUserData()
|
||||
│ │ └─ Access item.UserData (memory - NO DATABASE)
|
||||
│ │
|
||||
│ ├─ AttachStudios() (memory)
|
||||
│ ├─ AttachBasicFields() (memory)
|
||||
│ └─ Etc.
|
||||
│
|
||||
├─ SetItemByNameInfo() (POTENTIAL DATABASE if requested)
|
||||
│
|
||||
└─ GetChildCount() (POTENTIAL DATABASE if requested)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Query Optimization Opportunities
|
||||
|
||||
### Optimization 1: Batch ItemCounts
|
||||
**Current Cost**: N queries (one per item)
|
||||
**Optimized Cost**: 1 query (batch operation)
|
||||
|
||||
```csharp
|
||||
// BEFORE (Current - N queries)
|
||||
foreach (var item in items)
|
||||
{
|
||||
SetItemByNameInfo(dto, user); // Database query per item
|
||||
}
|
||||
|
||||
// AFTER (Optimized - 1 query)
|
||||
var itemIds = items.Select(i => i.Id).ToArray();
|
||||
var allCounts = await GetItemCountsBatch(itemIds, user);
|
||||
foreach (var dto in dtos)
|
||||
{
|
||||
if (allCounts.TryGetValue(dto.Id, out var counts))
|
||||
{
|
||||
dto.ItemCounts = counts;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Optimization 2: Cache ChildCount
|
||||
**Current Cost**: N queries (one per folder per request)
|
||||
**Optimized Cost**: 1 query (one-time cache)
|
||||
|
||||
```csharp
|
||||
// Use a memory cache keyed by FolderId + UserId
|
||||
private readonly IMemoryCache _childCountCache;
|
||||
|
||||
private int GetChildCount(Folder folder, User user)
|
||||
{
|
||||
var cacheKey = $"ChildCount_{folder.Id}_{user.Id}";
|
||||
if (_childCountCache.TryGetValue(cacheKey, out int count))
|
||||
{
|
||||
return count;
|
||||
}
|
||||
|
||||
count = folder.GetChildCount(user);
|
||||
_childCountCache.Set(cacheKey, count, TimeSpan.FromMinutes(5));
|
||||
return count;
|
||||
}
|
||||
```
|
||||
|
||||
### Optimization 3: Consolidate UI Requests
|
||||
**Current Cost**: 5+ simultaneous requests = 10+ queries
|
||||
**Optimized Cost**: 1 batch request = 2 queries
|
||||
|
||||
```javascript
|
||||
// BEFORE (Current - 5 API calls)
|
||||
Promise.all([
|
||||
fetch('/Items?includeItemTypes=Movie&limit=16'),
|
||||
fetch('/Items?includeItemTypes=Series&limit=16'),
|
||||
fetch('/Items?filters=IsResumable&limit=16'),
|
||||
fetch('/Users/{userId}/Items/Resume'),
|
||||
fetch('/Playlists')
|
||||
]);
|
||||
|
||||
// AFTER (Optimized - 1 API call to get all)
|
||||
fetch('/Items?batch=true&includeItemTypes=Movie,Series&limit=16')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Capture query statistics during page load
|
||||
|
||||
echo "Query execution started at $(date)"
|
||||
tail -f /var/log/jellyfin/log_*.log | \
|
||||
grep -E "Database.Command|Executing|milliseconds" | \
|
||||
while read line; do
|
||||
if [[ $line == *"FROM"* ]]; then
|
||||
echo "📊 QUERY: $(echo $line | grep -oE 'FROM [^ ]+')"
|
||||
elif [[ $line == *"Executing"* ]]; then
|
||||
echo "⏱️ TIMING: $line"
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
Usage:
|
||||
```bash
|
||||
chmod +x monitor_queries.sh
|
||||
./monitor_queries.sh &
|
||||
|
||||
# In another terminal, load the page
|
||||
firefox http://localhost:8096
|
||||
|
||||
# Stop monitoring
|
||||
kill %1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- **Main Analysis**: [QUERY_EXECUTION_ANALYSIS.md](QUERY_EXECUTION_ANALYSIS.md)
|
||||
- **Remediation Guide**: [QUERY_ISSUES_REMEDIATION.md](QUERY_ISSUES_REMEDIATION.md)
|
||||
- **Source Files**:
|
||||
- BaseItemRepository: [Jellyfin.Server.Implementations/Item/BaseItemRepository.cs](Jellyfin.Server.Implementations/Item/BaseItemRepository.cs)
|
||||
- DtoService: [Emby.Server.Implementations/Dto/DtoService.cs](Emby.Server.Implementations/Dto/DtoService.cs)
|
||||
- ItemsController: [Jellyfin.Api/Controllers/ItemsController.cs](Jellyfin.Api/Controllers/ItemsController.cs)
|
||||
Reference in New Issue
Block a user