wjones
e80dbd757b
- Added QUERY_PATH_MAP.md to document query execution paths and analysis. - Created QUICK_START.md for a quick guide on N+1 optimization implementation. - Introduced RESPONSE_CACHING_STRATEGY.md outlining caching strategies for API endpoints. - Developed TECHNICAL_REFERENCE.md detailing changes made in DtoService.cs for N+1 optimization. - Optimized item counts retrieval by batching queries, reducing database load significantly. - Implemented caching for child counts to minimize repeated database queries. - Enhanced performance metrics showing substantial improvements in query counts and page load times.
480 lines
17 KiB
Markdown
480 lines
17 KiB
Markdown
# Query Execution Analysis: Web UI Page Load - Multiple Query Issue
|
|
|
|
## Executive Summary
|
|
|
|
The query is NOT executed multiple times due to N+1 problems during **eager loading** of the initial results. Instead, UserData **should be** eagerly loaded by the main query via `.Include(e => e.UserData)`. If multiple queries are observed, the issue is likely:
|
|
|
|
1. **UI making multiple API calls** to different endpoints (different item types/collections)
|
|
2. **Multiple simultaneous requests** for different pages or items
|
|
3. **DbContext disposal** causing lazy loading fallback
|
|
4. **Optional field requests** triggering separate queries (like ItemCounts, ChildCount, etc.)
|
|
|
|
---
|
|
|
|
## Flow Diagram: Web UI Page Load → Database Queries
|
|
|
|
```
|
|
┌─────────────────────────────────────────┐
|
|
│ Web UI Page Load │
|
|
│ (Browser renders views/components) │
|
|
└─────────────────────┬───────────────────┘
|
|
│
|
|
┌─────────────▼─────────────┐
|
|
│ Multiple Simultaneous API │
|
|
│ Calls May Occur Here: │
|
|
│ • GetItems (movies) │
|
|
│ • GetItems (series) │
|
|
│ • GetItems (resume) │
|
|
│ • Other endpoints │
|
|
└─────────────┬─────────────┘
|
|
│
|
|
┌─────────────▼──────────────────────────┐
|
|
│ ItemsController.GetItems() │
|
|
│ [Jellyfin.Api/Controllers/..cs:166] │
|
|
└─────────────┬──────────────────────────┘
|
|
│
|
|
┌─────────────▼──────────────────────────────┐
|
|
│ 1. Build InternalItemsQuery with: │
|
|
│ - Filters, sorting, paging │
|
|
│ - DtoOptions.EnableUserData = true │
|
|
└─────────────┬──────────────────────────────┘
|
|
│
|
|
┌─────────────▼──────────────────────────────┐
|
|
│ 2. LibraryManager.GetItemsResult() │
|
|
│ [Emby.Server.Implementations/..:1647] │
|
|
│ → ItemRepository.GetItems(query) │
|
|
└─────────────┬──────────────────────────────┘
|
|
│
|
|
┌─────────────▼────────────────────────────────┐
|
|
│ 3. BaseItemRepository.GetItemsAsync() │
|
|
│ [Jellyfin.Server.Implementations/..:421] │
|
|
│ │
|
|
│ Step A: Build query (filtering) │
|
|
│ Step B: Apply ApplyNavigations() │
|
|
│ → .Include(e => e.UserData) │
|
|
│ Step C: Get paged IDs │
|
|
│ Step D: Load full items with UserData │
|
|
│ (ALL UserData is eagerly loaded) │
|
|
└─────────────┬────────────────────────────────┘
|
|
│
|
|
┌─────────────▼──────────────────────────────┐
|
|
│ 4. DtoService.GetBaseItemDtos() │
|
|
│ [Emby.Server.Implementations/..:160] │
|
|
│ │
|
|
│ FOR EACH ITEM: │
|
|
│ → GetBaseItemDtoInternal() │
|
|
│ → AttachUserSpecificInfo() │
|
|
│ → GetUserDataDto() │
|
|
│ → GetUserData() │
|
|
│ (NO QUERY - UserData in │
|
|
│ memory from Include) │
|
|
└─────────────┬──────────────────────────────┘
|
|
│
|
|
┌─────────────▼──────────────────┐
|
|
│ Response sent to browser │
|
|
│ (DTOs with UserData included) │
|
|
└───────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Detailed Analysis
|
|
|
|
### 1. ItemsController.GetItems() - Parameter Setup
|
|
**File**: [Jellyfin.Api/Controllers/ItemsController.cs](Jellyfin.Api/Controllers/ItemsController.cs#L166)
|
|
|
|
```csharp
|
|
public ActionResult<QueryResult<BaseItemDto>> GetItems(
|
|
[FromQuery] Guid? userId,
|
|
[FromQuery] bool? enableUserData, // Line 219
|
|
[FromQuery] int? imageTypeLimit,
|
|
[FromQuery] bool? enableImages = true) // Line 253
|
|
{
|
|
// Line 274-275: Create DtoOptions with defaults
|
|
var dtoOptions = new DtoOptions { Fields = fields }
|
|
.AddAdditionalDtoOptions(enableImages, enableUserData, imageTypeLimit, enableImageTypes);
|
|
```
|
|
|
|
**Key Point**: If `enableUserData` parameter is null, it uses the DtoOptions default value of `true`.
|
|
|
|
---
|
|
|
|
### 2. DtoOptions Default Value
|
|
**File**: [MediaBrowser.Controller/Dto/DtoOptions.cs](MediaBrowser.Controller/Dto/DtoOptions.cs#L32)
|
|
|
|
```csharp
|
|
public DtoOptions()
|
|
: this(true)
|
|
{
|
|
}
|
|
|
|
public DtoOptions(bool allFields)
|
|
{
|
|
ImageTypeLimit = int.MaxValue;
|
|
EnableImages = true;
|
|
EnableUserData = true; // Line 36 - DEFAULT IS TRUE
|
|
AddCurrentProgram = true;
|
|
Fields = allFields ? AllItemFields : [];
|
|
ImageTypes = AllImageTypes;
|
|
}
|
|
```
|
|
|
|
**Conclusion**: UserData eager loading is enabled by default.
|
|
|
|
---
|
|
|
|
### 3. BaseItemRepository.GetItemsAsync() - Eager Loading
|
|
**File**: [Jellyfin.Server.Implementations/Item/BaseItemRepository.cs](Jellyfin.Server.Implementations/Item/BaseItemRepository.cs#L421)
|
|
|
|
```csharp
|
|
public async Task<QueryResult<BaseItemDto>> GetItemsAsync(
|
|
InternalItemsQuery filter,
|
|
CancellationToken cancellationToken = default)
|
|
{
|
|
// Lines 435-445: Build initial query with filters
|
|
IQueryable<BaseItemEntity> dbQuery = PrepareItemQuery(context, filter);
|
|
dbQuery = TranslateQuery(dbQuery, context, filter);
|
|
|
|
// Lines 449-451: Apply ordering
|
|
dbQuery = ApplyOrder(dbQuery, filter, context);
|
|
|
|
// Lines 453-455: Get IDs with SELECT (doesn't load UserData yet)
|
|
var allIds = await dbQuery
|
|
.Select(e => new { e.Id, e.PresentationUniqueKey,
|
|
SeriesPresentationUniqueKey = e.TvExtras!.SeriesPresentationUniqueKey })
|
|
.ToListAsync(cancellationToken);
|
|
|
|
// Lines 467-473: Load FULL entities with navigations
|
|
var dbQueryWithNavs = ApplyNavigations(
|
|
context.BaseItems.Where(e => pagedIds.Contains(e.Id)),
|
|
filter);
|
|
var items = await dbQueryWithNavs
|
|
.ToListAsync(cancellationToken);
|
|
|
|
// Items now have UserData loaded in memory
|
|
result.Items = MaterializeOrderedDtos(items, pagedIds, filter.SkipDeserialization);
|
|
}
|
|
```
|
|
|
|
**Critical Section - ApplyNavigations()** at lines 468-473:
|
|
|
|
```csharp
|
|
private static IQueryable<BaseItemEntity> ApplyNavigations(
|
|
IQueryable<BaseItemEntity> dbQuery,
|
|
InternalItemsQuery filter)
|
|
{
|
|
if (filter.TrailerTypes.Length > 0 || filter.IncludeItemTypes.Contains(BaseItemKind.Trailer))
|
|
{
|
|
dbQuery = dbQuery.Include(e => e.TrailerTypes);
|
|
}
|
|
|
|
if (filter.DtoOptions.ContainsField(ItemFields.ProviderIds))
|
|
{
|
|
dbQuery = dbQuery.Include(e => e.Provider);
|
|
}
|
|
|
|
if (filter.DtoOptions.ContainsField(ItemFields.Settings))
|
|
{
|
|
dbQuery = dbQuery.Include(e => e.LockedFields);
|
|
}
|
|
|
|
if (filter.DtoOptions.EnableUserData) // Line 797
|
|
{
|
|
dbQuery = dbQuery.Include(e => e.UserData); // Line 798
|
|
}
|
|
|
|
if (filter.DtoOptions.EnableImages)
|
|
{
|
|
dbQuery = dbQuery.Include(e => e.Images);
|
|
}
|
|
|
|
dbQuery = dbQuery
|
|
.Include(e => e.TvExtras)
|
|
.Include(e => e.LiveTvExtras)
|
|
.Include(e => e.AudioExtras);
|
|
|
|
return dbQuery;
|
|
}
|
|
```
|
|
|
|
**Result**: UserData is loaded as part of the single query that retrieves the paged items.
|
|
|
|
---
|
|
|
|
### 4. DtoService.GetBaseItemDtos() - Per-Item Processing
|
|
**File**: [Emby.Server.Implementations/Dto/DtoService.cs](Emby.Server.Implementations/Dto/DtoService.cs#L160)
|
|
|
|
```csharp
|
|
public IReadOnlyList<BaseItemDto> GetBaseItemDtos(
|
|
IReadOnlyList<BaseItem> items,
|
|
DtoOptions options,
|
|
User? user = null,
|
|
BaseItem? owner = null)
|
|
{
|
|
var accessibleItems = user is null ? items : items.Where(x => x.IsVisible(user)).ToList();
|
|
var returnItems = new BaseItemDto[accessibleItems.Count];
|
|
|
|
for (int index = 0; index < accessibleItems.Count; index++)
|
|
{
|
|
var item = accessibleItems[index];
|
|
var dto = GetBaseItemDtoInternal(item, options, user, owner);
|
|
|
|
if (item is LiveTvChannel tvChannel)
|
|
{
|
|
(channelTuples ??= []).Add((dto, tvChannel));
|
|
}
|
|
else if (item is LiveTvProgram)
|
|
{
|
|
(programTuples ??= []).Add((item, dto));
|
|
}
|
|
|
|
if (options.ContainsField(ItemFields.ItemCounts))
|
|
{
|
|
SetItemByNameInfo(dto, user); // ⚠️ POTENTIAL N+1: Per-item operation
|
|
}
|
|
|
|
returnItems[index] = dto;
|
|
}
|
|
|
|
return returnItems;
|
|
}
|
|
```
|
|
|
|
**Note**: Line 182 shows `SetItemByNameInfo()` is called per-item if ItemCounts field is requested.
|
|
|
|
---
|
|
|
|
### 5. GetBaseItemDtoInternal() - User-Specific Info Attachment
|
|
**File**: [Emby.Server.Implementations/Dto/DtoService.cs](Emby.Server.Implementations/Dto/DtoService.cs#L222)
|
|
|
|
```csharp
|
|
private BaseItemDto GetBaseItemDtoInternal(
|
|
BaseItem item,
|
|
DtoOptions options,
|
|
User? user = null,
|
|
BaseItem? owner = null)
|
|
{
|
|
var dto = new BaseItemDto { ServerId = _appHost.SystemId };
|
|
|
|
// ... various field attachments ...
|
|
|
|
if (user is not null)
|
|
{
|
|
AttachUserSpecificInfo(dto, item, user, options); // Line 259
|
|
}
|
|
|
|
// ... more field attachments ...
|
|
|
|
return dto;
|
|
}
|
|
```
|
|
|
|
### 6. AttachUserSpecificInfo() - Where UserData is Accessed
|
|
**File**: [Emby.Server.Implementations/Dto/DtoService.cs](Emby.Server.Implementations/Dto/DtoService.cs#L465)
|
|
|
|
```csharp
|
|
private void AttachUserSpecificInfo(
|
|
BaseItemDto dto,
|
|
BaseItem item,
|
|
User user,
|
|
DtoOptions options)
|
|
{
|
|
if (item.IsFolder)
|
|
{
|
|
var folder = (Folder)item;
|
|
|
|
if (options.EnableUserData)
|
|
{
|
|
dto.UserData = _userDataRepository.GetUserDataDto(item, dto, user, options);
|
|
// Line 473: ✓ This should NOT query - UserData already loaded
|
|
}
|
|
|
|
if (!dto.ChildCount.HasValue && item.SourceType == SourceType.Library)
|
|
{
|
|
if (item is MusicAlbum || item is Season || item is Playlist)
|
|
{
|
|
dto.ChildCount = dto.RecursiveItemCount;
|
|
var folderChildCount = folder.LinkedChildren.Length;
|
|
if (folderChildCount > 0)
|
|
{
|
|
dto.ChildCount ??= folderChildCount;
|
|
}
|
|
}
|
|
|
|
if (options.ContainsField(ItemFields.ChildCount))
|
|
{
|
|
dto.ChildCount ??= GetChildCount(folder, user);
|
|
// ⚠️ POTENTIAL N+1: Per-item database call
|
|
}
|
|
}
|
|
|
|
// ... more operations ...
|
|
}
|
|
else
|
|
{
|
|
if (options.EnableUserData)
|
|
{
|
|
dto.UserData = _userDataRepository.GetUserDataDto(item, user);
|
|
// ✓ This should NOT query - UserData already loaded
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### 7. UserDataManager.GetUserData() - Accesses In-Memory Collection
|
|
**File**: [Emby.Server.Implementations/Library/UserDataManager.cs](Emby.Server.Implementations/Library/UserDataManager.cs#L247)
|
|
|
|
```csharp
|
|
public UserItemData? GetUserData(User user, BaseItem item)
|
|
{
|
|
return item.UserData?.Where(e => e.UserId.Equals(user.Id)).Select(Map).FirstOrDefault()
|
|
?? new UserItemData()
|
|
{
|
|
Key = item.GetUserDataKeys()[0],
|
|
};
|
|
}
|
|
```
|
|
|
|
**Key Point**: This accesses `item.UserData` collection directly. Since it was eagerly loaded via `.Include(e => e.UserData)`, this is an in-memory LINQ query with NO database access.
|
|
|
|
---
|
|
|
|
## Why Multiple Queries Are Observed
|
|
|
|
### Issue #1: Multiple API Endpoints Called Simultaneously
|
|
**Most Likely Cause**
|
|
|
|
The web UI likely calls multiple endpoints during page load:
|
|
```
|
|
GET /Items?userId=...&includeItemTypes=Movie&startIndex=0&limit=20
|
|
GET /Items?userId=...&includeItemTypes=Series&startIndex=0&limit=20
|
|
GET /Items?userId=...&filters=IsResumable&startIndex=0&limit=20
|
|
GET /Users/{userId}/Items/Resume
|
|
GET /Playlists
|
|
```
|
|
|
|
Each call results in a separate query to the database.
|
|
|
|
**Evidence**: Check browser DevTools Network tab to see all API calls during page load.
|
|
|
|
---
|
|
|
|
### Issue #2: ItemCounts Field Requested (N+1 Risk)
|
|
**Possible But Less Likely**
|
|
|
|
If `ItemFields.ItemCounts` is included in the fields, `SetItemByNameInfo()` is called per-item:
|
|
|
|
```csharp
|
|
if (options.ContainsField(ItemFields.ItemCounts))
|
|
{
|
|
SetItemByNameInfo(dto, user); // Called for EACH item!
|
|
}
|
|
```
|
|
|
|
This method might query for each item if not properly optimized.
|
|
|
|
---
|
|
|
|
### Issue #3: ChildCount Field Requested (Potential N+1)
|
|
**Possible But Optimized**
|
|
|
|
If `ItemFields.ChildCount` is requested for folders, `GetChildCount()` is called:
|
|
|
|
```csharp
|
|
if (options.ContainsField(ItemFields.ChildCount))
|
|
{
|
|
dto.ChildCount ??= GetChildCount(folder, user);
|
|
}
|
|
```
|
|
|
|
However, there's optimization:
|
|
- Collection folders and user views return random count (NO query)
|
|
- Other folders call `folder.GetChildCount(user)` (could query)
|
|
|
|
---
|
|
|
|
### Issue #4: DbContext Disposal Causing Lazy Loading
|
|
**Possible If Data Access Layer Issue**
|
|
|
|
If the DbContext is disposed before accessing the UserData collection:
|
|
```csharp
|
|
var dbQueryWithNavs = ApplyNavigations(...);
|
|
var items = await dbQueryWithNavs.ToListAsync(); // DbContext disposed here?
|
|
|
|
// If accessed later, UserData lazy-loads = NEW QUERY
|
|
_dtoService.GetBaseItemDtos(items, ...);
|
|
```
|
|
|
|
---
|
|
|
|
## Recommendations
|
|
|
|
### 1. **Capture Network Traffic**
|
|
Use browser DevTools Network tab to see exactly which API endpoints are called and in what order:
|
|
```
|
|
Network tab → Filter by XHR/Fetch → Sort by Name
|
|
Look for multiple GetItems calls with different parameters
|
|
```
|
|
|
|
### 2. **Enable Detailed Query Logging**
|
|
Modify logging.json to see all queries:
|
|
```json
|
|
{
|
|
"Microsoft.EntityFrameworkCore.Database.Command": "Debug"
|
|
}
|
|
```
|
|
Then check the logs to see:
|
|
- How many separate queries are executed
|
|
- When they're executed (timing)
|
|
- What parameters they have
|
|
|
|
### 3. **Check Specific Fields Being Requested**
|
|
If the same query is logged multiple times, check if UI is requesting expensive fields:
|
|
- `ItemFields.ItemCounts` - calls `SetItemByNameInfo()` per-item
|
|
- `ItemFields.ChildCount` - calls `GetChildCount()` per-item
|
|
- `ItemFields.People` - calls `AttachPeople()` per-item
|
|
|
|
### 4. **Batch Related Queries**
|
|
If multiple similar GetItems calls are made, consider consolidating them or adding caching.
|
|
|
|
### 5. **Verify Include() is Working**
|
|
Add logging to confirm `ApplyNavigations()` is being called with correct flags:
|
|
```csharp
|
|
_logger.LogInformation("EnableUserData: {EnableUserData}, UserData navigation included",
|
|
filter.DtoOptions.EnableUserData);
|
|
```
|
|
|
|
---
|
|
|
|
## Files Involved in Query Flow
|
|
|
|
| File | Line | Function | Purpose |
|
|
|------|------|----------|---------|
|
|
| ItemsController.cs | 166 | GetItems() | HTTP endpoint, builds DtoOptions |
|
|
| ItemsController.cs | 527 | folder.GetItems(query) | Calls repository |
|
|
| LibraryManager.cs | 1647 | GetItemsResult() | Builds InternalItemsQuery |
|
|
| BaseItemRepository.cs | 421 | GetItemsAsync() | Main query execution |
|
|
| BaseItemRepository.cs | 468-473 | ApplyNavigations() | Eager loads UserData via Include() |
|
|
| BaseItemRepository.cs | 476 | MaterializeOrderedDtos() | Converts entities to DTOs |
|
|
| DtoService.cs | 160 | GetBaseItemDtos() | Per-item DTO conversion loop |
|
|
| DtoService.cs | 222 | GetBaseItemDtoInternal() | Builds individual DTO |
|
|
| DtoService.cs | 465 | AttachUserSpecificInfo() | Adds user-specific data |
|
|
| UserDataManager.cs | 247 | GetUserData() | Accesses UserData collection (in-memory) |
|
|
| UserDataManager.cs | 260 | GetUserDataDto() | Converts UserData to DTO |
|
|
|
|
---
|
|
|
|
## Conclusion
|
|
|
|
**UserData is properly eagerly loaded** via `.Include(e => e.UserData)` in the main query. The apparent "multiple queries" during page load is almost certainly due to:
|
|
|
|
1. **Multiple simultaneous API calls** from the web UI (different item types, collections)
|
|
2. **Expensive fields requested** (ItemCounts, ChildCount) triggering per-item operations
|
|
3. **Normal pagination** requiring separate queries for different pages
|
|
|
|
**Action Items**:
|
|
- [ ] Capture network traffic to verify which endpoints are called
|
|
- [ ] Check if UserData is being accessed in the response (if not requested, don't Include it)
|
|
- [ ] Review UI for opportunities to consolidate API calls
|
|
- [ ] Consider caching for frequently requested data
|