pgsql-jellyfin/QUERY_EXECUTION_ANALYSIS.md

Query Execution Analysis: Web UI Page Load - Multiple Query Issue

Executive Summary

The query is NOT executed multiple times due to N+1 problems during eager loading of the initial results. Instead, UserData should be eagerly loaded by the main query via .Include(e => e.UserData). If multiple queries are observed, the issue is likely:

1. UI making multiple API calls to different endpoints (different item types/collections)
2. Multiple simultaneous requests for different pages or items
3. DbContext disposal causing lazy loading fallback
4. Optional field requests triggering separate queries (like ItemCounts, ChildCount, etc.)

---

Flow Diagram: Web UI Page Load → Database Queries

┌─────────────────────────────────────────┐
│  Web UI Page Load                       │
│  (Browser renders views/components)     │
└─────────────────────┬───────────────────┘
                      │
        ┌─────────────▼─────────────┐
        │ Multiple Simultaneous API │
        │ Calls May Occur Here:     │
        │ • GetItems (movies)       │
        │ • GetItems (series)       │
        │ • GetItems (resume)       │
        │ • Other endpoints         │
        └─────────────┬─────────────┘
                      │
        ┌─────────────▼──────────────────────────┐
        │ ItemsController.GetItems()             │
        │ [Jellyfin.Api/Controllers/..cs:166]   │
        └─────────────┬──────────────────────────┘
                      │
        ┌─────────────▼──────────────────────────────┐
        │ 1. Build InternalItemsQuery with:          │
        │    - Filters, sorting, paging             │
        │    - DtoOptions.EnableUserData = true      │
        └─────────────┬──────────────────────────────┘
                      │
        ┌─────────────▼──────────────────────────────┐
        │ 2. LibraryManager.GetItemsResult()         │
        │    [Emby.Server.Implementations/..:1647]  │
        │    → ItemRepository.GetItems(query)       │
        └─────────────┬──────────────────────────────┘
                      │
        ┌─────────────▼────────────────────────────────┐
        │ 3. BaseItemRepository.GetItemsAsync()        │
        │    [Jellyfin.Server.Implementations/..:421] │
        │                                              │
        │    Step A: Build query (filtering)          │
        │    Step B: Apply ApplyNavigations()         │
        │            → .Include(e => e.UserData)     │
        │    Step C: Get paged IDs                    │
        │    Step D: Load full items with UserData    │
        │            (ALL UserData is eagerly loaded) │
        └─────────────┬────────────────────────────────┘
                      │
        ┌─────────────▼──────────────────────────────┐
        │ 4. DtoService.GetBaseItemDtos()            │
        │    [Emby.Server.Implementations/..:160]    │
        │                                             │
        │    FOR EACH ITEM:                          │
        │    → GetBaseItemDtoInternal()              │
        │       → AttachUserSpecificInfo()           │
        │          → GetUserDataDto()                │
        │             → GetUserData()                │
        │                (NO QUERY - UserData in    │
        │                 memory from Include)      │
        └─────────────┬──────────────────────────────┘
                      │
        ┌─────────────▼──────────────────┐
        │ Response sent to browser       │
        │ (DTOs with UserData included)  │
        └───────────────────────────────┘

---

Detailed Analysis

1. ItemsController.GetItems() - Parameter Setup

File: Jellyfin.Api/Controllers/ItemsController.cs

public ActionResult<QueryResult<BaseItemDto>> GetItems(
    [FromQuery] Guid? userId,
    [FromQuery] bool? enableUserData,      // Line 219
    [FromQuery] int? imageTypeLimit,
    [FromQuery] bool? enableImages = true) // Line 253
{
    // Line 274-275: Create DtoOptions with defaults
    var dtoOptions = new DtoOptions { Fields = fields }
        .AddAdditionalDtoOptions(enableImages, enableUserData, imageTypeLimit, enableImageTypes);

Key Point: If enableUserData parameter is null, it uses the DtoOptions default value of true.

---

2. DtoOptions Default Value

File: MediaBrowser.Controller/Dto/DtoOptions.cs

public DtoOptions()
    : this(true)
{
}

public DtoOptions(bool allFields)
{
    ImageTypeLimit = int.MaxValue;
    EnableImages = true;
    EnableUserData = true;              // Line 36 - DEFAULT IS TRUE
    AddCurrentProgram = true;
    Fields = allFields ? AllItemFields : [];
    ImageTypes = AllImageTypes;
}

Conclusion: UserData eager loading is enabled by default.

---

3. BaseItemRepository.GetItemsAsync() - Eager Loading

File: Jellyfin.Server.Implementations/Item/BaseItemRepository.cs

public async Task<QueryResult<BaseItemDto>> GetItemsAsync(
    InternalItemsQuery filter, 
    CancellationToken cancellationToken = default)
{
    // Lines 435-445: Build initial query with filters
    IQueryable<BaseItemEntity> dbQuery = PrepareItemQuery(context, filter);
    dbQuery = TranslateQuery(dbQuery, context, filter);
    
    // Lines 449-451: Apply ordering
    dbQuery = ApplyOrder(dbQuery, filter, context);
    
    // Lines 453-455: Get IDs with SELECT (doesn't load UserData yet)
    var allIds = await dbQuery
        .Select(e => new { e.Id, e.PresentationUniqueKey, 
                          SeriesPresentationUniqueKey = e.TvExtras!.SeriesPresentationUniqueKey })
        .ToListAsync(cancellationToken);
    
    // Lines 467-473: Load FULL entities with navigations
    var dbQueryWithNavs = ApplyNavigations(
        context.BaseItems.Where(e => pagedIds.Contains(e.Id)), 
        filter);
    var items = await dbQueryWithNavs
        .ToListAsync(cancellationToken);
    
    // Items now have UserData loaded in memory
    result.Items = MaterializeOrderedDtos(items, pagedIds, filter.SkipDeserialization);
}

Critical Section - ApplyNavigations() at lines 468-473:

private static IQueryable<BaseItemEntity> ApplyNavigations(
    IQueryable<BaseItemEntity> dbQuery, 
    InternalItemsQuery filter)
{
    if (filter.TrailerTypes.Length > 0 || filter.IncludeItemTypes.Contains(BaseItemKind.Trailer))
    {
        dbQuery = dbQuery.Include(e => e.TrailerTypes);
    }

    if (filter.DtoOptions.ContainsField(ItemFields.ProviderIds))
    {
        dbQuery = dbQuery.Include(e => e.Provider);
    }

    if (filter.DtoOptions.ContainsField(ItemFields.Settings))
    {
        dbQuery = dbQuery.Include(e => e.LockedFields);
    }

    if (filter.DtoOptions.EnableUserData)              // Line 797
    {
        dbQuery = dbQuery.Include(e => e.UserData);    // Line 798
    }

    if (filter.DtoOptions.EnableImages)
    {
        dbQuery = dbQuery.Include(e => e.Images);
    }

    dbQuery = dbQuery
        .Include(e => e.TvExtras)
        .Include(e => e.LiveTvExtras)
        .Include(e => e.AudioExtras);

    return dbQuery;
}

Result: UserData is loaded as part of the single query that retrieves the paged items.

---

4. DtoService.GetBaseItemDtos() - Per-Item Processing

File: Emby.Server.Implementations/Dto/DtoService.cs

public IReadOnlyList<BaseItemDto> GetBaseItemDtos(
    IReadOnlyList<BaseItem> items, 
    DtoOptions options, 
    User? user = null, 
    BaseItem? owner = null)
{
    var accessibleItems = user is null ? items : items.Where(x => x.IsVisible(user)).ToList();
    var returnItems = new BaseItemDto[accessibleItems.Count];
    
    for (int index = 0; index < accessibleItems.Count; index++)
    {
        var item = accessibleItems[index];
        var dto = GetBaseItemDtoInternal(item, options, user, owner);
        
        if (item is LiveTvChannel tvChannel)
        {
            (channelTuples ??= []).Add((dto, tvChannel));
        }
        else if (item is LiveTvProgram)
        {
            (programTuples ??= []).Add((item, dto));
        }

        if (options.ContainsField(ItemFields.ItemCounts))
        {
            SetItemByNameInfo(dto, user);  // ⚠️ POTENTIAL N+1: Per-item operation
        }

        returnItems[index] = dto;
    }
    
    return returnItems;
}

Note: Line 182 shows SetItemByNameInfo() is called per-item if ItemCounts field is requested.

---

5. GetBaseItemDtoInternal() - User-Specific Info Attachment

File: Emby.Server.Implementations/Dto/DtoService.cs

private BaseItemDto GetBaseItemDtoInternal(
    BaseItem item, 
    DtoOptions options, 
    User? user = null, 
    BaseItem? owner = null)
{
    var dto = new BaseItemDto { ServerId = _appHost.SystemId };

    // ... various field attachments ...

    if (user is not null)
    {
        AttachUserSpecificInfo(dto, item, user, options);  // Line 259
    }

    // ... more field attachments ...

    return dto;
}

6. AttachUserSpecificInfo() - Where UserData is Accessed

File: Emby.Server.Implementations/Dto/DtoService.cs

private void AttachUserSpecificInfo(
    BaseItemDto dto, 
    BaseItem item, 
    User user, 
    DtoOptions options)
{
    if (item.IsFolder)
    {
        var folder = (Folder)item;

        if (options.EnableUserData)
        {
            dto.UserData = _userDataRepository.GetUserDataDto(item, dto, user, options);
            // Line 473: ✓ This should NOT query - UserData already loaded
        }

        if (!dto.ChildCount.HasValue && item.SourceType == SourceType.Library)
        {
            if (item is MusicAlbum || item is Season || item is Playlist)
            {
                dto.ChildCount = dto.RecursiveItemCount;
                var folderChildCount = folder.LinkedChildren.Length;
                if (folderChildCount > 0)
                {
                    dto.ChildCount ??= folderChildCount;
                }
            }

            if (options.ContainsField(ItemFields.ChildCount))
            {
                dto.ChildCount ??= GetChildCount(folder, user);  
                // ⚠️ POTENTIAL N+1: Per-item database call
            }
        }
        
        // ... more operations ...
    }
    else
    {
        if (options.EnableUserData)
        {
            dto.UserData = _userDataRepository.GetUserDataDto(item, user);
            // ✓ This should NOT query - UserData already loaded
        }
    }
}

7. UserDataManager.GetUserData() - Accesses In-Memory Collection

File: Emby.Server.Implementations/Library/UserDataManager.cs

public UserItemData? GetUserData(User user, BaseItem item)
{
    return item.UserData?.Where(e => e.UserId.Equals(user.Id)).Select(Map).FirstOrDefault() 
        ?? new UserItemData()
    {
        Key = item.GetUserDataKeys()[0],
    };
}

Key Point: This accesses item.UserData collection directly. Since it was eagerly loaded via .Include(e => e.UserData), this is an in-memory LINQ query with NO database access.

---

Why Multiple Queries Are Observed

Issue #1: Multiple API Endpoints Called Simultaneously

Most Likely Cause

The web UI likely calls multiple endpoints during page load:

GET /Items?userId=...&includeItemTypes=Movie&startIndex=0&limit=20
GET /Items?userId=...&includeItemTypes=Series&startIndex=0&limit=20
GET /Items?userId=...&filters=IsResumable&startIndex=0&limit=20
GET /Users/{userId}/Items/Resume
GET /Playlists

Each call results in a separate query to the database.

Evidence: Check browser DevTools Network tab to see all API calls during page load.

---

Issue #2: ItemCounts Field Requested (N+1 Risk)

Possible But Less Likely

If ItemFields.ItemCounts is included in the fields, SetItemByNameInfo() is called per-item:

if (options.ContainsField(ItemFields.ItemCounts))
{
    SetItemByNameInfo(dto, user);  // Called for EACH item!
}

This method might query for each item if not properly optimized.

---

Issue #3: ChildCount Field Requested (Potential N+1)

Possible But Optimized

If ItemFields.ChildCount is requested for folders, GetChildCount() is called:

if (options.ContainsField(ItemFields.ChildCount))
{
    dto.ChildCount ??= GetChildCount(folder, user);
}

However, there's optimization:

• Collection folders and user views return random count (NO query)
• Other folders call folder.GetChildCount(user) (could query)

---

Issue #4: DbContext Disposal Causing Lazy Loading

Possible If Data Access Layer Issue

If the DbContext is disposed before accessing the UserData collection:

var dbQueryWithNavs = ApplyNavigations(...);
var items = await dbQueryWithNavs.ToListAsync();  // DbContext disposed here?

// If accessed later, UserData lazy-loads = NEW QUERY
_dtoService.GetBaseItemDtos(items, ...);

---

Recommendations

1. Capture Network Traffic

Use browser DevTools Network tab to see exactly which API endpoints are called and in what order:

Network tab → Filter by XHR/Fetch → Sort by Name
Look for multiple GetItems calls with different parameters

2. Enable Detailed Query Logging

Modify logging.json to see all queries:

{
    "Microsoft.EntityFrameworkCore.Database.Command": "Debug"
}

Then check the logs to see:

• How many separate queries are executed
• When they're executed (timing)
• What parameters they have

3. Check Specific Fields Being Requested

If the same query is logged multiple times, check if UI is requesting expensive fields:

• ItemFields.ItemCounts - calls SetItemByNameInfo() per-item
• ItemFields.ChildCount - calls GetChildCount() per-item
• ItemFields.People - calls AttachPeople() per-item

4. Batch Related Queries

If multiple similar GetItems calls are made, consider consolidating them or adding caching.

5. Verify Include() is Working

Add logging to confirm ApplyNavigations() is being called with correct flags:

_logger.LogInformation("EnableUserData: {EnableUserData}, UserData navigation included", 
    filter.DtoOptions.EnableUserData);

---

Files Involved in Query Flow

File	Line	Function	Purpose
ItemsController.cs	166	GetItems()	HTTP endpoint, builds DtoOptions
ItemsController.cs	527	folder.GetItems(query)	Calls repository
LibraryManager.cs	1647	GetItemsResult()	Builds InternalItemsQuery
BaseItemRepository.cs	421	GetItemsAsync()	Main query execution
BaseItemRepository.cs	468-473	ApplyNavigations()	Eager loads UserData via Include()
BaseItemRepository.cs	476	MaterializeOrderedDtos()	Converts entities to DTOs
DtoService.cs	160	GetBaseItemDtos()	Per-item DTO conversion loop
DtoService.cs	222	GetBaseItemDtoInternal()	Builds individual DTO
DtoService.cs	465	AttachUserSpecificInfo()	Adds user-specific data
UserDataManager.cs	247	GetUserData()	Accesses UserData collection (in-memory)
UserDataManager.cs	260	GetUserDataDto()	Converts UserData to DTO

---

Conclusion

UserData is properly eagerly loaded via .Include(e => e.UserData) in the main query. The apparent "multiple queries" during page load is almost certainly due to:

1. Multiple simultaneous API calls from the web UI (different item types, collections)
2. Expensive fields requested (ItemCounts, ChildCount) triggering per-item operations
3. Normal pagination requiring separate queries for different pages

Action Items:

• Capture network traffic to verify which endpoints are called
• Check if UserData is being accessed in the response (if not requested, don't Include it)
• Review UI for opportunities to consolidate API calls
• Consider caching for frequently requested data