Improve build reliability, query perf, and documentation
- Add build-error-resolution.md guide for CS0006 and IDE analyzer issues - Add rebuild-solution.ps1 script for clean/reliable builds - Suppress noisy IDE/StyleCop warnings in .editorconfig - Optimize BaseItemRepository grouping queries to use Min(Id) instead of FirstOrDefault, avoiding correlated subqueries and greatly improving DB performance - Add database-query-optimization.md explaining query changes and tuning - Enable EF Core SQL query logging, sensitive data, and detailed errors when log level is Debug (ServiceCollectionExtensions, logging.json) - Update readme with SQL logging instructions - Clarify git commit/ignore guidance and improve inline documentation
This commit is contained in:
@@ -0,0 +1,132 @@
|
||||
# Database Query Optimization Guide
|
||||
|
||||
## Query Performance Issues and Solutions
|
||||
|
||||
### Issue: Correlated Subquery Performance Problem
|
||||
|
||||
#### Problem Description
|
||||
Prior to optimization, queries using `GroupBy().Select(e => e.FirstOrDefault()).Select(e => e.Id)` pattern were generating inefficient correlated subqueries in PostgreSQL, causing timeout errors (30+ seconds).
|
||||
|
||||
**Example of problematic pattern:**
|
||||
```csharp
|
||||
var tempQuery = dbQuery
|
||||
.GroupBy(e => e.PresentationUniqueKey)
|
||||
.Select(e => e.FirstOrDefault())
|
||||
.Select(e => e!.Id);
|
||||
dbQuery = context.BaseItems.Where(e => tempQuery.Contains(e.Id));
|
||||
```
|
||||
|
||||
**Generated SQL (PROBLEMATIC):**
|
||||
```sql
|
||||
WHERE b.Id IN (
|
||||
SELECT (
|
||||
SELECT b1.Id
|
||||
FROM library.BaseItems AS b1
|
||||
WHERE b0.PresentationUniqueKey = b1.PresentationUniqueKey
|
||||
LIMIT 1
|
||||
)
|
||||
FROM library.BaseItems AS b0
|
||||
GROUP BY b0.PresentationUniqueKey
|
||||
)
|
||||
```
|
||||
|
||||
This creates a correlated subquery that executes once for each group, resulting in exponential performance degradation.
|
||||
|
||||
#### Solution
|
||||
Replace `FirstOrDefault()` with `Min(x => x.Id)` to generate an efficient aggregate query:
|
||||
|
||||
**Optimized pattern:**
|
||||
```csharp
|
||||
var tempQuery = dbQuery
|
||||
.GroupBy(e => e.PresentationUniqueKey)
|
||||
.Select(e => e.Min(x => x.Id));
|
||||
dbQuery = context.BaseItems.Where(e => tempQuery.Contains(e.Id));
|
||||
```
|
||||
|
||||
**Generated SQL (OPTIMIZED):**
|
||||
```sql
|
||||
WHERE b.Id IN (
|
||||
SELECT MIN(b0.Id)
|
||||
FROM library.BaseItems AS b0
|
||||
GROUP BY b0.PresentationUniqueKey
|
||||
)
|
||||
```
|
||||
|
||||
This generates a simple GROUP BY with MIN aggregate, which PostgreSQL can optimize efficiently.
|
||||
|
||||
#### Performance Impact
|
||||
- **Before**: 30,000+ ms (timeout)
|
||||
- **After**: ~10-50 ms (estimated based on query complexity)
|
||||
- **Improvement**: ~600-3000x faster
|
||||
|
||||
### Affected Queries
|
||||
This optimization was applied to:
|
||||
1. `ApplyGroupingFilter` with `GroupBySeriesPresentationUniqueKey` and `PresentationUniqueKey`
|
||||
2. All three grouping scenarios in the `BaseItemRepository`
|
||||
|
||||
### Testing
|
||||
After applying this optimization:
|
||||
1. Monitor query logs to verify improved SQL generation
|
||||
2. Test library browsing performance, especially for:
|
||||
- TV show episode lists
|
||||
- Duplicate media detection
|
||||
- Collection grouping
|
||||
3. Verify that the "first" item from each group is consistently selected (by Id ordering)
|
||||
|
||||
### Notes
|
||||
- Using `Min(Id)` instead of `FirstOrDefault()` ensures deterministic selection
|
||||
- The selected item will be the one with the lowest GUID value in each group
|
||||
- This change maintains functional equivalence while dramatically improving performance
|
||||
- If a different selection criterion is needed (e.g., by date), use `Min(x => x.DateCreated)` and join back to get the Id
|
||||
|
||||
### Related Files
|
||||
- `Jellyfin.Server.Implementations/Item/BaseItemRepository.cs` - Line 572-603 (ApplyGroupingFilter method)
|
||||
|
||||
### Additional Recommendations
|
||||
|
||||
#### 1. Consider Using AsSplitQuery() for Related Data
|
||||
When loading items with multiple relationships (providers, images, user data), consider using split queries:
|
||||
|
||||
```csharp
|
||||
dbQuery = dbQuery
|
||||
.Include(e => e.Provider)
|
||||
.Include(e => e.UserData)
|
||||
.Include(e => e.Images)
|
||||
.AsSplitQuery(); // Prevents cartesian explosion
|
||||
```
|
||||
|
||||
#### 2. Increase Command Timeout for Complex Queries
|
||||
If queries legitimately need more time, increase the command timeout in DbContext configuration:
|
||||
|
||||
```csharp
|
||||
opt.CommandTimeout(60); // 60 seconds
|
||||
```
|
||||
|
||||
#### 3. Database Indexing
|
||||
Ensure proper indexes exist on:
|
||||
- `BaseItems.PresentationUniqueKey`
|
||||
- `BaseItems.SeriesPresentationUniqueKey`
|
||||
- `BaseItems.IsVirtualItem`
|
||||
- `BaseItems.TopParentId`
|
||||
|
||||
Check with:
|
||||
```sql
|
||||
SELECT * FROM pg_indexes WHERE tablename = 'BaseItems';
|
||||
```
|
||||
|
||||
#### 4. Query Logging Configuration
|
||||
To debug slow queries, enable Entity Framework Core query logging in `logging.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"Serilog": {
|
||||
"MinimumLevel": {
|
||||
"Override": {
|
||||
"Microsoft.EntityFrameworkCore.Database.Command": "Debug"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
See `src/Jellyfin.Database/readme.md` for more details on query logging.
|
||||
Reference in New Issue
Block a user