Commit Graph

18 Commits

Author SHA1 Message Date
wjones e67c191843 Add comprehensive documentation for PostgreSQL schema conversion
- Introduced DATABASE_SCHEMA_MISMATCH_REPORT.md to detail critical mismatches between SQL schema and C# code configurations, highlighting issues with naming conventions and potential runtime failures.
- Created SCHEMA_COLUMN_CONVERSION_REFERENCE.md to provide a reference for column name conversions from PascalCase to snake_case across all tables.
- Developed SCHEMA_CONVERSION_GUIDE.md outlining the conversion strategy for the SQL schema, including authoritative table mappings, column naming rules, and a validation checklist to ensure accuracy in the conversion process.
2026-05-04 10:16:17 -04:00
wjones 3e5d29225a Refactor SQLite Database Provider
- Removed unused classes and files related to SQLite database provider, including SqliteDesignTimeJellyfinDbFactory, ModelBuilderExtensions, PragmaConnectionInterceptor, AssemblyInfo, SqliteDatabaseProvider, DateTimeKindValueConverter.
- Updated tests to remove dependencies on removed classes and adjusted mocking for configuration sections.
- Added Microsoft.EntityFrameworkCore.Sqlite package reference to test project.
- Improved string handling in tests for better consistency and clarity.
- Refactored logging methods in JellyfinApplicationFactory for better readability and maintainability.
2026-05-03 09:39:00 -04:00
wjones 57d6342524 Optimize PostgreSQL dead tuple handling for media scans
Introduce bulk operation extensions and transaction helpers for efficient, batched EF Core operations. Refactor MediaStreamInfo schema by splitting audio/video details into dedicated tables, reducing table bloat and improving update performance. Add migration for data movement and schema changes. Update EF Core models and context for new structure. Enhance PostgreSQL provider configuration and add comprehensive documentation and monitoring guides. Includes minor middleware and code cleanups.
2026-04-30 15:52:37 -04:00
wjones e81c127514 Add JSON config, DB-backed library options, and docs
- Add JSON-based config loading with XML fallback for DB and library options
- Implement LibraryOptionsRepository with EF Core, migrations, and entity
- Update CollectionFolder to use DB-backed options with XML fallback/backfill
- Register repository in DI and initialize at startup
- Use EF execution strategy for transactional DB operations
- Suppress code analysis warnings in .csproj and test files
- Add DATABASE_MIGRATION.md, LIBRARY_OPTIONS_DB_DESIGN.md, and WEBSOCKET_AUTHENTICATION.md
- Add database.json.example and improve migration docs
- Add tests for JSON config loader and update test naming warnings
2026-04-30 12:25:24 -04:00
wjones ee2aa8c13c Merge pull request 'Add placeholder item handling and related unit test for deletion' (#2) from pgsql_testing_branch into main
Reviewed-on: #2
2026-04-13 19:10:24 +00:00
wjones eccb1359c8 Add placeholder item handling and related unit test for deletion 2026-04-13 11:01:59 -04:00
wjones e7aa3024cb Refactor BaseItemRepository for ordered DTO materialization
Refactored BaseItemRepository to use a new helper method, MaterializeOrderedDtos, ensuring DTOs are returned in the order of precomputed IDs and improving code reuse. Updated comments to clarify ordering. Cleaned up scenario.json properties and reformatted rebuild-solution.ps1 without changing its logic.
2026-03-07 11:32:56 -05:00
wjones 8b0503f398 Fix EF Core DistinctBy query translation errors
Refactor all item query methods to avoid untranslatable DistinctBy
in IQueryable expressions. Replace ApplyGroupingFilter with a
two-phase pattern: load IDs and grouping keys to memory, apply
grouping and paging in-memory, then reload full entities by ID.
Add ApplyGroupingInMemory<T> and ApplyPagingToIds helpers.
Use AsEnumerable() before DistinctBy in GetItemValues to force
client-side evaluation. Resolves InvalidOperationException on
/Items, /Items/Filters, /Users/{id}/Items/Latest, and /Studios
endpoints across all supported databases. See
EF_CORE_QUERY_TRANSLATION_FIX.md for details.
2026-03-05 17:17:16 -05:00
wjones 7eb2b445cb Postgres perf, path migration, and shutdown race fixes
- Major query performance boost: replaced correlated subqueries with DistinctBy() in BaseItemRepository, added episode deduplication index, and increased default command timeout to 120s.
- Fixed LibraryMonitor shutdown race: added disposal checks and exception handling in FileRefresher to prevent ObjectDisposedException.
- Added PowerShell and SQL utilities for fixing Linux paths in config files and database; new scripts for WebDir and path migration.
- Auto-fix for WebDir in startup.json on load; new helper scripts and docs.
- Added automated weekly DB performance monitoring scripts and reporting.
- Expanded documentation and README for all fixes, scripts, and migration steps.
- Updated SQL scripts for index creation and diagnostics; improved output handling.
- Updated db-config defaults and added new diagnostic/report files.
2026-03-05 08:56:42 -05:00
wjones c76853a442 PostgreSQL: Production fixes—UPSERT, remote backup, auth logs
- Fix: Atomic UPSERT for BaseItemProviders (resolves duplicate key errors during concurrent metadata refresh; clears navigation property to prevent EF Core tracking conflicts)
- Add: Remote PostgreSQL backup support (removes localhost-only restriction; works with pg_dump/pg_restore for both local and remote DBs)
- Add: Configurable backup disable option (`disable-backups`)
- Fix: Query timeout and performance (documented `command-timeout` config, added performance index scripts)
- Fix: Authentication errors now log as warnings with clear messages (ExceptionMiddleware), reducing log noise
- Fix: SyncPlay authorization handler validates user before lookup, logs warnings for unauthenticated/unknown users (returns 403/404)
- Fix: Database deadlock detection logs warnings and allows EF Core auto-retry
- Add: Configurable LibraryMonitorDelay (min 30s, default 60s)
- Fix: SQLite migration filtering—skip SQLite-only migrations on PostgreSQL
- Chore: Suppress StyleCop warnings (SA1137, etc.) for project consistency
- Docs: 21 documentation files added/updated (config, backup, performance, troubleshooting, session summary)
- All changes are backward compatible and production-ready
2026-03-03 16:35:27 -05:00
wjones 623af06e46 Optimize BaseItemRepository docs for grouping performance
Extensively document query grouping performance issues and optimization attempts in BaseItemRepository. Add guides on PostgreSQL UUID aggregation limitations, recommended indexes, increasing database timeouts, and monitoring query performance. Current code reverts to original (slow) implementation due to EF Core/Npgsql limitations, with detailed comments and references to new docs. Provides clear workarounds and upgrade path for future `.DistinctBy()` support.
2026-03-01 11:33:16 -05:00
wjones 23ed81b6b1 Improve build reliability, query perf, and documentation
- Add build-error-resolution.md guide for CS0006 and IDE analyzer issues
- Add rebuild-solution.ps1 script for clean/reliable builds
- Suppress noisy IDE/StyleCop warnings in .editorconfig
- Optimize BaseItemRepository grouping queries to use Min(Id) instead of FirstOrDefault, avoiding correlated subqueries and greatly improving DB performance
- Add database-query-optimization.md explaining query changes and tuning
- Enable EF Core SQL query logging, sensitive data, and detailed errors when log level is Debug (ServiceCollectionExtensions, logging.json)
- Update readme with SQL logging instructions
- Clarify git commit/ignore guidance and improve inline documentation
2026-03-01 09:49:30 -05:00
wjones 78bc22db79 Update BaseItemProviders after deletion in repository
Ensure entity.Provider is re-added to context after deleting
existing BaseItemProviders, preserving updated provider info.
2026-02-27 17:09:06 -05:00
wjones f833f0ceeb BaseItemRepository: 100% async conversion complete
All 21 public methods in BaseItemRepository are now fully async, including complex query, retrieval, write, delete, and aggregation operations. Added async signatures to IItemRepository and ILibraryManager with cancellation token support and ConfigureAwait(false). Sync wrappers retained for backward compatibility. Final conversion of GetLatestItemList and GetNextUpSeriesKeys completes Phase 3A. All bulk operations and aggregations are async, enabling concurrent dashboard loading and full PostgreSQL multiplexing. Migration routines fixed for Npgsql "command in progress" errors. Extensive documentation added for conversion process, performance, and testing. Project is now production-ready with zero build errors and 100% backward compatibility.
2026-02-23 12:36:29 -05:00
wjones 86883cd5c6 Refactor PostgreSQL provider: multi-schema & async prep
- Refactor migrations and provider to use multiple PostgreSQL schemas, each matching a legacy SQLite database (activitylog, authentication, displaypreferences, library, users).
- All tables, foreign keys, and indexes are now schema-qualified; Down migration drops tables by schema.
- Provider ensures schemas exist before migrations; entities are mapped to correct schemas in OnModelCreating.
- Add support for max-pool-size, min-pool-size, and multiplexing connection options; update logging accordingly.
- VACUUM ANALYZE now runs per schema during scheduled optimization.
- TruncateAllTablesAsync now truncates tables with schema qualification.
- README updated with schema structure, new options, and multiplexing warnings.
- CacheDecorator now calls async repository methods using .GetAwaiter().GetResult(), with documentation.
- Lays groundwork for full async/await and multiplexing support in the database layer.
2026-02-23 09:38:22 -05:00
wjones af1152b001 Refactor: standardize namespace and using directive style
Refactored all C# test files to use explicit namespace declarations and moved all using directives inside the namespace block for consistency. Updated assembly info and cache files to reflect the new build. Adjusted MvcTestingAppManifest.json to use fully qualified assembly names. No functional changes; all updates are related to code style, organization, and project metadata.
2026-02-20 16:26:53 -05:00
wjones 1c6730e2ad creating pgsql development branch 2026-02-19 09:20:41 -05:00
wjones 460884fea3 repo creation with initial code after cloning public repo 2026-02-19 07:36:25 -05:00