wjones
bbf92af79f
update gitignore file
2026-02-05 07:31:43 -05:00
wjones
26a7e94b4e
Update previous updates file naming to include current timestamp for versioning
2026-01-22 14:53:40 -05:00
wjones
968a3a2949
Refactor update table logging and processing; enhance clarity in output messages.
2026-01-22 14:50:30 -05:00
wjones
f6254ee42e
Update psycopg2 dependency to psycopg2-binary in requirements.txt; refine timestamp handling in main function of update_mongo.py; add new cache.tgz file.
2026-01-22 14:25:53 -05:00
wjones
f5627e58a8
Add function to update update table
2026-01-22 12:36:47 -05:00
wjones
5c97eb3fb4
Add skip_ids functionality to main for conditional JSON downloads
2025-12-13 14:21:53 -05:00
wjones
f79f8772ae
Update .gitignore to include additional log files and temporary directories for better cleanup
2025-12-12 14:21:34 -05:00
wjones
99a2d78032
Comment out unused code in CreateTempTables and update_tvupdates methods for future verification; streamline set_apienv function by removing jget parameter and related logic.
2025-12-12 13:48:05 -05:00
wjones
3447ba369f
Remove unused jprocessor module files to streamline the codebase.
2025-12-12 11:22:30 -05:00
wjones
eb1615ea3d
Remove unused tableddl_from_yaml.py and its test file to clean up the repository.
2025-12-12 11:04:45 -05:00
wjones
4f0267ecf2
Refactor update logic in Dbexec class; adjust logging level in settings; enhance JSON downloader function call in update_mongo.py for improved error handling.
2025-12-12 07:46:45 -05:00
wjones
097225c942
Add logging support to JSON download function in update_mongo.py
2025-12-12 07:37:50 -05:00
wjones
24203aff66
Comment out SQL update logic in Dbexec class for verification; adjust sleep duration in JSON downloader to reduce server load; enhance logging in update_mongo.py for clarity on processing updates.
2025-12-12 07:37:19 -05:00
wjones
3bc35a08f9
Fix typo in JSON download message to reflect the number of files instead of series IDs
2025-12-12 07:15:11 -05:00
wjones
7dfe22fe71
Add guest cast and crew directories to cache structure; update file check logic
2025-12-12 07:13:40 -05:00
wjones
e975b947fa
Refactor update_mongo.py to enhance JSON downloading and file handling; update MongoDB host in test_tableddl_from_yaml.py
2025-12-11 17:53:41 -05:00
wjones
a18c6a7386
Add PostgreSQL virtual table extension and SQLite utilities
...
- Implement pg_vtab as a minimal SQLite virtual table module backed by PostgreSQL.
- Create sqlite2pgsql utility for copying tables from SQLite to PostgreSQL.
- Add sqlite_pg_plugin to expose PostgreSQL tables to SQLite.
- Include Makefile for building the pg_vtab shared object.
- Update README with usage instructions and limitations.
- Remove obsolete cache.zip file.
2025-12-11 18:45:49 +00:00
wjones
2c2d6da194
Merge branch 'main' of https://gitea.wpjones.com/wjones/new_dbsync
2025-12-11 15:23:13 +00:00
wjones
96bd323567
Implement bulk upsert functionality in dbmongo class and update configuration for MongoDB database
2025-12-11 15:20:50 +00:00
wjones
197a81d8fb
Refactor dbmongo class to improve code structure and readability
2025-12-09 16:28:35 -05:00
wjones
dfeb5cb36e
Add timestamp conversion for updated fields in dbmongo class and clean up json_downloader.py
2025-12-09 16:00:25 -05:00
wjones
aa2001bc87
Merge branch 'main' of https://gitea.wpjones.com/wjones/new_dbsync
2025-12-09 13:01:25 -05:00
wjones
c5502711e7
Refactor update_mongo.py for improved readability and maintainability
2025-12-09 13:00:48 -05:00
wjones
c51150396c
Refactor json_downloader.py to correct directory usage for JSON downloads
2025-11-26 08:07:25 -05:00
wjones
b390fc9f9a
Add function to read configuration from file or directory
...
- Introduced `read_config_file` function to handle reading configuration settings.
- Supports both direct file paths and project root directory structure.
- Extracts various database and update type settings into a dictionary.
- Handles timestamp calculations for update types.
2025-11-25 08:13:59 -05:00
wjones
7837ca2dd4
Remove unused functions and clean up json_downloader.py for better maintainability
2025-11-17 11:37:46 -05:00
wjones
271bf606ca
Refactor code structure for improved readability and maintainability
2025-11-14 16:24:06 -05:00
wjones
1d14e97637
Merge pull request 'removing cache folder and view and json files are no longer used.' ( #1 ) from development into main
...
Reviewed-on: #1
2025-06-27 15:46:19 -04:00
wjones
b9f135e325
removing cache folder and view and json files are no longer used.
2025-06-27 15:08:11 -04:00
wjones
c947f7a6db
fix code for all tables in json2postgres.py so all tables get processed now. Update requirements.txt to include pandas.
2025-06-25 16:13:34 -04:00
wjones
f719155a35
working on json2postgres.py file
2025-06-25 09:17:45 -04:00
wjones
29814092b7
repo migration
2025-05-20 12:02:07 -04:00
wjones