wjones/new_dbsync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Commit Graph

  • 0fa8b83f2d add logging functionality to json downloading file. development wjones 2026-02-11 08:20:23 -05:00
  • afe2dfa1c8 Update: Implement bulk update functionality using list of document IDs and updated data wjones 2026-02-10 12:01:22 -05:00
  • c6f9d1e617 Update .gitignore and settings configuration: add archived directory and settings template wjones 2026-02-07 12:29:55 -05:00
  • 179fb4fddb Refactor code structure for improved readability and maintainability wjones 2026-02-07 11:57:07 -05:00
  • 8ed2a77365 Add MongoDB query utilities and update batch size for document insertion wjones 2026-02-06 14:49:56 -05:00
  • 4f7c88e96e Merge pull request 'Refactor README.md for clarity and detail: enhance project description, add features, usage examples, and troubleshooting sections; remove outdated mongo2pgsql.py and search_mongodb.py files' (#6) from development into main main wjones 2026-02-06 18:05:01 +00:00
  • 1283bd282a Refactor README.md for clarity and detail: enhance project description, add features, usage examples, and troubleshooting sections; remove outdated mongo2pgsql.py and search_mongodb.py files wjones 2026-02-06 12:23:03 -05:00
  • a80c692786 Merge pull request 'development' (#5) from development into main wjones 2026-02-06 17:15:34 +00:00
  • 9fbd5dc8b9 Refactor MongoDB to PostgreSQL data synchronization: remove json2postgres.py, add mongodb2postgres.py for improved schema generation and data insertion wjones 2026-02-06 12:11:46 -05:00
  • 613a52d3da Merge pull request 'development' (#4) from development into main wjones 2026-02-06 15:39:04 +00:00
  • a52bd5571e Refactor database metadata initialization and update JSON download logic for improved clarity and efficiency wjones 2026-02-06 10:36:20 -05:00
  • a49354bbcb add search_mongodb.py and change seriesid in update_mongodb.py from a string to an int. wjones 2026-02-06 08:44:37 -05:00
  • 66cb23bb6d file not executable wjones 2026-02-05 16:58:30 -05:00
  • 0f0f9078c6 Refactor load_json_to_mongodb function to include updateDict for change detection and enhance logging with screenprint method wjones 2026-02-05 16:53:10 -05:00
  • 7e98f01cd2 commit gitignore file wjones 2026-02-05 16:12:09 -05:00
  • c6038c0b01 Add /cache/aliases/*.json to .gitignore wjones 2026-02-05 14:10:45 -05:00
  • 2d414ecbfc updated gitignore file wjones 2026-02-05 14:00:47 -05:00
  • 332ad8e5d8 Enhance load_json_to_mongodb function to support change detection and statistics tracking for inserted and updated documents. wjones 2026-02-05 10:37:17 -05:00
  • df89aad7e6 Add MongoDB integration and JSON download functionality wjones 2026-02-05 09:02:37 -05:00
  • bbf92af79f update gitignore file wjones 2026-02-05 07:31:43 -05:00
  • 26a7e94b4e Update previous updates file naming to include current timestamp for versioning wjones 2026-01-22 14:53:40 -05:00
  • 968a3a2949 Refactor update table logging and processing; enhance clarity in output messages. wjones 2026-01-22 14:50:30 -05:00
  • f6254ee42e Update psycopg2 dependency to psycopg2-binary in requirements.txt; refine timestamp handling in main function of update_mongo.py; add new cache.tgz file. wjones 2026-01-22 14:25:53 -05:00
  • f5627e58a8 Add function to update update table wjones 2026-01-22 12:36:47 -05:00
  • 5c97eb3fb4 Add skip_ids functionality to main for conditional JSON downloads wjones 2025-12-13 14:21:53 -05:00
  • f79f8772ae Update .gitignore to include additional log files and temporary directories for better cleanup wjones 2025-12-12 14:21:34 -05:00
  • 99a2d78032 Comment out unused code in CreateTempTables and update_tvupdates methods for future verification; streamline set_apienv function by removing jget parameter and related logic. wjones 2025-12-12 13:48:05 -05:00
  • 3447ba369f Remove unused jprocessor module files to streamline the codebase. wjones 2025-12-12 11:22:30 -05:00
  • eb1615ea3d Remove unused tableddl_from_yaml.py and its test file to clean up the repository. wjones 2025-12-12 11:04:45 -05:00
  • 4f0267ecf2 Refactor update logic in Dbexec class; adjust logging level in settings; enhance JSON downloader function call in update_mongo.py for improved error handling. wjones 2025-12-12 07:46:45 -05:00
  • 097225c942 Add logging support to JSON download function in update_mongo.py wjones 2025-12-12 07:37:50 -05:00
  • 24203aff66 Comment out SQL update logic in Dbexec class for verification; adjust sleep duration in JSON downloader to reduce server load; enhance logging in update_mongo.py for clarity on processing updates. wjones 2025-12-12 07:37:19 -05:00
  • 3bc35a08f9 Fix typo in JSON download message to reflect the number of files instead of series IDs wjones 2025-12-12 07:15:11 -05:00
  • 7dfe22fe71 Add guest cast and crew directories to cache structure; update file check logic wjones 2025-12-12 07:13:40 -05:00
  • e975b947fa Refactor update_mongo.py to enhance JSON downloading and file handling; update MongoDB host in test_tableddl_from_yaml.py wjones 2025-12-11 17:53:41 -05:00
  • a18c6a7386 Add PostgreSQL virtual table extension and SQLite utilities wjones 2025-12-11 18:45:49 +00:00
  • 2c2d6da194 Merge branch 'main' of https://gitea.wpjones.com/wjones/new_dbsync wjones 2025-12-11 15:23:13 +00:00
  • 96bd323567 Implement bulk upsert functionality in dbmongo class and update configuration for MongoDB database wjones 2025-12-11 15:20:50 +00:00
  • 197a81d8fb Refactor dbmongo class to improve code structure and readability wjones 2025-12-09 16:28:35 -05:00
  • dfeb5cb36e Add timestamp conversion for updated fields in dbmongo class and clean up json_downloader.py wjones 2025-12-09 16:00:25 -05:00
  • aa2001bc87 Merge branch 'main' of https://gitea.wpjones.com/wjones/new_dbsync wjones 2025-12-09 13:01:25 -05:00
  • c5502711e7 Refactor update_mongo.py for improved readability and maintainability wjones 2025-12-09 13:00:48 -05:00
  • c51150396c Refactor json_downloader.py to correct directory usage for JSON downloads wjones 2025-11-26 08:07:25 -05:00
  • b390fc9f9a Add function to read configuration from file or directory wjones 2025-11-25 08:13:59 -05:00
  • 7837ca2dd4 Remove unused functions and clean up json_downloader.py for better maintainability wjones 2025-11-17 11:37:46 -05:00
  • 271bf606ca Refactor code structure for improved readability and maintainability wjones 2025-11-14 16:24:06 -05:00
  • 1d14e97637 Merge pull request 'removing cache folder and view and json files are no longer used.' (#1) from development into main wjones 2025-06-27 15:46:19 -04:00
  • b9f135e325 removing cache folder and view and json files are no longer used. wjones 2025-06-27 15:08:11 -04:00
  • c947f7a6db fix code for all tables in json2postgres.py so all tables get processed now. Update requirements.txt to include pandas. 20250627_main wjones 2025-06-25 16:13:34 -04:00
  • f719155a35 working on json2postgres.py file wjones 2025-06-25 09:17:45 -04:00
  • 29814092b7 repo migration wjones 2025-05-20 12:02:07 -04:00