Commit Graph

10 Commits

Author SHA1 Message Date
wjones afe2dfa1c8 Update: Implement bulk update functionality using list of document IDs and updated data
This commit introduces a new function, `insert_mongo_documents_to_postgres`, which allows for the bulk updating of documents in a PostgreSQL table. It takes a list of document IDs and a dictionary containing the updated data as input. This method simplifies the process by reducing the number of individual insert statements required.

The changes include:

1. Adding a new function `insert_mongo_documents_to_postgres` to handle the bulk update.
2. Updating the existing code to call this new function when required.
3. Refactoring the `MongoDocumentInserter` class to encapsulate common functionality for document insertion and updating.

This refactoring enhances maintainability and efficiency by reducing redundancy in the codebase.
2026-02-10 12:01:22 -05:00
wjones 8ed2a77365 Add MongoDB query utilities and update batch size for document insertion 2026-02-06 14:49:56 -05:00
wjones 9fbd5dc8b9 Refactor MongoDB to PostgreSQL data synchronization: remove json2postgres.py, add mongodb2postgres.py for improved schema generation and data insertion 2026-02-06 12:11:46 -05:00
wjones a52bd5571e Refactor database metadata initialization and update JSON download logic for improved clarity and efficiency 2026-02-06 10:36:20 -05:00
wjones 99a2d78032 Comment out unused code in CreateTempTables and update_tvupdates methods for future verification; streamline set_apienv function by removing jget parameter and related logic. 2025-12-12 13:48:05 -05:00
wjones 4f0267ecf2 Refactor update logic in Dbexec class; adjust logging level in settings; enhance JSON downloader function call in update_mongo.py for improved error handling. 2025-12-12 07:46:45 -05:00
wjones 24203aff66 Comment out SQL update logic in Dbexec class for verification; adjust sleep duration in JSON downloader to reduce server load; enhance logging in update_mongo.py for clarity on processing updates. 2025-12-12 07:37:19 -05:00
wjones 96bd323567 Implement bulk upsert functionality in dbmongo class and update configuration for MongoDB database 2025-12-11 15:20:50 +00:00
wjones dfeb5cb36e Add timestamp conversion for updated fields in dbmongo class and clean up json_downloader.py 2025-12-09 16:00:25 -05:00
wjones 29814092b7 repo migration 2025-05-20 12:02:07 -04:00