Commit Graph

13 Commits

Author SHA1 Message Date
wjones df89aad7e6 Add MongoDB integration and JSON download functionality
- Created a new script `update_mongodb.py` to handle downloading JSON data from APIs and loading it into MongoDB.
- Implemented functions for downloading JSON files, loading data into MongoDB collections, and managing updates.
- Updated `update_json.py` to instantiate the `dbmongo` class for MongoDB operations.
- Enhanced logging for better tracking of the download and load processes.
2026-02-05 09:02:37 -05:00
wjones bbf92af79f update gitignore file 2026-02-05 07:31:43 -05:00
wjones 4f0267ecf2 Refactor update logic in Dbexec class; adjust logging level in settings; enhance JSON downloader function call in update_mongo.py for improved error handling. 2025-12-12 07:46:45 -05:00
wjones 097225c942 Add logging support to JSON download function in update_mongo.py 2025-12-12 07:37:50 -05:00
wjones 24203aff66 Comment out SQL update logic in Dbexec class for verification; adjust sleep duration in JSON downloader to reduce server load; enhance logging in update_mongo.py for clarity on processing updates. 2025-12-12 07:37:19 -05:00
wjones e975b947fa Refactor update_mongo.py to enhance JSON downloading and file handling; update MongoDB host in test_tableddl_from_yaml.py 2025-12-11 17:53:41 -05:00
wjones dfeb5cb36e Add timestamp conversion for updated fields in dbmongo class and clean up json_downloader.py 2025-12-09 16:00:25 -05:00
wjones c51150396c Refactor json_downloader.py to correct directory usage for JSON downloads 2025-11-26 08:07:25 -05:00
wjones b390fc9f9a Add function to read configuration from file or directory
- Introduced `read_config_file` function to handle reading configuration settings.
- Supports both direct file paths and project root directory structure.
- Extracts various database and update type settings into a dictionary.
- Handles timestamp calculations for update types.
2025-11-25 08:13:59 -05:00
wjones 7837ca2dd4 Remove unused functions and clean up json_downloader.py for better maintainability 2025-11-17 11:37:46 -05:00
wjones 271bf606ca Refactor code structure for improved readability and maintainability 2025-11-14 16:24:06 -05:00
wjones f719155a35 working on json2postgres.py file 2025-06-25 09:17:45 -04:00
wjones 29814092b7 repo migration 2025-05-20 12:02:07 -04:00