fix: Optimize GECloud memory usage by clearing cached data after fetch #2897
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Fixes OOM crashes by preventing duplicate data storage in RAM cache. Previously, GECloud data was stored in three places: 1. self.mdata (needed for processing) 2. self.ge_url_cache[url]["data"] (RAM cache - unnecessary duplication) 3. YAML file on disk (for persistence) This change clears the cached data from RAM after accumulating all data into self.mdata, keeping only metadata (stamp, next) in the RAM cache. The data is still saved to disk via save_ge_cache() for future runs. Memory savings: ~2,304 data points × 5 fields × 8 days of data no longer duplicated in RAM. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
| # This prevents duplicate storage - data is only in self.mdata and disk cache | ||
| for url_key in list(self.ge_url_cache.keys()): | ||
| if "data" in self.ge_url_cache[url_key]: | ||
| del self.ge_url_cache[url_key]["data"] |
Owner
There was a problem hiding this comment.
This isn't right, it will not save the data to disk either as you deleted it before the save
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Summary
Fixes OOM (Out of Memory) crashes in GECloud data fetching by eliminating duplicate data storage in RAM cache.
Problem
GECloud was storing fetched data in three separate locations:
self.mdata- needed for processing byminute_data()self.ge_url_cache[url]["data"]- RAM cache (unnecessary duplication)For typical usage (8 days of historical data at ~5-minute intervals = ~2,304 data points), this resulted in significant memory overhead from the duplicate RAM cache storage.
Solution
This PR clears the cached data from RAM after all data has been accumulated into
self.mdata:stamp,next) in the RAM cachesave_ge_cache()for future runsself.mdataChanges
apps/predbat/gecloud.py:1538- Added comment noting cached data is temporaryapps/predbat/gecloud.py:1541- Added comment on temporary data storageapps/predbat/gecloud.py:1606-1610- Added cleanup loop to clear cached data after fetchTesting
get_data()Memory Savings
Eliminates ~2,304 × 5 fields × 8 bytes per field = ~92 KB per fetch cycle from RAM cache duplication (actual savings may be higher depending on pagination and data density).
🤖 Generated with Claude Code