[ODSC-52993] Set up caching for listing models and compartments #592
Conversation
oracle-contributor-agreement bot added the OCA Verified 
|
|
| 📌 Cov diff with main: No success to gather report. 😿 📌 Overall coverage: No success to gather report. 😿 |
| 📌 Cov diff with main: 📌 Overall coverage: |
| ] | ||
| llm = ["langchain>=0.0.295", "evaluate>=0.4.0"] | ||
| aqua = ["fire"] | ||
| aqua = ["fire", "cachetools"] |
There was a problem hiding this comment.
We need to check license for this.
There was a problem hiding this comment.
did a search on slack and seems a lot of projects are using it, still need to confirm.
| """Read the information of an Aqua model.""" | ||
| return self.finish(AquaModelApp().get(model_id)) | ||
| | ||
| @handle_exceptions |
There was a problem hiding this comment.
Not sure if we need this :) User will not be able to reach out this end point. If only we add supporting refresh button in UI which might be a good idea.
There was a problem hiding this comment.
agree- depends on what we want to start with first. If we just use it for service models, then it makes sense not to have it. For custom models, we'll need it.
ads/aqua/model.py Outdated
| ), | ||
| ) | ||
| | ||
| @cached( |
There was a problem hiding this comment.
I think the problem with this approach is that we want different caching for the Service models and Custom models. Maybe we can start only with the service models?
There was a problem hiding this comment.
In that case, the parameters values for the cache key can be set as None, that way results are cached only when the function is called without specific compartment id and project id. Would using a cached decorator still be a good idea or is it better to explicitly maintain a cache in the class?
There was a problem hiding this comment.
There are pros and cons. Does it support saving cache into the file? ALso does it support updating cache explicitly? For example if we return user the cashed data but in the parallel thread update the case with the fresh data?
There was a problem hiding this comment.
having different cache for service vs. custom models isn't supported via the decorator directly. It's doable with custom wrappers but makes it more complex. Moved the cache outside and now we're caching service models and compartments only.
Need more testing on our end but UI team can use it in the meanwhile to that they don't have to wait longer times for testing.
There was a problem hiding this comment.
Let us start with the service models for now. The custom models anyway goes through RQS, which is makes it slightly faster.
Let us also add it to the compartment listing as well. I think these two will improve the overall experience.
|
| 📌 Cov diff with main: 📌 Overall coverage: |
|
| 📌 Cov diff with main: 📌 Overall coverage: |
VipulMascarenhas changed the title | """Handles DELETE request for clearing cache""" | ||
| url_parse = urlparse(self.request.path) | ||
| paths = url_parse.path.strip("/") | ||
| if paths.startswith("aqua/model/cache"): |
There was a problem hiding this comment.
Wouldn't it be better to move the paths into the const variable? At least all of them will be in one place.
There was a problem hiding this comment.
agree. Added this to pending items and will update all paths in a separate PR.
| ) | ||
| compartments.insert( | ||
| 0, | ||
| Compartment(id=TENANCY_OCID, name=" ** Root - Name N/A **"), |
There was a problem hiding this comment.
That what we want to show to user if they don't have permissions?
There was a problem hiding this comment.
I wasn't sure if it should be either empty or return the tenancy ocid. My thinking was that they should be able to see the root compartment at least.
|
| 📌 Cov diff with main: 📌 Overall coverage: |
|
| 📌 Cov diff with main: 📌 Overall coverage: |
3 participants
To cache model list API results, we can use TTL cache from cachetools. docs
We have set up TTL cache for model listing (5 hrs) and compartment listing (2 hrs) APIs. Since we need an option to clear/invalidate the cache, we need to set
lockas well.Note that caching is only enabled for models in the service compartment and not in the custom compartment.
To invalidate service model cache, user can call:
To invalidate compartment listing cache, call:
Time to list models locally:
Time to list compartments locally:
To do: