Profiling and misc #10
Conversation
…n function, editable transformers in docker image, cleanup
| input_tokens[t] = input_tokens[t].to(self.device) | ||
| | ||
| t1 = time.perf_counter() | ||
| with torch.no_grad(): |
There was a problem hiding this comment.
Is this needed?
The generate method already runs in torch.no_grad context by default
There was a problem hiding this comment.
It's probably useless but a bit of extra safety can't hurt. Same with the model.eval() in __init__.
There was a problem hiding this comment.
Any reason not to use inference_mode()?
There was a problem hiding this comment.
I didn't know about this one, thanks for pointing it out. I will leave it for future work though.
There was a problem hiding this comment.
Sure, I'm not sure of the context here but it should be a harmless replacement ... it includes everything no_grad does.
| # BLOOM AliBi | ||
| hf-1b-bloom-fp32: | ||
| python src/main.py --hidden_size 2048 --n_head 16 --n_layer 24 --pipeline_class HF_GPU_Pipeline --model_class BLOOM --dtype float32 --batch_size ${batch_size} | ||
| python3 src/main.py --hidden_size 2048 --n_head 16 --n_layer 24 --pipeline_class HF_Pipeline --model_class BLOOM --dtype float32 --batch_size ${batch_size} |
There was a problem hiding this comment.
@mayank31398 I don't see why we should have it as a module. It's a script and not installed as a package.
| return config, tokenizer, model_class | ||
| return sum(p.numel() for p in self.model.parameters()) | ||
| | ||
| def aggregate_and_format_metrics(self, metrics: List[Dict[str, Any]]): |
There was a problem hiding this comment.
@mayank31398 It's better to leave this encapsulated with the class that generates these metrics.
This reverts commit 4be387d.
3 participants
Pipelineclass