Sumedh A Sontakke

I am an Applied Scientist at Amazon Robotics where I work on robot learning. I'm interested in autonomous decision-making, causality, and physical artificial intelligence. My doctoral thesis investigated foundation models for embodied AI.

I received my PhD in 2024 from USC, where I was fortunate to be advised by Prof Laurent Itti and Prof Erdem Bıyık. In an old life, I helped build the Facebook-start funded Skyline Labs. I have also spent time at the Adolphs Lab at Caltech, the University of Oxford as a SENS Scholar and at the Max Planck Institute for Intelligent systems in Tubingen, Germany. In the industry, I've done internships at Google Brain (now Google Deepmind), Bell Labs, Adobe Research and at the Robot Learning Group at Microsoft Research. I received my bachelor's in electrical engineering at the College of Engineering, Pune in 2019.

Service: I review for ICLR, AISTATS, ICML, NeurIPS and CoRL.

Email / CV / Biography / Google Scholar / Twitter

Preprints

Video2Skill: Adapting Events in Demonstration Videos to Skills in an Environment using Cyclic MDP Homomorphisms
Sumedh A. Sontakke, Sumegh Roychowdhury, Mausoom Sarkar, Nikaash Puri, Laurent Itti, Balaji Krishnamurthy
Preprint. Under Review.

We teach robots to learn skills from demonstrations by humans using real-life video data in a self-supervised manner. Watch our robot stirring by imitating human demonstrations!

Model2Detector:Widening the Information Bottleneck for Out-of-Distribution Detection using a Handful of Gradient Steps
Sumedh A. Sontakke, Buvaneswari Ramanan, Laurent Itti, Thomas Woo
Proceedings of the Robust Artificial Intelligence System Assurance (RAISA) Workshop, AAAI 2022.

We convert a fully trained classification model into an Out-of-Distribution Detector for Safe ML.

Publications

	RoboCLIP:One Demonstration is Enough to Learn Robot Policies Sumedh A. Sontakke, Jesse Zhang, Sébastien M. R. Arnold, Karl Pertsch, Erdem Bıyık, Dorsa Sadigh, Chelsea Finn, Laurent Itti Thirty-seventh Conference on Neural Information Processing Systems, NeurIPS 2023 Website We teach a robot how to perform a task using a single visual demonstration or a textual description of the task.
	RT-1: Robotics Transformer for Real-World Control at Scale. Google Deepmind (Internship Project) Robotics Science and Systems, 2023. Best Demo Paper Finalist Website We build a Foundation Model for Manipulation solving more than 700 tasks.
	SHERLock: Self-Supervised Hierarchical Event Representation Learning Sumegh Roychowdhury, Sumedh A. Sontakke, Nikaash Puri, Mausoom Sarkar, Milan Aggarwal, Pinkesh Badjatiya, Balaji Krishnamurthy, Laurent Itti International Conference on Pattern Recognition 2022 Code We generate hierarchical concepts from long horizon video demonstrations without supervision.
	GalilAI: Out-of-Task Distribution Detection using Causal Active Experimentation for Safe Transfer RL Sumedh A. Sontakke, Stephen Iota, Zizhao Hu, Arash Mehrjou, Laurent Itti, Bernhard Schölkopf The 25th International Conference on Artificial Intelligence and Statistics, 2022.* Website We teach RL agents to detect Out-of-Distribution Tasks. Our agents differentiates between the effects of Mass and Gravity, like Galileo!
	Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning Sumedh A. Sontakke, Arash Mehrjou, Laurent Itti, Bernhard Schölkopf Thirty-eighth International Conference on Machine Learning (ICML) Spotlight, 2021. Website and Code We teach RL agents to perform self-supervised experiments to discover the causal processes like gravity and friction that affect their environment. Check out our Ant pirouetting to discover how heavy it is!
	Unsupervised Hierarchical Concept Learning Sumegh Roychowdhury, Sumedh A. Sontakke, Nikaash Puri, Mausoom Sarkar, Milan Aggarwal, Pinkesh Badjatiya, Balaji Krishnamurthy, Laurent Itti BabyMind Workshop at Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS) 2020. Code We generate hierarchical concepts from long horizon video demonstrations without supervision.
	Classification of Cardiotocography Signals Using Machine Learning Sumedh A. Sontakke, Jay Lohokare, Reshul Dani, Pranav Shivagaje IntelliSys 2018. Advances in Intelligent Systems and Computing. Springer, 869, 2019 We use deep-learning techniques to improve CTG based diagnosis.
	Acquiring Domain Knowledge for Cardiotocography: A Deep Learning Approach Priyamvada Huddar, Sumedh A. Sontakke IEEE International Conference on Informatics and Computational Sciences 2019 We use multi-task learning techniques to improve CTG representation learning.
	Emergency services platform for smart cities Jay Lohokare, Reshul Dani, Sumedh A. Sontakke , Ameya Apte and Rishab Sahni 2017 IEEE Region 10 Symposium (TENSYMP)
	Diagnosis of liver diseases using machine learning Sumedh A. Sontakke, Jay Lohokare, Reshul Dani 2017 IEEE International Conference on Emerging Trends & Innovation in ICT (ICEI) (pp. 129-133)
	Scalable tracking system for public buses using IoT technologies Jay Lohokare, Reshul Dani, Sumedh A. Sontakke 2017 IEEE International Conference on Emerging Trends & Innovation in ICT (ICEI) (pp. 104-109)
	Automated data collection for credit score calculation based on financial transactions and social media Jay Lohokare, Reshul Dani, Sumedh A. Sontakke 2017 IEEE International Conference on Emerging Trends & Innovation in ICT (ICEI) (pp. 134-138)