Download as PDF, PPTX
Uploaded byMongoDB
MongoDB World 2019: MongoDB in Data Science: How to Build a Scalable Product Using MongoDB
The document discusses the importance of MongoDB in transforming a pandas proof-of-concept into a scalable product for data science projects. It outlines the limitations of pandas, such as lack of persistence and indexing, and highlights MongoDB's benefits including schema-less design and horizontal scalability, which allow for efficient real-time and batch processing. Additionally, it addresses best practices for utilizing MongoDB in a data pipeline and emphasizes its role in enhancing speed and resource efficiency.
MongoDB World 2019: MongoDB in Data Science: How to Build a Scalable Product Using MongoDB
- 1. MongoDB in DataScience How to convert a Pandas Proof-of-Concept to a scalable product and why MongoDB is the key to success !
- 2. Who I am SoftwareEngineer Compiler Engineer Compiler Engineer LLVM contributor Software Engineer R/D Lead ML Engineer Backend Infrastructure Sr. ML Engineer
- 3. What will welearn ? ● Understand existing tools for delivering Data Science projects and when to use them. ● Why MongoDB could be crucial for your product and business ● How to easily productionize a Pandas Proof-of-Concept ● How to use MongoDB while being open to other technologies.
- 4.
- 5.
- 6.
- 7. Research Data Scientist Productionization Data/ML Engineer Speedof development Key factors
- 8. What is Pandas? Mostpopular Python framework for data manipulation and data wrangling in Data Science community.
- 9. What is Pandas? Mostpopular Python framework for data manipulation and data wrangling in Data Science community. Source: numpy.org, scipy.org, matplotlib.org, scikit-learn.org, pandas.pydata.org
- 10. Source: Stackoverflow postby David Robinson
- 11. Why use PandasDataframes ?
- 12. Why use PandasDataframes ?
- 13. Why use PandasDataframes ?
- 14. Why use PandasDataframes ?
- 15. Why use PandasDataframes ?
- 16. Drawbacks of Pandas ●Doesn’t have persistence layer ● Doesn’t support primary and secondary indexes ○ As a result, not efficient for querying ● Doesn’t support multi-threading
- 17.
- 18. Productionization options Real time service BatchJob Slow Inference
- 19. Productionization options Real time service BatchJob Slow Inference Fast Inference
- 20. Real time servicedemo (recommendation) Event Store
- 21. Real time servicedemo (recommendation) Event Store Model Training Job
- 22. Real time servicedemo (recommendation) Event Store Model Training Job Model store
- 23. Real time servicedemo (recommendation) Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 24. Real time servicedemo (recommendation) Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 25. Real time servicedemo (recommendation) Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 26. Real time servicedemo (recommendation) Event Store Feature Aggregation Model Inference Inference Service request respond
- 27. Real time servicedemo (recommendation)
- 28. Real time servicedemo (recommendation)
- 29. Real time servicedemo (recommendation)
- 30. Things to avoid ●Don’t forget to put indexes on your collection ● Don’t put indexes on every field ● Don’t read and write from the same replica
- 31. But… we generatea tons of user events! Is this solution going to work for us?
- 32. user events Consumer 1 Consumer2 Consumer N MongoDB Postgres DFS Typical data pipeline
- 33. user events Consumer 1 Consumer2 Consumer N MongoDB Postgres DFS Typical data pipeline
- 34. MongoDB TTL index Filters event_type ... Consumer Shrink downthe amount of data
- 35. Real time servicedemo (recommendation) Inference 1 Event Store Inference 2 Inference N Model Training Job Model store
- 36. Training Job Inference 1 Event Store Inference2 Inference N Model Training Job Model store
- 37.
- 38.
- 39.
- 40.
- 41. Batch Job versusReal Time Service Real Time Service Batch Job Pros On demand (scales as needed) Easier to develop and maintain Cons Harder to develop and maintain Constantly utilizing resources
- 42. Benefits of MongoDB ●Schema-Less ● Horizontally scalable ● Available as PaaS from many vendors. ● Has a huge community ● Easier to hire people
- 43. Summary ● Allows toprovide a real time experience ● Could help save expensive computational resources ● Provides a way to do real time as well as batch inference
- 44. We are hiring!!! careers.shopbonsai.ca
- 45. References ● https://stackoverflow.blog/2017/09/14/python-growing-quickly/ ● https://www.mongodb.com/products/spark-connector ●https://pandas.pydata.org/ ● https://scikit-learn.org/ ● https://matplotlib.org/ ● https://www.scipy.org/ ● https://www.numpy.org/ ● https://iconscout.com/icon/device-management-mobile-computer-seo-tool-analyze-7
- 46.