Downloaded 62 times
![Optimization process Recode CRAN R to Rx R Before trainit <- glm(as.formula(specs[[i]]), data = training.data, family='binomial', maxit=iters) fits <- predict(trainit, newdata=test.data, type='response') After trainit <- rxGlm(as.formula(specs[[i]]), data = training.data, family='binomial', maxIterations=iters) fits <- rxPredict(trainit, newdata=test.data, type='response') Copyright © 2014 Accenture. All rights reserved. 7](https://image.slidesharecdn.com/accentureandrevolutionanalyticsonteradatav31-140917130022-phpapp02/75/Quick-and-Dirty-Scaling-Out-Predictive-Models-Using-Revolution-Analytics-on-Teradata-7-2048.jpg)
Uploaded byRevolution Analytics
Quick and Dirty: Scaling Out Predictive Models Using Revolution Analytics on Teradata
The document discusses the rapid productionalization of predictive models using in-database modeling with Revolution Analytics on Teradata, led by Skylar Lyon from Accenture Analytics. Key achievements include optimizing a predictive modeling process from over 5 hours to just 40 seconds by moving computation closer to the data. The new method allows for larger training datasets and supports Hadoop, enhancing the capabilities of the data science team.
Quick and Dirty: Scaling Out Predictive Models Using Revolution Analytics on Teradata
- 1. Rapid Productionalization ofPredictive Models In-database Modeling with Revolution Analytics on Teradata Skylar Lyon Accenture Analytics
- 2. Introduction Skylar Lyon Accenture Analytics • 7 years of experience with focus on big data and predictive analytics - using discrete choice modeling, random forest classification, ensemble modeling, and clustering • Technology experience includes: Hadoop, Accumulo, PostgreSQL, qGIS, JBoss, Tomcat, R, GeoMesa, and more • Worked from Army installations across the nation and also had the opportunity to travel twice to Baghdad to deploy solutions downrange. Copyright © 2014 Accenture. All rights reserved. 2
- 3. How we gothere Project background and my involvement • New Customer Analytics team for Silicon Valley Internet eCommerce giant • Data scientists developing predictive models • Deferred focus on productionalization • Joined as Big Data Infrastructure and Analytics Lead Copyright © 2014 Accenture. All rights reserved. 3
- 4. Colleague‘s CRAN Rmodel Binomial logistic regression • 50+ Independent variables including categorical with indicator variables • Train from small sample (many thousands) – not a problem in and of itself • Scoring across entire corpus (many hundred millions) – slightly more challenging Copyright © 2014 Accenture. All rights reserved. 4
- 5. We optimized thecurrent productionalization process We moved compute to data Before After Reduced 5+ hour process to 40 seconds Copyright © 2014 Accenture. All rights reserved. 5
- 6. Benchmarking our optimizedprocess 5+ hours to 40 seconds: Recommendation is that this now become the defacto productionalization process Copyright © 2014 Accenture. All rights reserved. 6 rows minutes
- 7. Optimization process RecodeCRAN R to Rx R Before trainit <- glm(as.formula(specs[[i]]), data = training.data, family='binomial', maxit=iters) fits <- predict(trainit, newdata=test.data, type='response') After trainit <- rxGlm(as.formula(specs[[i]]), data = training.data, family='binomial', maxIterations=iters) fits <- rxPredict(trainit, newdata=test.data, type='response') Copyright © 2014 Accenture. All rights reserved. 7
- 8. Additional benefits tonew process Technology is increasing data science team’s options and opportunities • Train in-database on much larger set – reduces need to sample • Nearly “native” R language – decrease deploy time • Hadoop support – score in multiple data warehouses Copyright © 2014 Accenture. All rights reserved. 8
- 9. Appendix Table ofContents • Technical Considerations Copyright © 2014 Accenture. All rights reserved. 9
- 10. Technical considerations Environmentsetup • Teradata environment – 4 node, 1700 series appliance server • Revolution R Enterprise – version 7.1, running R 3.0.2 Copyright © 2014 Accenture. All rights reserved. 10