Download as PDF, PPTX
Building a data pipeline to ingest data into Hadoop in minutes using Streamsets Data Collector
- 1. Building a datapipeline to ingest data into Hadoop in minutes using Streamsets Data Collector Guglielmo Iozzia, Big Data Infrastructure Engineer @ IBM Ireland
- 2. Data Ingestion forAnalytics: a real scenario In the business area (cloud applications) to which my team belongs there were so many questions to be answered. They were related to: ● Defect analysis ● Outage analysis ● Cyber-Security
- 3. “Data is thesecond most important thing in analytics”
- 4. Data Ingestion: multiplesources... ● Legacy systems ● DB2 ● Lotus Domino ● MongoDB ● Application logs ● System logs ● New Relic ● Jenkins pipelines ● Testing tools output ● RESTful Services
- 5. … and somany tools available to get the data
- 6. What are wegoing to do with all those data?
- 7. Issues ● The needto collect data from multiple sources introduces redundancy, which costs additional disk space and increases query times. ● A small team. ● Lack of skills and experience across the team (and the business area in general) in managing Big Data tools. ● Low budget.
- 8.
- 9. Alternatives #2 Cloning teammembers
- 10. Alternatives #3 Find asmart way to simplify the data ingestion process
- 11. A single toolneeded... ● Design complex data flows with minimal coding and the maximum flexibility. ● Provide real-time data flow statistics, metrics for each flow stage. ● Automated error handling and alerting. ● Easy to use by everyone. ● Zero-downtime when upgrading the infrastructure due to logical isolation of each flow stage. ● Open Source
- 12. … something likethis
- 13.
- 14.
- 15. Streamsets Data Collector:supported origins
- 16. Streamsets Data Collector:available destinations
- 17. Streamsets Data Collector:available processors ● Base64 Field Decoder ● Base64 Field Encoder ● Expression Evaluator ● Field Converter ● JavaScript Evaluator ● JSON Parser ● Jython Evaluator ● Log Parser ● Stream Selector ● XML Parser ...and many others
- 18.
- 19. Streamsets DC: performanceand reliability ● Two available execution modes: standalone or cluster ● Implemented in Java: so any performance best practice/recommendation for Java applications applies here ● REST services for performance monitoring available ● Rules and alerts (metric and data both)
- 20. Streamsets Data Collector:security ● You can authenticate user accounts based on LDAP ● Authorization: the Data Collector provides several roles (admin, manager, creator, guest) ● You can use Kerberos authentication to connect to origin and destination systems ● Follow the usual security best practices in terms of iptables, networking, etc. for Java web applications running on Linux machines.
- 21. Useful Links Streamsets DataCollector: https://streamsets.com/product/
- 22. Thanks! My contacts: Linkedin: https://ie.linkedin.com/in/giozzia Blog:http://googlielmo.blogspot.ie/ Twitter: https://twitter.com/guglielmoiozzia