Download to read offline
![streamnative.io Interested In Learning More? Flink SQL Cookbook The Github Source for Flink SQL Demo The GitHub Source for Demo Manning's Apache Pulsar in Action O’Reilly Book [11/8] PASS Data Community [11/18] Developer Week Austin [11/19] Porto Tech Hub Con [12/3] Data Science Camp Resources Free eBooks Upcoming Events](https://image.slidesharecdn.com/apiworld-apachenifi101-211028164604/75/Api-world-apache-nifi-101-31-2048.jpg)
Uploaded byTimothy Spann
Api world apache nifi 101
Timothy Spann provides an overview of Apache NiFi, an open source dataflow software. Some key points about NiFi include: - It provides guaranteed data delivery, buffering, prioritized queuing, and data provenance. - It supports over 60 source connectors and has hundreds of processors for handling different data formats. - The architecture includes repositories for storing metadata and provenance data, and supports clustering. - Spann discusses best practices for using NiFi such as avoiding spaghetti flows, leveraging parameters and templates, and upgrading to the latest version. He also demonstrates how to consume data from sources like MQTT and FTP.
Api world apache nifi 101
- 1. Apache NiFi 101 TimothySpann | Developer Advocate
- 2. streamnative.io Tim Spann Developer Advocate ●https://www.datainmotion.dev/ ● https://github.com/tspannhw/SpeakerProfile ● https://dev.to/tspannhw ● https://sessionize.com/tspann/ ● https://streamnative.io/ DZone Zone Leader and Big Data MVB Data DJay
- 3.
- 4. Don’t Be Afraid ofOpen Source
- 5. streamnative.io Why Apache NiFi? •Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Hundreds of processors • Visual command and control • Over a sixty sources • Flow templates • Pluggable/multi-role security • Designed for extension • Clustering • Version Control
- 6.
- 7.
- 8.
- 9.
- 10. streamnative.io Flow File https://nifi.apache.org/docs/nifi-docs/html/overview.html Flow Filesare content and key/value pairs for attributes that are each event/message/file that has been introduced into NiFi.
- 11. streamnative.io Version Control (Githuband Beyond)
- 12.
- 13. Record Processors https://www.datainmotion.dev/2019/03/advanced-xml-processing-with-apache.html ● XML,CSV, JSON, AVRO and more ● Schemas or Inferred Schemas ● Easily convert between them ● Support SQL with Apache Calcite
- 14.
- 15.
- 16.
- 17. DevOps https://www.datainmotion.dev/2021/01/automating-starting-services-in-apache.html https://nipyapi.readthedocs.io/en/latest/ nifi-toolkit/bin/cli.sh nifi list-param-contexts-u http:/ /edge2ai-1.dim.local:8080 nifi-toolkit/bin/cli.sh nifi pg-list -u http:/ /edge2ai-1.dim.local:8080 nifi-toolkit/bin/cli.sh nifi pg-set-param-context -u http:/ /edge2ai-1.dim.local:8080 ...
- 18. DevOps https://dev.to/tspannhw/automating-starting-services-in-apache-nifi-and-applying-parameters-5h4n https://github.com/tspannhw/ApacheConAtHome2020/blob/main/scripts/setupnifi.sh nifi pg-list nifi pg-status nifipg-get-services nifi pg-enable-services -u http:/ /edge2ai-1.dim.local:8080 --processGroupId root nifi pg-start -u http:/ /edge2ai-1.dim.local:8080 -pgid LOOKTHISUP nifi list-param-contexts -u http:/ /edge2ai-1.dim.local:8080 -verbose nifi create-reporting-task -u http:/ /edge2ai-1.dim.local:8080 -verbose -i
- 19. Consume MQTT This couldread from Apache Pulsar - MoP (MQTT on Pulsar)
- 20. Listen FTP Let ApacheNiFi be your FTP server
- 21. streamnative.io No More SpaghettiFlows - DO NOT https://dev.to/tspannhw/no-more-spaghetti-flows-2emd Do Not ● Do not Put 1,000 Flows on one workspace. ● If your flow has hundreds of steps, this is a Flow Smell. Investigate why. ● Do not Use ExecuteProcess, ExecuteScripts or a lot of Groovy scripts as a default, look for existing processors ● Do not Use Random Custom Processors you find that have no documentation or are unknown. ● Do not forget to upgrade, if you are running anything before Apache NiFi 1.14, upgrade now! ● Do not run on default 512M RAM. ● Do not run one node and think you have a highly available cluster. ● Do not split a file with millions of records to individual records in one shot without checking available space/memory and back pressure. ● Use Split processors only as an absolute last resort. Many processors are designed to work on FlowFiles that contain many records or many lines of text. Keeping the FlowFiles together instead of splitting them apart can often yield performance that is improved by 1-2 orders of magnitude.
- 22. streamnative.io No More SpaghettiFlows - DO https://dev.to/tspannhw/no-more-spaghetti-flows-2emd Do ● Reduce, Reuse, Recycle. Use Parameters to reuse common modules. ● Put flows, reusable chunks (write to Slack, Database, Kafka) into separate Process Groups. ● Write custom processors if you need new or specialized features ● Use Record Processors everywhere ● Read the Docs! ● Use the NiFi Registry for version control. ● Use NiFi CLI and DevOps for Migrations. ● Walk through your flow and make sure you understand every step and it’s easy to read and follow. Is every processor used? Are there dead ends? ● Do run Zookeeper on different nodes from Apache NiFi. ● Use routing based on content and attributes to allow one flow to handle multiple nearly identical flows is better than deploying the same flow many times with tweaks to parameters in same cluster. ● Use the correct driver for your database. There's usually a couple different JDBC drivers.
- 23. streamnative.io Apache MXNet NativeProcessor through DJL.AI for Apache NiFi This processor uses the DJL.AI Java Interface https://github.com/tspannhw/nifi-djl-processor https://dev.to/tspannhw/easy-deep-learning-in-apache-nifi-with-djl-2d79
- 25. streamnative.io What are theBenefits of Pulsar? Data Durability Scalability Geo-Replication Multi-Tenancy Unified Messaging Model
- 26.
- 27. streamnative.io A Unified MessagingPlatform Message Queuing Data Streaming
- 28.
- 29.
- 30. streamnative.io Founded by theoriginal developers of Apache Pulsar and Apache BookKeeper, StreamNative builds a cloud-native event streaming platform that enables enterprises to easily access data as real-time event streams.
- 31. streamnative.io Interested In LearningMore? Flink SQL Cookbook The Github Source for Flink SQL Demo The GitHub Source for Demo Manning's Apache Pulsar in Action O’Reilly Book [11/8] PASS Data Community [11/18] Developer Week Austin [11/19] Porto Tech Hub Con [12/3] Data Science Camp Resources Free eBooks Upcoming Events
- 32.
- 33. Let’s Keep in Touch! SpeakerName Speaker title @PassDev https://www.linkedin.com/in/timothyspann https://github.com/tspannhw