Download as PDF, PPTX
Uploaded byTimothy Spann
ApacheCon 2021: Apache NiFi 101- introduction and best practices
The document outlines a presentation on Apache NiFi by Timothy Spann, focusing on its introduction and best practices for real-time data flows. It covers how to get started with NiFi, including installation and configuration, and discusses core components such as processors, connections, and process groups. Further resources are provided for deeper understanding and additional content related to Apache NiFi.
ApacheCon 2021: Apache NiFi 101- introduction and best practices
- 1. Apache NiFi 101 TimothySpann Developer Advocate / StreamNative Thursday 23-Sept-2021
- 2. Agenda Thursday 14:10 UTC ApacheNiFi 101: Introduction and Best Practices Timothy Spann In this talk, we will walk step by step through Apache NiFi from the first load to first application. I will include slides, articles and examples to take away as a Quick Start to utilizing Apache NiFi in your real-time dataflows. I will help you get up and running locally on your laptop, Docker or in CDP Public Cloud.
- 3. Speaker Bio DZone ZoneLeader and Big Data MVB @PaasDev https://github.com/tspannhw https://www.datainmotion.dev/ https://github.com/tspannhw/SpeakerProfile https://dev.to/tspannhw https://sessionize.com/tspann/ https://www.slideshare.net/bunkertor Timothy Spann Developer Advocate @
- 4. Powered by ApachePulsar, StreamNative provides a cloud-native, real-time messaging and streaming platform to support multi-cloud and hybrid cloud strategies. Built for Containers Cloud Native StreamNative Cloud Flink SQL
- 5. StreamNative Solution Application MessagingData Pipelines Real-time Contextual Analytics Tiered Storage APP Layer Computing Layer Storage Layer StreamNative Platform IaaS Layer Micro Service Notification Dashboard Risk Control Auditing Payment ETL
- 7.
- 8. Flow File https://nifi.apache.org/docs/nifi-docs/html/overview.html Flow Filesare content and key/value pairs for attributes that are each event/message/file that has been introduced into NiFi.
- 9. Processor https://nifi.apache.org/docs/nifi-docs/html/overview.html A Java componentthat runs in NiFi to route, process or manipulate data. You can build your own if it is not included in Standard NiFi or not in the open source. Controller Like a connection pools, connections, processes that ingest or work with outside data.
- 10. Connection https://nifi.apache.org/docs/nifi-docs/html/overview.html These link togetherNiFi processors. Process Groups Groups of processors. These are versionable and reusable components/modules.
- 11. Record Processors https://www.datainmotion.dev/2019/03/advanced-xml-processing-with-apache.html ● XML,CSV, JSON, AVRO and more ● Schemas or Inferred Schemas ● Easily convert between them ● Support SQL with Apache Calcite
- 12.
- 13.
- 14.
- 15.
- 16. DevOps https://www.datainmotion.dev/2021/01/automating-starting-services-in-apache.html https://nipyapi.readthedocs.io/en/latest/ nifi-toolkit/bin/cli.sh nifi list-param-contexts-u http:/ /edge2ai-1.dim.local:8080 nifi-toolkit/bin/cli.sh nifi pg-list -u http:/ /edge2ai-1.dim.local:8080 nifi-toolkit/bin/cli.sh nifi pg-set-param-context -u http:/ /edge2ai-1.dim.local:8080 ...
- 17. DevOps https://dev.to/tspannhw/automating-starting-services-in-apache-nifi-and-applying-parameters-5h4n https://github.com/tspannhw/ApacheConAtHome2020/blob/main/scripts/setupnifi.sh nifi pg-list nifi pg-status nifipg-get-services nifi pg-enable-services -u http:/ /edge2ai-1.dim.local:8080 --processGroupId root nifi pg-start -u http:/ /edge2ai-1.dim.local:8080 -pgid LOOKTHISUP nifi list-param-contexts -u http:/ /edge2ai-1.dim.local:8080 -verbose nifi create-reporting-task -u http:/ /edge2ai-1.dim.local:8080 -verbose -i
- 18. Consume MQTT This couldread from Apache Pulsar - MoP (MQTT on Pulsar)
- 19. Listen FTP Let ApacheNiFi be your FTP server
- 20. All Data -Anytime - Anywhere - Multi-Cloud - Multi-Protocol Multi- inges t Multi- inges t Multi-ingest Merge Priority
- 21. ● https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html ● https://github.com/tspannhw/EverythingApacheNiFi ●https://www.datainmotion.dev/2019/03/apache-nifi-101.html ● https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html ● https://pierrevillard.com/best-of-nifi/ ● https://blogs.apache.org/nifi/ ● https://www.nifi.rocks/documents/nifi-expression-language-cheat-sheet.pdf ● https://dev.to/tspannhw/new-features-of-apache-nifi-1-13-0-45ln ● https://dev.to/tspannhw/tracking-satellites-with-apache-nifi-44o7 ● https://www.datainmotion.dev/2021/01/flank-using-apache-kudu-as-cache-for.html ● https://www.datainmotion.dev/2020/12/basic-understanding-of-cloudera-flow.html Deeper Content @PaasDev https://datainmotion.dev/ timothyspann
- 22. streamnative.io Pulsar Summit Europe October6, 2021 Pulsar Summit Asia November 20-21, 2021 Contact us at partners@pulsar-summit.org to become a sponsor or partner