Kafka Up And Running For Network Devops Set Your Network Data In Motion Eric Chou

Kafka Up And Running For Network Devops Set Your Network Data In Motion Eric Chou download https://ebookbell.com/product/kafka-up-and-running-for-network- devops-set-your-network-data-in-motion-eric-chou-36653854 Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be interested in. You can click the link to download. Kafka Up And Running For Network Devops Eric Chou https://ebookbell.com/product/kafka-up-and-running-for-network-devops- eric-chou-55476666 Apache Kafka Set Up Apache Kafka Clusters And Develop Custom Message Producers And Consumers Using Practical Handson Examples Nishant Garg https://ebookbell.com/product/apache-kafka-set-up-apache-kafka- clusters-and-develop-custom-message-producers-and-consumers-using- practical-handson-examples-nishant-garg-5476594 Turning Up The Heat Urban Political Ecology For A Climate Emergency Maria Kaika Editor https://ebookbell.com/product/turning-up-the-heat-urban-political- ecology-for-a-climate-emergency-maria-kaika-editor-50975850 Kafka On The Shore Haruki Murakami https://ebookbell.com/product/kafka-on-the-shore-haruki- murakami-46476270

Kafka Streams In Action Second Edition Meap Version 8 2nd Bill Bejeck https://ebookbell.com/product/kafka-streams-in-action-second-edition- meap-version-8-2nd-bill-bejeck-46595524 Kafka And The Universal 1st Edition Arthur Cools Vivian Liska https://ebookbell.com/product/kafka-and-the-universal-1st-edition- arthur-cools-vivian-liska-46609216 Kafka And The Traveling Doll Jordi Sierra I Fabra https://ebookbell.com/product/kafka-and-the-traveling-doll-jordi- sierra-i-fabra-47220834 Kafkas Stereoscopes The Political Function Of A Literary Style Isak Winkel Holm https://ebookbell.com/product/kafkas-stereoscopes-the-political- function-of-a-literary-style-isak-winkel-holm-50216298 Storm Bringer Volume 8 Kafka Asagiri Sango Harukawa https://ebookbell.com/product/storm-bringer-volume-8-kafka-asagiri- sango-harukawa-50511446

Kafka Up and Running for Network DevOps Set Your Network Data in Motion Eric Chou This book is for sale at http://leanpub.com/network-devops-kafka-up-and- running This version was published on 2021-11-12 * * * * * This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishing process. Lean Publishing is the act of publishing an in- progress ebook using lightweight tools and many iterations to get reader feedback, pivot until you have the right book and build traction once you do. * * * * * © 2021 Network Automation Nerds, LLC. ISBN for EPUB version: 978-1-957046-01-3 ISBN for MOBI version: 978-1-957046-02-0

For my family, you are my ‘why’ for everything I do. I would like to thank the open-source software community. My life would be very different without the many dedicated, talented individuals in the open- source community. Thank you all.

Table of Contents Introduction What is Kafka Why do we need Kafka Prerequisites for this book Who this book is for What this book covers Download the example code files Conventions used Get in touch Chapter 1. Kafka Introduction History of Kafka Kafka Use Cases Disadvantages of Kafka Kafka Concepts Conclusion Chapter 2. Kafka Installation and Testing Network Lab Setup Kafka Installation Overview Install Java Download Kafka Configure Zookeeper Configure Kafka Start Zookper and Kafka manually Test the Kafka operations Configure System Services Conclusion Chapter 3. Kafka Concepts and Examples Producers: Writing Messages Consumers: Receiving Messages Offsets in Action

Kafka Topic Administration Replication Conclusion Chapter 4. Hosted Kafka Services AWS Managed Kafka Service Amazon MSK Costs Launch Amazon MSK Cluster Client Setup Produce and Consume Data Conclusion Chapter 5. Cloud Provider Messaging Services Amazon Kinesis Amazon Kinesis Example Azure Event Hub Azure Event Hub Example Google Cloud Pub/Sub GCP Pub/Sub Python Example Conclusion Chapter 6. Network Operations with Kafka Install Docker Install Elasticsearch Install Kibana Network Data Feed Network Data Pipeline Network Log as a Service Conclusion Chapter 7. Other Kafka Considerations and Looking Ahead Hardware Considerations Kafka Broker and Topic Configurations Schema Registry Kafka Stream Processing Cross-Cluster Data Mirroring Additional Resources

Conclusion Appendix A. Installing Lab Instance in Public Cloud

Introduction Welcome to the world of data! Unless you have been living under a rock for the last few years, you know data processing, machine learning, and artificial intelligence are taking over the world. Data exists everywhere around us. We can now check real-time traffic information from online cameras before we even leave the house. We can connect to our thermometers remotely to automatically adjust house temperatures. Better yet, the thermometers can also be self-taught so that they can adjust the temperatures all by themselves. Before our family weekend movie nights, my kids love to leverage the WiFi-enabled lights to match the lighting with our mood. How do these cameras, lights, and thermometers able to take measurements and generate data? It turns out the cost of small sensors and tiny computing units have been coming down steadily since the early days and now can be integrated into everyday items. However, the generated data by one or two devices might not be sufficient enough to yield meaningful results. After all, traffic information on one street might only benefit a tiny fraction of people who travels on that street, but aggregated traffic information on all streets can help everyone. Generally, it is by aggregating all disperse data sets across hundreds of devices; we are able to derive useful information that helps us with our daily lives. The data are constantly flowing between producers and consumers of data. Have you ever wondered how these data are being exchanged between data producers and consumers? Does each of the devices provide an API (Application Programming Interface) to be queried? Do each of them have local databases that persist the data? What about data integrity, transmission latency, or scalability? There are many tools and projects that address these data streaming and exchange issues. One of the most popular open-source tools widely used by companies large and small alike is Apache Kafka.

What is Kafka You might be thinking, “Don’t we already have lots of data storage systems? Why do we need yet-another-storage-system?” You are right, and we do have lots of storage solutions such as relational and non-relational databases, cache systems, big data storage clusters, search solutions, and many more. But in most of the data storage cases, the data is entered in once, stored in the database, then retrieved later when needed. For example, when I visited my dentist for the first time, they asked for my personal information, entered them into a database so for my future visits, they could pull up my record. This is very different than the traffic sensor data example that we discussed. What sets Kafka apart is it was built from the ground up to treat data as continuous flows of information that are constantly being produced, enhanced, manipulated, and consumed. Instead of a focus on holding in data like databases, key-value stores, search indexes, or caches, Kafka architects itself as a system that allows data to be a continually evolving stream of information. According to the Apache Kafka project page: Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Companies known for a large amount of data, such as AirBnb, Datadog, Etsy, and many others across different industries, use Kafka to build their data pipeline. These data pipelines use a variety of services that both produce and consume data in a continuous format.

Figure Intro. 1: Powered by Apache Kafka (https://kafka.apache.org/powered-by) Don’t worry if you have not heard of Kafka before or are not sure how, as network DevOps engineers, this tool can help us. We will go a lot deeper into Kafka in this book.

Why do we need Kafka As a general overview, there are many uses cases for Kafka in network engineers: We can use Kafka to stream data, such as logs and NetFlow data, once and be consumed by multiple receivers. Kafka takes care of the ordering of messages, acknowledging receipt to producers, delivery confirmation to consumers, and balancing the data between different recipients. We can separate data into logical partitions called Topics in a single Kafka cluster. This allows subscribers to only receive the data they are interested in, so the log receiver will not need to receive flow data. Kafka allows for an event-driven architecture, such as triggering events based on different types of events. For example, a log receiver can page an on-call engineer if it notices a BGP neighbor of a core device going down. Kafka allows us to build a centralized pipeline for network data processing instead of having dispersed teams process bits and pieces of data separately. These are just some of the use cases of Kafka. By the end of this book, I am sure we will be able to find much more creative use cases.

Prerequisites for this book Basic knowledge of Linux command line is required to make the most out of this book. We would use command-line tools such as using cd for changing directories, ls for listing directories contents, and pwd to know where in the directory tree you are currently operating from. We will be using Python 3 as the programming language in this book. Python is a popular language amongst network engineers with a large ecosystem of tools and libraries. We will use Python to create Kafka producers, consumers and interface with public cloud providers. However, I do not believe you need to be an expert in Python 3 to understand the scripts in this book. If you need a refresher on Python, a good place to go would be the official Python Tutorial.

Who this book is for This book is ideal for IT professionals and engineers who want to take advantage of Kafka’s distributed, fault-tolerant streaming data platform. This book can also be used by management to gain a general understanding of Kafka and how it fits into the general IT infrastructure.

What this book covers Chapter 1. Kafka Introduction, In this chapter, we will cover the general concepts of Kafka. The core architecture, components, and tools. The idea behind Kafka, how it was built, and how the components can help maintain data streams at scale. Chapter 2. Kafka Installation and Testing, In this chapter, we will install Zookeeper and Kafka on a single Virtual Machine and configure both components. We will also prepare our network lab to be used for future examples. After installation, we will work on a few producer-consumer examples using Kafka command-line tools. Chapter 3. Kafka Concepts and Examples, In this chapter, we will provide examples of Kafka usage for Producers and Consumers. The producers will write messages to a Topic with consumers receiving the messages. We will look at examples of offset, commit, and acknowledgment for data in the topics. Chapter 4. Hosted Kafka Services, When we want to move Kafka from our lab setup into production, we can use the Kafka-hosting-as-a-service provided by various cloud providers, such as Amazon AWS or Confluent Cloud. In this chapter, we will provide a step-by-step guide to launch our Kafka cluster using Amazon Managed Streaming for Apache Kafka. Chapter 5. Cloud Providers Messaging Services, If we are not ready for a managed Kafka cluster, the top public cloud providers, Amazon AWS, Microsoft Azure, and Google Cloud, offer their adopted version of message streaming service. The messaging services have various degrees of Kafka compatibility. In this chapter, we will look at examples of AWS Kinesis, Azure Event Hub, and Google Pub/Sub. Chapter 6. Network Operations with Kafka, In this chapter, we will explore examples of Kafa in network engineering. We will look at data feeds, data enhancement, and Kafka Connect. The Kafka Connect reuses code provided

by the community. We will look at the File and Elasticsearch Kafka connect plugins. Chapter 7. Other Kafka Considerations and Looking Ahead, In this chapter, we will discuss other Kafka considerations, such as hardware requirements, Broker and Topic configuration, Schema registry, and many more. This chapter will provide additional resources for readers to explore Kafka.

Download the example code files The code examples used in this book can be downloaded from GitHub at https://github.com/ericchou1/network-devops-kafka-up-and-running.

Conventions used There are a number of text conventions used in this book to help organize the flow. Information in bold and italic are used to indicate important or special terms. Code blocks are shown below: 1 print('hello world') Command-line input or output will be shown as follows: 1 $ touch my_script.py 2 3 $ ls / 4 bin cdrom etc lib lib64 lost+found mnt proc run snap sw 5 apfile tmp var 6 boot dev home lib32 libx32 media opt root sbin srv sy 7 s usr 8 9 $ python 10 Python 3.8.10 (default, Jun 2 2021, 10:49:15) 11 [GCC 9.4.0] on linux 12 Type "help", "copyright", "credits" or "license" for more information. 13 >>> print('hello world') 14 hello world 15 >>> exit() Warning, tips, and information will be specified in their own special block: This is a tip section. It will include useful tips and tricks in relation to the topic discussed at hand. This is an information section. It will provide additional information to help you explore the topic further.

Another Random Scribd Document with Unrelated Content

containing a part of this work or any other work associated with Project Gutenberg™. 1.E.5. Do not copy, display, perform, distribute or redistribute this electronic work, or any part of this electronic work, without prominently displaying the sentence set forth in paragraph 1.E.1 with active links or immediate access to the full terms of the Project Gutenberg™ License. 1.E.6. You may convert to and distribute this work in any binary, compressed, marked up, nonproprietary or proprietary form, including any word processing or hypertext form. However, if you provide access to or distribute copies of a Project Gutenberg™ work in a format other than “Plain Vanilla ASCII” or other format used in the official version posted on the official Project Gutenberg™ website (www.gutenberg.org), you must, at no additional cost, fee or expense to the user, provide a copy, a means of exporting a copy, or a means of obtaining a copy upon request, of the work in its original “Plain Vanilla ASCII” or other form. Any alternate format must include the full Project Gutenberg™ License as specified in paragraph 1.E.1. 1.E.7. Do not charge a fee for access to, viewing, displaying, performing, copying or distributing any Project Gutenberg™ works unless you comply with paragraph 1.E.8 or 1.E.9. 1.E.8. You may charge a reasonable fee for copies of or providing access to or distributing Project Gutenberg™ electronic works provided that: • You pay a royalty fee of 20% of the gross profits you derive from the use of Project Gutenberg™ works calculated using the method you already use to calculate your applicable taxes. The fee is owed to the owner of the Project Gutenberg™ trademark, but he has agreed to donate royalties under this paragraph to the Project Gutenberg Literary Archive Foundation. Royalty

payments must be paid within 60 days following each date on which you prepare (or are legally required to prepare) your periodic tax returns. Royalty payments should be clearly marked as such and sent to the Project Gutenberg Literary Archive Foundation at the address specified in Section 4, “Information about donations to the Project Gutenberg Literary Archive Foundation.” • You provide a full refund of any money paid by a user who notifies you in writing (or by e-mail) within 30 days of receipt that s/he does not agree to the terms of the full Project Gutenberg™ License. You must require such a user to return or destroy all copies of the works possessed in a physical medium and discontinue all use of and all access to other copies of Project Gutenberg™ works. • You provide, in accordance with paragraph 1.F.3, a full refund of any money paid for a work or a replacement copy, if a defect in the electronic work is discovered and reported to you within 90 days of receipt of the work. • You comply with all other terms of this agreement for free distribution of Project Gutenberg™ works. 1.E.9. If you wish to charge a fee or distribute a Project Gutenberg™ electronic work or group of works on different terms than are set forth in this agreement, you must obtain permission in writing from the Project Gutenberg Literary Archive Foundation, the manager of the Project Gutenberg™ trademark. Contact the Foundation as set forth in Section 3 below. 1.F. 1.F.1. Project Gutenberg volunteers and employees expend considerable effort to identify, do copyright research on, transcribe and proofread works not protected by U.S. copyright

law in creating the Project Gutenberg™ collection. Despite these efforts, Project Gutenberg™ electronic works, and the medium on which they may be stored, may contain “Defects,” such as, but not limited to, incomplete, inaccurate or corrupt data, transcription errors, a copyright or other intellectual property infringement, a defective or damaged disk or other medium, a computer virus, or computer codes that damage or cannot be read by your equipment. 1.F.2. LIMITED WARRANTY, DISCLAIMER OF DAMAGES - Except for the “Right of Replacement or Refund” described in paragraph 1.F.3, the Project Gutenberg Literary Archive Foundation, the owner of the Project Gutenberg™ trademark, and any other party distributing a Project Gutenberg™ electronic work under this agreement, disclaim all liability to you for damages, costs and expenses, including legal fees. YOU AGREE THAT YOU HAVE NO REMEDIES FOR NEGLIGENCE, STRICT LIABILITY, BREACH OF WARRANTY OR BREACH OF CONTRACT EXCEPT THOSE PROVIDED IN PARAGRAPH 1.F.3. YOU AGREE THAT THE FOUNDATION, THE TRADEMARK OWNER, AND ANY DISTRIBUTOR UNDER THIS AGREEMENT WILL NOT BE LIABLE TO YOU FOR ACTUAL, DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE OR INCIDENTAL DAMAGES EVEN IF YOU GIVE NOTICE OF THE POSSIBILITY OF SUCH DAMAGE. 1.F.3. LIMITED RIGHT OF REPLACEMENT OR REFUND - If you discover a defect in this electronic work within 90 days of receiving it, you can receive a refund of the money (if any) you paid for it by sending a written explanation to the person you received the work from. If you received the work on a physical medium, you must return the medium with your written explanation. The person or entity that provided you with the defective work may elect to provide a replacement copy in lieu of a refund. If you received the work electronically, the person or entity providing it to you may choose to give you a second opportunity to receive the work electronically in lieu of a refund.

If the second copy is also defective, you may demand a refund in writing without further opportunities to fix the problem. 1.F.4. Except for the limited right of replacement or refund set forth in paragraph 1.F.3, this work is provided to you ‘AS-IS’, WITH NO OTHER WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY PURPOSE. 1.F.5. Some states do not allow disclaimers of certain implied warranties or the exclusion or limitation of certain types of damages. If any disclaimer or limitation set forth in this agreement violates the law of the state applicable to this agreement, the agreement shall be interpreted to make the maximum disclaimer or limitation permitted by the applicable state law. The invalidity or unenforceability of any provision of this agreement shall not void the remaining provisions. 1.F.6. INDEMNITY - You agree to indemnify and hold the Foundation, the trademark owner, any agent or employee of the Foundation, anyone providing copies of Project Gutenberg™ electronic works in accordance with this agreement, and any volunteers associated with the production, promotion and distribution of Project Gutenberg™ electronic works, harmless from all liability, costs and expenses, including legal fees, that arise directly or indirectly from any of the following which you do or cause to occur: (a) distribution of this or any Project Gutenberg™ work, (b) alteration, modification, or additions or deletions to any Project Gutenberg™ work, and (c) any Defect you cause. Section 2. Information about the Mission of Project Gutenberg™

Project Gutenberg™ is synonymous with the free distribution of electronic works in formats readable by the widest variety of computers including obsolete, old, middle-aged and new computers. It exists because of the efforts of hundreds of volunteers and donations from people in all walks of life. Volunteers and financial support to provide volunteers with the assistance they need are critical to reaching Project Gutenberg™’s goals and ensuring that the Project Gutenberg™ collection will remain freely available for generations to come. In 2001, the Project Gutenberg Literary Archive Foundation was created to provide a secure and permanent future for Project Gutenberg™ and future generations. To learn more about the Project Gutenberg Literary Archive Foundation and how your efforts and donations can help, see Sections 3 and 4 and the Foundation information page at www.gutenberg.org. Section 3. Information about the Project Gutenberg Literary Archive Foundation The Project Gutenberg Literary Archive Foundation is a non- profit 501(c)(3) educational corporation organized under the laws of the state of Mississippi and granted tax exempt status by the Internal Revenue Service. The Foundation’s EIN or federal tax identification number is 64-6221541. Contributions to the Project Gutenberg Literary Archive Foundation are tax deductible to the full extent permitted by U.S. federal laws and your state’s laws. The Foundation’s business office is located at 809 North 1500 West, Salt Lake City, UT 84116, (801) 596-1887. Email contact links and up to date contact information can be found at the Foundation’s website and official page at www.gutenberg.org/contact

Section 4. Information about Donations to the Project Gutenberg Literary Archive Foundation Project Gutenberg™ depends upon and cannot survive without widespread public support and donations to carry out its mission of increasing the number of public domain and licensed works that can be freely distributed in machine-readable form accessible by the widest array of equipment including outdated equipment. Many small donations ($1 to $5,000) are particularly important to maintaining tax exempt status with the IRS. The Foundation is committed to complying with the laws regulating charities and charitable donations in all 50 states of the United States. Compliance requirements are not uniform and it takes a considerable effort, much paperwork and many fees to meet and keep up with these requirements. We do not solicit donations in locations where we have not received written confirmation of compliance. To SEND DONATIONS or determine the status of compliance for any particular state visit www.gutenberg.org/donate. While we cannot and do not solicit contributions from states where we have not met the solicitation requirements, we know of no prohibition against accepting unsolicited donations from donors in such states who approach us with offers to donate. International donations are gratefully accepted, but we cannot make any statements concerning tax treatment of donations received from outside the United States. U.S. laws alone swamp our small staff. Please check the Project Gutenberg web pages for current donation methods and addresses. Donations are accepted in a number of other ways including checks, online payments and

credit card donations. To donate, please visit: www.gutenberg.org/donate. Section 5. General Information About Project Gutenberg™ electronic works Professor Michael S. Hart was the originator of the Project Gutenberg™ concept of a library of electronic works that could be freely shared with anyone. For forty years, he produced and distributed Project Gutenberg™ eBooks with only a loose network of volunteer support. Project Gutenberg™ eBooks are often created from several printed editions, all of which are confirmed as not protected by copyright in the U.S. unless a copyright notice is included. Thus, we do not necessarily keep eBooks in compliance with any particular paper edition. Most people start at our website which has the main PG search facility: www.gutenberg.org. This website includes information about Project Gutenberg™, including how to make donations to the Project Gutenberg Literary Archive Foundation, how to help produce our new eBooks, and how to subscribe to our email newsletter to hear about new eBooks.

Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com

Kafka Up And Running For Network Devops Set Your Network Data In Motion Eric Chou

More Related Content

Similar to Kafka Up And Running For Network Devops Set Your Network Data In Motion Eric Chou

Recently uploaded

Kafka Up And Running For Network Devops Set Your Network Data In Motion Eric Chou