Java Scanner tokens() Method

📘 Premium Read: Access my best content on Medium member-only articles — deep dives into Java, Spring Boot, Microservices, backend architecture, interview preparation, career advice, and industry-standard best practices.

🎓 Top 15 Udemy Courses (80-90% Discount): My Udemy Courses - Ramesh Fadatare — All my Udemy courses are real-time and project oriented courses.

▶️ Subscribe to My YouTube Channel (176K+ subscribers): Java Guides on YouTube

▶️ For AI, ChatGPT, Web, Tech, and Generative AI, subscribe to another channel: Ramesh Fadatare on YouTube

The tokens() method in Java, part of the java.util.Scanner class, is used to create a stream of tokens from the input. This method is useful for processing input data as a stream of strings.

Table of Contents

1. Introduction
2. tokens() Method Syntax
3. Understanding tokens()
4. Examples
   • Basic Usage
   • Processing Tokens as a Stream
5. Real-World Use Case
6. Conclusion

Introduction

The tokens() method returns a stream of tokens that the Scanner generates from the input. This method is useful when you want to process input data in a functional style using Java Streams.

tokens() Method Syntax

The syntax for the tokens() method is as follows:

public Stream<String> tokens() 

Parameters:

• This method does not take any parameters.

Returns:

• A Stream<String> of tokens.

Throws:

• IllegalStateException: If the scanner is closed.

Understanding tokens()

The tokens() method converts the input data into a stream of strings (tokens). This is useful for processing large amounts of data in a functional programming style, leveraging the power of Java Streams for efficient and concise data manipulation.

Examples

Basic Usage

To demonstrate the basic usage of tokens(), we will create a Scanner object and use it to create a stream of tokens from a string.

Example

import java.util.Scanner; import java.util.stream.Stream; public class TokensExample { public static void main(String[] args) { String input = "Hello world! Welcome to Java programming."; // Create Scanner object in try-with-resources to ensure it closes automatically try (Scanner scanner = new Scanner(input)) { Stream<String> tokenStream = scanner.tokens(); // Print each token tokenStream.forEach(System.out::println); } // Scanner is automatically closed here } } 

Output:

Hello world! Welcome to Java programming. 

Processing Tokens as a Stream

This example shows how to process tokens as a stream, such as filtering and mapping tokens.

Example

import java.util.Scanner; import java.util.stream.Collectors; import java.util.stream.Stream; public class ProcessTokensExample { public static void main(String[] args) { String input = "apple banana cherry date"; // Create Scanner object in try-with-resources to ensure it closes automatically try (Scanner scanner = new Scanner(input)) { Stream<String> tokenStream = scanner.tokens(); // Process tokens: filter and collect to a list var filteredTokens = tokenStream .filter(token -> token.length() > 5) .collect(Collectors.toList()); System.out.println("Filtered tokens: " + filteredTokens); } // Scanner is automatically closed here } } 

Output:

Filtered tokens: [banana, cherry] 

Real-World Use Case

Parsing a CSV File

In real-world applications, the tokens() method can be used to parse CSV data into individual tokens for further processing.

Example

import java.io.File; import java.io.FileNotFoundException; import java.util.Scanner; import java.util.stream.Stream; public class CSVParser { public static void main(String[] args) { File file = new File("data.csv"); try (Scanner scanner = new Scanner(file)) { scanner.useDelimiter(","); // Use comma as delimiter Stream<String> tokenStream = scanner.tokens(); // Process tokens: print each token tokenStream.forEach(System.out::println); } catch (FileNotFoundException e) { System.out.println("File not found: " + e.getMessage()); } // Scanner is automatically closed here } } 

Output (Assuming data.csv contains comma-separated values):

value1 value2 value3 ... 

Conclusion

The Scanner.tokens() method is used to create a stream of tokens from the input. This method is particularly useful for applications requiring functional-style data processing. By understanding and using this method, you can efficiently parse and handle input data as streams. Always close the Scanner using try-with-resources to ensure proper resource management.