DEV Community

Super Kai (Kazuya Ito)
Super Kai (Kazuya Ito)

Posted on • Edited on

Iterator in Python (3)

#python #generator #iterator #iterable

Buy Me a Coffee☕

*Memo:

A generator:

  • is the function with one or more yield statements:
    • A yield statement is a yield or yield from.
  • can return an iterator.
  • 's iterator can be created by a generator comprehension:
    • A generator comprehension is an expression.
  • terminates if there is no element to return, if close() is called or of course if error occurs.
  • 's iterator cannot be copied.

A generator can be created with a function and one or more yield statements and read with next() as shown below:

*Memo:

  • __next__() can also be used to read a generator.
  • A yield statements is a yield or yield from.
  • A yield can return any type of element.
  • A yield from can only return an iterable.

<yield>: 

def func(): yield 0 yield 1 yield 2 print(func) # <function func at 0x000001FCD2FF93A0> print(type(func)) # <class 'function'>  gen = func() print(gen) # <generator object func at 0x000001FCD3015220> print(type(gen)) # <class 'generator'>  print(next(gen)) # 0 print(next(gen)) # 1 print(next(gen)) # 2 print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode 
def func(): yield [0, 1, 2] yield [3, 4, 5] gen = func() print(next(gen)) # [0, 1, 2] print(next(gen)) # [3, 4, 5] print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

<yield from>: 

def func(): yield from [0, 1, 2] yield from [3, 4, 5] print(func) # <function func at 0x000001FCD640B1A0> print(type(func)) # <class 'function'>  gen = func() print(gen) # <generator object func at 0x000001FCD661DD80> print(type(gen)) # <class 'generator'>  print(next(gen)) # 0 print(next(gen)) # 1 print(next(gen)) # 2 print(next(gen)) # 3 print(next(gen)) # 4 print(next(gen)) # 5 print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode 
def func(): yield from 0 yield from 1 yield from 2 gen = func() print(next(gen)) # TypeError: 'int' object is not iterable
Enter fullscreen mode Exit fullscreen mode

<yield & yield from>: 

def func(): yield [0, 1, 2] yield from [3, 4, 5] gen = func() print(next(gen)) # [0, 1, 2] print(next(gen)) # 3 print(next(gen)) # 4 print(next(gen)) # 5 print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

This is how a generator works as shown below:

*Memo:

  • next() starts or resumes a generator, executes a yield statement to return a value and pauses the generator at a yield statement or raises StopIteration if a generator is terminated:
  • A generator is always paused at a yield statement.
  • __next__() also does the same things.

<yield>: 

def func(): print("func() starts.") yield "func() pauses." print("func() resumes.") yield "func() pauses again." print("func() resumes again.") yield "func() terminates." gen = func() # `func()` doesn't start yet.  print(next(gen)) # func() starts. # func() pauses.  print(next(gen)) # func() resumes. # func() pauses again.  print(next(gen)) # func() resumes again. # func() terminates.  print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

<yield from>: 

def func(): print("func() starts.") yield from ["func() pauses.", "func() resumes and pauses."] print("func() resumes.") yield from ["func() pauses.", "func() resumes and terminates."] gen = func() # `func()` doesn't start yet.  print(next(gen)) # func() starts. # func() pauses.  print(next(gen)) # func() resumes and pauses.  print(next(gen)) # func() resumes. # func() pauses.  print(next(gen)) # func() resumes and terminates.  print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

This is the generator with a for statement as shown below:

<yield>: 

def func(): for x in [0, 1, 2]: yield x gen = func() print(next(gen)) # 0 print(next(gen)) # 1 print(next(gen)) # 2 print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode 
def func(): for x in [[0, 1, 2], [3, 4, 5]]: yield x gen = func() print(next(gen)) # [0, 1, 2] print(next(gen)) # [3, 4, 5] print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

<yield from>: 

def func(): for x in [[0, 1, 2], [3, 4, 5]]: yield from x gen = func() print(next(gen)) # 0 print(next(gen)) # 1 print(next(gen)) # 2 print(next(gen)) # 3 print(next(gen)) # 4 print(next(gen)) # 5 print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode 
def func(): for x in [0, 1, 2]: yield from x gen = func() print(next(gen)) # TypeError: 'int' object is not iterable
Enter fullscreen mode Exit fullscreen mode

<yield & yield from>: 

def func(): for x in [[0, 1, 2], [3, 4, 5]]: yield x yield from x gen = func() print(next(gen)) # [0, 1, 2] print(next(gen)) # 0 print(next(gen)) # 1 print(next(gen)) # 2 print(next(gen)) # [3, 4, 5] print(next(gen)) # 3 print(next(gen)) # 4 print(next(gen)) # 5 print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

A generator comprehension can create a generator's iterator as shown below:

<1D iterator>: 

v = (x**2 for x in [0, 1, 2, 3, 4, 5, 6, 7]) for x in v: print(x) # 0 # 1 # 4 # 9 # 16 # 25 # 36 # 49
Enter fullscreen mode Exit fullscreen mode

<2D iterator>: 

sample = [[0, 1, 2, 3], [4, 5, 6, 7]] v = ((y**2 for y in x) for x in sample) for x in v: for y in x: print(y) # 0 # 1 # 4 # 9 # 16 # 25 # 36 # 49
Enter fullscreen mode Exit fullscreen mode

<3D iterator>: 

sample = [[[0, 1], [2, 3]], [[4, 5], [6, 7]]] v = (((z**2 for z in y) for y in x) for x in sample) for x in v: for y in x: for z in y: print(z) # 0 # 1 # 4 # 9 # 16 # 25 # 36 # 49
Enter fullscreen mode Exit fullscreen mode

A generator's iterator cannot be copied as shown below:

import copy def func(): yield 0 yield 1 yield 2 # yield from [0, 1, 2]  v1 = func() v2 = copy.copy(v1) v2 = copy.deepcopy(v1) # TypeError: cannot pickle 'generator' object
Enter fullscreen mode Exit fullscreen mode

throw() can raise an exception at the point where the generator is paused as shown below:

*Memo:

  • The 1st argument is value(Required-Type:BaseException):
    • Don't use value=.

<yield without a try statement>: 

def func(): yield 0 yield 1 yield 2 yield 3 gen = func() print(next(gen)) # 0 print(next(gen)) # 1 print(gen.throw(Exception)) # Exception:
Enter fullscreen mode Exit fullscreen mode

<yield from without a try statement>: 

def func(): yield from [0, 1, 2, 3] gen = func() print(next(gen)) # 0 print(next(gen)) # 1 print(gen.throw(Exception)) # Exception:
Enter fullscreen mode Exit fullscreen mode

<yield with a try statement>: 

def func(): yield 0 try: yield 1 except Exception: pass yield 2 yield 3 gen = func() print(next(gen)) # 0 print(next(gen)) # 1 print(gen.throw(Exception)) # 2 print(next(gen)) # 3
Enter fullscreen mode Exit fullscreen mode

<yield from with a try statement>: 

def func(): try: yield from [0, 1] except Exception: pass yield from [2, 3] gen = func() print(next(gen)) # 0 print(next(gen)) # 1 print(gen.throw(Exception)) # 2 print(next(gen)) # 3
Enter fullscreen mode Exit fullscreen mode

close() can terminate a generator as shown below:

*Memo:

  • It has no arguments.
  • It should be used in a finally clause to terminate a generator for if error occurs.

<yield without a finally clause>: 

def func(): yield 0 yield 1 yield 2 yield 3 gen = func() print(next(gen)) # 0 print(next(gen)) # 1  gen.close() print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

<yield with a finally clause>: 

def func(): yield 0 yield 1 yield 2 yield 3 gen = func() try: print(next(gen)) # 0  print(next(gen)) # 1  print(next(gen)) # 2  print(next(gen)) # 3 finally: gen.close()
Enter fullscreen mode Exit fullscreen mode

<yield from without a finally clause>: 

def func(): yield from [0, 1, 2, 3] gen = func() print(next(gen)) # 0 print(next(gen)) # 1  gen.close() print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

<yield from with a finally clause>: 

def func(): yield from [0, 1, 2, 3] gen = func() try: print(next(gen)) # 0  print(next(gen)) # 1  print(next(gen)) # 2  print(next(gen)) # 3 finally: gen.close()
Enter fullscreen mode Exit fullscreen mode

gi_yieldfrom can return an iterator if the generator is resumed at yield from otherwise it returns None as shown below:

def func(): yield from [0, 1] yield 2 yield from [3, 4] yield 5 gen = func() while True: try: print(gen.gi_yieldfrom, next(gen.gi_yieldfrom)) except: print(gen.gi_yieldfrom, next(gen)) # None 0 # <list_iterator object at 0x000001C66EBFE410> 1 # <list_iterator object at 0x000001C66EBFE410> 2 # None 3 # <list_iterator object at 0x000001C66EBFE410> 4 # <list_iterator object at 0x000001C66EBFE410> 5
Enter fullscreen mode Exit fullscreen mode

gi_running can check if the generator is currently running as shown below:

<yield>: 

def func(): print(gen.gi_running, "func") # True func  yield 0 print(gen.gi_running, "func") # True func  yield 1 print(gen.gi_running, "func") # True func  yield 2 print(gen.gi_running, "func") # True func  gen = func() print(gen.gi_running) # False print(next(gen)) # 0 print(gen.gi_running) # False print(next(gen)) # 1 print(gen.gi_running) # False print(next(gen)) # 2 print(gen.gi_running) # False print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

<yield from>: 

def func(): print(gen.gi_running, "func") # True func  yield from [0, 1] print(gen.gi_running, "func") # True func  yield from [2, 3] print(gen.gi_running, "func") # True func  yield from [4, 5] print(gen.gi_running, "func") # True func  gen = func() print(gen.gi_running) # False print(next(gen)) # 0 print(gen.gi_running) # False print(next(gen)) # 1 print(gen.gi_running) # False print(next(gen)) # 2 print(gen.gi_running) # False print(next(gen)) # 3 print(gen.gi_running) # False print(next(gen)) # 4 print(gen.gi_running) # False print(next(gen)) # 5 print(gen.gi_running) # False print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

gi_suspended can check if the generator is currently suspended(paused) as shown below:

<yield>: 

def func(): print(gen.gi_suspended, "func") # False func  yield 0 print(gen.gi_suspended, "func") # False func  yield 1 print(gen.gi_suspended, "func") # False func  yield 2 print(gen.gi_suspended, "func") # False func  gen = func() print(gen.gi_suspended) # False print(next(gen)) # 0 print(gen.gi_suspended) # True print(next(gen)) # 1 print(gen.gi_suspended) # True print(next(gen)) # 2 print(gen.gi_suspended) # True print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

<yield from>: 

def func(): print(gen.gi_suspended, "func") # False func  yield from [0, 1] print(gen.gi_suspended, "func") # False func  yield from [2, 3] print(gen.gi_suspended, "func") # False func  yield from [4, 5] print(gen.gi_suspended, "func") # False func  gen = func() print(gen.gi_suspended) # False print(next(gen)) # 0 print(gen.gi_suspended) # True print(next(gen)) # 1 print(gen.gi_suspended) # True print(next(gen)) # 2 print(gen.gi_suspended) # True print(next(gen)) # 3 print(gen.gi_suspended) # True print(next(gen)) # 4 print(gen.gi_suspended) # True print(next(gen)) # 5 print(gen.gi_suspended) # True print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

A yield statement can be assigned to a variable to be used with or without send() as shown below:

*Memo:

  • send() starts or resumes the generator, sends a value into the variable assigned a yield statement only when resuming the generator, executes a yield statement to return a value and pauses the generator or raises StopIteration if the generator is terminated:
    • The 1st argument is value(Required-Type:Any):
      • It must be None when starting the generator.
      • Don't use value=.
  • The variable assigned a yield statement has None by default.
  • yield from only with a generator(yield from generator) works with send() properly.

<yield without send()>: 

def func(): v1 = yield 0 print(v1, 'func') v2 = yield 1 print(v2, 'func') v3 = yield 2 print(v3, 'func') gen = func() print(next(gen)) # 0 print(next(gen)) # None func  # 1 print(next(gen)) # None func  # 2 print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

<yield with send()>: 

def func(): v1 = yield 0 print(v1, 'func') v2 = yield 1 print(v2, 'func') v3 = yield 2 print(v3, 'func') gen = func() print(gen.send(None)) # 0 print(gen.send('A')) # A func  # 1 print(gen.send('B')) # B func  # 2 print(gen.send('C')) # C func print(gen.send('D')) # StopIteration:
Enter fullscreen mode Exit fullscreen mode 
def func(): v1 = yield 0 print(v1, 'func') v2 = yield 1 print(v2, 'func') v3 = yield 2 print(v3, 'func') gen = func() print(gen.send('A')) # TypeError: can't send non-None value to a just-started generator
Enter fullscreen mode Exit fullscreen mode

<yield from without send()>: 

def func(): v1 = yield from [0, 1] print(v1, 'func') v2 = yield from [2, 3] print(v2, 'func') v3 = yield from [4, 5] print(v3, 'func') gen = func() print(next(gen)) # 0 print(next(gen)) # 1  # None func print(next(gen)) # 2 print(next(gen)) # 3  # None func print(next(gen)) # 4 print(next(gen)) # 5  # None func print(next(gen)) # StopIteration:
Enter fullscreen mode Exit fullscreen mode

<yield from with send()>: 

def sub_func(): v1 = yield 0 print(v1, 'sub_func') v2 = yield 1 print(v2, 'sub_func') v3 = yield 2 print(v3, 'sub_func') def func(): v = yield from sub_func() print(v, 'func') gen = func() print(gen.send(None)) # 0 print(gen.send('A')) # A sub_func  # 1 print(gen.send('B')) # B sub_func  # 2 print(gen.send('C')) # C sub_func  # None func  # StopIteration:
Enter fullscreen mode Exit fullscreen mode 
def func(): v1 = yield from [0, 1] print(v1, 'func') v2 = yield from [2, 3] print(v2, 'func') v3 = yield from [4, 5] print(v3, 'func') gen = func() print(gen.send('A')) # TypeError: can't send non-None value to a just-started generator
Enter fullscreen mode Exit fullscreen mode 
def func(): v1 = yield from [0, 1] print(v1, 'func') v2 = yield from [2, 3] print(v2, 'func') v3 = yield from [4, 5] print(v3, 'func') gen = func() print(gen.send(None)) # 0 print(gen.send('A')) # AttributeError: 'list_iterator' object has no attribute 'send'
Enter fullscreen mode Exit fullscreen mode

A yield statement cannot be assigned to a for statement as shown below:

def func(): for v in yield [0, 1, 2]: # for v in yield from [0, 1, 2]:  print(v) # SyntaxError: invalid syntax
Enter fullscreen mode Exit fullscreen mode

Top comments (0)