Improve public API type annotations & fix unit test type errors (#248)

* chore: improve typing of functions returning AnyCloudEvent kafka.conversion.from_binary() and from_structured() return AnyCloudEvent type var according to their event_type argument, but when event_type is None, type checkers cannot infer the return type. We now use an overload to declare that the return type is http.CloudEvent when event_type is None. Previously users had to explicitly annotate this type when calling without event_type. This happens quite a lot in this repo's test_kafka_conversions.py — this fixes quite a few type errors like: > error: Need type annotation for "result" [var-annotated] Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: type v1.Event chainable Set*() methods The v1.Event self-returning Set*() methods like SetData() were returning BaseEvent, which doesn't declare the same Set* methods. As a result, chaining more than one Set* method would make the return type unknown. This was causing type errors in test_event_pipeline.py. The Set*() methods now return the Self type. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: fix type errors in tests mypy was failing with lots of type errors in test modules. I've not annotated all fixtures, mostly fixed existing type errors. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: allow non-dict headers types in from_http() from_http() conversion function was requiring its headers argument to be a typing.Dict, which makes it incompatible with headers types of http libraries, which support features like multiple values per key. typing.Mapping and even _typeshed.SupportsItems do not cover these types. For example, samples/http-image-cloudevents/image_sample_server.py was failing to type check where it calls `from_http(request.headers, ...)`. To support these kind of headers types in from_http(), we now define our own SupportsDuplicateItems protocol, which is broader than _typeshed.SupportsItems. I've only applied this to from_http(), as typing.Mapping is OK for most other methods that accept dict-like objects, and using this more lenient interface everywhere would impose restrictions on our implementation, even though it might be more flexible for users. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * build: run mypy via tox Tox now runs mypy on cloudevents itself, and the samples. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * build(ci): run mypy in CI alongside linting Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: fix minor mypy type complaint in samples Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * feat: use Mapping, not Dict for input arguments Mapping imposes less restrictions on callers, because it's read-only and allows non-dict types to be passed without copying them as dict(), or passing dict-like values and ignoring the resulting type error. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: fix tests on py3.8 Tests were failing because the sanic dependency dropped support for py3.8 in its current release. sanic is now pinned to the last compatible version for py3.8 only. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * feat: support new model_validate_json() kwargs Pydantic added by_alias and by_name keyword arguments to BaseModel.model_validate_json in 2.11.1: pydantic/pydantic@acb0f10 This caused mypy to report that that the Pydantic v2 CloudEvent did not override model_validate_json() correctly. Our override now accepts these newly-added arguments. They have no effect, as the implementation does not use Pydantic to validate the JSON, but we also don't use field aliases, so the only effect they could have in the superclass would be to raise an error if they're both False. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: accept Mapping as well as SupportsDuplicateItems Although our types.SupportsDuplicateItems type is wider than Dict and Mapping, it's not a familar type to users, so explicitly accepting Mapping in the from_http() functions should make it more clear to users that a dict-like object is required for the headers argument. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * chore: constrain deps to maintain py 3.8 support Python 3.8 is unsupported and dependencies (such as pydantic) are now shipping releases that fail to type check with mypy running in 3.8 compatibility mode. We run mypy in py 3.8 compatibility mode, so the mypy tox environments must only use deps that support 3.8. And unit tests run by py 3.8 must only use deps that support 3.8. To constrain the deps for 3.8 support, we use two constraint files, one for general environments that only constrains the dependencies that python 3.8 interpreters use, and another for mypy that constraints the dependencies that all interpreters use. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> --------- Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk>

Author: h4l

.github/workflows/main.yml

@@ -17,7 +17,7 @@ jobs:
  - name: Install dev dependencies
  run: python -m pip install -r requirements/dev.txt
  - name: Run linting
- run: python -m tox -e lint
+ run: python -m tox -e lint,mypy,mypy-samples-image,mypy-samples-json
 
  test:
  strategy:

cloudevents/abstract/event.py

@@ -32,7 +32,7 @@ class CloudEvent:
  @classmethod
  def create(
  cls: typing.Type[AnyCloudEvent],
- attributes: typing.Dict[str, typing.Any],
+ attributes: typing.Mapping[str, typing.Any],
  data: typing.Optional[typing.Any],
  ) -> AnyCloudEvent:
  """

cloudevents/conversion.py

@@ -91,7 +91,9 @@ def from_json(
 
 def from_http(
  event_type: typing.Type[AnyCloudEvent],
- headers: typing.Mapping[str, str],
+ headers: typing.Union[
+ typing.Mapping[str, str], types.SupportsDuplicateItems[str, str]
+ ],
  data: typing.Optional[typing.Union[str, bytes]],
  data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
 ) -> AnyCloudEvent:
@@ -260,7 +262,7 @@ def best_effort_encode_attribute_value(value: typing.Any) -> typing.Any:
 
 def from_dict(
  event_type: typing.Type[AnyCloudEvent],
- event: typing.Dict[str, typing.Any],
+ event: typing.Mapping[str, typing.Any],
 ) -> AnyCloudEvent:
  """
  Constructs an Event object of a given `event_type` from

cloudevents/http/conversion.py

@@ -37,7 +37,9 @@ def from_json(
 
 
 def from_http(
- headers: typing.Dict[str, str],
+ headers: typing.Union[
+ typing.Mapping[str, str], types.SupportsDuplicateItems[str, str]
+ ],
  data: typing.Optional[typing.Union[str, bytes]],
  data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
 ) -> CloudEvent:
@@ -58,7 +60,7 @@ def from_http(
 
 
 def from_dict(
- event: typing.Dict[str, typing.Any],
+ event: typing.Mapping[str, typing.Any],
 ) -> CloudEvent:
  """
  Constructs a CloudEvent from a dict `event` representation.

cloudevents/http/event.py

@@ -34,11 +34,13 @@ class CloudEvent(abstract.CloudEvent):
 
  @classmethod
  def create(
- cls, attributes: typing.Dict[str, typing.Any], data: typing.Optional[typing.Any]
+ cls,
+ attributes: typing.Mapping[str, typing.Any],
+ data: typing.Optional[typing.Any],
  ) -> "CloudEvent":
  return cls(attributes, data)
 
- def __init__(self, attributes: typing.Dict[str, str], data: typing.Any = None):
+ def __init__(self, attributes: typing.Mapping[str, str], data: typing.Any = None):
  """
  Event Constructor
  :param attributes: a dict with cloudevent attributes. Minimally

cloudevents/kafka/conversion.py

@@ -111,11 +111,29 @@ def to_binary(
  return KafkaMessage(headers, message_key, data)
 
 
+@typing.overload
 def from_binary(
  message: KafkaMessage,
- event_type: typing.Optional[typing.Type[AnyCloudEvent]] = None,
+ event_type: None = None,
+ data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
+) -> http.CloudEvent:
+ pass
+
+
+@typing.overload
+def from_binary(
+ message: KafkaMessage,
+ event_type: typing.Type[AnyCloudEvent],
  data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
 ) -> AnyCloudEvent:
+ pass
+
+
+def from_binary(
+ message: KafkaMessage,
+ event_type: typing.Optional[typing.Type[AnyCloudEvent]] = None,
+ data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
+) -> typing.Union[http.CloudEvent, AnyCloudEvent]:
  """
  Returns a CloudEvent from a KafkaMessage in binary format.
 
@@ -144,10 +162,11 @@ def from_binary(
  raise cloud_exceptions.DataUnmarshallerError(
  f"Failed to unmarshall data with error: {type(e).__name__}('{e}')"
  )
+ result: typing.Union[http.CloudEvent, AnyCloudEvent]
  if event_type:
  result = event_type.create(attributes, data)
  else:
- result = http.CloudEvent.create(attributes, data) # type: ignore
+ result = http.CloudEvent.create(attributes, data)
  return result
 
 
@@ -210,12 +229,32 @@ def to_structured(
  return KafkaMessage(headers, message_key, value)
 
 
+@typing.overload
 def from_structured(
  message: KafkaMessage,
- event_type: typing.Optional[typing.Type[AnyCloudEvent]] = None,
+ event_type: None = None,
+ data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
+ envelope_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
+) -> http.CloudEvent:
+ pass
+
+
+@typing.overload
+def from_structured(
+ message: KafkaMessage,
+ event_type: typing.Type[AnyCloudEvent],
  data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
  envelope_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
 ) -> AnyCloudEvent:
+ pass
+
+
+def from_structured(
+ message: KafkaMessage,
+ event_type: typing.Optional[typing.Type[AnyCloudEvent]] = None,
+ data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
+ envelope_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
+) -> typing.Union[http.CloudEvent, AnyCloudEvent]:
  """
  Returns a CloudEvent from a KafkaMessage in structured format.
 
@@ -264,8 +303,9 @@ def from_structured(
  attributes["datacontenttype"] = val.decode()
  else:
  attributes[header.lower()] = val.decode()
+ result: typing.Union[AnyCloudEvent, http.CloudEvent]
  if event_type:
  result = event_type.create(attributes, data)
  else:
- result = http.CloudEvent.create(attributes, data) # type: ignore
+ result = http.CloudEvent.create(attributes, data)
  return result

cloudevents/pydantic/v1/conversion.py

@@ -21,7 +21,9 @@
 
 
 def from_http(
- headers: typing.Dict[str, str],
+ headers: typing.Union[
+ typing.Mapping[str, str], types.SupportsDuplicateItems[str, str]
+ ],
  data: typing.Optional[typing.AnyStr],
  data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
 ) -> CloudEvent:
@@ -63,7 +65,7 @@ def from_json(
 
 
 def from_dict(
- event: typing.Dict[str, typing.Any],
+ event: typing.Mapping[str, typing.Any],
 ) -> CloudEvent:
  """
  Construct an CloudEvent from a dict `event` representation.

cloudevents/pydantic/v1/event.py

@@ -100,7 +100,9 @@ class CloudEvent(abstract.CloudEvent, BaseModel): # type: ignore
 
  @classmethod
  def create(
- cls, attributes: typing.Dict[str, typing.Any], data: typing.Optional[typing.Any]
+ cls,
+ attributes: typing.Mapping[str, typing.Any],
+ data: typing.Optional[typing.Any],
  ) -> "CloudEvent":
  return cls(attributes, data)
 
@@ -155,7 +157,7 @@ def create(
 
  def __init__( # type: ignore[no-untyped-def]
  self,
- attributes: typing.Optional[typing.Dict[str, typing.Any]] = None,
+ attributes: typing.Optional[typing.Mapping[str, typing.Any]] = None,
  data: typing.Optional[typing.Any] = None,
  **kwargs,
  ):

cloudevents/pydantic/v2/conversion.py

@@ -22,7 +22,9 @@
 
 
 def from_http(
- headers: typing.Dict[str, str],
+ headers: typing.Union[
+ typing.Mapping[str, str], types.SupportsDuplicateItems[str, str]
+ ],
  data: typing.Optional[typing.AnyStr],
  data_unmarshaller: typing.Optional[types.UnmarshallerType] = None,
 ) -> CloudEvent:
@@ -64,7 +66,7 @@ def from_json(
 
 
 def from_dict(
- event: typing.Dict[str, typing.Any],
+ event: typing.Mapping[str, typing.Any],
 ) -> CloudEvent:
  """
  Construct an CloudEvent from a dict `event` representation.

cloudevents/pydantic/v2/event.py

@@ -44,7 +44,9 @@ class CloudEvent(abstract.CloudEvent, BaseModel): # type: ignore
 
  @classmethod
  def create(
- cls, attributes: typing.Dict[str, typing.Any], data: typing.Optional[typing.Any]
+ cls,
+ attributes: typing.Mapping[str, typing.Any],
+ data: typing.Optional[typing.Any],
  ) -> "CloudEvent":
  return cls(attributes, data)
 
@@ -103,7 +105,7 @@ def create(
 
  def __init__( # type: ignore[no-untyped-def]
  self,
- attributes: typing.Optional[typing.Dict[str, typing.Any]] = None,
+ attributes: typing.Optional[typing.Mapping[str, typing.Any]] = None,
  data: typing.Optional[typing.Any] = None,
  **kwargs,
  ):
@@ -173,6 +175,8 @@ def model_validate_json(
  *,
  strict: typing.Optional[bool] = None,
  context: typing.Optional[typing.Dict[str, Any]] = None,
+ by_alias: typing.Optional[bool] = None,
+ by_name: typing.Optional[bool] = None,
  ) -> "CloudEvent":
  return conversion.from_json(cls, json_data)