PYTHON-1819 Documentation & examples for custom type encoding/decoding

functionality

Author: prashantmital

bson/codec_options.py

@@ -41,6 +41,8 @@ class TypeEncoder(ABC):
 
  Codec classes must implement the ``python_type`` attribute, and the
  ``transform_python`` method to support encoding.
+
+ See :ref:`custom-type-type-codec` documentation for an example.
  """
  @abstractproperty
  def python_type(self):
@@ -59,6 +61,8 @@ class TypeDecoder(ABC):
 
  Codec classes must implement the ``bson_type`` attribute, and the
  ``transform_bson`` method to support decoding.
+
+ See :ref:`custom-type-type-codec` documentation for an example.
  """
  @abstractproperty
  def bson_type(self):
@@ -73,13 +77,15 @@ def transform_bson(self, value):
 
 class TypeCodec(TypeEncoder, TypeDecoder):
  """Base class for defining type codec classes which describe how a
- custom type can be transformed to/from one of the types BSON already
- understands, and can encode/decode.
+ custom type can be transformed to/from one of the types :mod:`bson`
+ can already encode/decode.
 
  Codec classes must implement the ``python_type`` attribute, and the
  ``transform_python`` method to support encoding, as well as the
  ``bson_type`` attribute, and the ``transform_bson`` method to support
  decoding.
+
+ See :ref:`custom-type-type-codec` documentation for an example.
  """
  pass
 
@@ -96,14 +102,19 @@ class TypeRegistry(object):
  >>> type_registry = TypeRegistry([Codec1, Codec2, Codec3, ...],
  ... fallback_encoder)
 
+ See :ref:`custom-type-type-registry` documentation for an example.
+
  :Parameters:
  - `type_codecs` (optional): iterable of type codec instances. If
  ``type_codecs`` contains multiple codecs that transform a single
  python or BSON type, the transformation specified by the type codec
- occurring last prevails.
+ occurring last prevails. A TypeError will be raised if one or more
+ type codecs modify the encoding behavior of a built-in :mod:`bson`
+ type.
  - `fallback_encoder` (optional): callable that accepts a single,
- unencodable python value and transforms it into a type that BSON can
- encode.
+ unencodable python value and transforms it into a type that
+ :mod:`bson` can encode. See :ref:`fallback-encoder-callable`
+ documentation for an example.
  """
  def __init__(self, type_codecs=None, fallback_encoder=None):
  self.__type_codecs = list(type_codecs or [])
@@ -217,6 +228,9 @@ class CodecOptions(_options_base):
  - `type_registry`: Instance of :class:`TypeRegistry` used to customize
  encoding and decoding behavior.
 
+ .. versionadded:: 3.8
+ `type_registry` attribute.
+
  .. warning:: Care must be taken when changing
  `unicode_decode_error_handler` from its default value ('strict').
  The 'replace' and 'ignore' modes should not be used when documents

doc/examples/custom_type.rst

@@ -7,11 +7,11 @@ codec, which is used to populate a :class:`~bson.codec_options.TypeRegistry`.
 The type registry can then be used to create a custom-type-aware
 :class:`~pymongo.collection.Collection`. Read and write operations
 issued against the resulting collection object transparently manipulate
-documents as they are saved or retrieved from MongoDB.
+documents as they are saved to or retrieved from MongoDB.
 
 
-Setup
------
+Setting Up
+----------
 
 We'll start by getting a clean database to use for the example:
 
@@ -26,10 +26,10 @@ We'll start by getting a clean database to use for the example:
 Since the purpose of the example is to demonstrate working with custom types,
 we'll need a custom data type to use. For this example, we will be working with
 the :py:class:`~decimal.Decimal` type from Python's standard library. Since the
-BSON library has a :class:`~bson.decimal128.Decimal128` type (that implements
-the IEEE 754 decimal128 decimal-based floating-point numbering format) which
-is distinct from Python's built-in :py:class:`~decimal.Decimal` type, when we
-try to save an instance of ``Decimal`` with PyMongo, we get an
+BSON library's :class:`~bson.decimal128.Decimal128` type (that implements
+the IEEE 754 decimal128 decimal-based floating-point numbering format) is
+distinct from Python's built-in :py:class:`~decimal.Decimal` type, attempting
+to save an instance of ``Decimal`` with PyMongo, results in an
 :exc:`~bson.errors.InvalidDocument` exception.
 
 .. doctest::
@@ -44,13 +44,13 @@ try to save an instance of ``Decimal`` with PyMongo, we get an
 
 .. _custom-type-type-codec:
 
-The Type Codec
---------------
+The :class:`~bson.codec_options.TypeCodec` Class
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. versionadded:: 3.8
 
-In order to encode custom types, we must first define a **type codec** for our
-type. A type codec describes how an instance of a custom type can be
+In order to encode a custom type, we must first define a **type codec** for
+that type. A type codec describes how an instance of a custom type can be
 *transformed* to and/or from one of the types :mod:`~bson` already understands.
 Depending on the desired functionality, users must choose from the following
 base classes when defining type codecs:
@@ -62,7 +62,7 @@ base classes when defining type codecs:
  decodes a specified BSON type into a custom Python type. Users must implement
  the ``bson_type`` property/attribute and the ``transform_bson`` method.
 * :class:`~bson.codec_options.TypeCodec`: subclass this to define a codec that
- can both encode from and decode to a custom type. Users must implement the
+ can both encode and decode a custom type. Users must implement the
  ``python_type`` and ``bson_type`` properties/attributes, as well as the
  ``transform_python`` and ``transform_bson`` methods.
 
@@ -93,14 +93,14 @@ interested in both encoding and decoding our custom type, we use the
 
 .. _custom-type-type-registry:
 
-The Type Registry
------------------
+The :class:`~bson.codec_options.TypeRegistry` Class
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. versionadded:: 3.8
 
 Before we can begin encoding and decoding our custom type objects, we must
-first inform PyMongo about our type codec. This is done by creating a
-:class:`~bson.codec_options.TypeRegistry` instance:
+first inform PyMongo about the corresponding codec. This is done by creating
+a :class:`~bson.codec_options.TypeRegistry` instance:
 
 .. doctest::
 
@@ -113,7 +113,7 @@ Once instantiated, registries are immutable and the only way to add codecs
 to a registry is to create a new one.
 
 
-Putting it together
+Putting It Together
 -------------------
 
 Finally, we can define a :class:`~bson.codec_options.CodecOptions` instance
@@ -201,35 +201,79 @@ This is trivial to do since the same transformation as the one used for
  information, it is impossible to discern which incoming
  :class:`~bson.decimal128.Decimal128` value needs to be decoded as ``Decimal``
  and which needs to be decoded as ``DecimalInt``. This example only considers
- the situation where a user wants to *encode* documents containing one or both
+ the situation where a user wants to *encode* documents containing either
  of these types.
 
-Now, we can create a new codec options object and use it to get a collection
-object:
+After creating a new codec options object and using it to get a collection
+object, we can seamlessly encode instances of ``DecimalInt``:
 
 .. doctest::
 
  >>> type_registry = TypeRegistry([decimal_codec, decimalint_codec])
  >>> codec_options = CodecOptions(type_registry=type_registry)
  >>> collection = db.get_collection('test', codec_options=codec_options)
  >>> collection.drop()
-
-
-We can now seamlessly encode instances of ``DecimalInt``. Note that the
-``transform_bson`` method of the base codec class results in these values
-being decoded as ``Decimal`` (and not ``DecimalInt``):
-
-.. doctest::
-
  >>> collection.insert_one({'num': DecimalInt("45.321")})
  <pymongo.results.InsertOneResult object at ...>
  >>> mydoc = collection.find_one()
  >>> pprint.pprint(mydoc)
  {u'_id': ObjectId('...'), u'num': Decimal('45.321')}
 
+Note that the ``transform_bson`` method of the base codec class results in
+these values being decoded as ``Decimal`` (and not ``DecimalInt``).
+
+
+.. _decoding-binary-types:
+
+Decoding :class:`~bson.binary.Binary` Types
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The decoding treatment of :class:`~bson.binary.Binary` types having
+``subtype = 0`` by the :mod:`bson` module varies slightly depending on the
+version of the Python runtime in use. This must be taken into account while
+writing a ``TypeDecoder`` that modifies how this datatype is decoded.
+
+On Python 3.x, :class:`~bson.binary.Binary` data (``subtype = 0``) is decoded
+as a ``bytes`` instance:
+
+.. code-block:: python
+
+ >>> # On Python 3.x.
+ >>> from bson.binary import Binary
+ >>> newcoll = db.get_collection('new')
+ >>> newcoll.insert_one({'_id': 1, 'data': Binary(b"123", subtype=0)})
+ >>> doc = newcoll.find_one()
+ >>> type(doc['data'])
+ bytes
+
+
+On Python 2.7.x, the same data is decoded as a :class:`~bson.binary.Binary`
+instance:
+
+.. code-block:: python
+
+ >>> # On Python 2.7.x
+ >>> newcoll = db.get_collection('new')
+ >>> doc = newcoll.find_one()
+ >>> type(doc['data'])
+ bson.binary.Binary
 
-The Fallback Encoder
---------------------
+
+As a consequence of this disparity, users must set the ``bson_type`` attribute
+on their :class:`~bson.codec_options.TypeDecoder` classes differently,
+depending on the python version in use.
+
+
+.. note::
+
+ For codebases requiring compatibility with both Python 2 and 3, type
+ decoders will have to be registered for both possible ``bson_type`` values.
+
+
+.. _fallback-encoder-callable:
+
+The ``fallback_encoder`` Callable
+---------------------------------
 
 .. versionadded:: 3.8
 
@@ -268,27 +312,110 @@ We can now seamlessly encode instances of :py:class:`~decimal.Decimal`:
  >>> pprint.pprint(mydoc)
  {u'_id': ObjectId('...'), u'num': Decimal128('45.321')}
 
-As you can tell, fallback encoders are a compelling alternative to type codecs
-when we only want to encode custom types due to their much simpler API.
-Users should note however, that fallback encoders cannot be used to modify the
-encoding of types that PyMongo already understands, as illustrated by the
-following example:
 
- >>> def fallback_encoder(value):
- ... """Encoder that converts floats to int."""
- ... if isinstance(value, float):
- ... return int(value)
- ... return value
- >>> type_registry = TypeRegistry(fallback_encoder=fallback_encoder)
- >>> codec_options = CodecOptions(type_registry=type_registry)
- >>> collection = db.get_collection('test', codec_options=codec_options)
- >>> collection.drop()
- >>> collection.insert_one({'num': 45.321})
- <pymongo.results.InsertOneResult object at ...>
- >>> mydoc = collection.find_one()
- >>> pprint.pprint(mydoc)
- {u'_id': ObjectId('...'), u'num': 45.321}
+.. note::
+
+ Fallback encoders are invoked *after* attempts to encode the given value
+ with standard BSON encoders and any configured type encoders have failed.
+ Therefore, in a type registry configured with a type encoder and fallback
+ encoder that both target the same custom type, the behavior specified in
+ the type encoder will prevail.
+
+
+Because fallback encoders don't need to declare the types that they encode
+beforehand, they can be used to support interesting use-cases that cannot be
+serviced by ``TypeEncoder``. One such use-case is described in the next
+section.
+
+
+Encoding Unknown Types
+^^^^^^^^^^^^^^^^^^^^^^
+
+In this example, we demonstrate how a fallback encoder can be used to save
+arbitrary objects to the database. We will use the the standard library's
+:py:mod:`pickle` module to serialize the unknown types and so naturally, this
+approach only works for types that are picklable.
+
+We start by defining some arbitrary custom types:
+
+.. code-block:: python
+
+ class MyStringType(object):
+ def __init__(self, value):
+ self.__value = value
+ def __repr__(self):
+ return "MyStringType('%s')" % (self.__value,)
+
+ class MyNumberType(object):
+ def __init__(self, value):
+ self.__value = value
+ def __repr__(self):
+ return "MyNumberType(%s)" % (self.__value,)
+
+We also define a fallback encoder that pickles whatever objects it receives
+and returns them as :class:`~bson.binary.Binary` instances with a custom
+subtype. The custom subtype, in turn, allows us to write a TypeDecoder that
+identifies pickled artifacts upon retrieval and transparently decodes them
+back into Python objects:
+
+.. code-block:: python
+
+ import pickle
+ from bson.binary import Binary, USER_DEFINED_SUBTYPE
+ def fallback_pickle_encoder(value):
+ return Binary(pickle.dumps(value), USER_DEFINED_SUBTYPE)
+
+ class PickledBinaryDecoder(TypeDecoder):
+ bson_type = Binary
+ def transform_bson(self, value):
+ if value.subtype == USER_DEFINED_SUBTYPE:
+ return pickle.loads(value)
+ return value
+
+
+.. note::
+
+ The above example is written assuming the use of Python 3. If you are using
+ Python 2, ``bson_type`` must be set to ``Binary``. See the
+ :ref:`decoding-binary-types` section for a detailed explanation.
+
+
+Finally, we create a ``CodecOptions`` instance:
+
+.. code-block:: python
+
+ codec_options = CodecOptions(type_registry=TypeRegistry(
+ [PickledBinaryDecoder()], fallback_encoder=fallback_pickle_encoder))
+
+We can now round trip our custom objects to MongoDB:
+
+.. code-block:: python
+
+ collection = db.get_collection('test_fe', codec_options=codec_options)
+ collection.insert_one({'_id': 1, 'str': MyStringType("hello world"),
+ 'num': MyNumberType(2)})
+ mydoc = collection.find_one()
+ assert isinstance(mydoc['str'], MyStringType)
+ assert isinstance(mydoc['num'], MyNumberType)
+
+
+Limitations
+-----------
+
+PyMongo's type codec and fallback encoder features have the following
+limitations:
 
-This is due to the fact that fallback encoders are invoked only after
-an attempt to encode the value with type codecs and standard BSON encoding
-routines has been unsuccessful.
+#. Users cannot customize the encoding behavior of Python types that PyMongo
+ already understands like ``int`` and ``str`` (the 'built-in types').
+ Attempting to instantiate a type registry with one or more codecs that act
+ upon a built-in type results in a ``TypeError``. This limitation extends
+ to all subtypes of the standard types.
+#. Chaining type encoders is not supported. A custom type value, once
+ transformed by a codec's ``transform_python`` method, *must* result in a
+ type that is either BSON-encodable by default, or can be
+ transformed by the fallback encoder into something BSON-encodable--it
+ *cannot* be transformed a second time by a different type codec.
+#. The :meth:`~pymongo.database.Database.command` method does not apply the
+ user's TypeDecoders while decoding the command response document.
+#. :mod:`gridfs` does not apply custom type encoding or decoding to any
+ documents received from or to returned to the user.

pymongo/database.py

@@ -616,6 +616,9 @@ def command(self, command, value=1, check=True,
  :attr:`read_preference` or :attr:`codec_options`. You must use the
  `read_preference` and `codec_options` parameters instead.
 
+ .. note:: :meth:`command` does **not** apply any custom TypeDecoders
+ when decoding the command response.
+
  .. versionchanged:: 3.6
  Added ``session`` parameter.