Add #[pyclass(generic)] (#4926)

* First attempt at supporting pyclass(generic) * Include class_geitem in macro * Minor formatting * Pass py argument to new() * Add tests covering pyclass(generic) * Add changelog entry * Add test covering __getitem__ * Fix compilation error tests * Fix Alias -> TypingAlias * Make test case slightly stronger * Add compilation error test * Do not allow pyclass(generic) for enums * Remove unused import * Add import only to 3.9+ * Add documentation * Ignore Python typing hints newly added Rust code * Fix clippy warnings for the test * Add newline * Update TRYBUILD tests * Move `generic` in table * TypingAlias is only available in 3.10+ * Update trybuild test

Author: IvanIsCoding

guide/pyclass-parameters.md

@@ -10,6 +10,7 @@
 | <span style="white-space: pre">`extends = BaseType`</span> | Use a custom baseclass. Defaults to [`PyAny`][params-1] |
 | <span style="white-space: pre">`freelist = N`</span> | Implements a [free list][params-2] of size N. This can improve performance for types that are often created and deleted in quick succession. Profile your code to see whether `freelist` is right for you. |
 | <span style="white-space: pre">`frozen`</span> | Declares that your pyclass is immutable. It removes the borrow checker overhead when retrieving a shared reference to the Rust struct, but disables the ability to get a mutable reference. |
+| `generic` | Implements runtime parametrization for the class following [PEP 560](https://peps.python.org/pep-0560/). |
 | `get_all` | Generates getters for all fields of the pyclass. |
 | `hash` | Implements `__hash__` using the `Hash` implementation of the underlying Rust datatype. |
 | `immutable_type` | Makes the type object immutable. Supported on 3.14+ with the `abi3` feature active, or 3.10+ otherwise. |

guide/src/python-typing-hints.md

@@ -41,7 +41,7 @@ As we can see, those are not full definitions containing implementation, but jus
 
 ### What do the PEPs say?
 
-At the time of writing this documentation, the `pyi` files are referenced in three PEPs.
+At the time of writing this documentation, the `pyi` files are referenced in four PEPs.
 
 [PEP8 - Style Guide for Python Code - #Function Annotations](https://www.python.org/dev/peps/pep-0008/#function-annotations) (last point) recommends all third party library creators to provide stub files as the source of knowledge about the package for type checker tools.
 
@@ -55,6 +55,8 @@ It contains a specification for them (highly recommended reading, since it conta
 
 [PEP561 - Distributing and Packaging Type Information](https://www.python.org/dev/peps/pep-0561/) describes in detail how to build packages that will enable type checking. In particular it contains information about how the stub files must be distributed in order for type checkers to use them.
 
+[PEP560 - Core support for typing module and generic types](https://www.python.org/dev/peps/pep-0560/) describes the details on how Python's type system internally supports generics, including both runtime behavior and integration with static type checkers.
+
 ## How to do it?
 
 [PEP561](https://www.python.org/dev/peps/pep-0561/) recognizes three ways of distributing type information:
@@ -165,3 +167,77 @@ class Car:
  :return: the name of the color our great algorithm thinks is the best for this car
  """
 ```
+
+### Supporting Generics
+
+Type annotations can also be made generic in Python. They are useful for working
+with different types while maintaining type safety. Usually, generic classes
+inherit from the `typing.Generic` metaclass.
+
+Take for example the following `.pyi` file that specifies a `Car` that can
+accept multiple types of wheels:
+
+```python
+from typing import Generic, TypeVar
+
+W = TypeVar('W')
+
+class Car(Generic[W]):
+ def __init__(self, wheels: list[W]) -> None: ...
+
+ def get_wheels(self) -> list[W]: ...
+
+ def change_wheel(self, wheel_number: int, wheel: W) -> None: ...
+```
+
+This way, the end-user can specify the type with variables such as `truck: Car[SteelWheel] = ...`
+and `f1_car: Car[AlloyWheel] = ...`.
+
+There is also a special syntax for specifying generic types in Python 3.12+:
+
+```python
+class Car[W]:
+ def __init__(self, wheels: list[W]) -> None: ...
+
+ def get_wheels(self) -> list[W]: ...
+```
+
+#### Runtime Behaviour
+
+Stub files (`pyi`) are only useful for static type checkers and ignored at runtime. Therefore,
+PyO3 classes do not inherit from `typing.Generic` even if specified in the stub files.
+
+This can cause some runtime issues, as annotating a variable like `f1_car: Car[AlloyWheel] = ...`
+can make Python call magic methods that are not defined. 
+
+To overcome this limitation, implementers can pass the `generic` parameter to `pyclass` in Rust:
+
+```rust ignore
+#[pyclass(generic)]
+```
+
+#### Advanced Users
+
+`#[pyclass(generic)]` implements a very simple runtime behavior that accepts
+any generic argument. Advanced users can opt to manually implement
+[`__class_geitem__`](https://docs.python.org/3/reference/datamodel.html#emulating-generic-types)
+for the generic class to have more control.
+
+```rust ignore
+impl MyClass {
+ #[classmethod]
+ #[pyo3(signature = (key, /))]
+ pub fn __class_getitem__(
+ cls: &Bound<'_, PyType>,
+ key: &Bound<'_, PyAny>,
+ ) -> PyResult<PyObject> {
+ /* implementation details */
+ }
+}
+```
+
+Note that [`pyo3::types::PyGenericAlias`][pygenericalias] can be helfpul when implementing
+`__class_geitem__` as it can create [`types.GenericAlias`][genericalias] objects from Rust.
+
+[pygenericalias]: {{#PYO3_DOCS_URL}}/pyo3/types/struct.pygenericalias
+[genericalias]: https://docs.python.org/3/library/types.html#types.GenericAlias

newsfragments/4926.added.md

@@ -0,0 +1,4 @@
+Introduced a new optional parameter with `#[pyclass(generic)]`. This new
+parameter makes classes support generic typing at the runtime following
+[PEP 560](https://peps.python.org/pep-0560/). It is an alternative
+to inheriting from [typing.Generic](https://docs.python.org/3/library/typing.html#typing.Generic).

pyo3-macros-backend/src/attributes.rs

@@ -46,6 +46,7 @@ pub mod kw {
  syn::custom_keyword!(transparent);
  syn::custom_keyword!(unsendable);
  syn::custom_keyword!(weakref);
+ syn::custom_keyword!(generic);
  syn::custom_keyword!(gil_used);
 }
 

pyo3-macros-backend/src/pyclass.rs

@@ -83,6 +83,7 @@ pub struct PyClassPyO3Options {
  pub subclass: Option<kw::subclass>,
  pub unsendable: Option<kw::unsendable>,
  pub weakref: Option<kw::weakref>,
+ pub generic: Option<kw::generic>,
 }
 
 pub enum PyClassPyO3Option {
@@ -107,6 +108,7 @@ pub enum PyClassPyO3Option {
  Subclass(kw::subclass),
  Unsendable(kw::unsendable),
  Weakref(kw::weakref),
+ Generic(kw::generic),
 }
 
 impl Parse for PyClassPyO3Option {
@@ -154,6 +156,8 @@ impl Parse for PyClassPyO3Option {
  input.parse().map(PyClassPyO3Option::Unsendable)
  } else if lookahead.peek(attributes::kw::weakref) {
  input.parse().map(PyClassPyO3Option::Weakref)
+ } else if lookahead.peek(attributes::kw::generic) {
+ input.parse().map(PyClassPyO3Option::Generic)
  } else {
  Err(lookahead.error())
  }
@@ -232,6 +236,7 @@ impl PyClassPyO3Options {
  );
  set_option!(weakref);
  }
+ PyClassPyO3Option::Generic(generic) => set_option!(generic),
  }
  Ok(())
  }
@@ -429,6 +434,21 @@ fn impl_class(
  }
  }
 
+ let mut default_methods = descriptors_to_items(
+ cls,
+ args.options.rename_all.as_ref(),
+ args.options.frozen,
+ field_options,
+ ctx,
+ )?;
+
+ let (default_class_geitem, default_class_geitem_method) =
+ pyclass_class_geitem(&args.options, &syn::parse_quote!(#cls), ctx)?;
+
+ if let Some(default_class_geitem_method) = default_class_geitem_method {
+ default_methods.push(default_class_geitem_method);
+ }
+
  let (default_str, default_str_slot) =
  implement_pyclass_str(&args.options, &syn::parse_quote!(#cls), ctx);
 
@@ -443,21 +463,9 @@ fn impl_class(
  slots.extend(default_hash_slot);
  slots.extend(default_str_slot);
 
- let py_class_impl = PyClassImplsBuilder::new(
- cls,
- args,
- methods_type,
- descriptors_to_items(
- cls,
- args.options.rename_all.as_ref(),
- args.options.frozen,
- field_options,
- ctx,
- )?,
- slots,
- )
- .doc(doc)
- .impl_all(ctx)?;
+ let py_class_impl = PyClassImplsBuilder::new(cls, args, methods_type, default_methods, slots)
+ .doc(doc)
+ .impl_all(ctx)?;
 
  Ok(quote! {
  impl #pyo3_path::types::DerefToPyAny for #cls {}
@@ -472,6 +480,7 @@ fn impl_class(
  #default_richcmp
  #default_hash
  #default_str
+ #default_class_geitem
  }
  })
 }
@@ -514,6 +523,10 @@ pub fn build_py_enum(
  bail_spanned!(enum_.brace_token.span.join() => "#[pyclass] can't be used on enums without any variants");
  }
 
+ if let Some(generic) = &args.options.generic {
+ bail_spanned!(generic.span() => "enums do not support #[pyclass(generic)]");
+ }
+
  let doc = utils::get_doc(&enum_.attrs, None, ctx);
  let enum_ = PyClassEnum::new(enum_)?;
  impl_enum(enum_, &args, doc, method_type, ctx)
@@ -2008,6 +2021,46 @@ fn pyclass_hash(
  }
 }
 
+fn pyclass_class_geitem(
+ options: &PyClassPyO3Options,
+ cls: &syn::Type,
+ ctx: &Ctx,
+) -> Result<(Option<syn::ImplItemFn>, Option<MethodAndMethodDef>)> {
+ let Ctx { pyo3_path, .. } = ctx;
+ match options.generic {
+ Some(_) => {
+ let ident = format_ident!("__class_getitem__");
+ let mut class_geitem_impl: syn::ImplItemFn = {
+ parse_quote! {
+ #[classmethod]
+ fn #ident<'py>(
+ cls: &#pyo3_path::Bound<'py, #pyo3_path::types::PyType>,
+ key: &#pyo3_path::Bound<'py, #pyo3_path::types::PyAny>
+ ) -> #pyo3_path::PyResult<#pyo3_path::Bound<'py, #pyo3_path::types::PyGenericAlias>> {
+ #pyo3_path::types::PyGenericAlias::new(cls.py(), cls.as_any(), key)
+ }
+ }
+ };
+
+ let spec = FnSpec::parse(
+ &mut class_geitem_impl.sig,
+ &mut class_geitem_impl.attrs,
+ Default::default(),
+ )?;
+
+ let class_geitem_method = crate::pymethod::impl_py_method_def(
+ cls,
+ &spec,
+ &spec.get_doc(&class_geitem_impl.attrs, ctx),
+ Some(quote!(#pyo3_path::ffi::METH_CLASS)),
+ ctx,
+ )?;
+ Ok((Some(class_geitem_impl), Some(class_geitem_method)))
+ }
+ None => Ok((None, None)),
+ }
+}
+
 /// Implements most traits used by `#[pyclass]`.
 ///
 /// Specifically, it implements traits that only depend on class name,

tests/test_class_basics.rs

@@ -714,3 +714,34 @@ fn test_unsendable_dict_with_weakref() {
  );
  });
 }
+
+#[cfg(Py_3_9)]
+#[pyclass(generic)]
+struct ClassWithRuntimeParametrization {
+ #[pyo3(get, set)]
+ value: PyObject,
+}
+
+#[cfg(Py_3_9)]
+#[pymethods]
+impl ClassWithRuntimeParametrization {
+ #[new]
+ fn new(value: PyObject) -> ClassWithRuntimeParametrization {
+ Self { value }
+ }
+}
+
+#[test]
+#[cfg(Py_3_9)]
+fn test_runtime_parametrization() {
+ Python::with_gil(|py| {
+ let ty = py.get_type::<ClassWithRuntimeParametrization>();
+ py_assert!(py, ty, "ty[int] == ty.__class_getitem__((int,))");
+ py_run!(
+ py,
+ ty,
+ "import types;
+ assert ty.__class_getitem__((int,)) == types.GenericAlias(ty, (int,))"
+ );
+ });
+}

tests/test_compile_error.rs

@@ -10,6 +10,10 @@ fn test_compile_errors() {
  t.compile_fail("tests/ui/invalid_pyclass_args.rs");
  t.compile_fail("tests/ui/invalid_pyclass_enum.rs");
  t.compile_fail("tests/ui/invalid_pyclass_item.rs");
+ #[cfg(Py_3_9)]
+ t.compile_fail("tests/ui/invalid_pyclass_generic.rs");
+ #[cfg(Py_3_9)]
+ t.compile_fail("tests/ui/pyclass_generic_enum.rs");
  t.compile_fail("tests/ui/invalid_pyfunction_signatures.rs");
  t.compile_fail("tests/ui/invalid_pyfunction_definition.rs");
  #[cfg(any(not(Py_LIMITED_API), Py_3_11))]