【Hackathon 6th No.2】【Typing】为 Paddle 新增 cholesky_inverse API -part #64519
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| 你的PR提交成功，感谢你对开源项目的贡献! |
megemini commented May 22, 2024
Comment on lines +4418 to +4422
| ( | ||
| Q[:, i:] - (Q[:, i:] @ w @ w.T * tau[i]) | ||
| if x.dtype in [paddle.complex128, paddle.complex64] | ||
| else Q[:, i:] - (Q[:, i:] @ w @ w.T * tau[i]) | ||
| ), |
Contributor Author
There was a problem hiding this comment.
这里是 pre-commit 自动修改的格式 ～
Charles-hit reviewed May 24, 2024
| from paddle.base import core | ||
| from paddle.pir_utils import test_with_pir_api as _test_with_pir_api | ||
| | ||
| RTOL = {'float32': 1e-7, 'float64': 1e-11} |
Contributor Author
There was a problem hiding this comment.
我本地测试 linux 能到 1e-13 ，window 下面精度到 1e-11 ，所以这里写的 1e-11 ～
这个函数精度貌似达不到这么高，对比 torch :
In [34]: import torch ...: import numpy as np ...: x = np.array([[3.0, 0.0, 0.0], [2.0, 3.0, 0.0], [-1.0, 1.0, -2.0]]) ...: A = torch.tensor(x) ...: A = A @ A.T + torch.eye(3)*1e-3 ...: L = torch.linalg.cholesky(A) ...: out = torch.cholesky_inverse(L) ...: An = x @ x.T + np.eye(3)*1e-3 ...: on = np.linalg.inv(An) ...: np.testing.assert_allclose(out.numpy(), on, rtol=1e-15) --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) Cell In[34], line 10 8 An = x @ x.T + np.eye(3)*1e-3 9 on = np.linalg.inv(An) ---> 10 np.testing.assert_allclose(out.numpy(), on, rtol=1e-15) [... skipping hidden 1 frame] File /usr/lib/python3.8/contextlib.py:75, in ContextDecorator.__call__.<locals>.inner(*args, **kwds) 72 @wraps(func) 73 def inner(*args, **kwds): 74 with self._recreate_cm(): ---> 75 return func(*args, **kwds) File ~/venv38dev/lib/python3.8/site-packages/numpy/testing/_private/utils.py:862, in assert_array_compare(comparison, x, y, err_msg, verbose, header, precision, equal_nan, equal_inf, strict) 858 err_msg += '\n' + '\n'.join(remarks) 859 msg = build_err_msg([ox, oy], err_msg, 860 verbose=verbose, header=header, 861 names=('x', 'y'), precision=precision) --> 862 raise AssertionError(msg) 863 except ValueError: 864 import traceback AssertionError: Not equal to tolerance rtol=1e-15, atol=0 Mismatched elements: 9 / 9 (100%) Max absolute difference: 4.28335145e-12 Max relative difference: 2.79894493e-11 x: array([[ 0.237564, -0.120313, 0.138811], [-0.120313, 0.138848, -0.083284], [ 0.138811, -0.083284, 0.249911]]) y: array([[ 0.237564, -0.120313, 0.138811], [-0.120313, 0.138848, -0.083284], [ 0.138811, -0.083284, 0.249911]]) In [35]: out Out[35]: tensor([[ 0.2376, -0.1203, 0.1388], [-0.1203, 0.1388, -0.0833], [ 0.1388, -0.0833, 0.2499]], dtype=torch.float64) In [36]: on.dtype Out[36]: dtype('float64')差不多也是 1e-11 左右 ～
Contributor
There was a problem hiding this comment.
了解，在问最后一个问题，torch跟numpy的对比也跟我们比较接近嘛
Contributor Author
There was a problem hiding this comment.
测试用例里面是比对 paddle 与 numpy 的结果 ～
这里单独比对一下 paddle，torch 和 numpy 的结果：
In [5]: import paddle In [6]: import torch In [7]: import numpy as np In [8]: x = np.array([[3.0, 0.0, 0.0], [2.0, 3.0, 0.0], [-1.0, 1.0, -2.0]]) In [9]: A = torch.tensor(x) In [10]: A = A @ A.T + torch.eye(3)*1e-3 In [11]: L = torch.linalg.cholesky(A) In [12]: out = torch.cholesky_inverse(L) In [13]: A_paddle = paddle.to_tensor(x) In [14]: A_paddle = A_paddle @ A_paddle.T + paddle.eye(3)*1e-3 W0528 12:16:00.375560 12983 dygraph_functions.cc:71336] got different data type, run type promotion automatically, this may cause data type been changed. In [15]: L_paddle = paddle.linalg.cholesky(A_paddle) In [16]: out_paddle = paddle.cholesky_inverse(L_paddle) In [17]: An = x @ x.T + np.eye(3)*1e-3 In [18]: on = np.linalg.inv(An) In [19]: np.testing.assert_allclose(out.numpy(), on, rtol=1e-15) --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) Cell In[19], line 1 ----> 1 np.testing.assert_allclose(out.numpy(), on, rtol=1e-15) [... skipping hidden 1 frame] File /usr/lib/python3.8/contextlib.py:75, in ContextDecorator.__call__.<locals>.inner(*args, **kwds) 72 @wraps(func) 73 def inner(*args, **kwds): 74 with self._recreate_cm(): ---> 75 return func(*args, **kwds) File ~/venv38dev/lib/python3.8/site-packages/numpy/testing/_private/utils.py:862, in assert_array_compare(comparison, x, y, err_msg, verbose, header, precision, equal_nan, equal_inf, strict) 858 err_msg += '\n' + '\n'.join(remarks) 859 msg = build_err_msg([ox, oy], err_msg, 860 verbose=verbose, header=header, 861 names=('x', 'y'), precision=precision) --> 862 raise AssertionError(msg) 863 except ValueError: 864 import traceback AssertionError: Not equal to tolerance rtol=1e-15, atol=0 Mismatched elements: 9 / 9 (100%) Max absolute difference: 4.28335145e-12 Max relative difference: 2.79894493e-11 x: array([[ 0.237564, -0.120313, 0.138811], [-0.120313, 0.138848, -0.083284], [ 0.138811, -0.083284, 0.249911]]) y: array([[ 0.237564, -0.120313, 0.138811], [-0.120313, 0.138848, -0.083284], [ 0.138811, -0.083284, 0.249911]]) In [20]: np.testing.assert_allclose(out_paddle.numpy(), on, rtol=1e-15) --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) Cell In[20], line 1 ----> 1 np.testing.assert_allclose(out_paddle.numpy(), on, rtol=1e-15) [... skipping hidden 1 frame] File /usr/lib/python3.8/contextlib.py:75, in ContextDecorator.__call__.<locals>.inner(*args, **kwds) 72 @wraps(func) 73 def inner(*args, **kwds): 74 with self._recreate_cm(): ---> 75 return func(*args, **kwds) File ~/venv38dev/lib/python3.8/site-packages/numpy/testing/_private/utils.py:862, in assert_array_compare(comparison, x, y, err_msg, verbose, header, precision, equal_nan, equal_inf, strict) 858 err_msg += '\n' + '\n'.join(remarks) 859 msg = build_err_msg([ox, oy], err_msg, 860 verbose=verbose, header=header, 861 names=('x', 'y'), precision=precision) --> 862 raise AssertionError(msg) 863 except ValueError: 864 import traceback AssertionError: Not equal to tolerance rtol=1e-15, atol=0 Mismatched elements: 9 / 9 (100%) Max absolute difference: 4.28335145e-12 Max relative difference: 2.79894493e-11 x: array([[ 0.237564, -0.120313, 0.138811], [-0.120313, 0.138848, -0.083284], [ 0.138811, -0.083284, 0.249911]]) y: array([[ 0.237564, -0.120313, 0.138811], [-0.120313, 0.138848, -0.083284], [ 0.138811, -0.083284, 0.249911]]) In [21]: np.testing.assert_allclose(out_paddle.numpy(), out.numpy(), rtol=1e-15) In [22]: np.testing.assert_allclose(out_paddle.numpy(), out.numpy(), rtol=1e-19) --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) Cell In[22], line 1 ----> 1 np.testing.assert_allclose(out_paddle.numpy(), out.numpy(), rtol=1e-19) [... skipping hidden 1 frame] File /usr/lib/python3.8/contextlib.py:75, in ContextDecorator.__call__.<locals>.inner(*args, **kwds) 72 @wraps(func) 73 def inner(*args, **kwds): 74 with self._recreate_cm(): ---> 75 return func(*args, **kwds) File ~/venv38dev/lib/python3.8/site-packages/numpy/testing/_private/utils.py:862, in assert_array_compare(comparison, x, y, err_msg, verbose, header, precision, equal_nan, equal_inf, strict) 858 err_msg += '\n' + '\n'.join(remarks) 859 msg = build_err_msg([ox, oy], err_msg, 860 verbose=verbose, header=header, 861 names=('x', 'y'), precision=precision) --> 862 raise AssertionError(msg) 863 except ValueError: 864 import traceback AssertionError: Not equal to tolerance rtol=1e-19, atol=0 Mismatched elements: 2 / 9 (22.2%) Max absolute difference: 1.38777878e-17 Max relative difference: 1.15346891e-16 x: array([[ 0.237564, -0.120313, 0.138811], [-0.120313, 0.138848, -0.083284], [ 0.138811, -0.083284, 0.249911]]) y: array([[ 0.237564, -0.120313, 0.138811], [-0.120313, 0.138848, -0.083284], [ 0.138811, -0.083284, 0.249911]])可以看到，torch 与 numpy ，paddle 与 numpy ，两者 rtol 结果大约在 1e-11 ～
torch 与 paddle 两者 rtol 结果大约在 1e-16 ～
Charles-hit previously approved these changes May 28, 2024
jeff41404 reviewed May 29, 2024
python/paddle/__init__.py Outdated
| 'diagonal_scatter', | ||
| 'combinations', | ||
| 'signbit', | ||
| 'cholesky_inverse', |
Contributor
There was a problem hiding this comment.
Which path we recommend users to use? paddle.cholesky_inverse or paddle.linalg.cholesky_inverse? If it is the latter(like other API of linear algebra, @Charles-hit confirms?), do not add it here, just add it in __all__ list of python/paddle/linalg.py. If it is the former, keep it here and delete it in __all__ list of python/paddle/linalg.py
Contributor Author
There was a problem hiding this comment.
嗯，torch 也有类似的问题，linalg 有一些暴露到上层，有些没有暴露出去，不太清楚是怎么考虑的 ～
比如：
torch.cholesky() is deprecated in favor of torch.linalg.cholesky() and will be removed in a future PyTorch release.
但是 torch 的 torch.cholesky_inverse 没在 linalg 里面，不知道后面会不会也移到 linalg ～
@Charles-hit 请帮忙看看怎么处理？
Contributor
There was a problem hiding this comment.
@megemini 您好 内部沟通了一下，打算还是只提供paddle.linalg.cholesky_inverse调用路径，辛苦修改一下，另外需要将这个方法patch到tensor上去，感谢！
Contributor Author
There was a problem hiding this comment.
@megemini 您好 内部沟通了一下，打算还是只提供
paddle.linalg.cholesky_inverse调用路径，辛苦修改一下，
OK ～
另外需要将这个方法patch到tensor上去，感谢！
这个已经有了，可以参考测试用例
# test `Tensor.xxx` out = x.cholesky_inverse(self._upper).numpy()
Contributor
顺师傅，需要加大一点时间 |
SigureMo reviewed Jun 3, 2024
python/paddle/tensor/linalg.py Outdated
| return result | ||
| | ||
| | ||
| def cholesky_inverse(x, upper=False): |
Member
There was a problem hiding this comment.
Suggested change
| def cholesky_inverse(x, upper=False): | |
| def cholesky_inverse(x: Tensor, upper: bool = False): |
这个 API 现在是否可以直接开启类型提示试试呢？标题上可以添加 [Typing]，现在 CI 已经添加了类型提示 approval 检查，如果这个 API 前置标注工作尚不满足，可以暂时直接 approve
| x (Tensor): A tensor of lower or upper triangular Cholesky decompositions of symmetric matrix with shape `[N, N]`. | ||
| The data type of the `x` should be one of ``float32``, ``float64``. | ||
| upper (bool, optional): If `upper` is `False`, `x` is lower triangular matrix, or is upper triangular matrix. Default: `False`. | ||
| name (str, optional): For details, please refer to :ref:`api_guide_Name`. Generally, no setting is required. Default: None. |
megemini changed the title SigureMo previously approved these changes Jun 4, 2024
Member
SigureMo left a comment
SigureMo left a comment There was a problem hiding this comment.
LGTMeow 
for new API type annotation
SigureMo previously approved these changes Jun 6, 2024
luotao1 changed the title Contributor
| @megemini 冲突了 |
SigureMo approved these changes Jun 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
PR Category
User Experience
PR Types
New features
Description
NO.2 为 Paddle 新增 cholesky_inverse API
关联 RFC ： PaddlePaddle/community#896
@Charles-hit 请评审 ～