4.12.6 (2023-05-25)

Bug Fixes

• more EOF protection fixes (1506ef2)

4.12.5 (2023-05-13)

Bug Fixes

• update s3fs pvc to prevent its caching mechanism from consuming all available ephemeral storage (a88602b)

4.12.4 (2023-05-11)

Bug Fixes

• improve cpu utilization metrics, and retry on ray head initContainer (9f35d12)

4.12.3 (2023-05-09)

Bug Fixes

• multinic detection was broken; also was hard-wiring name of resource (da9341b)

4.12.2 (2023-05-05)

Bug Fixes

• avoid downloading helm chart on every run (cache it) (17ac5da)
• custodian pods can linger forever (9276869)

4.12.1 (2023-05-03)

Bug Fixes

• improve torchx support for running multiple gpus per pod (70d522e)

4.12.0 (2023-05-03)

Bug Fixes

• codeflare top may fail due to Array(fractionalnumber) (cee32ec)

Features

• improve multinic and NCCL performance (f2f9a56)

4.11.12 (2023-05-02)

Bug Fixes

• use initContainer to wait for ray workers (3abe1e3)

4.11.11 (2023-05-01)

Bug Fixes

• increase ray gcs rpc timeout to 30s (1c62309)

4.11.10 (2023-05-01)

Bug Fixes

• increase resilience to network disconnects for torchx (9097cd0)