Taskflow VS Pytorch

Martin, have you had a look at https://github.com/taskflow/taskflow ?

> The set-theory approach is hard to do, but promising. Each object that wants something has a small set of the things it wants. There's a big pool of such sets. There's also a big pool of the items you have, which changes constantly. It's easy to express what you need to fetch and which objects are now ready to go as set intersection and difference operations. But you need representations for big sparse sets which can do set operations fast. Probably B-trees, or something in that space.

Incremental updates to dynamic dependency graphs is a familiar problem for build tooling. I personally have used the taskflow C++ library (https://github.com/taskflow/taskflow) to great effect.

> Microsoft Research fooled around with this concept years ago in a different context. The idea was to have a database which supported pending SQL queries. The query would return new results when the database changed such that the results of the query changed. The goal was to to support that for millions of pending queries. Financial traders would love to have that. It's a very hard scaling problem. Don't know how that came out.

Incremental view maintenance is an active area of research. The likes of Noria and Materialize have done this with SQL, and the pg_ivm Postgres extension looks promising. Not sure if there is an equivalent implementation geared towards entity-component systems, though.

For parallel programming nowadays, personally I reach for C++ Taskflow when I really care about performance, or a mix of core.async and running multiple load balanced instances when I’m doing more traditional web backend stuff in Clojure.

Also, you can take a look to good libraries. The problem is that very often libraries are heavily templated, so It could be hard. For example, I like the style of the Taskflow library, I think is very clear, is relatively small, while makes use of more advanced techniques: https://github.com/taskflow/taskflow

Cool. Thanks! How does it compare to taskflow?

I've not seen yet, but it's been a bit since I looked last, any evidence of being able to build a computation graph and "save" it to re-run on new inputs. Something like https://github.com/taskflow/taskflow

It seems you're describing something a generic parallel task framework. Check taskflow for a production ready example https://github.com/taskflow/taskflow/blob/master/

- https://github.com/taskflow/taskflow (I recommend to learn it first since its API and documentation are excellent)

If you want some sort of "job" system, where you submit items to a some sort of queue to be processed in parallel, try searching for a thread pool - there isn't one in the standard library, but there's about a million implementations online. There are more complicated versions of that idea, that describe computation as a directed acyclic graph, such as taskflow.

> A running number also carries data. Before you know it, someone's relying on the ordering or counting on there not being gaps - or counting the gaps to figure out something they shouldn't.

For example, if https://github.com/pytorch/pytorch/issues/111111 can be seen but https://github.com/pytorch/pytorch/issues/111110 can't, someone might infer the existence of a hidden issue relating to a critical security problem.

Whereas if the URL was instead https://github.com/pytorch/pytorch/issues/761500e0-0070-4c0d... that risk would be avoided.

# let's now encode the entire text dataset and store it into a torch.Tensor import torch # we use PyTorch: https://pytorch.org data = torch.tensor(encode(text), dtype=torch.long) print(data.shape, data.dtype) print(data[:10]) # the 10 characters we looked at earier will to the GPT look like this

PyTorch: https://github.com/pytorch/pytorch

He's not a core maintainer and hasn't been for years - pytorch's contributors are completely public

https://github.com/pytorch/pytorch/graphs/contributors

Unmatched integration with ML/AI ecosystems through NumPy, TensorFlow, and PyTorch

Detailed release notes: https://github.com/pytorch/pytorch/releases/tag/v2.9.0

For contrast, we also built a no-limits version in PyTorch, using CUDA when it’s available. The network is straightforward -12 inputs, two hidden layers of 128 and 64 with ReLU, and 3 outputs for UP, HOLD, DOWN - so: [12] → [128] → [64] → [3].

machine learning (TensorFlow, PyTorch)

Today I got notified by GitHub stale bot that a PR of mine in the famous PyTorch repo got closed by stale bot. 🤖 Yes, by a bot; not by an author or repo owner but a bot. You may argue that this is normal for huge projects like this one but by looking deeper into the PR this reveals to me how much wasted effort is out there in open source!