Time to Implement Pub/Sub in Hologram - Looking for Your Input!

Elixir Framework Forums Hologram Forum
,
1

With the core component system and HTTP/WebSocket infrastructure solid, it’s time to tackle Pub/Sub support.

What We Have vs What’s Missing

Currently: Actions, Commands, WebSockets and HTTP transport, component communication

Missing: Real-time server-initiated updates

I Want Your Input First

I have ideas about how Pub/Sub should work in Hologram, but I want to hear from you before biasing the discussion.

How would you want to use Pub/Sub?

  • API design and DSL preferences (Phoenix PubSub-like? Something else? etc.)
  • Component/Page integration (how should subscriptions work?)
  • Use cases (chat, notifications, live dashboards, collaboration, etc.)

Thanks for helping shape this!

10 Likes
2

Well … if you would implement something like PubSub then people would have to write wrappers for simplest things like send/2 and Process.send_after/3. However actions and commands would loose it’s meaning if you would implement handle_* support.

How about we would have 2 PIDs in server and client state? One goes for server and one for client. Then we can simply call send and Process.send_after on those pids and use some DSL functions similar to actions and commands to handle them.

With that any time like on initialisation or in action/command we would be able to start a background process in either client or server that would send it’s results for server, client or both depending on needs. That would be very flexible.

1 Like
3

Interesting idea… Could you share some code snippets to help explore this?

We’re focusing on the ideal developer experience right now - internal implementation details can come later. I’d love to see:

  • What a simple example might look like (chat message, live counter, etc.)
  • How the app would keep track of these PIDs (per component? globally? lifecycle management?), etc.

Just pseudocode is fine - want to get a feel how this would play out from a DX perspective.

4

Actually I forgot about components … Ideally there should be 4 ways to get PIDs of 3 types.

A server PID would be a JS equivalent for Elixir process that works like a proxy with server process. The message would be send via WebSocket.

A client PID would be a JS equivalent for Elixir process that works like a proxy with client process. The message would be send to current page (client side).

A component PID would be a JS equivalent for Elixir’s self() process. We simply want to send message to current component (anonymous function, action or command).

The fourth way would be an easy way to get a parent component PID or page PID. This is for “just send to parent whoever it is”. This would be nice feature for a behaviour-like implementations of some actions and commands.

defmodule MyApp.GreetingPage do use Hologram.Page route "/hello/:username" layout MyApp.MainLayout def init(params, component, server) do component |> put_state(:client_pid, component.client_pid) |> put_state(:component_pid, component.pid) |> put_state(:server_pid, server.pid) end def template do ~HOLO""" <div>To communicate with app use following process identifiers:</div> <ul> <li>Client PID: {@client_pid}!</li> <li>Component PID: {@component_pid}!</li> <li>Server PID: {@server_pid}!</li> </ul> """ end end

With that we can use those PID to send a message:

# optional 3rd argument for passing data holo_send(client_pid, :action_name) holo_send(server_pid, :command_name) holo_send(component_pid, {:action, :action_name}) holo_send(component_pid, {:command, :command_name}) # raises no such action error: holo_send(client_pid, :command_name) # raises no such command error: holo_send(server_pid, :action_name) # same for holo_send_after

Yes, client_pid and component_pid may send same actions as well as server_pid and component_pid may send same commands. This would happen only if component is a page. However all of those PIDs needs to represent different process (or JS equivalent) so that we can send strict actions to client or commands to server and alternatively send generic actions and commands within current component regardless if current component is page or not.

This way client_pid and server_pid would be recommended to use directly within init, command or action and component_pid would be used in more generic cases like external hex packages. Look that some developers may want to use actions as equivalent of JS intervals - i.e. background process that send message to itself (same action name) after specified time and based on some condition based on send data (if any).

1 Like
5

I would strongly favor a declarative sync-based API for tracking server state on the client. I don’t know exactly how this would look (nor do I have a great understanding of Hologram’s existing APIs). But using Phoenix terminology, some way to specify that a given set of assigns should be automatically shipped to the client and kept up to date (with no tearing) would be a good path, I think.

I would not recommend venturing down the path of encouraging developers to maintain synchronization with their own imperative “glue” protocol (i.e. sending their own diffs down the wire by hand). This always ends badly.

4 Likes
6

I think it depends how far you want to abstract things. I think Hologram is in a unique position to make an amazing DX here.

If Hologram’s ultimate goal is to enable local-first apps, what if we started from the ideal end state. I imagine that would look something like this:

Declare reactive live queries that sync into local state. Using a chat app as an example, maybe the syntax would look something like this:

messages = sync do Message |> where([m], m.room_id == ^room_id) |> order_by([m], asc: m.inserted_at) |> Repo.all() end end

Or

messages = Message |> where([m], m.room_id == ^room_id) |> order_by([m], asc: m.inserted_at) |> sync_all() # sync_one also available

These could be declared in a component or page.

Hologram would automatically create client-side stores (I imagine at the table-level, not for each query, so you don’t have duplicated data). The queries would be kept alive even when leaving the page / component is destroyed for some period (default to 5 min) so that if a user navigates away and then back within that time period the data is still in sync. In the future these could be synced into indexeddb (or OPFS possibly, I only have experience with indexeddb) under the hood but it acts more of a sidecar. Keep active live queries in memory for optimal performance. All of the traditional complexity could be abstracted away.

This would be conceptually similar to Zero and Meteor but the syntax and concepts could be made even more straightforward. From the developer’s perspective, they are just writing queries and everything “just works”. For the first version, I’m not sure Zero’s IVM approach would be needed, could be left as a future endeavor.

2 Likes
7

Naturally I find this approach very enticing, but I think there is a real risk of complexity explosion and “accidentally building a whole database” here. I mean, what exactly are you querying? Rows? From where? Who is persisting those rows?

You could pare the complexity back a bit by just offering a set of APIs for syncing sets (the assigns like I described) knowing that you could then incrementalize query operations over those sets using something like differential dataflow. This would be much easier to implement and would give Hologram a base for developers to start experimenting on. Then maybe somebody comes along (maybe you!) and adds queries on top of the sets and implements the IVM stuff (“Incremental View Maintenance” for those unaware).

I’m not saying it wouldn’t be better to implement everything from the get-go BTW, it’s just that it would be an enormous undertaking.

2 Likes
8

Electric sql does exactly that with tanstack/db and d2ts.

2 Likes
9

This is exactly what I had in mind. I was going to link d2ts and forgot!

1 Like
10

Thanks everyone! I love how we’ve accidentally created a buffet of real-time patterns :smiley:

If I understood correctly:

  • @Eiji proposed process messaging (very Elixir-native!)
  • @garrison proposed state sync (declarative and clean)
  • @jam proposed reactive queries (ambitious and exciting - I’d definitely like to explore the local-first path you showcased! I have something similar in mind for next stages)

All fascinating approaches, but I should clarify - I’m specifically looking for Pub/Sub (Publish/Subscribe) patterns. Think broadcasting messages across channels/topics like Phoenix Channels and Phoenix PubSub, but reimagined for ideal DX.

The exciting part: Hologram could eventually work across clusters of devices (web/mobile/desktop), so we could target specific users, sessions, or even component IDs (cids). We could broadcast to “all users in room X,” “session Y,” or “component Z on any device.”. Eventually… :wink: For now let’s just make it work for common cases.

Forget implementation details or what Phoenix did - what would the ideal developer experience look like for publishing/subscribing to messages across this kind of distributed component system?

Let’s get creative! :rocket:

3 Likes
11

You know, this probably isn’t the answer you’re looking for but I’m not sure if PubSub is a good idea for exactly the same reasons I mentioned above.

I have experimented quite deeply with building “realtime” apps using Phoenix PubSub in the standard way and I have come to the conclusion that it’s just not good. Using a streaming system to build apps like this is rather fundamentally a bad idea I think. I mean, it can be done right but then you end up with differential dataflow and similar (this is the point of that article, which I recognize is a bit much for this discussion).

The ElectricSQL guys built Phoenix Sync for this reason and that presents a better path. Personally I decided to head down the path of building an entire DB from scratch to solve this problem, which I hope will soon be another helpful contribution in this area for the community.

Given the above, I’m not entirely sure this is something Hologram should do at all. Instead of PubSub, maybe integration with something like ElectricSQL would be better. This stuff is really hard to get right.

Now maybe there are some things better suited to a PubSub system rather than a “streaming database”, but honestly I am actually struggling to think of anything. I’m sure somebody else can come up with an example, though!

2 Likes
12

The PubSub works on self() process, so let’s say the holo_subscribe would be a short for sending a special message that instead of be send to action or command would call PubSub.subscribe/3. Also the Registry would need to be fully implemented in Hologram’s JS.

Anyway in very short PubSub would work rather easily based on my proposal. I have mentioned something more generic to give a easy way for non-PubSub implementations. There are lots of stuff working around Elixir processes, so sending and sending after are the absolute must haves here …

Oh well, very interesting. First would be a simple broadcast/4 equivalent, the second would be something like a local_broadcast/4 limited to current user and the third would be again a broadcast/4 equivalent limited to current user and component.

Those specific targets are interesting, but I would also add broadcast "on any device", so broadcast/4` equivalent limited to current user. I would add 4 different function names with same prefix to make those calls clear and explicit, something like:

  1. broadcast (all users in room X on all nodes)
  2. broadcast_session (session Y)
  3. broadcast_user (on any device)
  4. broadcast_user_component (component Z on any device)
2 Likes
13

I think the problem here is level of abstraction.

PubSub is not a means of “when provided you have all you need for realtime UI”. PubSub is a low level message passing primitive, which can be used for lots of things. Realtime UI can use PubSub, but it likely won’t be great unless you layer more architecture or tooling on top of it. There’s also quite a bit of making tradeoffs involved. E.g. phoenix presence was layered on top of phoenix presence, but it’s quite explicit about being eventually consistent and just being able to hold reasonable amounts of metadata per listed user.

Incremental view maintenance (IVM) on the other hand is a much higher level tool for a much more narrow usecase. It’s awesome to get incrementially updating UI and I’d argue it’s a way of building UI with less footguns, but it’s also not really a holy grail either. It’s quite useful for what it does, but e.g. electric sql only uses it on the client between it’s local db and live queries. Syncing the db state to the client happens by syncing transaction events (kind of a WAL).

So I guess my first question here would really be what hologram understands as “realtime UI”. At least when coming from a POV of what’s going on in JS land, there’s currently a lot of work going into CRDT based systems, where the server becomes a dumb sync server, the whole data transfer becomes completely abstracted away and you essentially just trust the system to give you the correct data client side. That’s not at all the level where something like PubSub would sit.

1 Like
14

I tend to agree with garrison’s take but maybe someone can change my mind. I’ve been working with phoenix pubsub and channels to wire up a svelte frontend. And while it works, the DX leaves a lot to be desired. I’ve created my own abstractions to make it nicer to work with and I think Hologram could do something similar / take it to the next level which is why I suggested live queries as a potential path instead of working with lower-level subscribe and broadcast. Of course it would be a lot more work and maybe it doesn’t belong in Hologram core but it would make Hologram even more compelling if it were imo.

If there is a compelling reason to surface lower-level pubsub, then I think all you’d need are:

  • subscribe
  • broadcast
  • broadcast_from
  • broadcast_to - can be used to send to the current user only or something else specifically

Maybe subscribe should include some way to authorize. I’m not sure if you’re thinking authorization would be elsewhere.

I’m not sure I buy into the use CRDT for all the things. I think using server reconciliation (used by Zero and apps like Linear) makes more sense for most apps. CRDTs are no doubt useful for collaborative text editing. But again maybe someone can enlighten me.

1 Like
15

I’d argue that it doesn’t matter in the end. All a user cares for it that somehow and with some guarantees data becomes available. The hard parts are handled by the system. But that also means the system becomes a harder thing to implement.

1 Like
16

Sure if someone can say here’s some magic that “just works”, that sounds wonderful in theory. However, I think there is still a lot to be determined, e.g. permissions. There’s also the issue of CRDTs will converge, but the result may not be what you’d hoped. I think for a wide range of apps, you’re going to need a central server authority, at which point server reconciliation is the easier and more flexible path imo. But happy to be proven wrong so that I can offload complexity :slightly_smiling_face:.

1 Like
17

I might be misunderstanding something here, or maybe there’s just a disconnect. For now, my focus is on a simple pub/sub primitive that enables sending and broadcasting messages.

It feels like the conversation is jumping straight into high-level application patterns such as data synchronization. While pub/sub can certainly play a role there, it’s not the same thing. I suspect part of the disconnect comes from past experiences with Phoenix PubSub, likely when trying to implement custom data sync solutions.

For proper data sync, we’ll almost certainly need more than just pub/sub. Pub/sub might be one building block, but it won’t be sufficient on its own - it could even lead us toward building something like a lightweight client-side database on top of OPFS. I’m not sure yet. I still need to explore the broader local-first space. But what is clear is that raw pub/sub alone can’t cover data synchronization.

That said, Hologram should still have pub/sub. It’s broadly useful. For instance, it wouldn’t surprise me if Electric SQL used pub/sub under the hood to sync transaction logs. Differential Dataflow’s operator coordination also looks a lot like pub/sub in concept. And even Zero/IVM systems rely on some messaging layer to propagate changes.

The likely endgame is a layered architecture:

  • Pub/Sub Layer (what I’m working on now): reliable message delivery, topic management, subscription handling
  • Coordination Layer: consistency, watermarks, transaction boundaries
  • Application Layer: live queries, state sync, real-time UI updates

Even if pub/sub isn’t the backbone of the eventual data sync design, it will still serve plenty of other purposes - component-to-component messaging across devices, system events, escape hatches, and more.

Some cases where plain pub/sub shines:

  • Event notifications where consistency isn’t critical (user joins, deployment complete, toast notifications)
  • Real-time updates that don’t require data integrity (progress ticks, typing indicators, mouse tracking in multiplayer)
  • Anything idempotent or ephemeral, where lost or duplicated messages don’t cause problems

So for now, I’d like to explore possible API/DSL patterns for this, including auth as well.

2 Likes
18

I trust your ideas will be great, since the DX for hologram in general feels great. My use case would be your example of svg drawing from 0.5.0 release - I want to broadcast changes to other players in the room. I’m making a scribble game in your framework - it’s fun! This is my missing piece. :ogre:

3 Likes
19

Actually, why should Hologram have PubSub? Come to think of it, why does Phoenix have PubSub? Why don’t we just have “Elixir PubSub”?

I like your layers, but I think they’re the wrong way around. Transactions and consistency are the foundation, and the application layer which consumes them is on top of that.

PubSub is fanout, and is an (optional) final layer that one might use to scale out for a particular use-case.

In the common case, a client would subscribe directly to the part of the database they are interested in consuming (their own tenant space). This is not a “topic” subscription, this interaction must be with the database because only the database understands how its keyspace is sharded.

In the less common “fanout” case, an (internal) client would subscribe to a popular keyspace and then fan that out to PubSub subscribers on some sort of topic. But this is not the foundational API, because this is not the common case. Most of the time most users will subscribe only to things they are interested in, and again PubSub is a poor primitive for this behavior because it has no interaction with the sharding mechanism and would become a scaling bottleneck.

This is analogous to streaming a particular show vs viewing a live sports game. I think most apps are the former, though I’m sure some constitute the latter.

I think consistency is critical for all of these things. It’s critical for nearly everything, but we’re all so used to the extraordinary negligence of mainstream databases that we’ve come to accept living without it.

1 Like
20

I missed this line and I want to respond to it specifically because it highlights why the points I’m trying to make here are actually on topic (I swear!).

If you are sending messages between components like “user created a post!” then you have already lost. This is actually exactly the same problem that came up in that thread we had on two-way binding, except in this case it’s over the network.

What you want is top-down, one-way dataflow. In some cases this may be local (like React controlled components), and in some cases it may be flowing from your central database into the client’s components. But it must be one-way. A component should update the database, and the database should update the other component. If you start going down this path of “I’ll just send a notification for this one little thing” you will get bugs and spaghetti. I have felt this pain. I think we all have.

Now CRDTs and local-first do fit into this discussion (the database becomes “local”) but I don’t want to descend into that whole thing. I’m only trying to make this one point :slight_smile:

2 Likes