Posted on May 16

But how does torrenting actually work?

If you prefer to consume this article as a video, check this out!

README

Also, a quick side note: this article intends to be fully language agnostic, meaning that we don't discuss how to do something torrent-related in a certain language.

Finally, if you're here because you're trying to implement a BitTorrent client yourself, you should probably check this diagram out (you do have to make an account, sorry).

You may have heard of BitTorrent or torrenting before, but you’ve probably never taken a look under the hood of how it all actually works. Torrenting is a surprisingly complex process, and in this article, we’re going to take a deep dive into the BitTorrent protocol and how it works.

WTF is BitTorrent? 🫤

If you're familiar, feel free to skip ahead

Firstly, let’s talk about how it and torrenting works. So BitTorrent is, as you might know, decentralized, and downloads occur between these decentralized nodes called peers. Why? Well, you can commonly get much better download speeds if you download from multiple “servers”, or in this case peers, at the same time. By the way, these peers are also called seeders sometimes, as they “seed” the file that you’re looking to download.

Let’s walk through a demo: To get a list of peers to download from, you contact what’s called a Tracker. A Tracker maintains a list of all available peers for a certain file (and yes, this is kinda centralized, but you can do this in a decentralized way that we’ll talk about later) and it updates that list when a new peer is available. So, we request this list from a Tracker, and move on to the next step.

With this list, we start requesting pieces of the file from the peers that we just acquired. These pieces are literal pieces – normally anywhere from 32 KB to 16 MB in size. Finally, when we have all the pieces, we assemble the file and disconnect from the network (actually, we can stay on the network and we’ll talk about that later.)

The difference between torrent and torrent 👁️‍🗨️

Let’s denote something important: .torrent refers to the torrent filetype that maintains all of the data needed to torrent, and BitTorrent refers to the protocol – that is, BitTorrent is the protocol that allows the process of torrenting to occur, and a .torrent file is just the file that contains the information needed to contact that tracker, get a list of peers, and contact those peers. More on that later.

Some confusion 🙉

But it gets even more confusing. BitTorrent also refers to the BitTorrent client, which is basically just a wrapper for the BitTorrent protocol. It has a GUI, and it allows you to easily torrent files. There are also a few other BitTorrent (again, referring to the protocol here) clients like uTorrent, that again, act as a wrapper for the BitTorrent protocol.

And to make it even more confusing, BitTorrent can operate on multiple different protocols like UDP and TCP. And if that confuses you – well, I don’t blame you. This might help you understand what we’re talking about though (OSI model). This is the OSI model. You don’t need to be aware of it or how it works – just know that the physical layer refers to the literal cable transmitting data, the transport layer refers to stuff like TCP and UDP, and the application layer refers to protocols like HTTP.

So, the BitTorrent protocol operates by being boxed up and sent by TCP or UDP. There’s also this thing called uTP, which is the uTorrent Transport Protocol, but don’t worry about that.

Trackers 👈

Ok, one final thing that is absolutely essential in understanding BitTorrent: everyone, except for the aforementioned Tracker servers, is considered to be equal – in other words, everyone carries their own weight. That means that if your client connects to a few peers and downloads a few files, your client is also expected to provide those files to other potential peers. Just keep that in mind as we walk through the lower levels of BitTorrent. And with that, we are also a peer in any BitTorrent network that we are connected to.

Trackers in depth 🫵

Basics first. We start with either a torrent file or this thing called a magnet URI. We extract the relevant information, and make a request to the tracker to get a list of peers to start making requests to. Let’s take a deeper dive into this.

As I previously mentioned, this is the very first part of the BitTorrent protocol’s process of downloading the files: that is, getting a list of peers to download from.

What we need 🕴️

Naturally, we need the IP address or URL of the tracker, as well as a few other things. This data can come in one of two forms, as just now mentioned: a .torrent file, or a magnet URI, which is somewhat like a .torrent file in the form of a URI – not exactly though.

Now what are these “other things” that we need for the tracker, you ask? Well, we need this thing called an info hash and our peer id. The peer id is just our client's id on a given BitTorrent network (randomly generated), and the info hash is a SHA-1 hash of a part of the .torrent file. We need these so the tracker can identify what list of peers to return.

If it’s a magnet URI, things are a bit simpler (or more complex, depending on how you look at it). A magnet URI has similar fields to a .torrent file, but not all fields are going to be there. If the address tracker (AKA the URL/IP to the tracker server) is there, we can just make a simple request to it, and move on with our lives. If it isn’t, we need to check if there is a list of peers to start with.

This is where that “more decentralized method of getting peers” that I previously mentioned plays in, and it’s called DHT. If we’re given a list of peers to start with, those peers may have another list of peers for us, and those peers might as well, and so on, so forth. In other words, if we don’t have a tracker, we can get peers from other peers.

Finally, if none of those fields are present, we can attempt to download from the “acceptable source”. This is just a standard download, but it ensures that the user can at least download a file if nothing else works.

Peer(s)? 🫥

Oh, and you may have noticed that I’ve been saying “peers”, plural. This is because there does need to be at least one peer, but we could be connected to 20, 50, or even more peers. This is incredibly important to note because throughout this article, I’ll be referring to just one peer for the sake of simplicity. Just know that this same process would occur for a lot of other peers.

The Handshake 🤝

Next, let’s take a look at the initial handshake made with a peer. A high level interpretation at this portion of the protocol would go something like this:

We want to connect with a peer (also known as a seeder in this case, as they’re providing pieces of the file to us), so we say to the peer “hey, I have this file that I’d like to download. Can you send me your info hash so I can see if your file is intact?”. Then, if the info hash the peer sends back is correct, we can continue with the process.

With that, let’s take a look at what the innards of this handshake actually look like.

Recall that I’ll refer to this whole list of peers that we got from the tracker or the magnet URI as a singular peer for the sake of simplicity. To connect to a peer, we need to start by handshaking with them.

Here’s what the header looks like. It’s 68 bytes long. It starts with the character 19 as a single byte, and the string “BitTorrent Protocol” as 19 bytes. There are then 8 bytes of padding, the 20 byte SHA-1 info hash that we just now discussed, and our 20 byte peer id.

We send this off to the peer, and await a response. The peer sends back what they think their SHA-1 hash is, and also ensures that everything else about the handshake is valid (is the right length, etc).

We validate this handshake by checking if general features about the handshake are correct (again, is it the right length), and then ensuring the peer’s SHA-1 hash matches ours. If it doesn’t, the peers' pieces of the file might be corrupted, or simply incorrect. Whatever the issue may be, if the peer’s SHA-1 hash does not match ours, we sever the connection to that peer.

If everything looks right, we move on to the next step.

Actual messages 👋

We’re able to start sending actual messages now. Do note that we’re also supposed to send keep-alive messages every two minutes.

I would jump directly into what messages we’re sending, but we need to talk about the format first. On a fundamental level, there are 9 different messages that we can send to a peer. They are all denoted by the first byte: 0 through 8, inclusive. Actually, that’s a bit of a lie, because there’s 4 bytes of padding before the “type” byte. These 4 bytes are reserved for denoting the length of the message.

After the “length” and the “type bytes, we can do… anything, at least anything that the BitTorrent protocol’s spec allows. You’ll see some examples of payloads soon, but the first 4 message types that we’ll talk about have no payload. They only represent the state of the peer. These types are choked, unchoked, interested, and not interested. The terminology behind these messages seems really weird initially, but trust me, it’ll make sense soon. For now, you can just think of these like messages that change the quote-unquote state of a peer (or us).

The messages ➡️

The first message with a payload is the “have” message. The payload for this message is a single byte representing the index of the piece that the downloader just completed and checked the hash of. Recall that pieces are simply separate parts of a given file that the peer has.

The next message with a payload is “bitfield”. This message is only ever sent as the first message, and the bytes of the bitfield correspond to the indices of pieces of the file (0-7, 8-15, and so on). Naturally, the payload is a bitfield (yes, a literal field of bits) with each index of the piece that the downloader has set to 1 and the rest set to 0. For example, this bitfield tells us that the peer has pieces 3, 5, and 6.

The next message is “request”. This message acts as… you guessed it, a request for a piece. It contains an index, begin, and length. The length refers to the length of the block we’re requesting and is usually a power of 2, the index refers to the literal index of the literal piece (think about the bitfield here), and begin refers to the offset on that piece. This is a good time to mention something weird: we don’t actually request pieces as an entire piece – we request each piece in blocks. To put it more aptly, we have the file, which is broken up into pieces, which are then requested in blocks. This is why we need an offset – to tell the peer or seeder where to start sending a block.

The next message is “piece”. This message is a response to the request message. For instance, imagine we request a piece from a peer with the “request” message. That peer would respond with a “piece” message. A piece message contains an index, an offset, and a block: part of the actual piece we requested. The index is again the literal index of the piece, and the offset is copied over from the request message.

The final message is “cancel”. The use case of this message is a little bit weird, but we’ll get to that later – it’s called “endgame mode”, by the way. Naturally, it cancels an outstanding request, and it contains an index, an offset, and a length. We’ve already covered the index and the offset, and the length simply refers to the length field of the request.

Let's send some messages! 🐱

Finally, finally, finally, we are able to start sending messages!

The first message that we send is a bitfield of the pieces we have, if we have any. Remember that in BitTorrent, everyone is treated as an equal, meaning that we’re expected to act as a seeder once we have pieces of the file. The first message that we receive is a “bitfield” message telling us what pieces the peer has. If the peer does not have any pieces, we can either sever the connection, or let the peer download files from us.

Choking 😮‍💨

This is a great time to talk about choking – what it is, and why is it implemented in the BitTorrent protocol?

The concept of choking is essentially an algorithm that helps block peers that are leeching – what is leeching, you ask? Well, it’s the inverse of seeding. Imagine if that peer we just talked about intentionally sent an empty bitfield.

This would mean that they did not intend to contribute to the network at all, and instead wanted only to download a file. Naturally, leeching goes directly against the principle of equality in a BitTorrent network, and we need an efficient way to deal with this. Choking does exactly this, and it also helps with general performance and download speeds.

All that being said, the choking algorithm is a bit too intricate for this article. Simply understand that it is a very useful tool for the BitTorrent protocol, and that any time you see a “choke”, “unchoke”, “interested”, or “not interested” message like you did earlier, we are just utilizing the choking algorithm.

You should really check out this video if you'd like a deep dive into choking

Anyways, once we have sent and received any and all “bitfield” messages that we need, we can send an “interested” message and wait for and receive an “unchoke” message. If we have any files to supply, we will do the inverse.

Actually torrenting ⏬

If these requests are successful, we can start working with the peer – both in uploading and downloading files.

If the peer sent a bitfield of pieces, we can start requesting those pieces with a “request” peer message, and wait for a “piece” peer message in response to each of those requests. If the peer asks for any pieces, we can do the same. As long as we have a file to download or upload and no connections are unintentionally severed, we can stay connected to the peer (or again, multiple peers) and continue to upload or download files.

Now would be a great time to mention what endgame mode is. I think the BitTorrent docs sum this up pretty well, so I’ll quote them here: “When a download is almost complete, there's a tendency for the last few pieces to all be downloaded off a single hosed modem line, taking a very long time. To make sure the last few pieces come in quickly, once requests for all pieces a given downloader doesn't have yet are currently pending, it sends requests for everything to everyone it's downloading from. To keep this from becoming horribly inefficient, it sends cancels to everyone else every time a piece arrives.”

In other words, the “cancel” message can be used to speed up downloads towards the end of the download process.

Cleaning up 🧹

And finally, when we’re all done, we can sever the connection to the peer (or again, any and all peers that we are connected to). As far as I’m aware, there’s not much manual cleanup that we need to do with the BitTorrent protocol. It’s kinda just “ok, I’m done, bye”.

Once we have all of the pieces, we combine them, and save the final file to disk.

So with that, I hope this article was able to give you a better understanding of the BitTorrent Protocol.

If there’s anything that I got wrong, please leave a comment and let me know. I'd be very happy to change anything that isn't correct.

Citations

Top comments (1)

Parker Waiters • May 22

Great breakdown of how torrenting works! The visuals and step-by-step explanations made things really clear. Thanks for putting this together.