The first time you try to write a network client or server directly on top of sockets, you do something like this:

for filename in filenames:
    with open(filename, 'rb') as f:
        sock.sendall(f.read())

And then, on the other side:

for i in count(0):
    msg = sock.recv(1<<32)
    if not msg: break
    with open('file{}'.format(i), 'wb') as f:
        f.write(msg)

At first, this seems to work, but it fails on larger files. Or as soon as you try to use it across the internet. Or 1% of the time. Or when the computer is busy.

In reality, it can't possibly work, except in special circumstances. A TCP socket is a stream of bytes. Every time you call send (or sendall), you put more bytes on that stream. Every time you call recv, you get some or all of the bytes on the stream.

Let's say you send 1000 bytes, then send 1000000 bytes, and the other side calls recv. It might get 1000 bytes—but it just as easily might get 1001000 bytes, or 7, or 69102. There is no way to guarantee that it gets just the first send.

Why does it seem to work in my initial tests?

If you send 10000 bytes to localhost while the other side is waiting to receive something, and there's enough idle time on your CPUs to run the other side's code, your OS will probably just copy your 10000-byte send buffer over to the other side's receive buffer and give it the data all at once. It's not guaranteed, but it will usually happen. But only if the other side is on the same machine (or at least in the same bridged LAN), and your data fits into a single buffer, and you don't finish two sends before it gets enough CPU time to do its receive, and so forth.

So, what's the solution?

The key is that the other side has to somehow know that the next 1000 bytes are the message it wants. This means that unless you have some out-of-band way of transmitting that information, or your messages are some type that inherently includes that information (e.g., JSON objects, or MIME messages), you have to create a byte stream that has not just your messages, but enough information to tell where one message ends and the next begins.

In other words, you have to design, and then implement, a protocol.

That sounds scary… but it really isn't.

A simple protocol

Assuming your messages can't be more than 4GB long, just send the length, packed into exactly 4 bytes, and then you send the data itself. So, the other side always knows how much to read: Read exactly 4 bytes, unpack it into a length, then read exactly as many bytes as that:

def send_one_message(sock, data):
    length = len(data)
    sock.sendall(struct.pack('!I', length))
    sock.sendall(data)

def recv_one_message(sock):
    lengthbuf = recvall(sock, 4)
    length, = struct.unpack('!I', lengthbuf)
    return recvall(sock, length)

That's almost a complete protocol. The only problem is that Python doesn't have a recvall counterpart to sendall, but you can write it yourself:

def recvall(sock, count):
    buf = b''
    while count:
        newbuf = sock.recv(count)
        if not newbuf: return None
        buf += newbuf
        count -= len(newbuf)
    return buf

Protocol design

Protocol implementation