for filename in filenames:
with open(filename, 'rb') as f:
sock.sendall(f.read())
And then, on the other side:
for i in count(0):
msg = sock.recv(1<<32)
if not msg: break
with open('file{}'.format(i), 'wb') as f:
f.write(msg)
At first, this seems to work, but it fails on larger files. Or as soon as you try to use it across the internet. Or 1% of the time. Or when the computer is busy.
In reality, it can't possibly work, except in special circumstances. A TCP socket is a stream of bytes. Every time you call send (or sendall), you put more bytes on that stream. Every time you call recv, you get some or all of the bytes on the stream.
Let's say you send 1000 bytes, then send 1000000 bytes, and the other side calls recv. It might get 1000 bytes—but it just as easily might get 1001000 bytes, or 7, or 69102. There is no way to guarantee that it gets just the first send.
Why does it seem to work in my initial tests?
If you send 10000 bytes to localhost while the other side is waiting to receive something, and there's enough idle time on your CPUs to run the other side's code, your OS will probably just copy your 10000-byte send buffer over to the other side's receive buffer and give it the data all at once. It's not guaranteed, but it will usually happen. But only if the other side is on the same machine (or at least in the same bridged LAN), and your data fits into a single buffer, and you don't finish two sends before it gets enough CPU time to do its receive, and so forth.So, what's the solution?
The key is that the other side has to somehow know that the next 1000 bytes are the message it wants. This means that unless you have some out-of-band way of transmitting that information, or your messages are some type that inherently includes that information (e.g., JSON objects, or MIME messages), you have to create a byte stream that has not just your messages, but enough information to tell where one message ends and the next begins.In other words, you have to design, and then implement, a protocol.
That sounds scary… but it really isn't.
A simple protocol
Assuming your messages can't be more than 4GB long, just send the length, packed into exactly 4 bytes, and then you send the data itself. So, the other side always knows how much to read: Read exactly 4 bytes, unpack it into a length, then read exactly as many bytes as that:def send_one_message(sock, data):
length = len(data)
sock.sendall(struct.pack('!I', length))
sock.sendall(data)
def recv_one_message(sock):
lengthbuf = recvall(sock, 4)
length, = struct.unpack('!I', lengthbuf)
return recvall(sock, length)
That's almost a complete protocol. The only problem is that Python doesn't have a recvall counterpart to sendall, but you can write it yourself:
def recvall(sock, count):
buf = b''
while count:
newbuf = sock.recv(count)
if not newbuf: return None
buf += newbuf
count -= len(newbuf)
return buf
Protocol design
There are a few problems with the protocol described above:
- Binary headers are hard to read, generate, or debug as a human.
- Headers with no redundancy are not at all robust. One mistake, and you get out of sync and there's no way to recover. Worse, there's no way to even notice that you've made a mistake. You could read 4 arbitrary bytes from the middle of a file as a header, and think it means there's a 3GB file coming up, at which point you may run out of memory, or wait forever for data that's never coming, etc.
- You have to pick some arbitrary limit, like 4GB. (Of course 640K ought to be enough for anyone…)
- There's no way to pass additional information about each file, like the type, or name.
- There's no way to extend the protocol in the future without making it completely incompatible.
Not all of these problems will be relevant to every use case. You can solve the first 3 by using something like netstrings. Or by using delimiters instead of headers (such as a newline, assuming you're sending text messages that can't contain newlines, or you're willing to escape your data). If you need to solve all 5, consider using something like RFC2822, the "Name: Value" format used by HTTP, email, and many other internet protocols.
Meanwhile, this is purely a data protocol. But often, you want to send commands or requests, and get back responses. For example, you might want to tell the server to "cd foo" before storing the next file. Or you may want to get a filter, and then send files matching that filter, instead of sending all of your files. Or you may want something you can build an interactive application around. This makes things more complicated, and there are many different ways to deal with the issues that arise. Look over HTTP, FTP, IMAP, SMTP, and IRC for some good examples. They're all well-known and well-documented, with both tutorials for learning and rigorous RFCs for reference. They have Python libraries to play with. They have servers and clients pre-installed or readily-available for almost every platform. And they're all relatively easy to talk with by hand, over a simple netcat connection.
Protocol implementation
There are also a few things that are less than ideal about the protocol handler above:
- It's directly tied to sockets; you can't use it with, say, an SSL transport, or an HTTP tunnel, or a fake transport that feeds in prepared testing data.
- It can only be used synchronously. Not a big deal for a urllib2-style client where it's appropriate to just block until the whole message is received, or even for a server that's only meant to handle a handful of simultaneous connections (just spawn a thread or child process for each one), but if you want to play a video as you receive it, or handle 10000 simultaneous clients, this is a problem.
- Receiving 4 bytes is a pretty slow way to use sockets. Also, trying to receive exactly what you want makes it easy to accidentally run into the same bug you started with, and not notice it until you try a larger file or communicating across the internet. So, you generally want to receive some multiple of 4K at a time, append onto a buffer, and pull messages out of the buffer.
Dealing with these problems can be complicated. For learning purposes, it's worth building a transport-independent protocol and a couple of transports from scratch, or an asynchronous server directly on top of select.select, etc. But for practical development, you don't want to do that.
That's why there are networking frameworks that do all the hard stuff for you, so you only have to write the interesting parts that are relevant to your application. In Python 3.4 and later, there's a framework built in to the standard library called asyncio (and, for 3.3, a backport on PyPI). If you're using an older version, you only have asyncore and asynchat, and you don't want to use those; instead, you probably want to install and use something like Twisted or Monocle. There will be a bit of a learning curve, but it's worth it.
If you're building on top of a higher-level protocol like HTTP or JSON-RPC, you can write even less code by using a higher-level framework. The standard library has clients for a number of major protocols, and servers for some. But it doesn't handle everything (e.g., JSON-RPC), and you still may want to reach for third-party libraries in some cases (e.g., for client-side HTTP, Requests is a lot easier to use than urllib2 for anything but the most trivial or most complex cases). And sometimes, the right answer is to go even higher-level and build a web service instead of a protocol, building on WSGI, or a server framework like Django.
View comments