Stupid Python Ideas: September 2014

Sockets and multiprocessing

If you want to write a forking server, one obvious way to do it is to use the multiprocessing module to create a pool of processes, then hand off requests as you get them.

The problem is that handling a request usually involves reading from and writing to a client socket. So, you need to send the socket object as part of the request. And you can't pickle sockets.

There are easy ways around this for some cases:

For protocols without much data per request and response, there's an easy solution: the main process does the reads, dispatches a buffer instead of a socket, receives back a buffer, and does the writes. But that won't work for, e.g., streaming large files.

Sep

Micro-optimization and Python

I was recently trying to optimize some C code using cachegrind, and discovered that branch misprediction in an inner loop was the culprit. I began wondering how much anything similar could affect Python code.

Sep

Why does my 100MB file take 1GB of memory?

A lot of people have questions like this:

I've got a 100MB CSV file. I read it in to a list, populated with a csv.DictReader, and my computer starts swapping. Why? Let's look at what it takes to store a 100MB file as a list of dicts of strings.

But first, let's look at how to solve this problem, instead of how to understand it.

Sep

How to edit a file in-place

There are hundreds of questions on StackOverflow from people who want to know something like, "How do I edit a file in-place, without creating a new file."

In general, the answer is, "You can't. You really do want to create a new file." (There is an exception, which I'll get to later.)

Why? Well, files are just streams of bytes—or, for text files, streams of characters.