I suspect Austin used the phrase as being more elegant than "diminishing nits".
It occurs, of course, at the opening of his essay, "Unfair to Facts":
This paper goes back to an old controversy between Strawson and me about truth. Of course comments on comments, criticisms of criticisms, are subject to the law of diminishing fleas, but I think there are here some misconceptions still to be cleared up, some of which seem to be still prevalent in generally sensible quarters.The phrase has since been picked up and used widely on the internet to refer to comments on comments on comments .... on a blog post. But what did it mean originally?
Some have supposed that Austin was referring to how some microscopic mites live on fleas, and maybe even other microscopic mites live on those mites, etc., "all the way down". The point would be that the importance of comments is miniscule when compared with the original paper commented upon, and thus comments upon comments are proportionately even smaller in importance. An iteration or two of comments and replies, and one's words will have almost no importance at all.
But it's unlikely that this was Austin's image. First of all, fleas don't live on other fleas, even if mites do. Morever, his "Unfair to Facts" was a reply to Strawson's reply to Austin's "Truth", which was itself a reply to something by Strawson. Thus, "Unfair to Facts" would have represented a mite, upon a mite, upon a flea, upon an animal, and Austin couldn't have supposed that that essay was so insignificant. (Look at his words, and you'll see that he doesn't.)
A student in my J. L. Austin graduate seminar last night, himself raised on a farm, proposed a much more satisfactory explanation. He pointed out that fleas are removed in successive 'passes' over an animal, and that each pass yields fewer and fewer fleas. One certainly wants to remove all of them all, so that the animal is clean, but eventually one has to give up as having done well enough-- even though some fleas may remain, and if so they will return eventually in force. (Thus also our 'nit picking', but for lice, not fleas.)
Not only does this image make more sense, it also fits Austin's understanding of philosophy perfectly. For him, philosophy is simply a matter of removing fleas, and it requires the same attention to detail. One is fixed on removing them all--that's the goal of the 'treatment'--but one is likely to fail, and then the infestation, and 'illness', will return in full force.
The one objection to this interpretation, as I see it, is that it does not attribute to Austin any sense of reciprocity in his exchange with Strawson. Strawson has got the fleas, and Austin is removing them. Strawson's own comments don't count as removing anyone's fleas at all, except perhaps in intention only and misguidedly.
And yet I think a careful attention to the tone of Austin's essay confirms that that was how he conceived of the exchange.