Mill fails to acknowledge that there are rights to be left alone to one's liberty that hold independently of projected net utility calculations. It's no surprise that a utilitarian fails to do so, yet it's no use pretending that the fact that your life is yours and not anyone else's is not a very good reason you should be left alone to do as you like. Yet so Mill must pretend for never even mentioning this reason.
The lacuna bespeaks a stilted strategy of Mill's: to discover principles of morality and to conduct moral reasoning by proving these and then applying them. In this case the principle is "the general principle of liberty" which is that one's actions insofar as they don't concern others' interests cannot be wrongful to others, whereas actions which harm others are punishable. He wants to prove this principle and establish it as an applicable tool of reasoning. He has to scrape together utilitarian excuses where this principle seems to fail, and he must ignore moral factors that don't fit into the simple model. This is not how moral reasoning should proceed. Moral reasoning should embrace any moral considerations which obtain in the situation under scrutiny. Mill does some of this case-based reasoning in the book, but his goal is to establish his utilitarian liberty.
Over the course of the book, Mill deems justified many specific infringements of liberty on the grounds that only negligible liberty is lost or no harm done in these cases. He sees certain cases very clearly but he bungles the account of them up because he chooses to serve his general principle rather than to fit the facts. Another woeful lacunae in the book is that Mill doesn't recognize that the government interference with individuals' actions which he says doesn't really infringe upon liberty or inhibit utility requires taxation of third parties and this taxation is a considerable infringement of liberty.
This is not to say that some of his observations on the disutility of liberty infringement aren't spot on. While one waits in vain to hear Mill acknowledge that taxation is liberty-infringing he sees that the
most cogent reason for restricting the interference of government is the great evil of adding unnecessarily to its power. Every function superadded to those already exercised by the government causes its influence over hopes and fears to be more widely diffused, and converts, more and more, the active and ambitious part of the public into hangers-on of the government, or of some party which aims at becoming the government. If the roads, the railways, the banks, the insurance offices, the great joint-stock companies, the universities, and the public charities were all of them branches of the government; if, in addition, the municipal corporations and local boards, with all that now devolves on them, became departments of the central administration; if the employees of all these different enterprises were appointed and paid by the government and looked to the government for every rise in life, not all the freedom of the press and popular constitution of the legislature would make this or any other country free otherwise than in name.And
A government cannot have too much of the kind of activity which does not impede, but aids and stimulates, individual exertion and development. The mischief begins when, instead of calling forth the activity and powers of individual and bodies, it substitutes its own activity for theirs; when, instead of informing, advising, and, upon occasion, denouncing, it makes them work in fetters, or bids them stand aside and does their work instead of them.
These are insights of a man during the 1850's. The book is a hodge-podge of such insights, classic defenses of freedom of thought and debate, and also contortions of moral theory. It's classical liberalism, which its admiration of progressively-vectored moral theories and its solicitous regard for the sanctity of individual liberty. It would have done better to dispose of the progressively-vectored moral theories, but then its central argument - the defense of a general principle - would not have been a non-starter. It's better to preserve the manifold values we cherish than to choose a certain way to progress beyond them with new principles of liberty and utility.
