Scott Aaronson has this list of Eleven Objections, involving both mathematics and physics arguments.
What I did is to write out every skeptical argument against the possibility of quantum computing that I could think of. We'll just go through them, and make commentary along the way. Let me just start by saying that my point of view has always been rather simple: it's entirely conceivable that quantum computing is impossible for some fundamental reason. If so, then that's by far the most exciting thing that could happen for us. That would be much more interesting than if quantum computing were possible, because it changes our understanding of physics. To have a quantum computer capable of factoring 10000-digit integers is the relatively boring outcome -- the outcome that we'd expect based on the theories we already have.
So, on to the Eleven Objections:
- Works on paper, not in practice.
- Violates Extended Church-Turing Thesis.
Perhaps the most precise objection from a mathematical point of view. Here is an extended discussion. - Not enough "real physics."
- Small amplitudes are unphysical.Leonid Levin's objection
- Exponentially large states are unphysical.Paul Davies' objection
- Quantum computers are just souped-up analog computers.Robert Laughlin's objection
- Quantum computers aren't like anything we've ever seen before.Dyakonov's objection
- Quantum mechanics is just an approximation to some deeper theory.
- Decoherence will always be worse than the fault-tolerance threshold.Gil Kalai's objection
- We don't need fault-tolerance for classical computers.
- Errors aren't independent.This is (part of) Gil Kalai's objection
UPDATE (September 2019): objection 2, the violation of the extended Church-Turing thesis, may now have been blown awayremoved, as discussed by Scott Aaronson.