There are other reasons too. If someone says “I can conjure a rabbit out of a hat — seriously!”, everyone knows that it’s a common magic trick, and they can avoid spending effort on wondering whether it might be possible according to some unknown laws of physics. Reading a long book in a few minutes is a common enough mentalist trick that I think we can say it’s “impossible”, or at least requires an extremely high standard of proof that has never been demonstrated by anyone who has made the claims.
Nodas wrote a good summary of what is possible with speed-reading at the top levels of speed-reading competition. If someone can read a 250-page book in four minutes, that competition would be a good place to demonstrate it and have skeptics immediately accept that it is real. I’d be happy if it were real, but 10 years in the memory scene has made me skeptical enough to view these things as rabbit-in-hat feats by default. 