2
Well, like most anyone here, I'd recommend XOOPS. A few years ago, I looked under the hood of a number of them (XOOPS, phpNuke, postNuke, Drupal, Joomla, WordPress, etc.), and I feel that XOOPS is the best of the lot from a technology standpoint. It's certainly the speediest (one of your criteria) if you take advantage of the caching mechanism which allows a site to serve more people with less CPU power (read shared hosting).
In the end, though, you will have to make a decision and run with it because no matter which one you pick, there is a learning curve with a time investment involved. You should pick a CMS that has the capabilities you need; a solid strong community of developers, users and supporters; good underlying technology; a long-term roadmap; and such. I would also caution you about picking the prettiest one OOB (out of the box). XOOPS doesn't have the initial curbside appeal that Drupal or Joomla have upon fresh installation. And XOOPS comes with just the basic framework and no longer has "core" applications like Drupal or Joomla. But, don't let that deter you. There are many excellent pieces here contributed by the community.
There are a couple of CMS review sites out there that highlight some of these things, and maybe that will help you in your decision, but in the end, of the 200+ CMS engines out there, likely there is really only a few worth investing time in. Jump in and try it. There is plenty of help here. I can attest to getting assistance as I've needed it.
Good luck!
