7
Naturally XOOPS is a little bit slower than IPB because it has more information to handle. For example, when running an IPB site, with GZIP compression, your average pages uses 6 queries. XOOPS on the other hand, running the IPB forum module (for the sake of fair comparison), uses 12-14 queries.
You have to remember that the banners, and the left and right blocks all use SQL querries and take time to render, but this is the price you pay for power. In my personal experience, with a reasonable host, you shouldn't really notice the difference in speed. When people talk about speed differences on this level, they are looking at things statistically, not from a users' perspective.
I very much doubt if, on the whole, your users will notice any drop in speed by moving to Xoops, and if they do - you can always consider moving to a faster host

Gareth