I develop for both, and others...but unless you are a developer, the differences can't really be appreciated. Suffice it to say that you have more initial work in planning a proper module for XOOPS that in the long-run should equate to more stable code. In Joomla, you have less up-front planning because everything is encapsulated, but you have longer debug times getting everything to work the way you want.
Joomla was Mambo -which didn't have a "huge" user base compared to others. Since becoming Joomla, it seems larger, but appearances could be deceiving. The biggest thing I notice on the surface is there is a lot more commercialization of extensions. It seems to be 50/50, but there are a lot of cool things out there to purchase, not download for free. My opinion is, Joomla is solid, and yet another CMS. Is it worth jumping over and working - don't know, that's an individual decision. The only reason I play with so many CMS's is because I am a developer first, CMS zealot second.
Okay, you brought up security. First and foremost, no PHP based CMS is without security worries. I monitor logs on a very frequent basis watching what folks try to do on my servers. Any PHP based CMS will suffer the same fate - global polution, POST-Payload injection, SQL injection, etc., etc. Unfortuantely, it is a catch-up world for most CMS. Further, for the average site sitting on todays shared environments - you can only do so much! That's why it's important to pay attention to what is going on in your logs.
Okay, enough of that - I have seen an equal amount of injection occur under Joomla and I see in XOOPS - PHPNuke get's the prize though! You want to know what the number one injection hack to any site is? BCC injection in your contact page!
Number 2? SQL injection to try to steal account info - specifically administration info. Number 3? I call it /tmp ownership - but basically the script-kiddies love to try to run "wget" "curl" fetch" and "stat" on your server to copy and run commands from your server. It could be worms, it could be email exploits so they end up relaying off your site, it could be rootkits...ah, the life of a server master!
Anyway, as with anything, there are all kinds of outside influences. Don't make decisions based on the superficial. Are you really tired of the perceived bickering amoungst creative folks, or are you just ready to throw this girl away for a new one?
Good Luck with your decision!