No this is hard coded into the system.
When XOOPS interpets bb code links it adds target="_blank". So it reads:
[URL=yourlink]LinkText[/URL]
and changes it to:
LinkText in html
This forces a new window to open.
It is possible to edit the file /html/class/module.textsanitizer.php. There is a small function in there called &xoopsCodeDecode that handles this. You can just remove everything that referes to target="blank". (Note: always make a backup before editing code.)
The best way to get around this is to make any custom blocks with links that you want to open in _self or _top in html. Then save the blocak as html.