Once in a few weeks I have to empty all three core caches manually to bring my site back to life again. This really is necessary because the site gets deadly slow, waiting about 30 seconds before the entry page loads. After the first page has loaded the next pages load with a bit more speed but also slowly. Only a total cache wipe cures this problem for a while.

Is this normal behavior with xoops 2.5.5 or is it me doing something wrong? Also gzip compression is turned on.

After some very bad experienced like you now it is 4 years that I dont use cache for any block or module and my big site works fine in a poor shared host (more than 100Gig traffic per month)
i recommend:
1- turn caches off for all blocks and modules
2- turn gzip cache off
3- set custom session to NO.

IMO xoops have problems in handling caches and sessions.