2
some new update:
we now have our own dedicate server that is 2.4GHz Celeron, 512mb RAM, and 80G IDE on FreeBSD, no WHM no cpanel, so no garbage applications that comes with it. Basically it's just a FreeBSD 5.1 box that is primarily used as a web server.
I have done excessive research and have personally optimized the kernel, system parameters, MySql (4.1.3), PHP (4.3.7), Apache (1.31). All applications are compiled using optimization and Zend 2.5.1 is installed to accelerate PHP. I have tweaked every performance related parameters of Apache and increased buffer of MySql. I have configured XOOPS to cache most of the blocks thus cut database queries on our main page from 90 to 60. I have even conducted an experiment of redirecting all static file request to thttpd running on another port and let Apache only handle .php requests, though this configuration failed to boost performance as expected.
Alright, our site can handle about 150 concurrent users during peak time but still will freeze sometimes and we are stuck now.
Here are some obervations we had on XOOPS and performance tweaking.
1. turn off Keep-Alive on the server will help quite a bit
2. I think there are bugs in database access of XOOPS, during off load time it may be OK but it seems like some dead-lock or race condition will occur if some condition is met. I am not an expert of PHP so I didn't bother to look into the code, the below finding is purely based on observation of the behavior of our server.
When our server is frozen (load is more than 10), most of the Apache process is serving either main page (index.php) or news module (article.php). It seems like tow or more processes started the lock down and the lock caused new httpd and MySQL to spawn for requests coming in, eventually every httpd process is at "W" stage and eats up memory thus crippled the server. Below is a capture of Apache status.
9.65 requests/sec - 25.4 kB/second - 2699 B/request
143 requests currently being processed, 7 idle servers
W_WW_WW_WWWWWWWWWWWWWWWWWWWWWWWW_WWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWW___..................
........................
During this stage the established TCP connection is about 100 and also about 100 MySQL is spawned as well. Basically we have eliminated the possibility of being attacked and think this problem is probably caused by XOOPS.
Since most of the resource XOOPS required is used on the generation of contents for different group of users (like module permissions), is there a functionality or module in XOOPS that can pre-generate static file (html) for all viewers (much like weblog applications)? Even doing so would pretty much abandon the flexablity of managed interface XOOPS has, however it will be a great feature to have in performance perspective.