Check your CPU usage when you see long load times like that, if it is running near 100% you likely have a similar problem. I terminaled in and killed all my http processes and restarted them, thereby killing all the rogue processes in the background and my site instantly went from a 15 second load time to a consistent 2 second load time.
Set up a static 50kb or so webpage somewhere on the server with a url you can call directly so you can gauge how fast that page loads vs. php pages that have to access the database. That can maybe tell you if something related to your DB is the problem, or just the host in general.