8
Is this a local or hosted server and what are the specs (OS, server versions, apps etc.)?
Assuming that there isn't any external content being loaded it sounds like something has to wake up after an idle period. Harddisk, server threads, screensaver, etc.
need more data.....