Alexa says this about my site at http://www.aviary.info
Speed: Slow (77% of sites are faster), Avg Load Time: 2.9 Seconds

Should I be concerned? Anyone give me some pointers on speeding it up some? Maybe I need to trim back on some of the top page information?
I am using a fairly agressive cache right now with almost all modules using cache. Most blocks are also set to cache except the obvious no-nos (why they allow cache on these I don't know) such as main menu, user menu, who's online.

Though, cache only solved part of the problem. The pages still have to be created and served.

WebSiteOptimization.com has a free web page analyzer that measures the output of your site and gives recommendations on improving end-user performance.

http://www.websiteoptimization.com/services/analyze/