Posted on: 12/11/08 11:35am
By: Chrispcritters
I'll give the background first:
*
whatismyipaddress.com[*1]
* 500+ concurrent users during the day
* Running on a dedicated box
* Local MySQL installation
* Mac OS X Server 10.5
* Multi-Processor (almost never more than 20% CPU usage)
* 6GB of RAM (almost never more than 2GB used)
* Mirrored Drive RAID
* Net connection usually at 50% utilization during peak traffic
What I am seeing is apache becoming non-responsive, memory usage climbs to 100%, and the drives start thrashing (swapping memory out assumedly). Once it's in this mode I've got three options: 1) change DNS entries to direct traffic to a different machine (different site), 2) power cycle the machine and usually have to repair the gl_sessions table, or 3) drop the network connection and wait 30 minutes+ for the machine to get back to normal.
Needing some help in trying to figure out what direction to start looking. I've tried to lower the MySQL queries by removing polls from the blocks and removing the forum centerblock from the homepage. Does not seem to help. Upped the MySQL querycache size to 500MB this morning. Any suggestion on other things to look at?
Re: High traffic site problems
Posted on: 12/11/08 06:30pm
By: Laugh
I wish I had your problem!
Do you think maybe it is one your IP tools instead of Geeklog?
Tom
Re: High traffic site problems
Posted on: 12/11/08 06:48pm
By: Chrispcritters
Yeah, it's a nice problem to have.
Honestly, I don't know where the problem arises so I'm going after this on multiple fronts (of which geeklog is one). If I knew that someone else was exceeding the traffic I have (without problems) then I'd focus more of my attention elsewhere.
I previously had a problem with one of the IP tools' db table reaching 1.5 million records and selects and inserts would suffer. That problem has been resolved.
Another thing that I cannot do is batch delete users. If I try to delete more than maybe ten people at a time the site hangs. That makes me think more along the lines of MySQL than geeklog itself.
I've spend tweaked some of the Apache settings today so I'll see if that helps.
Re: High traffic site problems
Posted on: 12/11/08 06:49pm
By: Chrispcritters
As a side note, page creation times remain low up to the time where the site become unresponsive.
Re: High traffic site problems
Posted on: 12/12/08 02:39am
By: Anonymous (Richard.bkk)
I have a similar problem, I run a popular Thai motorcycle website (motorcycle.in.th)
+2000 users a day
* Running on a shared hosting (but as I'm also the hosting it is only shared with fasthosting.in.th)
* Linux Hosting (RedHat) and lots of memory...
The website sometimes just doesn't react on clicks and response time is in the worst moments down to a minute...
I installed Bad Behavior2 to filter out the bad weed, but with Bad Behavior2 saying that it blocked 24,000 hits (I was wondering is that not also 24,000 SQL activities, or does Bad Behavior write it to a log file?)
I have to tell that it is based on glFusion, but as glFusion has the same background I follow this posting with lost of interest.
Re: High traffic site problems
Posted on: 12/12/08 03:20am
By: Dirk
The first thing I'd do is to actually check the webserver's log and see where that traffic is coming from. You may be targetted by a spam bot or script kiddies. If so, I'd block IP addresses or
typical requests[*2] at the webserver level, so that PHP and MySQL (and Geeklog) don't even come into play.
As I said in a
recent discussion[*3] , Bad Behavior will help a little bit, but as far as actual server load goes, it's kicking in too late to make a difference.
bye, Dirk
Re: High traffic site problems
Posted on: 12/12/08 04:18am
By: Anonymous (richardbkk)
Most of the hits that Bad Behavior2 is flagging as bad are trackbacks. I have the idea to switch of the trackback option... but then I'm sure they will target other parts of our website...
Re: High traffic site problems
Posted on: 12/12/08 09:52am
By: Laugh
Quote by: ChrispcrittersYeah, it's a nice problem to have.
Honestly, I don't know where the problem arises so I'm going after this on multiple fronts (of which geeklog is one). If I knew that someone else was exceeding the traffic I have (without problems) then I'd focus more of my attention elsewhere.
I previously had a problem with one of the IP tools' db table reaching 1.5 million records and selects and inserts would suffer. That problem has been resolved.
Another thing that I cannot do is batch delete users. If I try to delete more than maybe ten people at a time the site hangs. That makes me think more along the lines of MySQL than geeklog itself.
I've spend tweaked some of the Apache settings today so I'll see if that helps.
How long does it take before this problem appears after a server reboot (do you get at least a few days out of it)? I would think it would be a PHP bug or possibly a MySQL issue because of the memory usuage issues (leak).
Did this just start happening or has this always happened under these loads.
The deleting of users is a weird problem. Maybe concentrate on this since it is easy to reproduce. Any log information on this?
Re: High traffic site problems
Posted on: 12/12/08 10:35am
By: Chrispcritters
It's a new issue the corresponds with a new advertising campaign that has bumped traffic. Issue seems to occur when traffic is peaking to historically new highs. It happened again this morning.
I'm trying to go through the logs but they're 500MB a day so it takes some time to be able to identify issues.
I've made some more tweaks to some code that is used for proxy server detection to see if that is the issue.
Re: High traffic site problems
Posted on: 12/12/08 02:33pm
By: Chrispcritters
Argh, just happened again. This time I was able to get a mysql status before it system got unusable.
Before
Text Formatted Code
Uptime: 14801 Threads: 501 Questions: 1797849 Slow queries: 1285 Opens: 39201 Flush tables: 1 Open tables: 992 Queries per second avg: 121.468
After (restarted both Apache and MySQL)
Text Formatted Code
Uptime: 476 Threads: 3 Questions: 59522 Slow queries: 0 Opens: 81 Flush tables: 1 Open tables: 75 Queries per second avg: 125.046
Re: High traffic site problems
Posted on: 12/12/08 04:15pm
By: Chrispcritters
When it's misbehaving and I run the SHOW PROCESSLIST I see a whole bunch of processes that all look the same with varying "time"
Text Formatted Code
SELECT ug_main_grp_id,grp_name FROM gl_group_assignments,gl_groups WHERE grp_id = ug_main_grp_id AND
State is "Sending data".
Re: High traffic site problems
Posted on: 12/12/08 04:36pm
By: Dirk
That SQL request is from SEC_getUserGroups where it collects the list of groups the current user is in. Do you have a lot of groups? Or many groups that inherit from other groups?
bye, Dirk
Re: High traffic site problems
Posted on: 12/12/08 05:18pm
By: Chrispcritters
I've made no group changes since installation.
admin block says (18 ) next to groups but when I click there are only 8 groups.
There are 30k users.
(could this be tied in with problems does batch deletes of users?)
Re: High traffic site problems
Posted on: 12/15/08 03:24pm
By: Chrispcritters
Doubled all MySQL caches. Switched from using GeoIPCity DB to GeoIPCountry where applicable. Seems to have made a world of difference. Seeing record concurrent users and page creation time (when not using the GeoIPCity db) down to 0.07 seconds.
Looks like I need to switch over to the Apache2 mod_geoip to be able to push more performance out of it. Now to figure out how to do that on Mac OS X 10.5. What little reading I've found seems to indicate there may be issues there. (due to 64bit OS).
Thanks to everyone for their suggestions.
Re: High traffic site problems
Posted on: 12/15/08 07:26pm
By: jmucchiello
Quote by: ChrispcrittersSlow queries: 1285
How about turning on the slow query log and seeing if there's just a few specific queries that are slow.