Welcome to Geeklog, Anonymous Sunday, June 26 2022 @ 09:30 pm EDT

Geeklog Forums

High traffic site problems


Status: offline

Chrispcritters

Forum User
Chatty
Registered: 05/11/05
Posts: 49
Location:Tustin, CA
I'll give the background first:

* whatismyipaddress.com
* 500+ concurrent users during the day
* Running on a dedicated box
* Local MySQL installation
* Mac OS X Server 10.5
* Multi-Processor (almost never more than 20% CPU usage)
* 6GB of RAM (almost never more than 2GB used)
* Mirrored Drive RAID
* Net connection usually at 50% utilization during peak traffic

What I am seeing is apache becoming non-responsive, memory usage climbs to 100%, and the drives start thrashing (swapping memory out assumedly). Once it's in this mode I've got three options: 1) change DNS entries to direct traffic to a different machine (different site), 2) power cycle the machine and usually have to repair the gl_sessions table, or 3) drop the network connection and wait 30 minutes+ for the machine to get back to normal.

Needing some help in trying to figure out what direction to start looking. I've tried to lower the MySQL queries by removing polls from the blocks and removing the forum centerblock from the homepage. Does not seem to help. Upped the MySQL querycache size to 500MB this morning. Any suggestion on other things to look at?
 Quote

Status: offline

Laugh

Site Admin
Admin
Registered: 09/27/05
Posts: 1442
I wish I had your problem!

Do you think maybe it is one your IP tools instead of Geeklog?

Tom
One of the Geeklog Core Developers.
 Quote

Status: offline

Chrispcritters

Forum User
Chatty
Registered: 05/11/05
Posts: 49
Location:Tustin, CA
Yeah, it's a nice problem to have.

Honestly, I don't know where the problem arises so I'm going after this on multiple fronts (of which geeklog is one). If I knew that someone else was exceeding the traffic I have (without problems) then I'd focus more of my attention elsewhere.

I previously had a problem with one of the IP tools' db table reaching 1.5 million records and selects and inserts would suffer. That problem has been resolved.

Another thing that I cannot do is batch delete users. If I try to delete more than maybe ten people at a time the site hangs. That makes me think more along the lines of MySQL than geeklog itself.

I've spend tweaked some of the Apache settings today so I'll see if that helps.
 Quote

Status: offline

Chrispcritters

Forum User
Chatty
Registered: 05/11/05
Posts: 49
Location:Tustin, CA
As a side note, page creation times remain low up to the time where the site become unresponsive.
 Quote

Richard.bkk

Anonymous
I have a similar problem, I run a popular Thai motorcycle website (motorcycle.in.th)

+2000 users a day
* Running on a shared hosting (but as I'm also the hosting it is only shared with fasthosting.in.th)
* Linux Hosting (RedHat) and lots of memory...

The website sometimes just doesn't react on clicks and response time is in the worst moments down to a minute...

I installed Bad Behavior2 to filter out the bad weed, but with Bad Behavior2 saying that it blocked 24,000 hits (I was wondering is that not also 24,000 SQL activities, or does Bad Behavior write it to a log file?)

I have to tell that it is based on glFusion, but as glFusion has the same background I follow this posting with lost of interest.

 Quote

Status: offline

Dirk

Site Admin
Admin
Registered: 01/12/02
Posts: 13073
Location:Stuttgart, Germany
The first thing I'd do is to actually check the webserver's log and see where that traffic is coming from. You may be targetted by a spam bot or script kiddies. If so, I'd block IP addresses or typical requests at the webserver level, so that PHP and MySQL (and Geeklog) don't even come into play.

As I said in a recent discussion, Bad Behavior will help a little bit, but as far as actual server load goes, it's kicking in too late to make a difference.

bye, Dirk
 Quote

richardbkk

Anonymous
caffeinated
Most of the hits that Bad Behavior2 is flagging as bad are trackbacks. I have the idea to switch of the trackback option... but then I'm sure they will target other parts of our website...
 Quote

Status: offline

Laugh

Site Admin
Admin
Registered: 09/27/05
Posts: 1442
Quote by: Chrispcritters

Yeah, it's a nice problem to have.

Honestly, I don't know where the problem arises so I'm going after this on multiple fronts (of which geeklog is one). If I knew that someone else was exceeding the traffic I have (without problems) then I'd focus more of my attention elsewhere.

I previously had a problem with one of the IP tools' db table reaching 1.5 million records and selects and inserts would suffer. That problem has been resolved.

Another thing that I cannot do is batch delete users. If I try to delete more than maybe ten people at a time the site hangs. That makes me think more along the lines of MySQL than geeklog itself.

I've spend tweaked some of the Apache settings today so I'll see if that helps.



How long does it take before this problem appears after a server reboot (do you get at least a few days out of it)? I would think it would be a PHP bug or possibly a MySQL issue because of the memory usuage issues (leak).

Did this just start happening or has this always happened under these loads.

The deleting of users is a weird problem. Maybe concentrate on this since it is easy to reproduce. Any log information on this?
One of the Geeklog Core Developers.
 Quote

Status: offline

Chrispcritters

Forum User
Chatty
Registered: 05/11/05
Posts: 49
Location:Tustin, CA
It's a new issue the corresponds with a new advertising campaign that has bumped traffic. Issue seems to occur when traffic is peaking to historically new highs. It happened again this morning.

I'm trying to go through the logs but they're 500MB a day so it takes some time to be able to identify issues.

I've made some more tweaks to some code that is used for proxy server detection to see if that is the issue.
 Quote

Status: offline

Chrispcritters

Forum User
Chatty
Registered: 05/11/05
Posts: 49
Location:Tustin, CA
Argh, just happened again. This time I was able to get a mysql status before it system got unusable.

Before
Text Formatted Code
Uptime: 14801  Threads: 501  Questions: 1797849  Slow queries: 1285  Opens: 39201  Flush tables: 1  Open tables: 992  Queries per second avg: 121.468

After (restarted both Apache and MySQL)
Text Formatted Code
Uptime: 476  Threads: 3  Questions: 59522  Slow queries: 0  Opens: 81  Flush tables: 1  Open tables: 75  Queries per second avg: 125.046
 Quote

Status: offline

Chrispcritters

Forum User
Chatty
Registered: 05/11/05
Posts: 49
Location:Tustin, CA
When it's misbehaving and I run the SHOW PROCESSLIST I see a whole bunch of processes that all look the same with varying "time"

Text Formatted Code
SELECT ug_main_grp_id,grp_name FROM gl_group_assignments,gl_groups WHERE grp_id = ug_main_grp_id AND


State is "Sending data".
 Quote

Status: offline

Dirk

Site Admin
Admin
Registered: 01/12/02
Posts: 13073
Location:Stuttgart, Germany
That SQL request is from SEC_getUserGroups where it collects the list of groups the current user is in. Do you have a lot of groups? Or many groups that inherit from other groups?

bye, Dirk
 Quote

Status: offline

Chrispcritters

Forum User
Chatty
Registered: 05/11/05
Posts: 49
Location:Tustin, CA
I've made no group changes since installation.

admin block says (18 ) next to groups but when I click there are only 8 groups.

There are 30k users.

(could this be tied in with problems does batch deletes of users?)
 Quote

Status: offline

Chrispcritters

Forum User
Chatty
Registered: 05/11/05
Posts: 49
Location:Tustin, CA
Doubled all MySQL caches. Switched from using GeoIPCity DB to GeoIPCountry where applicable. Seems to have made a world of difference. Seeing record concurrent users and page creation time (when not using the GeoIPCity db) down to 0.07 seconds.

Looks like I need to switch over to the Apache2 mod_geoip to be able to push more performance out of it. Now to figure out how to do that on Mac OS X 10.5. What little reading I've found seems to indicate there may be issues there. (due to 64bit OS).

Thanks to everyone for their suggestions.
 Quote

Status: offline

jmucchiello

Forum User
Full Member
Registered: 08/29/05
Posts: 985
Quote by: Chrispcritters

Slow queries: 1285

How about turning on the slow query log and seeing if there's just a few specific queries that are slow.
 Quote

All times are EDT. The time is now 09:30 pm.

  • Normal Topic
  • Sticky Topic
  • Locked Topic
  • New Post
  • Sticky Topic W/ New Post
  • Locked Topic W/ New Post
  •  View Anonymous Posts
  •  Able to post
  •  Filtered HTML Allowed
  •  Censored Content