Welcome to Geeklog, Anonymous Thursday, March 28 2024 @ 01:28 pm EDT

Geeklog Forums

Possible memory leak


Status: offline

Marites

Forum User
Chatty
Registered: 02/04/04
Posts: 64
stressed
Hi we are currently investigating a memory leak on our server which we have narrowed down to MySQL. We run 8 instances of Geeklog on the server and 1 instance of phpAdsNew which is linked to Adlink in Germany. We have rebooted a couple of times and within hours the memory usage climbs to 99% plus the virtual memory. None of the GL's have extra plugins apart from what comes out of the box.

Our server is an IBM with Xeon 3.02 1 Gb RAM and dual scsi running RAID. The Geeklogs running are a mixture of 1.3.8.x and 1.3.9.x all with latest patches as appropriate. MySQL is 4.0.20 and Apache 2.0.5x. OS is RedHat Enterprise.

We are not saying it is anything to do GL but has anyone had any experience of memory leaks.

- Tess -
 Quote

Status: offline

tomw

Forum User
Full Member
Registered: 08/12/02
Posts: 300
I have a red hat (FEDORA CORE 2) server running 6 geeklog sites and there are no memory leak issues. My previous server (pentium 200 with 64 mb of memory!) ran 9 months once between boots, running http (6 geeklog sites) and mail servers.

Have you looked at the number of server instances running, etc?
 Quote

Status: offline

Remdotc

Forum User
Chatty
Registered: 06/12/02
Posts: 55
If you are using an RPM version of Apache / PHP, you may want to compile your own. Usually RPM distros are "best fit" and "most generic" pre-compiles and whilie they work most of the time, in production they tend to have issues.

As for trouble shooting your mem leak, I would first take a look at what services you have running, turn off any services and modules not needed. Finally, check out your php.ini and make sure its the latest version, along with apache and mysql.

If you still are having problems after that, run a free unix utility called mem-check. You can google it. It will check the physical sectors of your hardwares memory, and report if any are bad, and possiblity block access to them.

If that still doesnt work, disable php-ads, see what happens.
If that still doesnt work, you may need to recompile your own webserver as a needed file may be broken
 Quote

Status: offline

Marites

Forum User
Chatty
Registered: 02/04/04
Posts: 64
We built our own Apache, MySQL and PHP rather than using the RPM's. We have switched usages of MySQL off in bits and pieces including moving the phpadsnew to another rack.

When we reboot we have been starting 1 of the instances of GL running for 30 minutes, rebooting then starting a different instance of GL, switching of everything else no needed including Sendmail.

In each of our tests we have found the one instance builds up memory usage quite fast, the same in fact for each instance tested. When rebooting and starting 2 or three of the instances at the same time the memory build up is that much faster.

IBM who donated the unit are coming to check the hardware part and switch out the memory tomorrow.

At the moment it does look as if it is our install of GL that is the problem 3 of the site add 200 new items a day and receive about 10,000 visitors a day, the other sites are a 'little' quieter.

The server is managed by Verio-ATT and is only two months old, both of the Xeon's have been changed and senior engineers have examined the software installation .... we are all baffled.

I appreciate the responces from all - I do ot think it is a flaw with GL let me make that clear I feel it is some interaction with GL our software and hardware.

Regards

- Marites -
 Quote

Steve

Anonymous
Marites,

it could simple be normal cache memory grabbing all it can. How are you "reporting" that memory is nearly full?
What's swap use like? - if this is nearly untouched, you don't have a problem.

Have a look at tools like orca, vmstat and friends

You mention the problem is apparently with MySQL - on what basis is that claim made? I'd be loking at hard at your mysql config. You may have simply allowed it to grab as much memory as it can.


Cheers!


- Steve
 Quote

Status: offline

Marites

Forum User
Chatty
Registered: 02/04/04
Posts: 64
Steve the swap is also being eaten

We have been watching with ps and various other tools - I will hold tight until the memory is swapped out tomorrow and see if that makes any difference.

Many thanks

- Marites -


Quote by Steve: Marites,

it could simple be normal cache memory grabbing all it can. How are you "reporting" that memory is nearly full?
What's swap use like? - if this is nearly untouched, you don't have a problem.

Have a look at tools like orca, vmstat and friends

You mention the problem is apparently with MySQL - on what basis is that claim made? I'd be loking at hard at your mysql config. You may have simply allowed it to grab as much memory as it can.


Cheers!


- Steve
 Quote

Status: offline

Marites

Forum User
Chatty
Registered: 02/04/04
Posts: 64
We have isolated the problem to 2 sites running GL 1.3.8.x we are not sure if this is an issue relating to the versions of GL/MySQL/PHP in combination or individual problem within the combination.

We have turned off these two instances prior to upgrading to 1.3.9.x, which the rest of the site use with no severe problems hopefully that will cure the problem.

For some reason on our server the 1.3.8 versions were using 3 times as many resources as the 1.3.9.

I have no idea where to try and track down this issue and pinpoint exactly what is/was causing it but if we find anything more I will report back.

- Marites -
 Quote

Status: offline

vinny

Site Admin
Admin
Registered: 06/24/02
Posts: 352
Location:Colorado, USA
Quote by Marites:
For some reason on our server the 1.3.8 versions were using 3 times as many resources as the 1.3.9.


It might have something to do with the fact we made some major improvements to how Geeklog queries the Database in 1.3.9 (by reducing DB calls and making the remaining ones more efficient). More of these (though probably not as significant) will be coming in 1.3.10.

You're memory problems sound like a MySQL caching/buffer issue. Check your config options looking specifically at places that deal with memmory. Check out mysql config options for more information.

Good Luck,
Vinny
 Quote

Status: offline

Marites

Forum User
Chatty
Registered: 02/04/04
Posts: 64
freakingout
We have had the server checked inside out by an IBM, RedHat and network engineer.

MySQL checks out fine we have the correct configs in my.cnf applicable for the memory we have.

It does sadly seem as if there is something in our hardware / applications / Geeklog that is not compatible. Everyone except our group is pointing the finger at GL

To reiterate we are running on an IBM x306 server with 3.02 GHz processor, normally 1 Mb RAM, (currently trying more), with RedHat Enterprise 3 Server, latest php, mysql.

I should say the memory has been replaced, along with the CPU - It is not so bad with 1.3.9 as it was with 1.3.8 but it is bad enough to cause concern. This is what mysql is using at this minute ...

1821 mysql 15 0 15540 15M 1944 S 0.4 1.5 0:14 0 mysqld

Just don't know where else to look.

Marites

Quote by vinny:It might have something to do with the fact we made some major improvements to how Geeklog queries the Database in 1.3.9 (by reducing DB calls and making the remaining ones more efficient). More of these (though probably not as significant) will be coming in 1.3.10.

You're memory problems sound like a MySQL caching/buffer issue. Check your config options looking specifically at places that deal with memmory. Check out mysql config options for more information.

Good Luck,
Vinny
 Quote

Status: offline

Remdotc

Forum User
Chatty
Registered: 06/12/02
Posts: 55
I suggest adding more memory, as its cheap. 1 gig of mem sounds like a lot, but in a application server roll, thats nothing (I have a gig of mem in my laptop)
Really, if you have extensive use on that box, up the memory.

Second, if your IBM box is SMP, You may want to check their site for updates, as i have had experience with IBM 3XX SMP boxes and redhat choking, for simular issues.

Finally, if you have custom coded your 1.3.8 plugins. Check php. there was a bug with my_sql_connect, or simular statement , last year. Had everything to do with leaving connections open. (thus chewing up memory)


 Quote

Status: offline

Marites

Forum User
Chatty
Registered: 02/04/04
Posts: 64
Hi thanks

We have exclusive and etensive use of the box. We have actually asked the server managers earlier today to upgrade the RAM to 2 Gb. It does take a maximum of 4 so still have leeway yet.

We are finding the sql_connect is effecting 1.3.8 and 1.3.9 but strangely it seems to have been closed in 1.3.10. Chancing our arm we have put 1.3.10 rc2 on a high usage site and the difference was noticable immediately with the amount the same site was using on 1.3.9.

So it looks like the problem may go away when 1.3.10 is released.

Many thanks

Marites

Quote by Remdotc: I suggest adding more memory, as its cheap. 1 gig of mem sounds like a lot, but in a application server roll, thats nothing (I have a gig of mem in my laptop)
Really, if you have extensive use on that box, up the memory.

Second, if your IBM box is SMP, You may want to check their site for updates, as i have had experience with IBM 3XX SMP boxes and redhat choking, for simular issues.

Finally, if you have custom coded your 1.3.8 plugins. Check php. there was a bug with my_sql_connect, or simular statement , last year. Had everything to do with leaving connections open. (thus chewing up memory)


 Quote

All times are EDT. The time is now 01:28 pm.

  • Normal Topic
  • Sticky Topic
  • Locked Topic
  • New Post
  • Sticky Topic W/ New Post
  • Locked Topic W/ New Post
  •  View Anonymous Posts
  •  Able to post
  •  Filtered HTML Allowed
  •  Censored Content