Subject: New Version of Visitor Stats almost ready.

Posted on: 03/09/03 02:27pm
By: tomw

At Simon's prodding I almost have a new version of the Visitor Stats Plugin ready for the world. I am doing final testing and need to update the install and docs. Here is what I have changed/added.

1) To make it work on high volume sites.
Added several indexes to speed things up.
Broken several of the slow pages up to make them faster.
(Tested on one of my sites that has about 10,000 visitors and 40,000 page views per month with 9 months of data).
2) Fixes
Worked on the permissions so other than 'root' can access stats
Fixed JLH's lookup feature at his suggestion.
Made it so user totals are more accurate
3) Additions
Added ignore pages like ignore users and ips
Added rudimentary net block ignore
Provide list of popular spider ip's you can import
Provide sql to clean you database of spider accesses (too slow for web page).

If you have any burning things that need fixed or added, speak up now or hold your peace.

TomW

Re:New Version of Visitor Stats almost ready.

Posted on: 03/09/03 05:20pm
By: samstone

Good to see you back Tom. It has been a while since you last made the oink that blesses the Geeklog community. If it is at all possible. A feature that deletes the old stats would be great. Or may be a backup system that ignore the stats. The reason is because last time when I restore a site, 95% of my bauckup data belonged to the vistor stat. Sam

Re:New Version of Visitor Stats almost ready.

Posted on: 04/09/03 06:56am
By: Robin

Is it possible to exlude bots from stats. Like e.g. crawl24.googlebot.com they have new ips all the time.

Re:New Version of Visitor Stats almost ready.

Posted on: 04/09/03 09:48am
By: Anonymous

Deleting stats by month would be easy -- I will put that in.

Blocking bots is already included by the ability to block ranges. I have included the most common bot ip ranges that you can import.

TomW

Re:New Version of Visitor Stats almost ready.

Posted on: 04/09/03 11:21am
By: bcbrock

An addition to ignore specific referrers would be nice. Especially with the Chatterblock installed, each time it refreshes, visitor stats makes a new entry.

Re:New Version of Visitor Stats almost ready.

Posted on: 04/09/03 07:04pm
By: Anonymous

Hi Tom, If you need a french (canadian) translation of the language file I\'m available! A module to select data within a date range and export it to CSV or any other delimited text file that could be imported in Excel or Access would be nice. This selection could also be \"dropped\" to reduce the database size. And why not use \"gzip\" before sending the file ! Presently to speed up my sites I dump the data in a text file using phpMyAdmin and load it in Excel and then drop the selected data, so I have a \"zipped\" backup of all the database since the beginning, and a smaller and faster site. I\'m looking forward to this new version. If you need my email add. let me know. Alain Ponton

Re:New Version of Visitor Stats almost ready.

Posted on: 06/09/03 11:41pm
By: Anonymous

Would it be possible to show what each registered user did? Like, show the URLs that he visited? I\'m guessing no, but just thought I\'d ask. Also, another thing I\'d like to see would be the amount of hits an article got that day. Just so I know that any of my obscure articles may be getting hits.

Re:New Version of Visitor Stats almost ready.

Posted on: 07/09/03 01:47pm
By: samstone

Arn\'t these already part to your current Visitor Stats? Mine is working that way. It records everyone, registered or not, regarding the pages the person visited and the url that s/he came from. And the articles s/he read. Sam

Re:New Version of Visitor Stats almost ready.

Posted on: 07/09/03 04:20pm
By: geKow

I\'m sure you noticed, that the \"GeekTools Whois Proxy\" link (the one clicking on an IP links to) has changed? Greetings, Georg

Re:New Version of Visitor Stats almost ready.

Posted on: 07/09/03 08:23pm
By: Anonymous

No my stats don\'t list the articles people read. For example, I\'d like to see a list of articles/pages Username1 accessed that day.Is that implemented because its not shown up in mine

Re:New Version of Visitor Stats almost ready.

Posted on: 08/09/03 12:44pm
By: tomw

[QUOTE BY= bcbrock] An addition to ignore specific referrers would be nice. Especially with the Chatterblock installed, each time it refreshes, visitor stats makes a new entry.[/QUOTE] Hmm, see what you mean here. Then you would have to enter your own site. It seems that then you would loose all access after a visitor hit your site. This is probably not what you want. Chatterblock does not make a unique url with its refreshes. Don\'t see how I could implement this. TomW

Re:New Version of Visitor Stats almost ready.

Posted on: 08/09/03 12:53pm
By: tomw

[QUOTE BY= Alain] Hi Tom, A module to select data within a date range and export it to CSV or any other delimited text file that could be imported in Excel or Access would be nice. This selection could also be \"dropped\" to reduce the database size. And why not use \"gzip\" before sending the file ! Presently to speed up my sites I dump the data in a text file using phpMyAdmin and load it in Excel and then drop the selected data, so I have a \"zipped\" backup of all the database since the beginning, and a smaller and faster site. [/QUOTE] I have looked into exporting the data and after seeing all the problems that a simple thing like backups has caused the Geeklog Team, I have shelved the idea. For those who want to it is a simple thing to use phpmyadmin or just the command line to save the data and then delete it from the database. Maybe someday when I get brave I will do this. With the new version I am running a site with a 25mb userstats file that is working ok. I will contact you Alain to update the translation when I am done. TomW

Re:New Version of Visitor Stats almost ready.

Posted on: 08/09/03 12:57pm
By: tomw

[QUOTE BY= Cheese] Would it be possible to show what each registered user did? Like, show the URLs that he visited? I\'m guessing no, but just thought I\'d ask. Also, another thing I\'d like to see would be the amount of hits an article got that day. Just so I know that any of my obscure articles may be getting hits.[/QUOTE] Both of these are already in the current version. From the main index page click on an individual month to show the days. From the day screen you can click on the Registered Visitors link for each to see what each visitor did. From the day screen click on the Pages Viewed link for each day to see how many times each pages was viewed that day. Or from the month screen click on the Pages Viewed link to see how many times each page was viewed that month. Click around nearly all the items are clickable to get details. TomW

Re:New Version of Visitor Stats almost ready.

Posted on: 08/09/03 12:59pm
By: tomw

[QUOTE BY= geKow] I\'m sure you noticed, that the \"GeekTools Whois Proxy\" link (the one clicking on an IP links to) has changed? Greetings, Georg[/QUOTE] Yeah you can change the config.php variables in current version even to $_ST_Whois_URL_start = \'\'; and it will work again. TomW

Re:New Version of Visitor Stats almost ready.

Posted on: 08/09/03 09:16pm
By: Anonymous

Hi Tom It would be great if we were able to do a search (in a day, week or month) within the stats ie. finding a particular IP, or a particular user who assessed the site. Or for that matter, find the number of people or individuals assessing a particular URL or page. I know the data is there already. Trying to check who assessed is really time consuming because we cannot sort the info by IP or user or page. or categorize them within the day, week or month. In my opinion, it would be great if there is such a facility. Thanks

Re:New Version of Visitor Stats almost ready.

Posted on: 08/09/03 11:30pm
By: Anonymous

Sorry to keep bugging, but was just wondering if you had any idea when it would be released? I\'ve had to re-install a couple of times already, and am now getting 1034: Incorrect key file for table: \'gl_userstats\'. Try to repair it, so instead of the old un-install re-install biz, thought I might wait and see if you have the sucker ready. Lemme know :-)

Re:New Version of Visitor Stats almost ready.

Posted on: 08/09/03 11:37pm
By: Anonymous

BTW, thanks I never knew I could see what my readers were doing. Thank you very much. I love your plugin chief!

Re:New Version of Visitor Stats almost ready.

Posted on: 09/09/03 12:53am
By: Agent X20

I'd be quite keen to see the feature that Anon:Cheese is talking about... ie ability to filter the page views by a user so you can see a user's recent activity. This would list all of their page views (across all sessions) say within the month being viewed.

Also the initial stats page that lists the months (/vstats/index.php) should probably be listed in date descending order rather than alpha. At the moment I have:

August 03
July 03
June 03
Sep 03

... which is obviously a little unintuitive.

Great package btw!

Re:New Version of Visitor Stats almost ready.

Posted on: 09/09/03 10:19am
By: tomw

[QUOTE BY= Barrywong] Hi Tom It would be great if we were able to do a search (in a day, week or month) within the stats ie. finding a particular IP, or a particular user who assessed the site. Or for that matter, find the number of people or individuals assessing a particular URL or page. I know the data is there already. Trying to check who assessed is really time consuming because we cannot sort the info by IP or user or page. or categorize them within the day, week or month. In my opinion, it would be great if there is such a facility. Thanks [/QUOTE] Hi Barry, The ability to sort is already there. Just click the column head you want to sort by. Click again to sort the opposite way. I get the feeling many of you do not know half what the plugin will do. Like I have said almost everything is clickable including headings. Experiment a little. TomW

Re:New Version of Visitor Stats almost ready.

Posted on: 09/09/03 10:22am
By: tomw

[QUOTE BY= Agent X20] I\'d be quite keen to see the feature that Anon:Cheese is talking about... ie ability to filter the page views by a user so you can see a user\'s recent activity. This would list all of their page views (across all sessions) say within the month being viewed. Also the initial stats page that lists the months (/vstats/index.php) should probably be listed in date descending order rather than alpha. At the moment I have: August 03 July 03 June 03 Sep 03 ... which is obviously a little unintuitive. Great package btw![/QUOTE] The month sort is fixed in the new version. Right now I am down to final testing on different websites and updating the docs. If I had nothing else to do today I could get it out today, but I have lots to do today and will probably not have a chance to do much. Hopefully within a week it will be ready. TomW

Re:New Version of Visitor Stats almost ready.

Posted on: 09/09/03 01:31pm
By: tomw

[QUOTE BY= TomW] Deleting stats by month would be easy -- I will put that in. Blocking bots is already included by the ability to block ranges. I have included the most common bot ip ranges that you can import. TomW[/QUOTE] Unfortunately it also is not a quick operation on large sites. In my testing it took up to 4 minutes. That is way beyond the time out for php pages. I will include the sql to do this you can paste in yourself. TomW

Re:New Version of Visitor Stats almost ready.

Posted on: 10/09/03 09:34am
By: Anonymous

Hello, Me back with some suggestions again Smile I’m excited for the plugin and can’t wait. How about each day, a list of informative stats on the main stats page, that may or may not be able to be put on your website, that are updated during the day. For example: The most viewed article/page during the day was ________, which was viewed ____ times. The most active user today was _____, who viewed ____ different pages. Etc. Even have a “top ten” listing of most active users that day. It’d just be cool for me to see who was active, and be able to just click their usernames, and see where they’d been in an instant. This would be very handy for me especially, as I have over 5000 registered users, 3000 of whom visit each day. I’d like to be able to keep a track on the “newbies” better, to see if they sign up and read one article, or if they read everything or whatever. Also, would it be possible to have some of this information in the user profiles? Or, at least a button in there that lets you see the stats? For example, for user “Blaine”, I click on his profile and it’s the normal bla bla bla. But then I click another button “visitor stats” or whatever, and it takes you to a comprehensive list of what he’s basically done on the site for that day, or even better for the month. I just think that’d be a really cool option. Also, if based on the stats compile some comprehensive stats, for example take in the browsers that people use, the operating system, resolution etc, and output it in one handy graph/chart. It’d be cool to see that 56% of my users used 800x600 screen resolution, so I’d have to make sure the site looked good in that view too etc etc. Just some ideas to keep you busier than normal Smile

Re:New Version of Visitor Stats almost ready.

Posted on: 10/09/03 11:24am
By: tomw

[QUOTE BY= Cheese] Hello, Me back with some suggestions again Smile I’m excited for the plugin and can’t wait. How about each day, a list of informative stats on the main stats page, that may or may not be able to be put on your website, that are updated during the day. For example: The most viewed article/page during the day was ________, which was viewed ____ times. The most active user today was _____, who viewed ____ different pages. Etc. Even have a “top ten” listing of most active users that day. It’d just be cool for me to see who was active, and be able to just click their usernames, and see where they’d been in an instant. This would be very handy for me especially, as I have over 5000 registered users, 3000 of whom visit each day. I’d like to be able to keep a track on the “newbies” better, to see if they sign up and read one article, or if they read everything or whatever. Also, would it be possible to have some of this information in the user profiles? Or, at least a button in there that lets you see the stats? For example, for user “Blaine”, I click on his profile and it’s the normal bla bla bla. But then I click another button “visitor stats” or whatever, and it takes you to a comprehensive list of what he’s basically done on the site for that day, or even better for the month. I just think that’d be a really cool option. Also, if based on the stats compile some comprehensive stats, for example take in the browsers that people use, the operating system, resolution etc, and output it in one handy graph/chart. It’d be cool to see that 56% of my users used 800x600 screen resolution, so I’d have to make sure the site looked good in that view too etc etc. Just some ideas to keep you busier than normal Smile [/QUOTE] I already capture the browser and operating system, look on the main index page for that info. The other info you were wanting is easy to do but I would implement it as a seperate page (static with php) because it would really slow down the webpage load times to do it as a block. Here is the sql needed to get these totals. Why don\'t you or some industrious person make the page and donate it back to the community. \'most popular pages by month select count(*) as cnt, page from gl_userstats where monthn=MONTH(CURDATE()) and year=YEAR(CURDATE()) group by page order by cnt desc limit 10 \'most popular pages by day select count(*) as cnt, page from gl_userstats where monthn=MONTH(CURDATE()) and year=YEAR(CURDATE()) AND day=DAYOFMONTH(curdate()) group by page order by cnt desc limit 10 \'most active user by month select count(*) as cnt, uid from gl_userstats where monthn=MONTH(CURDATE()) and year=YEAR(CURDATE()) group by uid order by cnt desc limit 10 \'most active user by day select count(*) as cnt, uid from gl_userstats where monthn=MONTH(CURDATE()) and year=YEAR(CURDATE()) AND day=DAYOFMONTH(curdate()) group by uid order by cnt desc limit 10 All these queries return an array with the top ten sorted in descending order. \'to see what a particular user did on a day (change uid to match) select count(*) as cnt, page from gl_userstats where monthn=MONTH(CURDATE()) and year=YEAR(CURDATE()) AND day=DAYOFMONTH(curdate()) and uid = 1 group by page, uid order by cnt desc limit 10 Now you have the hard part. TomW

Re:New Version of Visitor Stats almost ready.

Posted on: 14/09/03 02:49pm
By: odDonnellE

any plans to include the \"Show Online Status\" features in the visitor stats block as well? Meaning, not show people that choose not to be shown? -adam

Re:New Version of Visitor Stats almost ready.

Posted on: 14/09/03 11:35pm
By: tomw

Just an update on the visitor stats progress. It is almost ready. Some testing is being done by others and I am waiting their results. I did figure out an easy way to do caching and implemented it. On stats pages for which the current days totals are not important, stats caches the results and gives this cached version the next access. This greatly speeds up browsing and reduces the load on the webserver. The pages are only cached when you browse, so only pages you have looked at are in the cache. I looked into caching each page, but the time required to generate all those pages and the load it would put on your server for that time were too great, so I abandoned that idea. (There would be a minimum of 300 pages per month). TomW

Re:New Version of Visitor Stats almost ready.

Posted on: 16/09/03 11:14am
By: tomw

I think the new version of visitor stats is ready. I have incorporated as many of the suggestions as was possible. I need a few experienced users to test the install, upgrade, new features and docs, before I release it out into the wild. If you are interested drop me an email and I will send you a copy. TomW

Re:New Version of Visitor Stats almost ready.

Posted on: 17/09/03 11:42am
By: Anonymous

[QUOTE BY= tomw] I think the new version of visitor stats is ready. I have incorporated as many of the suggestions as was possible. I need a few experienced users to test the install, upgrade, new features and docs, before I release it out into the wild. If you are interested drop me an email and I will send you a copy. TomW[/QUOTE] Could you please e-mail me at g69@kingston.net with this plugin? I e-mailed you but not suer if you got it. Thanks

Geeklog - Forum
https://www.geeklog.net/forum/viewtopic.php?showtopic=24357