.: BiTLOG Realtime Statistics :.

Author: Albert

Download: BiTLOG 1.56a [ 30-10-2007 ] 259.8 kB                 Globals OFF compat.     PHP5.x compat.   Download: IP to Country table [ 11-2-2008 ] 554.6 kB   Download: Country Flags [ 26-11-2006 ] 88 kB   Download: Robots [ 3-9-2007 ] 217.7 kB   Download: Bitlog Bandwidth [ 3-9-2007 ] 24.3 kB

Before using read the licence agreement go there also for commercial purchase of .
If you use BiTLOG for non-commercial purpose you can make a donation here

I wanted to know to which pages my visitors went, where they came from and how many times they return and in realtime.
Also the countries of origine (including com/net/edu and so on) and which bots went over my site. Should make use of PHP but be suitable for html pages as well.
First i tried it as an extension package for Les Visiteur, but the change was to drastic.
That is how i created the current program.
IP-to-Country table / Flags/Robots are an integral part of BiTLOG. It will not work when one of them is missing! The graphical statistics require GDlib to be compiled with PHP.

Latest changes since december 2004

Version 1.49
Build an extra section to defeat referer spam. This is trying to reach the top 10 in referers in
the statistics. Now has 'badreferers.php' to which you can add their names. Access will be denied.
Also implemented rel=nofollow for urls in day.php. Now bots who adhere to this new standard will
not follow the links. This is needed when you have day.php public, like mine.
Removed an error in the admin section of load_dbf.php.

Version 1.50
Added average totals to daily/weekly/monthly. Changed checking searched done by search engines.
Now displays the number of searches, so you get the top 25 searches.
Some layout changes and minor updates to speed up display.
Revised generation of GD lib img for lastvist.php
Changes in bit_include/bit to properly register new robots.
Overhaul of urlinfo.php, now also gives the referring website, so you can
drill right to any info you want on a site url.
Overhaul of queries.php thanks to a modification by Steve Malone.
Now you get the totals of the top 25 queries in a period.

Version 1.51
The changes are in the administration part. Added two separate scripts:
Add-ip.php and Clean-ip.php.
Clean ip will reduce the ip-to-country table (ips) dramtically and speed up BiTLOG.
Displays like day.php are speeded up because adding the countries is a lot faster.
Around 50% of the entries in ips can be combined into bigger ranges.
Add-IP gives the possibility to add ranges if you get unknown ip's.
The links are provided for ip checking in whois. If already existing, the range can be
changed for a different country if the country was wrong.
Direct ip2country check is now included, enter an ip, you get the country
Bandwidth for donloaded files is now included in the bandwidth package.
I have changed the rotation and included a testfile, so a double start is now impossible.

Version 1.52
This is a service release, it contains minor changes.
Only Apache log rotation has been changed if you use the bandwidth package

Version 1.53
Has a small change in bandwidth table (if you use bandwidth package).
Has more complete display of Apache logs.
Deletion of data changed to deletion + archiving data.
Now you can delete old data, but still have them backed-up in a sql file.
For the rest some small errors have been taken out, and some cosmetic changes added.
Page.php had an error, this has been changed.Bbit_include.php and bit.php have been
changed to accomodate bandwith calculation on robots.

Version 1.53A -- IF YOU USE MYSQL 4.1+ and/or APACHE2
This version has a small change in day.php.
You can now click the flags to get a summary of visitors from that country.
This is mainly a service release for MySQL 4.1++.
MySql 4.1 is interpreting timestamp differently.
On a date like 12/6/2006 12:24:13 < 4.1 would give 20061206122413
4.1++ gives 2006-06-12 12:24:13 which trashed date reading.
This version contains a workaround, so everything works ok again.
Also APACHE 2 has a different layout of error-log this version has an
adaption for Apache2 error-log (bandwidth package).
It also works around a PHP bug in < 4.3.6 that throws a PHP error in the
Apache log when a PHP file is not found on the server.

Version 1.54 june 2006
Added TraceRoute to menu, now you can trace an ip from your server.
Also added trace route to ip's in access-log in bandwidth section.
Bits.php has been changed, now it can retrieve a referrer!
The input must be done via javascript like:
<script language=javascript>
document.write('<img src=../bit.php?ref=' + document.referrer + ' width=1 height=1 border=0>');
</script>
The old method with <img> alone does not work anymore.

Version 1.55 October 2006.
Some changes to errorlog.php to cope better with the php-leak.
Added whois and dig to the menu if yoy have accesss to system() they can be used directly.
A separate test program for the port is included.
Robots.zip now also contains badbods.php in which all bad robots/grabbers are stored.
Day.php contains returning visitors in the graphs.
Small changes to bit_include.php and bits.php.

Version 1.56 January 2007.
This is more or less a service release.
Took out errors in browserlist.php and bit_include with time difference.
Lots of robots added and a few browsers.
See also remarks on robots and browsers in the download.

Version 1.56a Jne 2007.
This is more or less a service release.
Took out error in graphics caching.
Made everything W3C HTML1 compliant. Some files reduce in size.

Plenty of statistic programs available, but non featuring it all.

So... that leaves create your own and thats what i did.
The total file size is big because of the country table.
Now how it is working in order to have an idea of the contents.

• What do we register and what we do:
• Refering web site / requested_uri of page (only available from PHP pages)
• Agent - name of browser/webcrawler with version number.
• Requested page (available from PHP and non-PHP pages)
• Set a cookie on first visit, cookie has unique number
• See whether cookie exsists if so use cookie number in statistics.
• Register time, requested page, refering page and ip address.
• Check browser agent, if it is a webcrawler, dont add to visitors, but to robots.
• If a robot register the entry page and date.
• If a spambot throw it out, no access.
• If ip is in the banlist no access either
• Bandwith calculations / statistics if you have access to Apache log file.
• Search phrases used to search your site.
• Browser detection to see any change in browser brand use
  
• Some functions ONLY with the use of bit_include not with bit

If we have all the data we want. The ip addresses are translated into long ip addresses.
Than compared to the country database and the country is known.

If this is known we can start creating the various statistics and graphics.
All the statistics is a good SQL excercise.

The country database is a database that changes and there are monthly updates at:
www.ip-to-country.com.
Disclaimer:This 'work' uses the IP-to-Country Database
provided by Directi (http://directi.com), available from http://ip-to-country.directi.com.",
or provided by webnet 77
http://software77.net/cgi-bin/ip-country/geo-ip.pl
Download the delimited version, than you can import it into the table.
First kick out the 3 letter country code it is not used.
Than add long ip 2130706433 and give it country code XX it is used for localhost.
Take care some countries contain an ' and this can cause trouble in MySql.
Better just replace them with a blank. Or download from this site all cleaned up.

How do you call the statistics......... there are 2 ways, a 1 for PHP and a non-PHP one.
It makes use of cookies, but the cookies are hidden in a transparent gif with a 1x1 pixel size , of the bit script, except bit_include.php which has to be included.
So you can use it on standard HTML pages as well as long as you have PHP installed on the server.

PHP
include("path-to-root/bit_include.php");
For cached and non PHP pages (zipped output no problem).
The above line should be the FIRST in your page before anything else.

Non-PHP (changed since version 1.54!)
<script language=javascript>
document.write('<img src=path-to-root/bit.php?ref=' + document.referrer + ' width=1 height=1 border=0>');
</script>
make sure bits.php / bit_include.php is in your root and has it contents properly changed.
(The require statement should contain the proper path to the other files).

If bit.php now gives the referer, only badbots can not be implemented as all is done in an 'image'
Agent from a bot is allways available.
Browsers info is retrieved from the agent and saved..
Same for operating systems are not saved. But if you want them its all in the agent.

To get a complete view you will have to put this bits - <img> or include bits in every page on your site.
Now you will know which pages are in the highest demand, and also how many times people come back, and you have and estimate how long they spent on the site.
Please note: The info from PHP pages contains the most data.(bit_include grabs robots 100%)
From HTML pages you cannot grab the refering page and the agent, thus no robots.

The bit.php file will store the data in the bitlog tables.
Install bit.php / bit_include.php in your root.

1. Create bitlog dir
2. Copy bits.php/bit_include to your root and edit path to \bitlog in all of them
3. Edit the bitlog.inc and the bitlog.admin.inc in the include dir.

Get the statistics with \bitlog\index.php
You get statistics of:
- 20 most active ip's/url's/referers
- the most requested pages with %
- how many pages did a visitor go to
- how many times did he come back
- statistics per day + graph
- statistics per week
- statistics per month
- visitors tracking per date (first/last visit) + pages visited
- countries/visitors per month + graph
- countries/visitors per year + graph
- all graphs are cached to avoid unneeded server load.
- which robot visited my site, which pages he went over
- quick request for visits per url
- browsers used by visitors
- jumpes to outside links
- traceroute info

NEW badbots, bit_include will kick email harvesters straight away off you site!
Also useronline/pageview and userwhere which shows the whereabouts of online users.
Banning certain ip's from your website (since version 1.42)

The BiTLOG scripts are kept as small as possible to avoid server load with a high number of visitors. Therefore no ip lookups are done.
This is only done when displaying statictics.
You can select the method of retrieving lookups.

What else can i use for ?
Since BiTLOG keeps track of all visits to all pages, you have a build in page counter.
If you want to use these data on your page, you can.
just use the following MySql statement:
Select count(*) from bitlog where referer LIKE('path and name of the page%')
Note:% is the wildcard character.
The only problem might be pages like index.htm, index.html, index.php and so on.
If the are referered as mysite/mydir/ and no index.** is mentioned because Apache does it.
You might have to add a search for LIKE('path without name') and add that to the previous results.
Look for the path like: '/path to the lib/' do not forget the '/' and NO % (because no wildcard)
The result contains the page visits.
Count is very fast and puts no strain on MySql.

Samples for creation of pageview count, users-online and last-visit are provided, they are a easy to include:
"Currently there are xx users online" and "This page has been viewed xxx times since xx/xx/xx."
"Your last visit was: xx/xx/xxxx", "Your first visit was: xx/xx/xxxx", "You visited this site xx times"
Pageview, can handle index pages so you can cover '/mydir/index.php' and '/mydir/' in an easy way.
See readme file in the package.

For a working preview (statistics of this site) go here.
Below a sample of a statistics screen:

The section for displaying the countries uses a big table. The delimited version and an inclusion program are enclosed.
The sql for the table is in the sql file.
All COM/NET/ORG/GOV and so on extensions are resolved!!
Download the table and open this file in WinWord, remove all " and save as txt file.
Upload to your server, use load_dbf to load data into the table. Name the txt file countries.txt.
An output sample of resolved countries is below:

  

You like this script? Rate it now!

Select RatingExcellentGoodFairPoorRubbish   Number of ratings: 191

Rating: 4.32

RateIt! V2.4