[SOLVED] identifying web crawlers / spiders by PHP scripting

vba_php · 11-15-2019, 08:23 PM

in the attached image, i'm looking at the "who's online" page. I have a php traffic report page for a customer that, when accessed, echoes out database data that has been stored by way of another php script that captures geoLocation data (ip address of ISP, referrer page, date/time of visit) using PHP global variables.

my question is - how does this forum script know the identity of the google spiders? in my traffic report, i am only capturing the ip address of the ISP as the identifying information. from what I understand, it's not possible to capture the actual location of the visitor, only the ISP's location. if I look up the ip address on an ip lookup website, i can see that it is a google spider, but can this be done through PHP scripting?

vba_php · 11-16-2019, 01:16 PM

if anyone is interested, I found the best solution (although not perfect) via other PHP websites and knowledgebases. Here is the solution:

The report page:

Code:

<?php


$dbHost = "localhost";
$dbName= "rptDatabase";
$dbUsername = "username";
$dbPassword = "password";

	$conn = mysqli_connect($dbHost, $dbUsername, $dbPassword, $dbName);

	if (mysqli_connect_errno()) {
		echo "Failed to connect to MySQL: " . mysqli_connect_error();
}
    $sql = mysqli_query($conn, "SELECT ip
                                     , host
                                     , page 
                                     , DATE_FORMAT(date, '%m/%d/%y') as date
                                     , TIME_FORMAT(time, '%T') as time
                                FROM tblTraffic 
                                ORDER BY date DESC, time DESC");
?>

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>TRAFFIC REPORT</title>
<style type="text/css">
    table  { font-family: verdana, sans-serif; font-size: 10pt; width: 100%; border-collapse: collapse; }
    th, td { width: 20%; text-align: center; padding: 8px; }
    th     { background-color: #396; color: #FFF; }
</style>
</head>
<body>
    <table border='1'>
        <tr>
            <th>VISITOR IP ADDRESS, ISP NAME</th>
            <th>VISITOR DOMAIN ADDRESS</th>
            <th>PAGE VISITED</th>
            <th>DATE</th>
            <th>TIME</th>
        </tr>
        
        <?php
            // printing table rows
            while($row = mysqli_fetch_row($sql)) {
                echo '<tr>'; 
                foreach ($row as $key => $col) {
					echo "<td>$col</td>";
            	}
            		echo '</tr>';
            }
        ?>
    </table>
</body>
</html>

To capture visitor information:

Code:

<?php

$dbHost = "localhost";
$dbName= "rptDatabase";
$dbUsername = "username";
$dbPassword = "password";
$ip = gethostbyaddr($_SERVER['REMOTE_ADDR']);
$host = $_SERVER['HTTP_HOST'];
$page = isset($_SERVER['PHP_SELF']) ? $_SERVER['PHP_SELF'] : '';
$date = date("Y/m/d");
$time = date("H:i:s", strtotime("+19 hours"));

	mysqli_report(MYSQLI_REPORT_ERROR|MYSQLI_REPORT_STRICT);
	$conn = new mysqli($dbHost, $dbUsername, $dbPassword, $dbName);

	$stmt = $conn->prepare("INSERT INTO tblTraffic (ip, host, page, date, time) 
						  	VALUES (?, ?, ?, ?, ?)");
	$stmt->bind_param("sssss", $ip, $host, $page, $date, $time);
	$stmt->execute();
	$stmt->close();
	$conn->close();

?>