One improvement I made today to speed up spidering website domains, is looking up the domain for a DNS-A entry. Using Curl or Guzzle is slowed down when the website has no entry in the DNS. So i started with the basic php function dns_get_record, however this isnt very reliable as it often caused an error. The '@' sign is to prevent errors, but there are known bugs with this function. The DNS_ALL option does seem to help, but it is a lot slower than just the A record look up.

    $result = @dns_get_record($domain, DNS_A);

So I decided to use the Linux Command Line tool dig which has proved much more reliable.

try {
    $result = shell_exec("dig +short ". $domain . ' A');
} catch (Exception $e) {
    break;
}

if (isset($result)) {
    echo " exists" . PHP_EOL;
} else {
    echo "NO dns: " . $domain. PHP_EOL;
}

This way I can check domains quickly and efficiently and mark those that dont work without wasting time trying to index a website which hasnt even got a DNS entry.

If you would like to contact me with this form on londinium.com, ilminster.net or via Twitter @andylondon