Yesterday, I added Google Adsense to londinium.com which has changed quite a bit since I last used it. Part of the process ended up with me in the Google Webmaster Tools, another area I have neglected, but a useful tool for SEO (Search Engine Optimisation) purposes.

One of the things it suggested was to create, was a sitemap.xml file for all the links in the site. Using Laravel, there are often a number of tools to complete any job, and I soon found and setup github.com/spatie/laravel-sitemap as a command line tool. However this is a tool that is going to crawl the entire website and with over 2000 routes, is a slow way to build this sitemap.

Google allows both .xml and .txt files according to the docs, see https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap

So I changed tack and went back to using AWK. With this one script, I can extract all the routes and prefix them with the domain name and make the first one of these from the routes/web,php file with awk

awk -F"'" '$1 ~ /^Route::/ {print"http://londinium.com"$2}' routes/web.php > sitemap.txt

The second part is to create a list of the pages from the database. I have a table 'points' where I import OSM data which has a website. so I created an artisan console command which outputs these pages with the command php artisan sitemap:generate > sitemapNWR.txt. It limits the select to 50000 as that is a limit imposed by Google. The commented out command will offset that query by 50,000.

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;
use App\Models\Points;

class GenerateSitemap extends Command
{
    /**
     * The console command name.
     *
     * @var string
     */
    protected $signature = 'sitemap:generate';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Make a sitemap of all the db pages from the points table.';

    /**
     * Execute the console command.
     *
     * @return mixed
     */
    public function handle()
    {
        $points = Points::limit(50000)->get();
        //$points = Points::skip(50000)->take(50000)->get();

        $points->each(function ($point) {
            echo "http://londinium.com/". $point->nwr. "/" .  $point->id;
            echo PHP_EOL;
        });
    }
}

It is now 2 simple commands, which i can upload to Google Webmaster Search Console.

Another job done!

If you would like to contact me with this form on londinium.com, ilminster.net or via Twitter @andylondon