Today I decided to make screenshots of webpages with spatie/browsershot

To install I ran:

composer require spatie/browsershot

I also ensured that Puppeteer and chromium-browser were installed, on debian I used.

export PUPPETEER_SKIP_DOWNLOAD='true'
npm install puppeteer --global
apt install -y chromium-browser

The code is fairly straightforward, I added this to my existing spider and it worked.

use Spatie\Browsershot\Browsershot;
    try {       
        Browsershot::url($url)
        ->timeout(120)
        ->setNodeBinary('/usr/bin/node')
        ->setScreenshotType('jpeg', 100)
        ->save("/images/".$id.".jpg");
    } catch (\Exception $e) {
        info($e->getMessage());
    }

one idea I had is to change the user agent to

    ->userAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36')

I added it to a guzzle based spider, with a payload of 416 websites. This made 343 screenshots and the guzzle spider only 329! A number of these are 404 errors (not found) that I will fix and run it again, but it was nice to getting it work better than guzzle.

Now there is the issue of the cookie popups and dealing with them....

If you would like to contact me with this form on londinium.com, ilminster.net or via Twitter @andylondon