Web scraping with PHP in 2026: Guzzle, DomCrawler, headless browsers and Laravel queues

Web scraping with PHP in 2026 comes down to a small, reliable stack: Guzzle (or Laravel's HTTP client) to fetch pages, Symfony DomCrawler with CSS selectors to extract data, a headless browser only when a page truly needs JavaScript, and a queue to run it politely on a schedule. The code is the easy part. What separates a scraper that runs for years from one that breaks next week is rate limiting, retries, deduplication, monitoring, and checking first whether you should scrape at all. This guide covers all of it with code tested on current package versions.

Quick answer: the stack for web scraping with PHP

If you used Goutte in the past, it is abandoned. Its role is covered by Symfony's BrowserKit HttpBrowser or by the Guzzle plus DomCrawler setup below.

Before you scrape: API first, robots.txt and terms

Before any web scraping with PHP, check for an official route. Many sites offer a public API, a data export, an RSS or Atom feed, a sitemap, or a partner program. An API gives you stable fields, documented limits and permission. A scraper gives you HTML that can change any day.

Then check the rules:

• robots.txt. The Robots Exclusion Protocol, RFC 9309 (opens in a new tab), defines how crawlers should read robots.txt. The RFC itself notes that these rules are not a form of access authorization, so obeying them is a matter of good conduct, not a legal permission slip. Respect Disallow lines and any Crawl-delay for your user agent.
• Terms of service. Many sites restrict automated access in their terms. Read them, especially for anything behind a login.
• Personal data. Scraping names, emails or phone numbers brings in data protection law, such as the GDPR in the EU or India's Digital Personal Data Protection Act. The bar for lawful processing is high.
• Copyright and database rights. Facts like prices are treated differently from articles, photos or reviews, and the rules vary by country.

This is not legal advice. If the data matters to your business, ask a lawyer in your jurisdiction before you build. And never scrape behind a login, bypass a CAPTCHA, or work around blocking that a site has put in place on purpose.

Step 1: install the packages

composer require guzzlehttp/guzzle symfony/dom-crawler symfony/css-selector spatie/robots-txt

Composer picks versions that match your PHP. On PHP 8.3 this installs Guzzle 8.3 and DomCrawler 7.4. symfony/css-selector is what lets DomCrawler's filter() accept CSS selectors; without it you are limited to filterXPath().

Step 2: fetch pages with Guzzle, timeouts and retries

A scraper's HTTP client needs three things: an honest user agent, timeouts so one slow page does not hang the job, and retries for temporary failures only.

<?php

declare(strict_types=1);

require __DIR__ . '/vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\Exception\ConnectException;
use GuzzleHttp\HandlerStack;
use GuzzleHttp\Middleware;
use Psr\Http\Message\RequestInterface;
use Psr\Http\Message\ResponseInterface;

const USER_AGENT = 'AcmePriceBot/1.0 (+https://example.com/bot)';

$stack = HandlerStack::create();
$stack->push(Middleware::retry(
    // Decide: retry connection errors, 429 and 5xx, at most 3 times
    function (int $retries, RequestInterface $request, ?ResponseInterface $response = null, $reason = null): bool {
        if ($retries >= 3) {
            return false;
        }
        if ($reason instanceof ConnectException) {
            return true;
        }

        return $response !== null && in_array($response->getStatusCode(), [429, 500, 502, 503, 504], true);
    },
    // Delay in milliseconds: honor Retry-After, else 2s, 4s, 8s
    function (int $retries, ?ResponseInterface $response): int {
        $retryAfter = $response?->getHeaderLine('Retry-After') ?? '';

        return ctype_digit($retryAfter) ? (int) $retryAfter * 1000 : (2 ** $retries) * 1000;
    }
));

$client = new Client([
    'handler' => $stack,
    'timeout' => 20,
    'connect_timeout' => 5,
    'headers' => [
        'User-Agent' => USER_AGENT,
        'Accept' => 'text/html,application/xhtml+xml',
        'Accept-Language' => 'en',
    ],
]);

The retry middleware runs inside Guzzle's error handling, so it sees the raw 429 or 503 response before Guzzle turns it into an exception. A 404 is not retried: a missing page will not appear by asking again. The user agent names your bot and gives a URL where a site owner can find out who you are, which is the polite and transparent way to crawl.

Step 3: parse HTML with Symfony DomCrawler

Symfony DomCrawler turns HTML into something you can query with CSS selectors. Pass the page URL as the second argument so relative links resolve correctly.

This example uses books.toscrape.com (opens in a new tab), a sandbox site built for scraping practice, and follows pagination for three pages:

use Spatie\Robots\RobotsTxt;
use Symfony\Component\DomCrawler\Crawler;

// Read robots.txt once per host
$robotsResponse = $client->get('https://books.toscrape.com/robots.txt', ['http_errors' => false]);
$robots = new RobotsTxt($robotsResponse->getStatusCode() === 200 ? (string) $robotsResponse->getBody() : '');
$delaySeconds = max(1.0, (float) ($robots->crawlDelay('AcmePriceBot') ?? 0));

$url = 'https://books.toscrape.com/';
$books = [];
$pages = 0;

while ($url !== null && $pages < 3) {
    if (! $robots->allows($url, 'AcmePriceBot')) {
        echo "Blocked by robots.txt: {$url}\n";
        break;
    }

    $html = (string) $client->get($url)->getBody();
    $crawler = new Crawler($html, $url);

    $rows = $crawler->filter('article.product_pod')->each(function (Crawler $card): array {
        $link = $card->filter('h3 a');

        return [
            'title' => $link->attr('title'),
            'url' => $link->link()->getUri(),
            'price' => (float) preg_replace('/[^0-9.]/', '', $card->filter('.price_color')->text()),
            'in_stock' => str_contains($card->filter('.availability')->text(), 'In stock'),
        ];
    });

    if ($rows === []) {
        throw new RuntimeException("No products found on {$url}: the markup may have changed.");
    }

    array_push($books, ...$rows);
    $pages++;

    $next = $crawler->filter('li.next a');
    $url = $next->count() > 0 ? $next->link()->getUri() : null;

    usleep((int) ($delaySeconds * 1_000_000));
}

printf("Scraped %d books from %d pages\n", count($books), $pages);

Run against the live sandbox, this prints "Scraped 60 books from 3 pages", with absolute product URLs and prices as floats. A few DomCrawler details worth knowing:

• text() normalizes whitespace by default. attr('title') returns null if the attribute is missing.
• link()->getUri() turns a relative href into an absolute URL, using the base URL you passed to the constructor.
• text() throws if the selector matched nothing. For optional fields, check count() first or pass a default: text('').
• The "no products found" check matters more than it looks. When a site changes its markup, a scraper without it silently saves empty results.

Step 4: rate limiting, caching and being a good citizen

PHP web scraping gets you blocked, and burdens small sites, when it requests too much too fast. Basic rules I follow:

1. One request at a time per host, with a delay of at least one second, longer if Crawl-delay says so. Concurrency across different hosts is fine.
2. Back off on 429 and 503. Honor Retry-After. If errors keep coming, stop the run and alert someone instead of hammering.
3. Only fetch what changed. Store ETag and Last-Modified headers and send If-None-Match or If-Modified-Since next time. A 304 Not Modified costs the site almost nothing.
4. Prefer sitemaps and listing pages over crawling every link. They tell you what exists and often when it changed.
5. Run off-peak for the target site's time zone.
6. Keep the raw HTML for a while, so a parsing bug means re-parsing, not re-downloading.

JavaScript-rendered pages: Panther or Browsershot

If the HTML you fetch is an empty shell and the data appears only in the browser, first open your browser's developer tools, go to the Network tab and reload. Very often the page loads its data from a JSON endpoint. Calling that endpoint (within the site's terms) is faster and more stable than rendering the page. If there is no such endpoint, use a real browser.

Symfony Panther drives Chrome or Firefox through WebDriver and returns a DomCrawler-compatible crawler:

composer require symfony/panther dbrekelmans/bdi
vendor/bin/bdi detect drivers

use Symfony\Component\Panther\Client;

$client = Client::createChromeClient();
$client->request('GET', 'https://example.com/products');

// Wait until the JavaScript has rendered the product cards
$crawler = $client->waitFor('.product-card');

$titles = $crawler->filter('.product-card h3')->each(fn ($node) => $node->text());

$client->quit();

Spatie Browsershot uses Node.js and Puppeteer to render the page and hands you the final HTML. Its v5 docs list Node 22 or newer and Puppeteer 23 or newer:

use Spatie\Browsershot\Browsershot;
use Symfony\Component\DomCrawler\Crawler;

$url = 'https://example.com/products';

$html = Browsershot::url($url)
    ->userAgent(USER_AGENT)
    ->waitUntilNetworkIdle()
    ->timeout(60)
    ->bodyHtml();

$crawler = new Crawler($html, $url);

Headless browsers use far more memory and CPU than plain HTTP requests, and they need Chrome and its system libraries on the server. Run them on a dedicated queue with one or two workers, and keep plain Guzzle for every page that does not need JavaScript.

Laravel web scraping: queues, rate limits and the scheduler

For anything that runs regularly, Laravel web scraping is much easier to operate than a cron script. You get retries, throttling, failure logging and scheduling from the framework.

Define a per-host rate limiter in AppServiceProvider::boot():

use Illuminate\Cache\RateLimiting\Limit;
use Illuminate\Support\Facades\RateLimiter;

RateLimiter::for('scraper', function (object $job) {
    return Limit::perMinute(20)->by(parse_url($job->url, PHP_URL_HOST));
});

Then a job that fetches one listing page, saves the rows and queues the next page:

namespace App\Jobs;

use App\Models\Product;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Http\Client\ConnectionException;
use Illuminate\Http\Client\RequestException;
use Illuminate\Queue\Middleware\RateLimited;
use Illuminate\Support\Facades\Http;
use RuntimeException;
use Symfony\Component\DomCrawler\Crawler;
use Throwable;

class ScrapeListingPage implements ShouldQueue
{
    use Queueable;

    public int $tries = 10;        // rate-limit releases count as attempts
    public int $maxExceptions = 3; // real failures allowed before giving up

    public function __construct(public string $url, public int $page = 1)
    {
        $this->onQueue('scraping');
    }

    public function middleware(): array
    {
        return [new RateLimited('scraper')];
    }

    public function backoff(): array
    {
        return [60, 300, 900];
    }

    public function handle(): void
    {
        $response = Http::withUserAgent(config('services.scraper.user_agent'))
            ->timeout(20)
            ->connectTimeout(5)
            ->retry(3, 2000, function (Throwable $e): bool {
                return $e instanceof ConnectionException
                    || ($e instanceof RequestException
                        && in_array($e->response->status(), [429, 500, 502, 503, 504], true));
            })
            ->get($this->url);

        $crawler = new Crawler($response->body(), $this->url);

        $rows = $crawler->filter('article.product_pod')->each(function (Crawler $card): array {
            $link = $card->filter('h3 a');
            $url = $link->link()->getUri();
            $title = $link->attr('title');
            $price = (float) preg_replace('/[^0-9.]/', '', $card->filter('.price_color')->text());

            return [
                'source' => 'books-toscrape',
                'url_hash' => hash('sha256', $url),
                'url' => $url,
                'title' => $title,
                'price' => $price,
                'content_hash' => hash('sha256', $title . '|' . $price),
                'last_seen_at' => now(),
            ];
        });

        if ($rows === []) {
            throw new RuntimeException("No products parsed from {$this->url}");
        }

        Product::upsert(
            $rows,
            uniqueBy: ['source', 'url_hash'],
            update: ['title', 'price', 'content_hash', 'last_seen_at'],
        );

        $next = $crawler->filter('li.next a');

        if ($next->count() > 0 && $this->page < 100) {
            self::dispatch($next->link()->getUri(), $this->page + 1);
        }
    }
}

Schedule the first page in routes/console.php:

use App\Jobs\ScrapeListingPage;
use Illuminate\Support\Facades\Schedule;

Schedule::job(new ScrapeListingPage('https://books.toscrape.com/'))->dailyAt('02:30');

A few notes on how this behaves:

• When the rate limit is hit, RateLimited releases the job back to the queue with a delay. The Laravel queue docs (opens in a new tab) point out that each release counts as an attempt, which is why $tries is higher than $maxExceptions. Laravel 13 also offers #[Tries] style attributes; the properties above work on Laravel 12 and 13.
• retry() handles quick network blips inside one attempt. backoff() handles longer outages between attempts.
• Run a worker for the queue, for example php artisan queue:work --queue=scraping, under Supervisor or your host's process manager, and add the single scheduler cron entry. My guide to deploying Laravel on cPanel or a VPS covers both.
• Failed jobs land in the failed_jobs table, so a broken selector shows up as a visible failure, not missing data.

On the brokerage risk platform I worked on, scheduled data pipelines pull trading data on a timetable. The source there is an official API (the MT5 Manager Web API) rather than HTML, but the same operating principles apply to any scraper: run on a schedule, work through queues, and make failures visible instead of silent.

Storing and deduplicating scraped data

Scrapers see the same record many times. Design the table so repeats update instead of duplicating:

Schema::create('products', function (Blueprint $table) {
    $table->id();
    $table->string('source', 50);
    $table->char('url_hash', 64);
    $table->text('url');
    $table->string('title');
    $table->decimal('price', 10, 2)->nullable();
    $table->char('content_hash', 64);
    $table->timestamp('last_seen_at');
    $table->timestamps();

    $table->unique(['source', 'url_hash']);
});

• A stable natural key. Use the site's own product ID if the page exposes one. Otherwise hash the normalized URL (strip tracking parameters and fragments first). The unique index makes upsert() work and makes duplicates impossible even if two jobs overlap.
• created_at is "first seen". Eloquent's upsert() sets created_at on insert and only touches the columns you list (plus updated_at) on update.
• last_seen_at finds removed items. Anything not seen for several runs has probably been delisted.
• content_hash detects real changes. Compare it before writing if you want a price history table, and insert a history row only when the hash changes.
• Validate before saving. A price of zero or a title of "Add to cart" usually means a selector broke. Reject the row and alert.

Monitoring: scrapers break quietly

Every site redesign is a potential outage for your scraper. Build in:

• Count checks. If a listing page that usually has 20 items returns zero, fail the job.
• Saved HTML fixtures and tests. Keep a few real pages in your test suite and assert the parser output. When the site changes, update the fixture and the selectors together.
• Alerts on failed jobs and on runs that produce far fewer rows than usual.
• A run log with start time, pages fetched, rows saved and errors, so you can answer "when did this stop working?"

When an official API is better than scraping

If an API exists and covers your data, use it, even if it costs something. Scraping is the right tool when there is no API, when the data is public and the site allows it, or when you are monitoring your own sites. If you are weighing both, my API development and integration work covers the API route, including webhooks and sync jobs.

Frequently asked questions

Is PHP good for web scraping?

Yes. Guzzle and Symfony DomCrawler handle most scraping jobs well, and Laravel adds queues, retries and scheduling for running scrapers in production. Python has more scraping-specific libraries, but if your application is already PHP, keeping the scraper in the same codebase is usually simpler to deploy and maintain.

How do I scrape a website with PHP?

Fetch the page with Guzzle or Laravel's HTTP client, load the HTML into Symfony DomCrawler with the page URL, and extract fields with CSS selectors using filter(), text() and attr(). Check robots.txt first, add a delay between requests, and save results with a unique key so repeated runs do not create duplicates.

Can PHP scrape JavaScript-rendered websites?

Yes, with a headless browser. Symfony Panther controls Chrome or Firefox through WebDriver, and Spatie Browsershot uses Puppeteer to return the rendered HTML. Check first whether the page loads its data from a JSON endpoint, because that is usually faster and more reliable.

What replaced Goutte for PHP web scraping?

Goutte is abandoned. Use Symfony BrowserKit's HttpBrowser with Symfony HttpClient if you want a browser-like client with forms and links, or Guzzle plus Symfony DomCrawler for straightforward fetching and parsing, as shown in this guide.

Is web scraping legal?

It depends on the data, the site's terms, the country and what you do with the results. Public facts such as prices are generally treated differently from personal data or copyrighted content. This is not legal advice, so for anything commercial, ask a lawyer and prefer an official API where one exists.

Building a scraper that lasts

Web scraping with PHP is quick to start and easy to get wrong over time. Respect the site, throttle and retry sensibly, store data with stable keys, and alert on breakage, and the scraper becomes a dependable part of your system. If you want one built and run properly, I take on web scraping and automation projects in PHP and Laravel.