dealsforme
Switch theme
Sign in

Legal

Our requests to shop websites

Part of listing discounts is asking a shop’s server for a page, and every single one of those requests leaves a line in somebody’s access log. This page is written for the person who reads that line, and it ends with a form that stops the next request from being sent at all.

Last updated:

Two reasons for a request

The first is a source on our list. A source is read on a schedule only once it has been checked and approved in its own right; a scheduled job wakes every hour and reads each approved source whose own schedule says it is due. Now and then a person on our side also reads a source by hand, to check how it is read. Apart from approved sources, no site is read regularly. The second is a submitted link: when a signed-in visitor sends us a deal, we request that single address to find out whether it answers. Links on that page are not followed.

robots.txt

Every address taken from an approved source, listing and detail pages alike, is checked against your robots.txt first. We obey the group for dealsforme-fetch; if the file has none, the group for our earlier name, DealsForMeBot; failing both, the * group. A Crawl-delay longer than our own gap is honoured. We keep using our copy of the file for 24 hours before asking for it again. If the file cannot be fetched at all, nothing is read from that host on a schedule until the next try, 24 hours later; an error status such as 404 counts as a file with no rules. The check of a submitted link does not consult robots.txt — for that request, the do-not-fetch list below is the way to stop us.

Pace

Requests to the same host start at least 2 seconds apart. Each one is dropped after 15 seconds; for a submitted link the limit is 8 seconds and at most 3 redirects. Only the first 2 MB of a response is used. On a 429 or 503 we retry the same address up to 4 times, waiting 1, 2, 4, 8 seconds in turn and never starting sooner than 2 seconds after the previous attempt. After the last refusal we stop: a source is read again when its schedule next comes round, and a submitted link is recorded as not answering.

Which fields reach our records

From an approved source: the fields read for each listed deal — such as title, address, image address, price, earlier price, currency and discount — as the page gives them, and our copy of your robots.txt. From a submitted link: whether it answered, its status code and the address it finally led to.

The do-not-fetch list

Once a domain is on the list, none of the requests described on this page go to it or to its subdomains: no scheduled read, no link check, no robots.txt request and no redirect from some other address. The list is looked up right before each request goes out, so a new entry takes effect with the next one. It stops requests and nothing else: deals from your shop that are already published here are not removed by it. If you want those taken down as well, use the note field.

Proving the domain is yours

You give us a domain and an email address. If the address is on that domain, or on the domain above it (shop.example for eu.shop.example), we mail it a single-use link valid for 24 hours. What that link opens is a page holding a single button; the press on that button is what puts the domain on the list. An address on any other domain, or at a webmail or internet provider, proves nothing about the shop, so such a request waits until a person on our side decides. For each request we store the domain you gave and the domain above it (eu.shop.example alongside shop.example — the daily allowance counts the one above), the email address and whether it proved the domain, your note, the language of the page you sent it from, which is the language of the link mail and of our answer, the time, and a one-way hash of your IP address. Beside those sits the state of the request itself: the stage it has reached, when the domain actually joined the list, when a person on our side decided it, if one did, and a one-way hash of the single-use link with the hour it stops working. The link itself is never stored.

Put a domain on the list

One domain per form, subdomains included. An address on the domain itself receives the single-use link; any other address sends the request to a person on our side.