Stream's reader bot

Stream is a personal RSS reader. When one of our signed-in users subscribes to a feed or opens an article, Stream fetches that page on their behalf so they can read it in their own private library. This page explains what our bot does, how to recognise it, and how to opt your site out — partially with robots.txt, which covers article pages, or completely with a domain block, which covers feeds too.

1. What the bot is

Stream is a subscription RSS and Atom reader for individuals. It is not a search engine, a public archive, or a content-republishing service.

• The bot fetches a page only when a specific signed-in user asks for it — by subscribing to your feed, or by opening or saving one of your articles in their reader.
• Anything it fetches is shown only to that user, inside their private account. We do not publish your content on a public web page.
• We do not sell your content or use it for advertising.

2. How to identify it

Every request our bot makes carries this User-Agent:

stream-reader/1.0 (+https://stream-reader.com/bots)

In robots.txt rules, the bot's user-agent token (product name) is stream-reader.

3. How it behaves

• Respects robots.txt when fetching article pages. Before fetching the page of an article — for reader view, translation, PDF export, or a page saved by URL — the bot checks your robots.txt and skips anything disallowed for stream-reader or for *.
• Feeds are fetched as subscribed. A feed is requested at the address the user added themselves, and we do not apply robots.txt to that request — the same as any other RSS reader, desktop or hosted. If you would rather we stopped polling a feed altogether, write to us and we will remove it from our system.
• Fetches on demand. It only requests pages a user has actually subscribed to or opened, at a modest rate. It does not crawl your whole site or follow links across it.
• Doesn't bypass access controls. It makes no attempt to defeat paywalls, logins, or other access restrictions.
• Identifies itself honestly. It always sends the User-Agent above and never disguises itself as a browser to evade a block.

4. Opt out with robots.txt

To stop the bot from fetching some or all of your site, add a rule for our user-agent token to your robots.txt. To block your whole site:

User-agent: stream-reader
Disallow: /

To block only part of it, list the paths instead:

User-agent: stream-reader
Disallow: /premium/

These rules apply to article pages. Feeds themselves are fetched at the address a user subscribed to, as described in section 3 — to stop those as well, use the domain opt-out.

We cache robots.txt per site, so a change takes effect within 24 hours. If your robots.txt is temporarily unreachable when we check it, the bot may proceed with the fetch; a rule you have published is only guaranteed to be seen once we can read the file.

5. Full domain opt-out

If you'd prefer a hard, account-side block instead of (or in addition to) robots.txt, email us the domain or domains and we'll add them to our opt-out list. This does two things:

• it stops the bot from fetching article pages on those domains, and
• it stops Stream from displaying content we may have already fetched from those domains. The content isn't merely hidden from a menu — it is no longer served to anyone.

Feed polling is handled separately from this list: if you also want us to stop requesting a feed itself, say so in the same email and name the feed URLs — we remove them by hand, which ends the polling.

6. Content takedown

To request removal of specific content — rather than a whole domain — or to raise a copyright or DMCA concern, email abuse@stream-reader.com with the URL(s) and a short description of the issue. We acknowledge and remove contested content promptly, and keep a record of removals.

7. Contact

For anything related to our bot:

• Opt-out & takedown — abuse@stream-reader.com
• General enquiries — support@stream-reader.com