What is Wikipedia Stream event?

 

The [https://stream.wikimedia.org/v2/stream/recentchange](https://stream.wikimedia.org/v2/stream/recentchange) endpoint is part of the Wikimedia EventStreams service. It provides a real-time, public "firehose" of activity happening across all Wikimedia wikis (including all language versions of Wikipedia, Wiktionary, Commons, etc.).

Here are the key details about how it works and how to use it:

1. How It Works

Format: It uses Server-Sent Events (SSE), an HTTP-based standard that allows servers to "push" data to clients.

Protocol: You connect to it as a standard HTTP stream. In most programming languages, you use an EventSource client library.

Underlying Tech: The stream is backed by Apache Kafka, which aggregates events from various MediaWiki installations before broadcasting them.

2. What Data You Get

Every time an action occurs on a Wikimedia site, a JSON-formatted event is pushed through the stream. These events include:

Edits: The most common event, detailing changes to articles.

User actions: New page creations, user registrations, and other log actions.

Metadata: Information like the wiki (e.g., enwiki), the user (or IP for anonymous edits), the title of the page, a timestamp, and the comment (edit summary).

3. Best Practices for Implementation

Use a Custom User-Agent: Wikimedia strictly requires a descriptive User-Agent header for all requests to their APIs/streams to identify your project and contact information. Requests with default or missing headers are frequently blocked or rate-limited.

Handle "Canary" Events: Occasionally, you will receive "canary" events used for monitoring the stream itself. You should filter these out (they are usually identified by meta.domain === 'canary').

Client-Side Filtering: The stream is a massive firehose. Since server-side filtering is not natively supported by the HTTP endpoint, your client code must perform filtering (e.g., by checking server_name or wiki) to process only the data you care about.

Resilience: Because network connections to a live stream can drop, implement an infinite loop with a reconnection strategy (as you did in your code) to ensure your service remains unattended and active.

Summary of Resources

Official Documentation: Wikitech: EventStreams.

MediaWiki API Help: API:Recent changes stream.