The year was 2020, and news about COVID-19 came flooding in so quickly that everyone felt completely overwhelmed with similar news articles providing updates with varying degrees of accuracy.
But all I needed to know was when my official government guidelines changed. In the end, that's all that mattered to me.
Whether the concern is a pandemic or just the latest tech news, keeping ahead of changes in website content can be critical.
The changedetection.io project provides a simple yet highly capable, open source solution for website change detection and notification. It's easy to set up, and it can notify over 70 (and counting) different notification systems, such as Matrix, Mattermost, Nextcloud, Signal, Zulip, Home Assistant, email, and more. It also notifies proprietary applications like Discord, Office365, Reddit, Telegram, and many others.
But changedetection.io isn't just limited to watching web page content. You can also monitor XML and JSON feeds, and it will build an RSS feed of the websites that changed.
Thanks to its built-in JSON simple storage system, there's no need to set up complicated databases to receive and store information. You can run it as a Docker image or install it with pip. The project has an extensive wiki help section, and most common questions are covered there.
For sites using complex JavaScript, you can connect your changedetection.io installation to a Chromium or Chrome browser with the built-in Playwright content fetcher.
Once running, access the application in your browser (http://localhost:5000, by default). You can set a password in the Settings section if your computer can be reached from an outside network.
(Leigh Morresi, CC BY-SA 4.0)
Submit the URL of a page you want to monitor. There are several settings related to how the page is filtered. For example, you more than likely do not want to know when a company's stock price listed in their site footer has changed, but you may want to know when they post a news article to their blog.
Monitor a site
Imagine you want to add your favorite website, Opensource.com, to be monitored. You only want to know when the main call-out article contains the word "python" and you want to be notified over Matrix.
To do this, begin with the visual-selector tool. (This requires the playwright browser interface to be connected.)
(Leigh Morresi, CC BY-SA 4.0)
The visual-selector tool automatically calculates the best Xpath or CSS filter to target the content. Otherwise, you would get a lot of noise from the daily page updates.
Next, visit the Filters & Triggers tab.
(Leigh Morresi, CC BY-SA 4.0)
In CSS/JSON/XPATH Filter field (the blue circle), you can see the automatically generated CSS filter from the previous step.
There are several useful filters available, such as Remove elements (good for removing noisy elements), Ignore text, Trigger/wait for text, and Block change-detection if text matches (used for waiting for some text to disappear, like "sold out").
In Trigger/wait for text (the red circle), type in the keyword you want to monitor for. (That's "python" in this example.)
The final step is in the Notifications tab, where you configure where you want to receive your notification. Below I added a Matrix room as the notification target, using the Matrix API.
(Leigh Morresi, CC BY-SA 4.0)
The notification URL is in the format of matrixs://username:password@matrix.org/#/room/#room-name:matrix.org
However, t2Bot format is also supported. Here are more Matrix notification options.
And that's it! Now you'll receive a message over Matrix whenever the content changes.
There's more
There's so much more to changedetection.io. If you prefer calling a custom JSON API, you don't have to use an API for notifications (use jsons://). You can also create a custom HTTP request (POST and GET), execute JavaScript before checking (perhaps to pre-fill a username and password login field), and many more interesting features, with more to come.
Stop browsing the web and start watching the web instead!
Comments are closed.