There was a thread started today by Moshi, who had stumbled on a nice tool that could perhaps be used (at least until something better comes along) to replace the ability of Google Reader to provide "universal" feed capabilities for skins.
Now that Google Reader is a fond memory, I think this is an approach that is worth looking at.
The key to it is a public API provided by Yahoo! that will take as input on the URL any feed in RSS/ATOM/JSON format and provided a standard, "normalized" feed in any of those formats as output.
So for instance to access the Rainmeter Forum feed here, and return an XML formatted ATOM feed you can easily parse, you might use:
URL="http://query.yahooapis.com/v1/public/yql?q=select * from feednormalizer where url="http://rainmeter.net/forum/feed.php" and output="atom_1.0"
For the same thing with for instance Google News, you might use:
URL="http://query.yahooapis.com/v1/public/yql?q=select * from feednormalizer where url="http://news.google.com/?output=rss" and output="atom_1.0"
In any case, no matter what feed you access this way, a standard, reliable, feed in the ATOM format will be returned in the WebParser measure for you to parse.
A HUGE bonus with this is that the "dates" returned in the feed <pubdate> / <updated> etc. are also normalized to a standard date format of 2013-12-07T01:23:32Z and always in UTC (Universal Coordinated Time / GMT). What this means is that with just a handful of Lua code, you can easily compare the timestamp of any feed item to the current time, and flag "new" items as you like with a different color or an image, or whatever you like.
The example skin can be found here:
http://rainmeter.net/forum/viewtopic.php?f=27&t=17224
It is currently May 4th, 2024, 10:49 am
Example using YQL "universal" feed reader
-
- Developer
- Posts: 22631
- Joined: April 19th, 2009, 11:02 pm
- Location: Fort Hunt, Virginia, USA
Example using YQL "universal" feed reader
You do not have the required permissions to view the files attached to this post.
-
- Posts: 1173
- Joined: January 22nd, 2011, 4:23 pm
- Location: GMT +8
Re: Example using YQL "universal" feed reader
This works just fine on all the feeds I like to keep track of.
However, you'll need to read more than five items from the BBC news feed http://feeds.bbci.co.uk/news/rss.xml that doesn't necessarily post the most recent item at the top of the page. At the moment, the most recent item is number five on the list and the five most recent posts are muddled up in the top ten.
Having got the most recent posts, I'm sure it wouldn't take much more than another sprinkling of lua code to sort them into the right order and display the five most recent.
However, you'll need to read more than five items from the BBC news feed http://feeds.bbci.co.uk/news/rss.xml that doesn't necessarily post the most recent item at the top of the page. At the moment, the most recent item is number five on the list and the five most recent posts are muddled up in the top ten.
Having got the most recent posts, I'm sure it wouldn't take much more than another sprinkling of lua code to sort them into the right order and display the five most recent.
-
- Developer
- Posts: 22631
- Joined: April 19th, 2009, 11:02 pm
- Location: Fort Hunt, Virginia, USA
Re: Example using YQL "universal" feed reader
Yeah, Google News is pretty psychotic about how it arranges things in the feed as well.Mordasius wrote:This works just fine on all the feeds I like to keep track of.
However, you'll need to read more than five items from the BBC news feed http://feeds.bbci.co.uk/news/rss.xml that doesn't necessarily post the most recent item at the top of the page. At the moment, the most recent item is number five on the list and the five most recent posts are muddled up in the top ten.
Having got the most recent posts, I'm sure it wouldn't take much more than another sprinkling of lua code to sort them into the right order and display the five most recent.
-
- Posts: 1740
- Joined: November 13th, 2012, 9:53 pm
Re: Example using YQL "universal" feed reader
you can do sorting with the right YQL syntax. after all the whole thing is a kind of database. you know, all this creating tables in Lua thing can be done in the Webparser Url= option.Mordasius wrote:This works just fine on all the feeds I like to keep track of.
However, you'll need to read more than five items from the BBC news feed http://feeds.bbci.co.uk/news/rss.xml that doesn't necessarily post the most recent item at the top of the page. At the moment, the most recent item is number five on the list and the five most recent posts are muddled up in the top ten.
Having got the most recent posts, I'm sure it wouldn't take much more than another sprinkling of lua code to sort them into the right order and display the five most recent.
example for merging two RSS feeds and sorting them by pubdate:
Code: Select all
URL="http://query.yahooapis.com/v1/public/yql?q=select title,link,pubDate from rss where url in ("http://rt.com/news/today/rss/","http://english.ruvr.ru/rss/export_all.xml") | sort(field="pubDate",descending="true") | truncate(count=8)"
i can't test this weekend, but YQL makes things a lot easier here.
-
- Posts: 1740
- Joined: November 13th, 2012, 9:53 pm
Re: Example using YQL "universal" feed reader
have a look at the &scoring=n parameter.jsmorley wrote:
Google News
http://blog.slashpoundbang.com/post/12975232033/google-news-search-parameters-the-missing-manual
-
- Developer
- Posts: 22631
- Joined: April 19th, 2009, 11:02 pm
- Location: Fort Hunt, Virginia, USA
Re: Example using YQL "universal" feed reader
Not sure google news rss supports that any more.moshi wrote: have a look at the &scoring=n parameter.
http://blog.slashpoundbang.com/post/12975232033/google-news-search-parameters-the-missing-manual
http://query.yahooapis.com/v1/public/yql?q=select * from feednormalizer where url="http://news.google.com/?output=rss%26scoring=d" and output="atom_1.0"
Still seems to sort by "relevance" for me.
P.S. I found that I had to URL-encode the & or YQL really hated it.
-
- Developer
- Posts: 22631
- Joined: April 19th, 2009, 11:02 pm
- Location: Fort Hunt, Virginia, USA
Re: Example using YQL "universal" feed reader
Unless I miss my guess, that is going to make having a skin that is "universal" hard to do. Since we are setting feednormalizer to ATOM or RSS or some other single output format, having to sort on the presumably unknown input format fields would be a problem.moshi wrote: example for merging two RSS feeds and sorting them by pubdate:now from rss is a different beast than from feednormalizer, you would need to figure out the right syntax to select the needed fields under the <entry> tags, not from toplevel.Code: Select all
URL="http://query.yahooapis.com/v1/public/yql?q=select title,link,pubDate from rss where url in ("http://rt.com/news/today/rss/","http://english.ruvr.ru/rss/export_all.xml") | sort(field="pubDate",descending="true") | truncate(count=8)"
-
- Posts: 1740
- Joined: November 13th, 2012, 9:53 pm
Re: Example using YQL "universal" feed reader
jsmorley wrote: Not sure google news rss supports that any more.
Still seems to sort by "relevance" for me.
P.S. I found that I had to URL-encode the & or YQL really hated it.
yeah, seems like it. for a web search the current parameter is &tbs=sbd:1, but that doesn't do anything on a feed.
-
- Posts: 1740
- Joined: November 13th, 2012, 9:53 pm
Re: Example using YQL "universal" feed reader
no, i don't think so. the select happens after the normalizing. the only problem is to get the desired fields.jsmorley wrote: Unless I miss my guess, that is going to make having a skin that is "universal" hard to do. Since we are setting feednormalizer to ATOM or RSS or some other single output format, having to sort on the presumably unknown input format fields would be a problem.
the last example here: http://www.yqlblog.net/blog/2013/03/07/yql-feednormalizer-table/ uses entry.title, not item.title although an RSS feed is transformed to an ATOM feed.
i guess "input" means something different here and is rather related to the database operation than the used feed.
something like this:
Code: Select all
select entry.title,entry.link,entry.updated from feednormalizer from url="http://feeds.bbci.co.uk/news/rss.xml" and output="atom_1.0" | sort("entry.updated",descending="true")
-
- Developer
- Posts: 22631
- Joined: April 19th, 2009, 11:02 pm
- Location: Fort Hunt, Virginia, USA
Re: Example using YQL "universal" feed reader
Yeah, we can play more on Monday.
Your example "almost" worked, but there was a small syntax error. It should be:
If you use the descending="true", then you also need to use the "=" format for the sort entity, as in field="entry.updated".
The output seems to be valid ATOM formatting ok as far as I can tell. Certainly better than having to jump out to Lua, where sorting table entries and keeping all the table dimensions sync'd up can be a tad complicated.
The only issue is that we lose the overall "feed" info fields. I'll have to play with that. Think we need to extend the "select" to include top level fields outside of the <entry>'s.
Edit: Think I got it:
Your example "almost" worked, but there was a small syntax error. It should be:
Code: Select all
http://query.yahooapis.com/v1/public/yql?q=select entry.title,entry.link,entry.updated from feednormalizer where url="https://news.google.com/news/feeds?output=rss" AND output="atom_1.0" | sort(field="entry.updated", descending="true")
The output seems to be valid ATOM formatting ok as far as I can tell. Certainly better than having to jump out to Lua, where sorting table entries and keeping all the table dimensions sync'd up can be a tad complicated.
The only issue is that we lose the overall "feed" info fields. I'll have to play with that. Think we need to extend the "select" to include top level fields outside of the <entry>'s.
Edit: Think I got it:
Code: Select all
URL=http://query.yahooapis.com/v1/public/yql?q=select title,link,updated,entry.title,entry.link,entry.updated from feednormalizer where url="#FeedURL#" AND output="atom_1.0" | sort(field="entry.updated", descending="true")