Example using YQL "universal" feed reader

[jsmorley]

There was a thread started today by Moshi, who had stumbled on a nice tool that could perhaps be used (at least until something better comes along) to replace the ability of Google Reader to provide "universal" feed capabilities for skins.

Now that Google Reader is a fond memory, I think this is an approach that is worth looking at.

The key to it is a public API provided by Yahoo! that will take as input on the URL any feed in RSS/ATOM/JSON format and provided a standard, "normalized" feed in any of those formats as output.

So for instance to access the Rainmeter Forum feed here, and return an XML formatted ATOM feed you can easily parse, you might use:

URL="http://query.yahooapis.com/v1/public/yql?q=select * from feednormalizer where url="http://rainmeter.net/forum/feed.php" and output="atom_1.0"

For the same thing with for instance Google News, you might use:

URL="http://query.yahooapis.com/v1/public/yql?q=select * from feednormalizer where url="http://news.google.com/?output=rss" and output="atom_1.0"

In any case, no matter what feed you access this way, a standard, reliable, feed in the ATOM format will be returned in the WebParser measure for you to parse.

A HUGE bonus with this is that the "dates" returned in the feed <pubdate> / <updated> etc. are also normalized to a standard date format of 2013-12-07T01:23:32Z and always in UTC (Universal Coordinated Time / GMT). What this means is that with just a handful of Lua code, you can easily compare the timestamp of any feed item to the current time, and flag "new" items as you like with a different color or an image, or whatever you like.

The example skin can be found here:

http://rainmeter.net/forum/viewtopic.php?f=27&t=17224

1.jpg

2.jpg

3.jpg

[Mordasius]

This works just fine on all the feeds I like to keep track of.

However, you'll need to read more than five items from the BBC news feed http://feeds.bbci.co.uk/news/rss.xml that doesn't necessarily post the most recent item at the top of the page. At the moment, the most recent item is number five on the list and the five most recent posts are muddled up in the top ten.

Having got the most recent posts, I'm sure it wouldn't take much more than another sprinkling of lua code to sort them into the right order and display the five most recent.

[jsmorley]

This works just fine on all the feeds I like to keep track of.

However, you'll need to read more than five items from the BBC news feed http://feeds.bbci.co.uk/news/rss.xml that doesn't necessarily post the most recent item at the top of the page. At the moment, the most recent item is number five on the list and the five most recent posts are muddled up in the top ten.

Having got the most recent posts, I'm sure it wouldn't take much more than another sprinkling of lua code to sort them into the right order and display the five most recent.

Yeah, Google News is pretty psychotic about how it arranges things in the feed as well.

[moshi]

This works just fine on all the feeds I like to keep track of.

However, you'll need to read more than five items from the BBC news feed http://feeds.bbci.co.uk/news/rss.xml that doesn't necessarily post the most recent item at the top of the page. At the moment, the most recent item is number five on the list and the five most recent posts are muddled up in the top ten.

Having got the most recent posts, I'm sure it wouldn't take much more than another sprinkling of lua code to sort them into the right order and display the five most recent.

you can do sorting with the right YQL syntax. after all the whole thing is a kind of database. you know, all this creating tables in Lua thing can be done in the Webparser Url= option.


example for merging two RSS feeds and sorting them by pubdate:

Code: Select all

URL="http://query.yahooapis.com/v1/public/yql?q=select title,link,pubDate from rss where url in ("http://rt.com/news/today/rss/","http://english.ruvr.ru/rss/export_all.xml") | sort(field="pubDate",descending="true") | truncate(count=8)"

now from rss is a different beast than from feednormalizer, you would need to figure out the right syntax to select the needed fields under the <entry> tags, not from toplevel.

i can't test this weekend, but YQL makes things a lot easier here.

[moshi]

Google News

have a look at the &scoring=n parameter.
http://blog.slashpoundbang.com/post/12975232033/google-news-search-parameters-the-missing-manual

[jsmorley]

have a look at the &scoring=n parameter.
http://blog.slashpoundbang.com/post/12975232033/google-news-search-parameters-the-missing-manual

Not sure google news rss supports that any more.

http://query.yahooapis.com/v1/public/yql?q=select * from feednormalizer where url="http://news.google.com/?output=rss%26scoring=d" and output="atom_1.0"

Still seems to sort by "relevance" for me.

P.S. I found that I had to URL-encode the & or YQL really hated it.

[jsmorley]

example for merging two RSS feeds and sorting them by pubdate:

Code: Select all

URL="http://query.yahooapis.com/v1/public/yql?q=select title,link,pubDate from rss where url in ("http://rt.com/news/today/rss/","http://english.ruvr.ru/rss/export_all.xml") | sort(field="pubDate",descending="true") | truncate(count=8)"

now from rss is a different beast than from feednormalizer, you would need to figure out the right syntax to select the needed fields under the <entry> tags, not from toplevel.

Unless I miss my guess, that is going to make having a skin that is "universal" hard to do. Since we are setting feednormalizer to ATOM or RSS or some other single output format, having to sort on the presumably unknown input format fields would be a problem.

[moshi]

Not sure google news rss supports that any more.

Still seems to sort by "relevance" for me.

P.S. I found that I had to URL-encode the & or YQL really hated it.

yeah, seems like it. for a web search the current parameter is &tbs=sbd:1, but that doesn't do anything on a feed.

[moshi]

Unless I miss my guess, that is going to make having a skin that is "universal" hard to do. Since we are setting feednormalizer to ATOM or RSS or some other single output format, having to sort on the presumably unknown input format fields would be a problem.

no, i don't think so. the select happens after the normalizing. the only problem is to get the desired fields.
the last example here: http://www.yqlblog.net/blog/2013/03/07/yql-feednormalizer-table/ uses entry.title, not item.title although an RSS feed is transformed to an ATOM feed.
i guess "input" means something different here and is rather related to the database operation than the used feed.

something like this:

Code: Select all

select entry.title,entry.link,entry.updated from feednormalizer from url="http://feeds.bbci.co.uk/news/rss.xml" and output="atom_1.0" | sort("entry.updated",descending="true")

the result will no longer be an ATOM feed though i guess and need a new RegExp, more on monday, this kind of stuff isn't really fun on an iPad

[jsmorley]

Yeah, we can play more on Monday.

Your example "almost" worked, but there was a small syntax error. It should be:

Code: Select all

http://query.yahooapis.com/v1/public/yql?q=select entry.title,entry.link,entry.updated from feednormalizer where url="https://news.google.com/news/feeds?output=rss" AND output="atom_1.0" | sort(field="entry.updated", descending="true")

If you use the descending="true", then you also need to use the "=" format for the sort entity, as in field="entry.updated".

The output seems to be valid ATOM formatting ok as far as I can tell. Certainly better than having to jump out to Lua, where sorting table entries and keeping all the table dimensions sync'd up can be a tad complicated.

The only issue is that we lose the overall "feed" info fields. I'll have to play with that. Think we need to extend the "select" to include top level fields outside of the <entry>'s.

Edit: Think I got it:

Code: Select all

URL=http://query.yahooapis.com/v1/public/yql?q=select title,link,updated,entry.title,entry.link,entry.updated from feednormalizer where url="#FeedURL#" AND output="atom_1.0" | sort(field="entry.updated", descending="true")