RSSReader

[jrlewis]

Having sparked my interest I put together an RSS Reader. It is a little rough around the edges and is likely to be a little buggy, but I wanted to get it out for folks to test. I have used it with RSS 2.0 feeds only. It won't work with ATOM feeds. I will probably update it to handle ATOM feeds by just simply translating the given elements to be compatible with RSS. The driver doesn't do anything fancy with respect to XML. It basically just eats through the feed and parses out the necessary data. If you run into feeds that cause it to hiccup post them here so I can take a look and see what the issue is.

The driver includes a CMLBin macro (\CMLBin\Drivers\RSSReader\Items) that provides another way to view ItemDescriptions if you have feeds that use HTML markup in their feeds. The updated template makes use of this. When installing the driver give it a moniker of RSSReader. For the uninitiated you will want to install the driver first and get it up and going before importing the template pack.

Posted Version 0.94

0.1 Beta
0.2 Added feed update tracking and ability to update individual feed
0.3 Changed Update fields and TTL to ReadWrite,Added support for QueryTextVal, Added persistance to Update fields and TTL fields
0.4 Support added for 0.91, 1.0, and Atom feeds, Changed some nomenclature, ability to override encoding, alias for the feed title, quotetoken for querytextval
0.5 Added FeedTicker and TickerSpace, improved/cleaned up code
0.51 Missing EndCase in CardFldChanged
0.6 Fixed Accept header in Process Feed (*.* -> */*)...hats of to Wuench, Added support for Google Calendar RSS feeds
0.7 Added FeedQuery and support for ClientCmd method
0.8 Added FeedTickerOrder, Changed name of TickerSpace > FeedTickerSpace,Added FeedTimeout,Removed QuoteToken and using BuildQuotedCommaList,Added user action event for updated feeds,Added FeedLastUpdate and support for conditional SendGET based on last-modified
0.9 Fixed issues with some feeds causing an exception for the conditional SendGET instead of returning a 304 status code, Modified ProcessFeed to handle all possible failure situations...I think, version 0.9b rolls back changes using the BuildQuotedCommaList so it will work for CQC versions 2.4 to current beta
0.91 Modified GetFeedType for ATOM feeds
0.92 Hacked in a little better support for Google Calendars
0.93 Provided better support for Google Calendars
0.94 Added CurItem and CurFeed for 1 based index of ItemIndex and FeedIndex
1.0 Fixes for Out/InOut parameter problems now being caught by the CML compiler

[anogee]

Wow, its great to have this. Thanks. Now I just have to figure out what a template pack is. :confused:

[jrlewis]

A template pack is just a template someone has creeated and packaged up so you can import it into your system. If user images that aren't included in the release are used in the template they will be packaged up along with it.

I updated the html description as I realized I left out the two stringlist fields from the field table.

[anogee]

It looks very good. I will try it tomorrow.

I have some questions, but be gentle, I'm a newbe.

I see FeedTTL to read the time between feed updates, but how do you set this value? (It seems marked as read only)

How do you trigger when the feed has just been read? And how do you trigger if a feed has changed? Do I have to store each feed, and compare it to the next one myself?

Thanks.

Allen

[jrlewis]

Currently I have the FeedTTL as read only. It gets this value from the feed. I'm thinking it is best to honor the feed publisher's wishes on how long to cache data before updating it. Some feeds don't specify a TTL and I'm using 15 minutes as the default. If you really need to update a feed for some reason just use the ReloadFeeds and this will update all the feeds.

This isn't going to be as fancy as a desktop rss reader. If you want that level of complexity you are better off using a full blown reader. I wouldn't try getting too complex with this. This driver is a very dumb reader. It justs provides you with the current feed data. Each feed has a FeedPubDate associated with it and this will tell you when the feed was last updated if the feed provides it. I guess I could provide an updated field that is true when the PubDate changes. The problem is how you reset it. You can't use the FeedIndex because the act of selecting the feed would reset it and you would never see it. It could be cleared when you set the ItemIndex that way a feed would show it has been updated until you select an item from that particular feed.

Edit: Actually I guess it is the lastBuildDate that specifies when the feed was updated. I will need to look into this.

Edit2: Doesn't look like the dates are dealt with in a predictable manner so the best option would probably be just checking to see if the first item in the feed has changed to determine if it has been updated. I will add that to the driver.

[jrlewis]

Updated the driver to track updates for individual feeds. It just looks at the first item in the list when a feed is updated to see if it has changed since the last time. It is set to false when you select an item in the given feed. Another variable tracks all the feeds so you know if one of your subscribed feeds still has new messages. I also have included a CMLBin macro that will display the ItemDescription in a web browser widget. Some feeds contain HTML markup in their item descriptions and it doesn't present well in a text widget.

[anogee]

First, thanks again, it seems to work great. I have it set to this feed:
http://rss.cnn.com/rss/cnn_topstories.rss

I got back an "Item Count" of 10, so 10 headlines. "Items" looks like its all the items separated by commas. Set "Item Index" to 0 to 9, and then "Item" contains that text. FeedTTL = 5, so it sounds like you are going to check every 5 minutes. Great.

I see you have FeedUpdated, and FeedsUpdated. I understand how they work, "true" if there is change, "false" otherwise. Unfortunately I still have a problem of how to trigger when you read the feeds. In other words, if I have an LED sign with headlines, I can trigger when FeedsUpdated goes to "true" but when I'm done, don't I need a way to set it "FeedsUpdated" back to false so I can trigger when it goes true? For example, lets say the feed changes every read, it will always be true, so I need it to trigger each read, but it won't because its always true. Am I making any sense?

Also, I think I might have come across a bug. I entered 3 feeds. In addition to the first one, I added:
http://rssfeeds.usatoday.com/UsatodaycomNation-TopStories
http://online.wsj.com/xml/rss/0,,3_7011,00.xml

I set the FeedIndex to 1, and USA Today worked fine. I set it to 2, and I got back an error that it couldn't be set to 2, but it did set to 2, although it displayed the data for 0 which was CNN.

I like it. Thanks for your hard work.

Allen

[jrlewis]

Last things first. The last feed you list actually redirects to http://feeds.wsjonline.com/wsj/xml/rss/3_7011.xml. You should verify that the feed you are entering is the actual feed URL before adding it to the list of feeds for the driver. I should probably add a status code field for each feed so you can get some information there if things go wrong. I also need to take a look and make sure I'm handling bad updates from the feed source.

Both the Feeds and Items fields are StringList fields. A StringList is a comma separated list of quoted strings. I have kept the zero based referencing for these lists, but it may make more sense to have them as 1 based for the user side of things. I have also been thinking about the TTL. What I will do is have that field as ReadWrite and also have an UpdateTTL field which is also ReadWrite. That way you can decide to set your own TTL or just let the reader use the TTL specified by the feed. This value will default to False so you will need to set it to True if you want it handled otherwise. This is something I will need to make sure is persistent so you don't have to reconfigure everything every time the cqc service restarts.

I could make the FeedsUpdated and FeedUpdated ReadWrite as well to make them easier to reset if you aren't directly interacting with the driver. The way it works now is if a feed is updated and it changes the FeedUpdate is set to true. To reset this you just have to set the ItemIndex to any valid index for that feed and it will reset to False. The FeedsUpdated resets when the FeedUpdate for all the feeds is False.

I will get the updates out this evening.

[Dean Roddey]

I should expose the redirect handling version of SendGET at the CML level. But you can handle redirects fairly easily. If you get a moved permanent or moved temporary code back, look for a LOCATION header line that got returned, and that's the new URL, just issue a new GET with that one. In theory you should do this in a loop (though with a max loop count) since you might be multiply redirected.

[jrlewis]

As a side effect implementing it helped with the structure for dealing with the return from the SendGET. It now handles redirects...just need to take care of the other stuff.

[anogee]

I've been using that RSS feed for about 2 years now with a simple VB reader I wrote and I never realized it redirected. I understand how that might cause a problem. The other proposed changes sound real good.

This driver will be very useful. Seems like everything has an RSS part to it nowadays.

[jrlewis]

Got sidetracked a bit poking around. Posted a new version that expands the capabilites of the driver a bit. You can now reset the FeedUpdate and FeedUpdates directly. The FeedTTL can now be set and it has an accompanying FeedDefaultTTL field. When FeedDefaultTTL is True the driver gets the FeedTTL from the feed and when False you can specify a FeedTTL. Both the FeedTTL and FeedDefaultTTL are persistent so they will not change when the cqc service is cycled or when using UpdateFeeds assuming you don't remove the feed from the config file. One other field added is the ItemIndexUpdated field. When a feed has been udpated this value will give you the last index of the newest item in the feed. For example if a feed updates and adds 5 new items this value will be 4.

In preparation to support the proposed "static" list browsers a fully functional QueryTexVal has been implemented for the driver. This will allow you to run a single version of the driver and interact with it through the normal field widgets as well as query the underlying data for all the current feeds without disturbing the driver field data. The new CMLBin macro (\CMLBin\Drivers\RSSReader\Items) used to display data in a web widget in the included template pack takes advantage of the new functionality of the QueryTextVal.

As an example the CNET News.Com (http://news.cnet.com/2547-1_3-0-20.xml) feed is the same feed data displayed by this CMLBin web page (http://florabamabob.dyndns.org/CMLBin/User/Drivers/RSSReader/Items.html?Moniker=RSSReader&fi=0&ii=All). Note for the CMLBin page it has the parameters ?Moniker=RSSReader&fi=0&ii=All. You can play around with this to get different feed data. If you wanted only items with index 0-3 for the current feed you would use ?Moniker=RSSReader&fi=&ii=0,3 and you would get this page (http://florabamabob.dyndns.org/CMLBin/User/Drivers/RSSReader/Items.html?Moniker=RSSReader&fi=&ii=0,3). Using Moniker=RSSReader&fi=&ii= gets you the current feed item (http://florabamabob.dyndns.org/CMLBin/User/Drivers/RSSReader/Items.html?Moniker=RSSReader&fi=&ii=).

[jrlewis]

The driver now supports RSS 0.91, RSS 1.0, RSS 2.0, and Atom feeds. I changed the nomenclature for one of the fields (FeedDefaultTTL -> OverrideTTL) to make it easier to understand and added a few other fields including the ability to use an alias for the feed title and override the encoding use to xcode the feed into text. With the new feed formats you should be able to use the driver to mine just about any feed.

The code for processing the feeds isn't pretty, but it does the job. Without support for dealing with XML it will do for now. The driver basically looks for some patterns in the returned feed and if it drops all the way through it gets a 2.0 designation. Based on the type of feed it basically scrapes the feed for the data. The only real trouble I ran into with feeds is that sometimes the encoding specified in the raw xml didn't work when trying to xcode the feed. I started out using ISO-8859-1 as the default when the encoding can't be determined by using the ParseTextEncoding method of the HTTPClient class. Searches on Craig's list is the model I used for 1.0 feeds and if you view the source it shows the encoding as ISO-8859-1. However, this encoding would often fail to xcode the whole feed. Setting the encoding to utf-8 seems to work fine. Also, for 1.0 feeds that have the .rdf extension the HTTPClient spits out a 406 status code.

Another interesting aspect is dealing with the SetConfigStr in Terminate and and GetConfigStr in WaitConfig. In the test harness you have to fake it out at times because it never gets to the Terminate naturally. It also appears that the ConfigStr is persistent even after a driver is removed from the driver's list. This caused some minor problems when adding more values to the string and the next time the driver was loaded the pattern in the configstr didn't quite match what it was looking for when it loaded up. At what point does a configstr for a removed driver cease to exist?

[Dean Roddey]

It won't ever cease to exist right now. At some point I'll put in an 'age out' feature into the config server to get rid of config for drivers that are not longer configured, but for right now it doesn't remove that stuff, on the assumption you might want to reload the driver again.

For development purposes you kind of have to deal with this. For real world purposes, you should probably put some version indicator into the string, to allow you to deal with upgrades in the future.

[anogee]

The driver works great. My only question is what happens if a site goes down? How does the program using the driver know this?

[jrlewis]

Dean...I had thought about that, but what I think I may do is structure the data being stored a little better so the data is stored in a key-value type pairing for each feed. That way any changes will be handled naturally and there won't be a need to keep track of the version. I currently just add them in a particular order and pull them off in the same order. I don't see too many other things that need to be stored though, but who knows. Does the config server store the configstr by moniker or some combination of that and the make/model of the driver?

anogee...I still have a little code to add for handling those cases where it doesn't return a 200 status code. Probably what I will do is just have it include 1 item where the item is the status code returned and the descprition is the reply text. That or I could also add a StatusCode field so it could be monitored as well.

[Dean Roddey]

It's stored by moniker.

[gacevich]

incorporating RSS feeds into an IV sounds interesting, but i am having trouble picturing in my head how it might be used in practice. if anyone develops a screen that they are comfortable enough to post screenshot of, i would appreciate it.

[anogee]

RSS Feeds are great. Check this link for some ideas:

http://www.accrediteddldegrees.com/2008/100-cool-things-you-can-do-with-rss/

I have an Alpha LED sign in my kitchen, and I currently scroll news headlines across it from CNN, USAToday, and the WSJ. I'm switching over to CQC now, and I'll probably do that on CQC as well. I also have reminders, weather, forecast, stock qoutes, and caller ID go to the sign. The wife loves it.

Other interesting stuff on RSS is traffic conditions, weather storm warnings, new DVDs released, new Movies, sports scores, US threat level, etc. you name it. Sure you can get this stuff from the web, but RSS is packaged is a simple way so you can extract it as you see fit, and its perfect for home automation. For example, when there is a weather warning in my area, I deactivate my outside motion detectors because I know they false from the wind. You can even get the new posts from this board as an RSS feed, then you can have your house tell you when someone replies to your post. :tounge There really are no limits on what it can do.

[personalt]

I want a few alerts when I wake up and a few when I come home and sit in the living room.

Mostly I am interested in text to speach from CQC. In the morning start playing stuff when we get up and in the evening use the living room motion sensor to start reading me new stuff.

In morning I am thinking mostly alerts for weather and traffic.

On sport sunday's RSS could pull in sports scores and test to speak them over the whole house audio speakers.

It live really close to a major tunnel to NYC... if there is a change in the threat level should always generate a warning.

[jrlewis]

Updated version posted. Most noteable is added a FeedTicker field for ease of use of feed items in a marquee widget.

If you donwload the included templates (they aren't pretty) you will see the a marquee widget scrolling items and also a button to open the current item in a popup web browser. For things like news and sports feeds you could have a marquee with the latest feeds scrolling on the screen. In cases for feeds where you may want to read the whole thing you can open up the url in a web browser popup... say a recipe from a recipe feed. The only downside with this is the info you are provided in the web browser widget is controlled by the website. I have some ideas about incorporating the rss reader into the QookBook driver as a means to provide the central recipe database so you can get recipes in a form that would fit in seamlessly with your template styles. That will have to be a little ways out though.

[anogee]

I like the new version. I noticed you changed the location of the "feeds" file. Just wondering, if you reinstall this, does it overwrite your "feeds" file with your sample feed info? Hopefully not. That wouldn't be nice.

[jrlewis]

The location shouldn't have changed. Did you install the driver with a different moniker? It should sit under \MacroFileRoot\Drivers\RSSReader\Moniker since you could have more than one installed each having its own feeds.txt file. It only creates the file if it isn't there. Otherwise it just uses what is there.

[anogee]

Makes sense. I probably did change the Moniker. That is something I have to get used to with CQC. If you change a Moniker by one letter, it screws up everything.

[jrlewis]

Here is a test macro that attempts to get an rss feed from a google calendar. Pasting the url into a browser works fine, but using the SendGet returns a Not Acceptable and gives and exceptioin error text of "The other side has closed down normally?". Any clues?

[anogee]

I just tried an RSS feed from Google Calendar, and it told me that the calendar had to be marked public for this to work. I'm not sure I would want my calendar public. Is there anyway to get this to work with non-public calendars?

[wuench]

I think you're going to have to catch that exception and ignore it. It isn't really an error, just means the HTTP connection closed down properly. I had the same issue in the ISY driver.

[jrlewis]

Wuench...
Unfortunately the returned Content buffer is empty so it is something more than just simply ignoring the error.
Edit: Looking at it again the content returned is "No acceptable type available".

Anogee...
There is a private and public address for the calendar. I don't know alot about it, but I think the public one can be found by a search, but provides you with the capability to control what a person sees. The private address allows someone to see everything. The macro I attached has my private url to the rss feed. I don't have anything in it so no big deal, but the address can be reset if need be.

[anogee]

So the macro has YOUR private feed to your calendar, but how does one determine their private URL?

When I click on the RSS icon at the top, I get this message...
Sorry, this calendar does not have public access enabled. If you are the calendar owner, you can make this calendar public on the calendar sharing settings page

[beelzerob]

So the macro has YOUR private feed to your calendar, but how does one determine their private URL?

Is that answer here? (http://www.google.com/support/calendar/bin/answer.py?answer=37648&cbid=69jfllmyn5to&src=cb&lev=answer)

[anogee]

Is that answer here? (http://www.google.com/support/calendar/bin/answer.py?answer=37648&cbid=69jfllmyn5to&src=cb&lev=answer)

Thanks. Even with those instructions, it wasn't obvious to me. You have to actually click on "Calendar Settings" then click on "Calendars," then click on your e-mail address next to the picture of a calendar to actually see the page with the XML/HTML buttons. Not incredibly intuitive, but I got it.

O.K. so I'm closer. Now, the problem is I notice the RSS feed returns EVERY appointment that you have. How do I screen them so I only have events which occur in the next 7 days?

[jrlewis]

Try this http://blog.lib.umn.edu/ayubx003/dividebyzero/2008/08/18/customize_google_calendar_rss.html.

[anogee]

Try this http://blog.lib.umn.edu/ayubx003/dividebyzero/2008/08/18/customize_google_calendar_rss.html.
Cool. :-D

[jrlewis]

You might try using /full at the end of the private feed. It gives you a different take on the calendar that is more in line with what you would want to see. If it can be determined what the issue is for a SendGet for this feed than it would provide a conduit for getting the google calendar information into CQC.

Having looked at the API's got me thinking about a specific driver for Google Calendars that would allow all the functionality the API provides for established calendars so you could edit,add,delete events. It could also provide a way to display calendar data in a more granular fashion than the RSSReader would allow. The API allows for a dogs dinner of other things as well that could be implemented on a desired basis. Using the API would require authentication via SSL, but thanks to Wuench I found I could use stunnel to authenticate so no SSL support in CML isn't a problem.

Edit: A little more thought and a read-only driver version for google calendars would be pretty quick to do and is part of a full blown driver any how. I know what I'm going to be doing for at least a few hours this weekend.

[sic0048]

A google calendar driver would be great. It has been discussed before, but nothing moved forward. I am currently using a web widget to display the regular google calendar, but it would be much nicer to have a CQC driver.

The onther hitch I can see is that Dean has yet to create a calendar widget or something similar to display all the information on. It's on the list, but it hasn't been a big priority.

Perhaps getting a Google driver could help move the calendar widget farther up on the list. ;-)

[anogee]

I tried it all, but it seemed to fail with the RSS driver. Here is what I did.
1) Created a calendar and put some appointments in it.
2) Found my private XML feed. This works because I can put it in a browser window and it shows the appointments.
3) Now, I added the full URL to the Feeds file after a few other CNN and USAToday feeds that I had.
4) Did a Refresh Feeds in the RSS Driver.

Now, before I did this, the RSS Reader was working great, and I was getting news from CNN, WSJ, and USAToday. After I added the Google Calendar feed, I started to have problems started showing up as if the RSS Driver didn't read it correctly. Google Calendar was Feed 0, CNN Feed 1, etc. When I set it to FeedIndex 0, it would spit out the correct path for Google Calendar, but still give me news feeds for CNN. Things like Copyright, feed description, etc. were all blank. Seems like the Reader got mixed up at that point. When I did an "Update Feeds" I got an error back that the driver had rejected the request.

What am I doing wrong? Do I have to change the type from ISO-8859-1 or UTF-8?

You might try using /full at the end of the private feed. It gives you a different take on the calendar that is more in line with what you would want to see. If it can be determined what the issue is for a SendGet for this feed than it would provide a conduit for getting the google calendar information into CQC.

When I use /Full, it returns the appointments and the date and time I entered them. Don't I want the date and time of each appointment instead?

[jrlewis]

Brian,
A homebrew setup for display very similar to what Rhamer does would have to be used, but you wouldn't have to configure the text file. The intial setup of the layout would of course be quite tedious, but I would need to do that to test the driver. I will start a new thread on it later so that type of discussion can go there. I need to look a a failed load again because it isn't handling it properly.

Anogee,
The RSSReader will not process the google calendar RSS feed. To do that or even the calendar driver it needs to be determined what is going on with that feed and the SendGET method of the HTTPClient class.I need to look a a failed load again because it isn't handling it properly...actually I know what the problem is. I have to add some elements to vectors and I'm not removing them from the mix on a failed load. For now just remove it from the feeds and reload and it will be fine.

[wuench]

Seems to be working well with Netflix feeds.

What are your thoughts about possibly providing an option to strip all HTML tags from the fields? They are in the descriptions from Netflix, but I am sure it is a pretty common thing...

[jrlewis]

That is a needed addition. Probably will have to be done on a feed by feed basis as some feeds are completely html based while others are pure text. It is all confounded by the fact that not all feeds can be guaranteed to have their contents escaped properly. Using ExpandBodyText from the HTTPClient class is hit and miss. Probably the biggest culprit is unescaped ampersands. I will add the capability to the next version.

[Squintz]

Yup, I was able to pull up my private calendars private address. Read only is all I need for a CQC Calendar. I just want to be able to trigger events based on my google calendar.

[anogee]

Yup, I was able to pull up my private calendars private address. Read only is all I need for a CQC Calendar. I just want to be able to trigger events based on my google calendar.

Yours read with the RSS reader? It didn't read correctly for me, but every other RSS feed I have tried works with the reader.

[LittleJohnny]

I just tried to install this driver for the first time with CQC 2.4.15.
(So, I made have done something wrong). I left everything with the defaults and used the moniker of RSSReader. The driver does not initialize and I see the following error in the logs:


09/26 14:32:59-dellsc420, CQCServer, CQCDrv_RSSReader_Thread7
{
CQCKit, MEng.User.CQC.Drivers.RSSReader.DriverImpl.2145, API Failed/Data Format
A macro parse event occured
Previous control flow was a(n) Case on line 2142, so a(n) Case was unexpected
}
09/26 14:32:59-dellsc420, CQCServer, CQCDrv_RSSReader_Thread7
{
CIDMacroEng, CIDMacroEng_Parser.cpp.1879, Status/App Status, Error: 6505/0/0
An unrecoverable error occured while parsing
}
09/26 14:32:59-dellsc420, CQCServer, CQCDrv_RSSReader_Thread7
{
CQCKit, CQCKit_MEngErrHandler.cpp.237, Status/Data Format
A C++ exception occured during macro parsing
MEng.User.CQC.Drivers.RSSReader.DriverImpl Line=2145, Col=17)
}
09/26 14:32:59-dellsc420, CQCServer, CQCDrv_RSSReader_Thread7
{
CIDMacroEng, CIDMacroEng_Parser.cpp.1879, Status/App Status, Error: 6505/0/0
An unrecoverable error occured while parsing
}

[jrlewis]

Been bit by my own bug. Give me a minute and I will have a fix up. It's missing an EndCase.

Edit: Try 0.51.

[LittleJohnny]

Been bit by my own bug. Give me a minute and I will have a fix up. It's missing an EndCase.

Edit: Try 0.51.

Yes, this version fixed the issue I was seeing. The driver does initialize. FYI - I upgraded to 2.4.16 because I was seeing another problem after I posted the message.

Thanks for the quick response and RSS driver!

[jrlewis]

Well I took my own advice and got out SmartSniff and poked around trying to figure out why the google calendar RSS feed won't work. The first item shows the stream data from using Internet Explorer to access the feed. The second comes from using the test macro and including all the same header information. For whatever reason the SendGET cannot handle this stream. Any insight would be greatly appreciated. Is it at all possible that the SendGET is using HTTP 1.0 and this feed has content that isn't supported?

GET /calendar/feeds/florabamabob%40gmail.com/private-fd7ff448df495937fb9053419777b2e5/basic HTTP/1.1
Accept: */*
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Sat, 27 Sep 2008 06:29:11 GMT
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)
Host: www.google.com
Connection: Keep-Alive


HTTP/1.1 304 Not Modified
Date: Sun, 28 Sep 2008 00:50:40 GMT
Server: GFE/1.3

GET /calendar/feeds/florabamabob%40gmail.com/private-fd7ff448df495937fb9053419777b2e5/basic HTTP/1.0
Host: www.google.com
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)
Accept: *.*
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Sat, 27 Sep 2008 06:29:11 GMT
Connection: Keep-Alive
Host: www.google.com


HTTP/1.0 406 Not Acceptable
Content-Type: text/plain; charset=UTF-8
Content-Encoding: gzip
Date: Sun, 28 Sep 2008 01:04:16 GMT
Expires: Sun, 28 Sep 2008 01:04:16 GMT
Cache-Control: private, max-age=0
Server: GFE/1.3
Connection: Close

............WHLNN-(IL.IU(.,HUH,K...q..../....

[Dean Roddey]

It may be that our guy only claims to support HTTP 1.0, and it might consider that not acceptable.

[jrlewis]

Is this something that can easily be tested on your end?

[Dean Roddey]

I'm not sure it would help, because if we claim to support 1.1, then we have to actually implement 1.1, which I couldn't do right now. I assume if it won't accept 1.0 clients, then it probably uses something specific to 1.1.

[jrlewis]

OK. I will have to implement a google calendar driver with the help of an external script for now then. If and when the time comes transferring the data retrieval inside of CQC will be pretty painless.

[wuench]

The accept lines are also different in the headers. *.* on one and */* on the other. You might want to check that first just in case.

It's definitely not the HTTP 1.0 thing. I just did a "telnet www.google.com 80" and pasted
GET /calendar/feeds/florabamabob%40gmail.com/private-fd7ff448df495937fb9053419777b2e5/basic HTTP/1.0 from above and I got back the following. It's something in your headers...


HTTP/1.0 200 OK
Content-Type: application/atom+xml; charset=UTF-8
Cache-Control: max-age=0, must-revalidate, private
GData-Version: 1.0
Last-Modified: Sun, 28 Sep 2008 12:45:51 GMT
Date: Sun, 28 Sep 2008 13:15:36 GMT
Server: GFE/1.3
Connection: Close
...


Edit: I did the telnet again with the following results:
Accept: *.* (Failed 406 Error)
Accept: * (Failed 406 Error)
Accept: */* (HTTP 200 OK)

[jrlewis]

You da man! I completely missed that. Its the little things...This just made things much easier. Very much appreciated. The google calendar feed is an atom feed, but it has slightly different elements and they use single quotes instead of double quotes for everything so I just need to add support to get those into the current scheme for the reader. To take full advantage of the feed will take the google calendar driver which I'm setting up the field structure for now. I will be posting something on that in the not to distant future.

Edit: New version up that now supports Google Calendar RSS Feeds.

[jrlewis]

I added another download to the first post. It is an example of how you could use a script to modify feeds that use time dependent filters, in this case a googlle calendar feed. There is a CMLBin macro and an accompanying vbscript. To use this import the CMLBin macro and put the vbscript in the same folder as the feeds.txt. Open up the script in a text editor and modify the lines shown below. To test the script open up a command prompt and run the script. You will get a message box pop up with a response from the CMLBin macro telling you if the reload was completed or not. Once you are sure it works remove the line in the last section that says WScript.Echo objXML.responseText. Next set up a scheduled event in windows to run the script daily at sometime after midnight (you could also use a scheduled event in CQC to do the same). The result will be your google calendar feed will be updated on a daily basis with the time interval you have specified in the script file. Note that the driver will perceive this as a new feed since the url has changed so all the defaults including the alias will be reset. The script and CMLBin macro could be customized to add the alias,TTL, etc. if desired.

For the time span of calendar events you would like change the interval (d,ww,m,y,...) and number of units in the interval. Below is a 7 day interval.
strInterval = "d"
intInterval = 7
This section is where you specify your feeds. For each feed set a different WriteLine. You can place the WriteLine("EOF") wherever you want the active feeds to end. For you google calendar feed change the strMagicCookie to match your feed.
objTSO.WriteLine("http://news.cnet.com/2547-1_3-0-5.xml")
objTSO.WriteLine("http://www.recipezaar.com/rss?categ=cajun")
objTSO.WriteLine("http://feeds.feedburner.com/cocoontech")

' Add calendar feed with sprecified time span
strMagicCookie = "florabamabob%40gmail.com/private-fd7ff448df495937fb9053419777b2e5/full"
Lastly you need to change the strMoniker to match that of your RSSReader.

' Notifies CQC CMLBin macro to reload the feeds
strMoniker = "RSSReader"

Edit: After giving this a little thought a better solution might be to include a FeedFilter field that will allow you to tack on any additional parameters to a base URL so you can do everything inside of CQC. The google calendar is a good example as are some of the recipe feeds where you may have the same base URL and it just has an additional query parameter that is passed like the recipezaar feed above. I would need to add some more back door commands so you can set these without directly interacting with the driver fields. Setting a filter directly would require that feed be the active feed, but that might not be a problem for some folks.

[jrlewis]

New version up that has a FeedQuery field and also support for setting some feed parameters using the ClientCmd instead of directly through the field interface. The FeedQuery seemed like a simple little addition at first, but I found out quickly it can be very powerful. For instance if you install this driver and add the feed http://www.recipezaar.com/rss?categ= you will get the newest recipes for this feed. Feel like italian? Set the FeedQuery to italian and it will requery the feed and give you the latest italian recipes. How about french?german?desserts?...? You name it. So with one base URL you can have access to hundreds of feeds. Now the feed/feed alias is going to seem a bit wierd. The feed alias defaults to the feed title at inizialization so depending on the feed and how you are using the FeedQuery you may need to make allowances for that to keep the FeedAlias in synch. This type of logic coupled with a static list browser for the FeedQuery options could be very powerful.

Another benefit of this is it still sees this as being one feed since the base URL is the same. This means that all the encoding and ttl data will be persistent where using the script method above wouldn't preserve that naturally. In the case of a Google Calendar you can run a scheduled event to run once a night and set the FeedQuery using the SendDrvCmd. In the action system the form is SendDrvCmd(Driver,ID,ToSet). The ID would be the index of the calendar feed and ToSet would have the form "FeedQuery:?start-min=yyyy-mm-dd&start-max=yyyy-mm-dd". There is a little logic involved to get the index and ToSet in the proper format, but it is definitely doable. You could also do it by setting the FeedIndex to that of the calendar feed and then write to the FeedQuery field directly. Either way this is a much better solution than using a script to update the feeds. This will have to tide folks over who are waiting for a full fledged google calendar driver as I wion't have any time this week to work on it.

[anogee]

I can't say I "digested" all this yet, but it looks very powerful and helpful. My problem, being new to CQC is the generating of dates on the fly. Lets say I want all the calendar events in the next 5 days. How do I go about generating "start-max=yyyy-mm-dd" if I know that today is 2008-09-29? How do I determine date plus 5 days? It was easy in VB, but how do you do it in a CQC action?

[jrlewis]

Look at the Action System documentation for FormatTimeVal. To get the current date in YYYY-MM-DD format you would use this.
FormatTimeVal([local],LVar:CurDate,Pattern,"\%(Y,4,0)-\%(M,2,0)-\%(D,2,0)")

For the day + 5 it is a little trickier, but I think about the only way to do it is set a local variable to %(RTV:TimeStamp) which you can get by right clicking in the Variable Value text box. This represents the number of 100ns intervals that have occurred since 1970. You will need to add 5 days to this in terms of 100ns intervals. To get 5 days in this form you would use 5 days x (24 hours/1day) x (60 min/1hour) x (60 secs/1 min) x (1 interval/1e-7 secs). It ain't pretty, but it works. If you work it out it is # days * 8.64e11. After that you would just use that variable in the FormatTimeVal as above.

[anogee]

O.K. formatting is good. How about a CQC function to add 5 days to a date?

[Dean Roddey]

I put that on the list o' things to add. It's easy to do in CML, but there's no clean way to do it at the action level.

[beelzerob]

I've had to do that recently...just adding 5 minutes to a date. Is the "easy" way to do that to get the stamp, add 5 minutes worth of 100 nsecs to the stamp, and then set the stamp back to the time object? Because that's how I do it...

[jrlewis]

That's the only way to do it for an absolute time. Relative times can be done much easier.

[jrlewis]

Rob,

I missed that you were talking about CML. One way to get a time five minutes from right now would be to use GetCurStampPlusSecs(300) and it would return a Card8 five minutes from now. For a date other than now you would need to add it to the stamp, but you have the Card8.kOneMinute to use so it makes it easy.

[jrlewis]

Anogee,
You can't do what I had suggested about creating the time for + 5 days since the action system uses integer math so the largest number you can get is the maximum for an Int 4 which is 2147483647 (0x7FFFFFFF). Anytime you try to use token expansion to get a value you are going to get an error. So basically I was full of it. You are going to need to create a script to do this. Since you may run into wanting time strings for other things you could create a script that looks something like GetFormattedTime(TimeStamp,Interval,Magnitude,Patt ern). You could pass it something like GetFormattedTime(%(stdRTV:TimeStamp),"d",5,"%(Y,4,0)-%(M,2,0)-%(D,2,0)") to have it return your desired time stamp. If you made it generic like this you can feed it any invterval/magnitude/pattern combination.

[jrlewis]

Anogee,
Just so I don't get to have all the fun here is a macro that will take a time, an interval (second,hour,...), the magnitude, and the pattern and it will return a formatted time string. I didn't add comments just so you could try to figure out what's what. Handling months and years could be handled differently, but this was just a quick and easy hack. Enjoy...

Class=[NonFinal]
ClassPath MEng.User.CQC.Macros.GetFormattedTime;
ParentClass MEng.Object;
EndClass;

Types=
Enum=TimeInterval
Sec : "s";
Min : "u";
Hour : "h";
Day : "d";
Month : "m";
Year : "y";
EndEnum;
EndTypes;

Members=
Card8 m_TimeStamp;
Card8 m_Factor;
Time m_Time;
TimeInterval m_TimeInterval;
EndMembers;


Methods=[Public,Final]

Constructor()
Begin
EndConstructor;

Method Start([In] Card8 ToFormat
,[In] String Interval
,[In] Int4 Magnitude
,[In] String Pattern
,[Out] String Formatted) Returns Int4
Begin

Try

m_TimeInterval.FromText(Interval);

Switch(m_TimeInterval)

Case TimeInterval.Sec :
m_Factor := Time.kOneSecond;
EndCase;

Case TimeInterval.Min :
m_Factor := Time.kOneMinute;
EndCase;

Case TimeInterval.Hour :
m_Factor := Time.kOneHour;
EndCase;

Case TimeInterval.Day :
m_Factor := Time.kOneDay;
EndCase;

Case TimeInterval.Month :
m_Factor := Time.kOneDay * 31;
EndCase;

Case TimeInterval.Year :
m_Factor := Time.kOneDay * 365;
EndCase;

Default :
EndCase;

EndSwitch;

m_TimeStamp := ToFormat;
If (Magnitude < 0)
m_TimeStamp -= m_Factor * TypeCast(MEng.Card8,Magnitude.GetNegated());
Else
m_TimeStamp += m_Factor * TypeCast(MEng.Card8,Magnitude);
EndIf;

m_Time.SetFromStamp(m_TimeStamp);
m_Time.SetDefFmt(Pattern);

Formatted.Clear();
Formatted.AppendFmt(m_Time);

EndTry;
Catch
Return -1;
EndCatch;

Return 0;

EndMethod;

EndMethods;

[zaccari]

Hmm, WebCalendar supports RSS feeds now. I thought I'd give the driver a whirl to see how my calendar fills fields and if it'd be usable. Unfortunately, the driver is returning 302.

Russ...

[jrlewis]

Russ,
A 302 status code is a Found which indicates a redirect. The driver will make a maximum of 3 "hops" to try and get to the end of the redirect chain. Let me take a look at that feed and I will get back to you. Is this an online feed you are accessing?

[zaccari]

Boy, I'm wearing egg on my face. Should have tried it with a browser and I could have seen that "You're not authorized" is a good reason things aren't working. Back to the docs to figure out how you authorize.

Russ...

[jrlewis]

You could make omelets with my egg incidents. With the feed you sent me it returns "login.php?" for the location so it errors out at that point because it can't assign that to a URL object as a fully qualified path. You may try using the query paramters ?user=Me&password=you at the end of the URL for the feed instead of just the ?user=

[anogee]

I will try this soon. My first Macro project. ;-)

Thanks.

[Dean Roddey]

See the action reference guide for how you would get that value back out of that macro. When you invoke a macro from an action, any [Out] or [InOut] parameters to the macro's Start method need to be passed variables. The output parameters will fill in the provided variables.

[anogee]

I'll check that out. Thanks.

Allen

[wuench]

BTW, I am not sure the guide points this out but macro parameters are space delimited. So if you have multiple parameters in your Start([InOut] String parm1, [InOut] String parm2, [InOut] String parm3) method you would call it from an action as Macro parm1 parm2 parm3. It took me a while to figure that out...

[jrlewis]

That's the fun part I was talking about...figuring out how the heck to use it...

[jrlewis]

New version posted...There was some discussion in this thread (http://www.charmedquark.com/vb_forum/showthread.php?t=7205) that it was possible the driver was causing an error related to a driver being running for too long or even stuck I guess. Good discussion and thanks for all the suggestions. What I did was to allow the timeout for the SendGET that retreives the thread to be user adjustable. The field is FeedTimeout and the default is 2 seconds and it can be set from 1-5. I decided not to try and stagger the feeds at startup because they could eventually end up on top of each other again. Instead I just limit the Poll() to only processing 1 feed per poll cycle of 1 minute and this will insure they are always staggered. Another new feature I added as a FeedTickerOrder field. This allows you to select the order (Oldest, Newest) of items in the feed ticker. And finally, when a feed is updated the driver sends out a user action event in the form of FeedUpdated:FeedIndex (e.g., FeedUpdated:2). For anogee if you are still getting the error you mentioned above after installing this check back in and see if you are seeing the same error or not.

[anogee]

I will try it. I'm not sure if this was causing the errors or not, and they don't happen all the time. I might need your help on trapping the event. I did this once to capture a UPB link, so I'm guessing its similar.

Thanks for the update, by the way.

[jrlewis]

For trapping the event you would want to set up a triggered event and for the filter you would use "Is User Action For" and in the compare value use the moniker of the driver. This will trap any user action events coming from the driver. If you only wanted to trap specific message types you would use "Is User Action From" where you can specify the moniker and the message type. In the action you would use TrigEvent::GetUserActionVals and set the event type and event value to local variables. Then you could use the value which will be the feed that was updated to send a Devices::QueryDrvText(Moniker,%(LVar:Value),"FeedTicker",LVar:FeedTicker) to get the FeedTicker for the updated feed. If you wanted you could use an If/Else to check if the feed is the active one and if so just get the feed ticker directly from the field value. If it isn't then you use the backdoor query to get the feed ticker.

[anogee]

Hi:

O.K. Had some problems. Here is what I did. Removed the older driver from the CQCServer. Imported the new driver. Added the new driver in using the same Moniker as before. When I imported, I noticed that it said Ver. 0.7 and not Ver 0.8 Seemed to go O.K. but now the driver is sitting in a "Wait for Initialization" state.

So, did I replace it correctly? I wasn't sure if I had to remove the old one first, but I figured it can't hurt.

Thanks.

Allen

I PM'd you my error log.

[jrlewis]

I got rid if the quote token and the methods I was using to build up some quote strings and replaced it with the new BuildQuotedCommaList method of the StringTokenizer class. That didn't get introduced until 2.4.15. You must be running an earlier version. Let me put together a 0.8b that doesn't use this method so you can still use the driver until you get up to a version that supports that method. I also checked the manifest and see I did forget to update the version there.

[anogee]

Yeah, I generally stick to the .0 builds. I'm on 2.4.0

[jrlewis]

Try the 0.9b driver pack.

[anogee]

For the uninitiated you will want to install the driver first and get it up and going before importing the template pack.

For the uninitiated, like me, what does the template pack actually do? Does this provide the widget with the scrolling headings?

[jrlewis]

When you create a template in the IV editor and you want to export it so other folks can use it you create a template pack. The template contains static and field based elements. For the field based elements when you import the template pack it looks for drivers with the same make/model. If it doesn't find a driver installed with the same make/model it warns you. You can either ignore the warning and continue the import or abort.

If you ignore and import the template when you install the driver that the field widgets reference you have to give the driver the exact same name as the driver the template was developed with. Otherwise you would have to go in and change all the widget field references. You also would need to go into all the actions and change any references to the old driver moniker. If you install the driver first before importing the template you can name it whatever you want. When you import the template pack it will find the driver of the same make/model and replace all the widget references. You still have to replace the driver moniker references in the actions though.

In most cases when using a template created for a specific driver it is easier to just use the same name unless it is something you are incorporating into your template namespace and it is worth the time to do the work. Many folks who create templates will create a variables driver that mimics the names that a series of templates would contain for a particular system configuration. I believe Ellisr63 (Ron) does this. I have written a few drivers that don't control a piece of hardware so it would be a bit of work to just take the driver and html doc and create an interface for it. For things like the RSSReader and recipe drivers I needed to create the interface to test them out so I just pass them along so folks can easily try out a driver to see if it is something they would want to use.

[anogee]

Tried 9b, and it connects, but anytime I try to write to it in the Field Browser, I get an error that it was rejected by the driver.

[jrlewis]

What could have happened is the configuration data got buggered. Try this. Remove the driver and put the EOF above all the feeds and then add one feed above the EOF that wasn't already there. If you have a google calendar feed just change the time limit. Install the driver and it should be fine. Add your old feeds and issue a ReloadFeeds and it should fix the problem. Unfortunately you will lose the old settings for encoding, tickerspace, etc. If it doesn't than put the driver verbosity into low before you issue the command and check the logs for the error that was issued.

[anogee]

O.K. Did it, and that seemed to have fixed things. Thanks.

Allen

[anogee]

O.K. Here is what I setup. Does this look O.K.?

Created an action. Filters with "Is User Action For" and "CQC_RSS_Reader" for the Comp Value. (To trap all events.)

In the Action I do this:

TrigEvent::GetUserActionVals(LVar:EventType, LVar:EventValue)
If System::Equals(%(LVar:EventType), FeedUpdated)
If System:Equals(%LVar:EventValue), 0)
Devices:FieldWrite(CQC_RSS_Reader.FeedIndex, 0)
Stuff when feed 0 is updated.........
End
If System:Equals(%LVar:EventValue), 1)
Devices:FieldWrite(CQC_RSS_Reader.FeedIndex, 1)
Stuff when feed 1 is updated.........
End
End

Its a bit different than you described, but should work, right? (Sorry, doesn't seem like the BBS honors leading spaces or tabs, so used periods.)

Allen

Update: It working. It seems to stagger updates like it should. Thanks.

[Dean Roddey]

Use 'code' tags around stuff that you want to maintain the spaces in.

[anogee]

Yes, makes sense. Thanks.

So is there any way to copy from the action directly? That has been very frustrating.

[jrlewis]

If you copy from the action and then paste into a text editor you get the format that looks like an action trace. You don't get the nice formatted code like you entered above.

Most likely this will be the only user action sent from this driver so you could skip checking the event type, but doesn't hurt to future proof now. If you have specific tasks you want to perform based on the feed then what you have is fine. However, you don't need to set the feedindex in the driver if you don't have to. You can manipulate all the writeable fields (other than the feedindex and item index) using the Devices::ClientCmd. You can query all the readable fields (except feedindex and itemindex) in the Devices::QueryTextVal. If the only way in which you interact with the driver is through actions and never directly in the IV than it would be easiest to do it the way you are doing it by directly setting the feed index. Working with the fields directly is a little easier than using the backdoor query/commands. Those exist so you could drive several IV's off the same driver without them fighting for control of the driver.

[anogee]

O.K. That is handy to know.

What is the easiest way to temporarily suspend updating, if I'm away or asleep? It looks like I set the Override TTL to true, then set Feed TTL to a large number, and repeat for all feeds. To reverse this, change the Override TTL to true, then anything else to get it to refresh? Is that the best way?

[jrlewis]

I could add a field that allows you to turn the polling on or off. The other option would be for you to create a variable in the variables driver and use that to set responding to user action events on or off.

[anogee]

I already have a variable that represents sign-on/sign-off, but the problem is the driver still polls, and thus wastes some Internet traffic. Not a big deal, but if you happen to create a polling on/off switch, that would be great.

I guess another option is to always leave it on, so when the sign turns on, the information is uptodate. I set the sign to an unused message when I'm away, which basically blanks the sign.

[jrlewis]

Unless you are running a ton of feeds I wouldn't worry about it. The traffic is minimal unless you have an active feed. Even at that you are probably only going to get a poll every 15+ minutes per feed. If a feed supports last-modified then things are slimmed down even more. However, I will put a poll switch into the driver.

[anogee]

Anyway you could send me version 0.8 again? I'm still not having good luck with 0.9b It seems to stop updating, with many of the fields being empty, and when I force an update, it tells it was rejected by the driver.

Thanks.

[jrlewis]

When I rolled stuff back is was a quick cut and paste so I could have missed something. First lets see if we can figure out the problem. If a feed doesn't appear to be updating check the FeedStatusCode and FeedLastUpdate. This will tell you the status code for the last request and the last time the feed got a successful update by either replying with new data or responding with a 304 which signifies "Not Modified". I ran into a couple of feeds that would throw an exception and not return a 304, but the RepText would contain "Not Modified" so I could catch that as well. If a feed isn't updating and it errors when you issue an update feed for that feed it is most likely a problem in the process feed method. I tried to catch every possible scenario where it could fail and I don't even clear out the item data for a feed until a successful response with new data is returned. Set the verbosity to high for the driver and issue the command that is causing the error and look to the log and send me the error.

[anogee]

I think (but I'm not positive) that the problem occurs during the course of starting and stopping CQC. When its running, and if I don't touch CQC, I'm pretty sure it keeps running, but I do need to do more testing.

When I find it and it has stopped working, most of the fields are blank, with the exception of the true/false feeds, and they are false, maybe how it might look if there were no feeds at all. If I attempt to change the FeedIndex or do an Update Feeds or Feed, I get an error that tells me that the driver rejected my attempt. If I go through the proceedure of changing the feeds, then doing a reload, it seems to work. I change my feeds back, do a reload, and it still works.

I can do more debugging if you like. Its happened maybe three times, so not often.

[jrlewis]

That is a ConfigStr issue. I mussed something up when recreating the old way of setting up the ConfigStr. Let me take a look at it and I will see what's wrong.

[jrlewis]

Since you are probably the only one using it I didn't rename it. The 0.9b up there will work. During the transition to get back inline with consistent field naming I renamed TickerSpace to FeedTickerSpace. It wrote out TickerSpace in Terminate(), but was looking for FeedTickerSpace in WaitConfig(). Load it up with a feed that isn't in the current list and then you won't have any problems. I have it set up the driver to get defaults for feeds in WaitConfig, but I have it in the processing loop and I'm not handling the error gracefully there.

[anogee]

O.K. so try the new 9b? I will do that.

Thanks.

[beelzerob]

Well, I installed it, and I copied what I THOUGHT was the RSS feed string...but I'm getting "Failed to retrieve thread" everywhere. Where do I start tackling this?

[beelzerob]

Here was the feed I was using:

http://www.weather.gov/alerts-beta/wwaatmget.php?x=PAZ036

Also, I encountered something rather weird....I installed the driver as RSSReader, and gave it the above feed. That didn't work. So I removed the driver, and during the install I gave it an RSS string I knew worked (for craigslist), and when the driver came up...it still had the weather feed on it. So I removed it again, reinstalled it again...and it still comes up with the weather feed no matter what feed string I give it. I guess this is related to the config file?

[jrlewis]

Since this is just basic scraping it can run into issues when folks decide to format their feeds just a little different. Version 0.91 should handle these feeds. As far as the feed not clearing out at one time there was an issue with the config file. There is a lot of data being persisted across startup/shutdowns and the config file was not being hanlded gracefully for invalid feeds. I'm pretty sure it is fixed and I just tested it for the situation you describe and it is working fine on my end. Hard to say without knowing exactly the steps you took.

[anogee]

Here was the feed I was using:

http://www.weather.gov/alerts-beta/wwaatmget.php?x=PAZ036

Also, I encountered something rather weird....I installed the driver as RSSReader, and gave it the above feed. That didn't work. So I removed the driver, and during the install I gave it an RSS string I knew worked (for craigslist), and when the driver came up...it still had the weather feed on it. So I removed it again, reinstalled it again...and it still comes up with the weather feed no matter what feed string I give it. I guess this is related to the config file?

Why did you use that location and not this one?

http://www.weather.gov/alerts/wwarssget.php?zone=PAZ036

It seems like this one would work with the RSS reader better.

[beelzerob]

Here's what I'm currently doing:

1) Remove the RSSReader-named driver. (it was currently fed the weather RSS, which was showing "Failed to retrieve")
2) Install the RSSReader driver.
3) Paste http://altoona.craigslist.org/search/sss?query=roofing&minAsk=min&maxAsk=max&format=rss for the initial feed.
4) Leave the rest of the options and finish the install.
5) Up it comes, but all the fields still show info from the weather feed it had before I uninstalled it.

I was just about to do it again.

1) Install the driver, picking the moniker RSS1.
2) Paste in a specific RSS feed URL.
3) Complete the install, then remove the driver.
4) Install the driver, picking the same moniker.
5) Choose the default RSS feed (CQC).
6) Complete the install.

The RSS1 driver will still show the RSS feed you chose in step 2. So it's like it's stuck on the initial RSS feed you give it.

I'm guessing I could go into the macroFileroot and delete the RSSReader directories there and be returned to sanity...so not a big deal. Might just be a quirk that has to be listed so people don't drive themselves insane.

If this is just an XML scrapper, then I guess that raises the question...what about creating a java reader (since there are so many of them out there already) and then using the field generator to keep it populated? I guess I don't know how mature this driver is, so I don't know if it was thrown together and works for most feeds or if maybe there'd be a more robust solution from utilizing an external java server. I'm always loath to use 3rd party programs for CQC drivers, but in this case we're not communicating to hardware anyway, so it doesn't bother me near as much.

Ultimately I'm just trying to get reliable (and reliability is key!) severe weather info available in CQC, so I can trigger off of specific weather conditions. Any suggestions on how best to use this driver to do that?

[beelzerob]

Thanks ano, that one works! I just copied the one I thought they wanted me to (I claim at least some ambiguity on their site when it comes to WHAT you're supposed to copy at the RSS feed)

And of COURSE it's calm right now....I need to find an area with good weather alerts to see what use I can make of that data.

[beelzerob]

Oh...the reason I was using the other one is because if you follow the link on the one you sent me, it says:

This page is expected to be phased out and replaced by the new version available here (County listing) by December of 2009. Please update your bookmarks and feed readers!;

I was using the one linked there to the "new" one. And for some reason that's the one that doesn't work.

I could use this old one that appears to work well up until Dec I guess....

[jrlewis]

The reason the old feed was still there is because you weren't removing the feeds.txt file (or just editing the existing one). The driver looks to see if a file already exists for that moniker and if it does it uses that one. If it doesn't exist it creates a new file with the feed you specify when installing. The description for that prompt is shown below and the behavior for the feeds.txt file is spelled out in the html file for the driver.
Enter the initial RSS Feed for the driver. Other feeds can be added later via a configuration file. If you have previously installed this driver with the same Moniker a configuration file may already exist for this driver.

Given the way this driver needs to operate I would never consider it to be a candidate for an official release (even though it will handle most anyting you throw at it). This is a case where having a middleware app would make sense... at least until xml is exposed to CML. I wouldn't do it in java and I wouldn't use the field generator either. I would keep as much of the automation logic as possible in a native CML driver and only use the middleware to do the heavy lifting. The field generator has its place, but it currently lacks the ability to provide fine grained control a native cml driver can provide, especially for a data driven driver like this.

[beelzerob]

Well, to clarify...the "reliable" part was in reference to parsing feeds successfully when the source might go and change something or be all naughty. It's a concern for this particular application, since this could be your only warning at night that a tornado is coming...so to suddenly go dead because of some strange character in the feed...well, that's what I'm trying to avoid. *shrug* I don't know that anything even in java would be any better at parsing if they go and omit some key character.

I had wanted to do an RSS reader before this, but I was also waiting for XML in CML...didn't want to do all the work that you ended up doing. So, since that's still not on the horizon...what would you use for middleware if not java?

I guess the real question is...how would the middleware improve on what you've already done, since yours seems to "get the job done" as it were. What is it still lacking?

[jrlewis]

My guess is these feed formats remain pretty stable so the odds of it changing format at a mission critical time is unlikely. That being said if CQC is your last line of defense in such a situation I would be a bit leary regardless. In times of high probability of those types of weather conditions I would want to have at least a couple of other warning systems available. At least one that was battery operated.

For a middleware app I would write it in C# if for no other reason its what I know and don't have a need/desire to learn java at this point.

[beelzerob]

Oh, hehe..ok. I'm the same way with java (it's what I know...what I use at work, too). I didn't know if there was specifically better-suited to the task.

I wouldn't rely on stuff like this for my smoke alarms or something like that...it's just that at this point, I don't HAVE any means to be notified of some imminent bad weather, especially at night while we're sleeping. Our first summer here, we were watching TV late one night (actually, I was using the PC and I think she was reading, but the TV was on), and the weather alert scrolled along the bottom saying there was a tornado nearby. We didn't realize it was even a possibility around here...we're tornado warning newbies. (Our second realization was that we had no place to hide...our rental had no basement).

So, something through CQC is better than what we have now.

[anogee]

And of COURSE it's calm right now....I need to find an area with good weather alerts to see what use I can make of that data.

Pick Dallas. Its busy this time of year. I got to go there next week. :-(

[klindy]

Pick Dallas. Its busy this time of year. I got to go there next week. :-(

Weather here keeps us on our toes for sure....:-)

[anogee]

Oh, hehe..ok. I'm the same way with java (it's what I know...what I use at work, too). I didn't know if there was specifically better-suited to the task.

I wouldn't rely on stuff like this for my smoke alarms or something like that...it's just that at this point, I don't HAVE any means to be notified of some imminent bad weather, especially at night while we're sleeping. Our first summer here, we were watching TV late one night (actually, I was using the PC and I think she was reading, but the TV was on), and the weather alert scrolled along the bottom saying there was a tornado nearby. We didn't realize it was even a possibility around here...we're tornado warning newbies. (Our second realization was that we had no place to hide...our rental had no basement).

So, something through CQC is better than what we have now.

You might just want to buy a cheap SAME Weather Radio. Maybe about $60 fro Radio Shack. Mine has relay contacts when an alert goes off and I pipe it through my house. The only problem is, sometimes the coverage area is pretty big, so you get woken up about a storm that is nowhere near your house.

[beelzerob]

Did something in this driver change recently? I just updated, and the driver won't get past initialization, apparently due to a compile-time issue:


05/06 22:52:43-cqc-server, CQCServer, CQCDrv_RSSReader_Thread12
{
CQCKit, MEng.User.CQC.Drivers.RSSReader.DriverImpl.2132, API Failed/Data Format
A macro parse event occured
The two sides of a RHS expression must be of the same type
}
05/06 22:52:43-cqc-server, CQCServer, CQCDrv_RSSReader_Thread12
{
CQCKit, MEng.User.CQC.Drivers.RSSReader.DriverImpl.2136, API Failed/Data Format
A macro parse event occured
Expected a method call, object reference, or flow control statement here
}
05/06 22:52:43-cqc-server, CQCServer, CQCDrv_RSSReader_Thread12
{
CIDMacroEng, CIDMacroEng_Parser.cpp.2184, Status/App Status, Error: 6505/0/0
An unrecoverable error occured while parsing
}

[jrlewis]

The issue is the ClientCmd method. It was changed along with the QueryTextVal. The ClientCmd in this case allows a user to set things like Encoding, TickerSpace, TickerOrder, etc. for a feed that isn't the active feed for the driver. There is another driver pack posted that should work, but it hasn't been compiled with the beta.

[eded9698]

The latest RSS reader isn't compiling with the latest beta.

[jrlewis]

Version up now fixed for latest beta. Fixed the ClientCmd, but forgot the QueryTextVal.

[eded9698]

Version up now fixed for latest beta. Fixed the ClientCmd, but forgot the QueryTextVal.

I've installed the driver from the first port saying yes to the defaults and I get "wait to initialization". I Check the log and it says it can not compile.

[jrlewis]

Are you sure you downloaded the 2_4_46 version? That should work just fine under the newest betas.

[beelzerob]

Well, it's workin' for me. But I'm still trying to figure out...

If I am subscribed to a weather alert RSS, and I want to have the latest alert info displayed on a marquee widget....which of the many fields do I use for that? I also want to use it to parse for keywords, like warning, watch, tsunami, etc.

[jrlewis]

Read the html file for the driver. It is pretty detailed. FeedTicker will give you a string field with all the items for the current feed separted by spaces specified in FeedTickerSpace. The driver also sends out a user event when a feed is updated with new information. So if you set up a triggered event filter to capture events for a particular feed you can do things like look for specific words in the FeedTicker. If the feed of interest isn't the current feed you can either use a QueryDrvText command to access the feed ticker that way or set the FeedIndex to that feed and just get at the ticker directly through the field interface.

[beelzerob]

Read the html file for the driver. It is pretty detailed.

:roll: I did read the html, and I agree, it IS pretty detailed...but for someone who has only the vaguest understanding of RSS, detailed doesn't exactly equate to intuitive. With no less than 21 fields that start with "Feed", it's pretty easy to lose track of the single Feed field that you want...though I do appreciate that it has the suggestion in the description that says "Use this with a marquee widget". That does help.

Didn't know about the event, though. Is that documented in the html? I couldn't find the event type and data mentioned.

Thanks!

[jrlewis]

There is definitely a lot of things going on for sure. Fields with the Feed prefix are related to feeds and fields with the Item prefix pertain to items. If you look at the FeedUpdated field description you will see the user event format. If you read through this thread from the start you will see some of the discussion about the evolution of the driver and it will give you a better understanding of the purpose of some of the fields. You might also want to download the templates from the first post. They are fugly, but they give you an idea of how to interact with the driver. I forgot to include the needed CMLBin macro that was updated for the new beta version so that has been added to the first post. In the template widgets with a red border should be able to be double clicked and edited (allow direct writes), but one of the upgrades toasted that at some point and I haven't gotten around to fixing that.

[eded9698]

Are you sure you downloaded the 2_4_46 version? That should work just fine under the newest betas.

Yes I download .46 first with no results, so I tried the older one. Same result. Sent Dean a dump file and he said the driver isn't compiling. Is there something I need to delete to do the upgrade?

[Dean Roddey]

I was seeing these errors:


Seq#497 - 05/13 12:30:36-mastercqcserver,CQCServer,CQCDrv_RSSReader_Thread1 6
{
CQCKit,MEng.User.CQC.Drivers.RSSReader.DriverImpl. 2178,API Failed/Data Format
A macro parse event occured
Overrides must have identical parameters, return, and visibility as the overriden method
}


Which would pretty likely be the backdoor command issue. However, Erik, if ever loaded up any modified version of this driver that someone provided, then you will still be getting that one, and that could explain why no change. Also, once you did load it, you will have to cycle the machine before the new driver will load, even if you didn't have some other modified version. So make sure you do that. Otherwise the old one is still running.

Look on the master server in [cqc]\CQCData\DataServer\Manifests\User and see if there are any .Manifest files in there. If so, are any of them for the RSS driver? The name will be obvious if so. If there is, delete that file and restart.

[beelzerob]

Ya, it's workin' good for me. finally got to see "Severe Thunderstorm Watch" scroll along the bottom...woot! Now to write a macro to find some key phrases, and key it off the event.

Thanks jr, it's workin' nice.

[eded9698]

Ya, it's workin' good for me. finally got to see "Severe Thunderstorm Watch" scroll along the bottom...woot! Now to write a macro to find some key phrases, and key it off the event.

Thanks jr, it's workin' nice.

beelzerob, when you get that marco written can you post it? I'm using the RSS reader for the same reason you are.

[eded9698]

I was seeing these errors:


Seq#497 - 05/13 12:30:36-mastercqcserver,CQCServer,CQCDrv_RSSReader_Thread1 6
{
CQCKit,MEng.User.CQC.Drivers.RSSReader.DriverImpl. 2178,API Failed/Data Format
A macro parse event occured
Overrides must have identical parameters, return, and visibility as the overriden method
}


Which would pretty likely be the backdoor command issue. However, Erik, if ever loaded up any modified version of this driver that someone provided, then you will still be getting that one, and that could explain why no change. Also, once you did load it, you will have to cycle the machine before the new driver will load, even if you didn't have some other modified version. So make sure you do that. Otherwise the old one is still running.

Look on the master server in [cqc]\CQCData\DataServer\Manifests\User and see if there are any .Manifest files in there. If so, are any of them for the RSS driver? The name will be obvious if so. If there is, delete that file and restart.

Thanks Dean, I deleted file earlier and its working.

[beelzerob]

absolutely.

My thought was that you'd pass in the weather string to the macro, and that the macro would write the resulting watch/warning/etc to a particular variables field. The variables driver moniker and string field are constants defined at the top of the macro, so you can change those to whatever you actually have.

I suppose it could otherwise just return the resulting advisory string as a parameter...

[beelzerob]

Hey Dean...I'm trying to pass in the string that's in FeedTicker...but when I pass it into my macro, I guess it's expanding out the field value and then parsing that based on the spaces...so instead of passing in a single string value, it thinks I'm passing in about 12 values.

So how do you pass to a macro a string parameter that has spaces in it?

[beelzerob]

*Weather parsing macro moved to its own thread*

[jrlewis]

If you want to trap the event and just parse it in the action system you could do what is shown below. In the "Do Stuff" sections you could do things like have a popup come up on various interfaces, use TTS to read out the FeedTicker that caused the warning, etc.

To catch the event you would use a "Is User Action From" for the filter and for the Field use the Monkier of the rssreader and for the Comp Value use FeedUpdated. That will catch the event. In the action you could then parse out the message. Note that the weather warning may come from your weather feed whilte that isn't the active feed in the driver. So you first need to determine if the updated event is for the feed you want. Probably best is to give the feed a distiinct FeedAlias that won't change over time and check that the update is from that particular feed.

TrigEvent::GetUserActionVals(LVar:Event,LVar:FeedI ndex)
Devices::QueryDrvText(Moniker,%(LVar:FeedIndex), FeedAlias, LVar:FeedAlias)
If System::Equals(%(LVar:FeedAlias), "My Weather Warning Feed")
Devices::QueryDrvText(Moniker, %(LVar:FeedIndex), FeedTicker, LVar:FeedTicer)
If LocalVars::Find(%(LVar:FeedTicker),0,"Warning",False)
// Do Stuff
End
If LocVars::Find(%(LVar:FeedTicker),0,"Extreme",False)
// Do Stuff
End
End


The other option is to change the feed to the feed that is the weather warning.

TrigEvent::GetUserActionVals(LVar:Event,LVar:FeedI ndex)
Devices::QueryDrvText(Moniker,%(LVar:FeedIndex), FeedAlias, LVar:FeedAlias)
If System::Equals(%(LVar:FeedAlias), "My Weather Warning Feed")
Devices::FieldWrite(Moniker.FeedIndex, %(LVar:FeedIndex))
LocalVars::SetVariable(LVar:FeedTicker,$(Moniker.F eedTicker))
If LocalVars::Find(%(LVar:FeedTicker),0,"Warning",False)
// Do Stuff
End
If LocVars::Find(%(LVar:FeedTicker),0,"Extreme",False)
// Do Stuff
End
End

[jrlewis]

Rob,

Your macro is only going to work if the feed that is associated with the weather warnings is the active feed. So if anyone uses that they have to make the weather feed the active one in the field interface to be able to read the FeedTicker directly from the field as I have shown above in the second scenario.

[beelzerob]

One possible problem with the code you showed for checking for the words "warning" is that when there is NO warning, it says "There are no active watches, warnings....". So that would match, right? That's why I check for that string first, so I don't trigger in Warning also.

I only use the RSSReader for just weather right now, so that's the only feed I have. I'm headscratching to figure out what you posted..that whole feed thing baffles me something fierce, and I have no idea why. But it seems to me like it'd actually be easier for me to load an RSSReader driver JUST for weather, and then if I want other more trivial RSS feeds, I'd just load another driver for those. Inefficient, ya, I know...but I'm not making this PC scream from the load yet anyway....

The macro as written also will only read from a field and write to a field, so it's pretty generic in its use. Hopefully people can make the weather the active feed and then trigger the macro, right? Because then it would read from the field correctly?

[jrlewis]

The code I showed was just an example. Whatever makes sense for parsing would need to be used. Thinking about this a little I wouldn't add too many layers on to this. I would probably stick with doing everything you need to do right from the event that captured the feed update. Having a macro parse is probably more efficient and it could be set up with an Out parameter so you could pass back an action to take if needed. Passing the event around may not be the best thing to do. I would have to think about it.

As far as the code above when you have multiple feeds each feed has its own TTL (time-to-live) that it uses to check for feed updates. This all goes on behind the scenes and even though a feed may not be the active feed in the driver as you could have many feeds, the underlying data is still updated and the FeedUpdated event is a method to let you know a feed has been updated even if it isn't active in the field interface for the driver. This driver being heavily data dependent having just the field interface would make it impractical/impossible to use on multiple clients including IV's, event clients, etc. The QueryDrvText is accessing the QueryTextVal method of the driver. The driver exposes every field value for a given feed even if it isn't the active feed through the QueryTextVal Method. The driver also makes use of the ClientCmd method which allows you to write to fields for a given feed even if it isn't the active feed. All this allows for a data driven driver such as this to be accessible and configurable from mulitple clients all at the same time.

[beelzerob]

Oh, I have every belief that the RSSReader is just like the other CQC capabilities and drivers I've run into....completely and utterly baffling at first, and then gradually you begin to understand and comprehend how darn USEFUL they are.

Honestly, I've been using CQC for...what, 5 years now? And writing drivers all that time. But I only discovered macros like 3 weeks ago. And MAN are those things great! So I usually tend to avoid like the plague things that are new/foreign/complicated, and eventually I get around to understanding them. I'm sure the RSSReader will be the same.

For now I'll probably use it only as my single feed weather string source supplier. Kinda like using an iPhone as a calculator just because that's all you know how to do with it....but that's where I'm at. :tounge

[jrlewis]

Having pondered and pontificated on this for this particular case given its use I think having a separate rssreader driver just for weather alerts would be the best route. I would still trigger off the user event just so you don't have to worry about setting the trigger on the field values.

[beelzerob]

I would still trigger off the user event just so you don't have to worry about setting the trigger on the field values.

Is that this one?


"Is User Action From" for the filter and for the Field use the Monkier of the rssreader and for the Comp Value use FeedUpdated

[beelzerob]

jr, when does the event for the feed happen? Does it happen when each update time occurs (the TTL time, right?), or does it only happen if the feed has actually changed?

My TTL is 4, but I know the event hasn't happened in the past 10 minutes. So I'm guessing the event only happens if the feed changes?

[jrlewis]

The "Is User Action From" event filter would be the one to use to capture the event. The event is only sent when a feed has been updated with new data. Internally the driver keeps track of the first item for the feed and after a feed has been processed it checks to see if the first item is different. If it is then the FeedUpdated field is set to True and it fires off the user event. If the weather feed is specifying a TTL than you probably shouldn't override that. It won't do any good to check for new feed data every 4 minutes if the feed only updates every 15 minutes. You should probably check on their site to see what they say about banging on the server with requests.

Edit: I see they say they are updated every 2 to 3 minutes so you could go with an even shorter TTL.

[beelzerob]

I never changed the TTL value, so I assumed it came from the site? I'm GLAD it's not every 15 minutes....that's way too much time for me to suddenly not be in Kansas anymore.

I'll have to find out tomorrow if the feed and event and all that updates it as necessary.

[jrlewis]

If your FeedTTL is 4 for that feed you had to have somehow set that. Looking at those feeds they don't provide a TTL so the driver would default to 15 minutes.

[beelzerob]

I just loaded a new instance of the driver and pasted in this feed URL:

http://www.weather.gov/alerts/wwarssget.php?zone=PAZ036

and it comes up again with a TTL of 4. I'm fairly sure I never changed it on the original instance, and obviously didn't with this new instance, so unless that TTL carries across all instances of your driver, it must be coming from them.

4 is about right I think.

[jrlewis]

That is a different version of the feeds than what I was looking at. They have several and that version does have a TTL. From the looks of it though none of these feeds will be able to be parsed for weather events from the FeedTicker unless the event comes up in the title for the feed entry. That title for an entry in a feed is what is used for the Item field and that is what the FeedTicker is built off of. The newer versions here (http://www.weather.gov/alerts-beta/) may be a better option as they provide extra information in what is known as the CAP (Common Alertings Protcol). It may be worth it to provide a driver specifically for this type of feeds.

[beelzerob]

Interesting...my macro updated correctly (wiping out the old Frost Advisory I had)...but looking at my marquee widget, it still says "Frost Advisory". And in the field browser, it's there too. So, at some point I guess it updated, which triggered the event, which wiped clean my weather text (and that's GOOD), but now the marquee appears to be out of date. Is there something I need to key off of to make the marquee stop showing stuff that's older? I guess with other RSS feeds, older stuff wouldn't matter so much, but with weather, that'd be kind of important.

[jrlewis]

The FeedTicker is just a collection of all the Items for a given feed so if a weather warning is still in effect it will likely remain active in the feed. The macro must be doing something other than just parsing out the FeedTicker because if it were you should be parsing that warning back out if it is still active in the feed.

[beelzerob]

Hmm...well, the macro is designed so that the only time it returns a blank string is if the feed string comes in as "There are no active watches...". Otherwise, it'll contain the string "Unknown" in it, so I know I'm not finding a term I should be.

It was kind of weird....I knew there WAS no more frost advisory, so it SHOULD have said "There are no active watches..", and from the results of the macro running, that appears to be what was sent, but when I looked on my marquee widget and the field browser, I saw "Frost Advisory". Even that was a little different..when the actual frost advisory was going, it was 2 items in the ticker..."Frost Advisory", and "Franklin Pennsylvannia 16603". All the warnings have been like that so far, but this time it just said "Frost Advisory". I knew it was different because it wasn't scrolling (it all fit in the screen).

I didn't see any warnings in the log that the macro didn't work. *shrug* I guess we'll have to wait until next time.

[beelzerob]

jrl, I'm seeing something strange...or more likely I'm confused.

I have only 1 feed going to this particular driver instance, it's my weather RSS feed. The item count is 1. So (correct me if I'm wrong)...."Items" contains all the RSS items for this feed...and "Item" contains the currently selected Item out of Items, as chosen by the ItemIndex. And FeedTicker is all of the Items appended to each other, separated by the chosen amount of spaces. Is all that correct?? If so, I'm amazed I've figured out that much!

Here's the part that's confusing me though....

Currently, for ItemCount I have 1...for ItemIndex I have 0. So that means that Item should be showing the single entry in Items, right?

But for Item, I have:
"Short Term Forecast - Franklin (Pennsylvannia)"

And for Items, I have:
"There are no active watches, warnings....etc".

And for FeedTicker, I have:
"Short Term Forecast - Franklin (Pennsylvannia) Short Term Forecast - Franklin (Pennsylvannia)"


I found this out because my weather warnings macro is based off the feed ticker, which if it held what was in Items, would have correctly deduced there are no warnings currently. But instead the ticker has 2 copies of "Item" in it, which gives me an "Unknown Unknown".

So do I still have this not figured out yet?

[jrlewis]

What you describe for item, items, and ticker is correct.

Is what you are showing for Item, Items, and FeedTicker from the driver or something you have parsed? I don't see any way that could happen even from the parsing of the feed going terribly wrong, but anything is possible. If you see something like that again grab the actual xml feed and post it so I can see where the problem might be.

One thing you can do to verify what is held internally is to access the fields using the QueryTexVal method to see if they give the same thing being shown in the field interface.

[beelzerob]

What I posted was verbatim out of the field browser...so no processing on my part.

Where do I find the actual XML feed, so I can post it?

[jrlewis]

Open the url you are using in the feeds.txt in a web browser and right click on it and select View Source. That will bring up the underlying source text that you can save to a text file.

[beelzerob]

Ok, will do tonight. Last I saw this morning it was still doing it.

[beelzerob]

Here's a shot of the field browser, as it currently exists:

http://i475.photobucket.com/albums/rr117/beelzerob/misc/rss_weather.jpg

and here is the XML feed:


<?xml version='1.0' encoding='UTF-8' standalone='yes'?>
<rss version='2.0' xmlns:dc='http://purl.org/dc/elements/1.1/'>

<channel>
<title>
Pennsylvania - (Franklin/PAZ036) - Current Watches, Warnings and Advisories for Pennsylvania Issued by the National Weather Service
</title>
<link>http://www.weather.gov/alerts/pa.html</link>
<lastBuildDate>Thu, 28 May 2009 19:33:55 </lastBuildDate>
<ttl>4</ttl>
<language>en-us</language>
<managingEditor>robert.bunge@noaa.gov</managingEditor>
<webMaster>w-nws.webmaster@noaa.gov</webMaster>
<description>
Current Watches, Warnings and Advisories for Pennsylvania (Franklin/PAZ036) Issued by the National Weather Service
</description>
<image>
<url>http://weather.gov/images/xml_logo.gif</url>
<title>NOAA - National Weather Service</title>
<link>http://weather.gov</link>
</image>

<item>
<title>There are no active watches, warnings or advisories in Pennsylvania (Franklin/PAZ036)</title>
<link>http://www.weather.gov/alerts/pa.html</link>
<description>There are no active watches, warnings or advisories</description>
</item>
</channel>
</rss>

[jrlewis]

Took at look at the driver and I can see why the current item info and feedticker is showing information from a previous update. When a feed is updated it is only updating the items, but not resetting the ItemIndex. There is also a problem with how the driver checks for an updated feed when it has only 1 item so the feedticker can lag as well. Not sure about the double item ticker you were seeing though. I couldn't see anything that would cause that. If you see the Items list being different then the current item and ticker set the FeedIndex to 0 and that should update everything. If I get some spare time this weekend I will make the needed updates.

[beelzerob]

Great! Thanks for lookin' at it. What I did last night was remove the driver and then reinstall it, and then the item, items and ticker were all in synch again. However, by this morning, the ticker was once again showing 2 copies of the single item. *shrug*

[jrlewis]

Thinking about it that makes sense now. What happened is the feed went from having 2 weather forecasts to the "no warnings" and the ticker got stuck at that point because it never got updated after that.

[znelbok]

I just upgraded to the latest beta and the driver is not working.

A macro parse event occurred.

driver version V0.91

Another comment, I don't find it very useful for the weather stuff because the feeds have html tags and they dont display very well (such as bold text and images) in a basic text format. I'd check which field that was but the driver is obviously not worked to find out, but I think it was the item description field

Mick

[jrlewis]

Make sure you grab the version that is built for the betas. There are 2 versions of the driver posted. HTML markup in the item descriptions is problematic. At some point I will add an option that will allow for the markup to be stripped. Not all feed sources properly escape the html markup so stripping it prior to expanding the text can be a bit hit and miss, but I will eventually add the option. I guess I can try and strip it out after it has been expanded out as well. In all cases you have to search through a given set of tags so differentiating what is html and what is simply meant to be text could be a pain as well.

[zaccari]

I think this is a Dean question, not for jrlewis as I originally thought. The manifest has a prompt asking for the initial feed. When I put in any feed with an & in it, everything after that is chopped from the end. Does it need to be escaped or is there something else we need to do? I haven't looked at the code but what is written to the config file doesn't have the &.

Thanks,
Russ...

[zaccari]

jrlewis, how does this driver handle no items in the feed? I manually edited the feeds.txt and put the correct parameter and I'm getting an exception "Index 0 is invalid for this collection object for http://www...

Russ...

[jrlewis]

It handles it fine as far as I know. When it gets a return from a feed it tries to parse out the feed. If no items are present then the item vectors are empty and setting the items for a feed internally is done based off of a count. Is that an unhandled exception or are you seeing that in the log file at a particular location?

[zaccari]

In AI I see an error in Item "Failed to retrieve feed." and an Item Count of 1. Feed Updated also remains False.

Russ...

[jrlewis]

That would happen if either the http request maxed out at 3 tries or there was an exception processing the feed. You can tell which happened a couple of ways. One is to look at the FeedStatusCode. That gives the last status code for the feed. If it is something other than 200 it likely couldn't get the feed in the allotted number of attempts. If it is 200 then try and put the driver in a high verbose mode and check the logs. You should see an exception that starts with "Exception_Moniker_ProcessFeed...". That would let me know exactly where the error is.

[anogee]

I think I have found a bug in the RSS Driver, but this may be a CQC 3.0.6 bug. CQC 3.0.5 works fine, BUT in CQC3.0.6, in the Admin Interface, if I attempt to modify the "ReloadFeeds" "UpdateFeed" or "UpdateFeeds" I get an "unhandled exception." Details are the "value must be a boolean" but I don't even get the chance to put true or false. Again in 3.0.5, it works fine. I verified this on several PCs.

Thanks.

[Dean Roddey]

This is a known issue which has been fixed in 3.0.7, the current beta release.

[anogee]

Hmmm, I'm not sure I want to go to 3.0.7 and its associated beta "experiences." Should I go back to 3.0.5 and where can I get it from? Sounds like your not going to post a 3.0.6a?

[Dean Roddey]

If you don't want to update to 3.0.7, just do without it for now I'd say. It only happens on write only boolean fields. You can easily set up a button or key mapped action to do the same in the meantime if you want to write to a boolean field. So it's probably not worth bothering with.

[anogee]

I have a minor RSS reader problem and its likely user error. Currently running 0.91 but I'll upgrade to 0.92 shortly.

I have 5 feeds, 4 news feeds and a Google Calendar feed. When a feed updates, it triggers an action using Is_User_Action_For: RSS_Reader

Once in the action I use GetUserActionVals(LVar:EventType, LVar:EventVal) to determine which feed is updated.

So the problem. I seem to be missing some feed updates, especially the Google Calendar feed. The RSS Reader Drive is getting the updates, but the problem is my action is not seeing it. I'm guessing maybe two feeds are being updated at once, and my action is only seeing one of them since my action can only process one feed update at a time. Its not a big problem with the news feeds since they are updated all the time, but the Google Calendar feeds doesn't change often, when I miss an update, I miss upcoming events.

So any ideas? Should I trigger differently?

[anogee]

Tried 0.92, and it broke my action. I see that the description field has changed. The problem for me is the description field contains the date and time span, which I need, but it now also contains the meeting text, location, attendees which I don't need. How do I separate the two?

How about, since the date has a "When:" before it, could you add "Desc:" after the date but before the description? That way I could detect the "Desc" text and delete everything after it. An idea at least.

Thanks.

[jrlewis]

I made some updates to try and make the Google Calendar feeds more friendly, but evidently not. I will have some time this afternoon so I can set it up so it is similar to the feed you posted in the other thread about integrating google calendar. For the feed updating I do use a Last-Modified in the header for the query so that may be messing something up for the google feed. Seems like I ran into something weird with updates and google calendar feeds but its been a long time since i looked at it. I will give it a gander.

[anogee]

As always, a big Thanks. :-D

[DaveB]

JR - Great job - works well. Have you ever considered adding a photo field? Several of the available feeds contain photos of the item.

[jrlewis]

Dave...any photos in the feeds are likely just links to photos and is part of the item description. What is in the item description is so varied it is hard to deal with things. What needs to be done is to have 2 description fields. One that is the raw description. That can be used in a cmlbin macro and displayed in a web widget. The other would be an attempted "clean" description that has any html formatting stripped out. With the new XML support in CML I will likely revisit this at some point and it will probably make things a little easier then what they are now.

[anogee]

I'm going to revamp my code this weekend, and I think I can work around the problems I have. To eliminate the Description, I can just search for the two spaces that are between the date and description in the ItemDescription field, and I can delete everything to the right of it. For the missing updates, I can just poll all the feeds on a regular interval instead of being triggered by changes.

One problem I can't get around, is in almost all feeds, I occasionally get non-printable characters mixed in, or at least non-US characters. Its usually that small "a" with the dots over it. I'm not sure where they are coming from but it would be great to have an option to filter them out. For Google Calendars, there sometimes seems to be this character right after the am or pm following an event time.

[jrlewis]

Polling the fields won't help if the feed itself doesn't get updated. I never got around to looking at that today...or was it yesterday...I will look at it this weekend. For the non-printable characters you can try overriding the encoding returned by the feed. I think I use UTF8 as default if the feed doesn't provide one, but you can override it by setting an OverrideFeedEncoding to true and then specify the FeedEncoding. Valid encodings are those listed in the TextXCoder class. The particulars are outlined in the rssreader.html file.

[DaveB]

JR; The feed actually contains a picture tag. The picture is not embedded in the field, it actually is a image tag in the field. The only feeds that are truly using this concept are Yahoo feeds.

<p><a href='http://us.rd.yahoo.com/dailynews/rss/topstories/*http://news.yahoo.com/s/ap/20081003/ap_on_go_co/meltdown_tax_breaks'><img src='http://d.yimg.com/us.yimg.com/p/ap/20081002/capt.085c0b3530644486915008002f6c7f8d.congress_fin ancial_meltdown_dclj110.jpg?x=130&y=86&q=85&sig=PXyOAeUwtiz4_UWX2gKSSA--'

You need to create code to separate this out and then display it on the screen via a field.

[anogee]

I worked on it this weekend, and learned a bit. I think I might be mistaken when I said the Google Calendar feed didn't update, although I do still see that occasionally. Two other problems lead me to think that. First, I'm using Google Calendar Sync at work to sync with Outlook, and there is a problem with that. If after syncing you attempt to remove an item from your Google Calendar, it will just magically reappear because the calendar sync is not "smart" enough to know to not resync it with your Outlook calendar. In other words, assuming Outlook and Google are synced, if I remove an event from Google, the next sync adds the event back to Google because its in Outlook, instead of Removing the event from Outlook.

The second thing I noticed is with the start-min parameter to get the RSS feeds. It seems to return events one day previous to the date you give it. For example, if start-min=2009-9-20 and start-max=2009-9-25, then Google Calendar returns any even on 9/19/2009 - 9/25/2009 I have no idea why that is, but it does do it that way for me.

So, even with the above info, I'm still occasionally seeing the Google Calendar feed either miss updates occasionally, or take much longer to update that it should. I have solved this for now by occasionally setting the UpdateFeed field.

[rocker]

Can somebody post a sample template pack so I can learn how to utilize the advanced features of the RSSReader Driver.

The subject can be of any type, I should be able to port it to sport and news for me and cooking and home decorating for my wife.

Thank you.

[anogee]

I'm not sure what a template pack it, but I use the reader for a variety of data. I haven't used the IV much, so maybe that is where the template pack fits in.

So lets take it one step at a time.

1) First figure out which RSS FEEDS you want. This doesn't require CQC at all. Safari, Firefox, and Internet Explorer can all display RSS feeds. For example, here is one I use for CNN.
http://rss.cnn.com/rss/cnn_topstories.rss
You'll typically see an item that is a link you can click on, a date it was posted, and a description. These are the items that CQC can read.

2) Install the CQC RSS driver, and find the "feeds" file it installs. These are where your feeds go. Put your feeds in there, one per line and end it with EOF.

3) Now, use the Admin Interface to look at the RSS Reader fields. The most important one is FeedIndex because it specifies which feed your looking at 0 to the number of feeds you have, minus one. If you just have one feed, set it to zero.

The next important field is ItemIndex, which specifies which item in the feed is of interest. Once that is set, there is Item, ItemPubDate, and ItemDescription which contain the info. That's pretty much it. The rest is just formatting to get it the way you want it.

FeedUpdated and FeedsUpdated tell you when the feeds get updated. Use those to trigger an event to feed your RSS feds from the driver.

Just take it one step at a time and let use know if you have problems.

[anogee]

Polling the fields won't help if the feed itself doesn't get updated. I never got around to looking at that today...or was it yesterday...I will look at it this weekend. For the non-printable characters you can try overriding the encoding returned by the feed. I think I use UTF8 as default if the feed doesn't provide one, but you can override it by setting an OverrideFeedEncoding to true and then specify the FeedEncoding. Valid encodings are those listed in the TextXCoder class. The particulars are outlined in the rssreader.html file.

Hmmmm in this document?
http://www.charmedquark.com/vb_forum/attachment.php?attachmentid=1334&d=1224095086

Anyway, I'm still getting strange characters instead of spaces on MSNBC feeds. They say UTF-8 like the RSS driver says says. Any other Encoding you could recommend?

[anogee]

And, if you use UTF-16, it crashes the driver. :-(

[sic0048]

I don't suppose you can use this driver to pull information out of an XML feed?

I'm thinking of this weather station feed as an example. (http://api.wunderground.com/weatherstation/WXCurrentObXML.asp?ID=KNYBUFFA16) It isn't in RSS format, so I am wondering if it can still work (or be modified to work).

Thanks,

[anogee]

RSS is XML so, yes they are both interchangable.

[potts.mike]

What is the easiest way to load an itemlink from a list into a web browser? I have a feed of footbal scores and I want to click on each game to load the espn gamecast into the browser.

[Dean Roddey]

BTW, XML support is now in CML, so at some point I guess this guy could be upgraded to use the real XML parser. Not sure if it's worth it or not.

[anogee]

What is the easiest way to load an itemlink from a list into a web browser? I have a feed of footbal scores and I want to click on each game to load the espn gamecast into the browser.

Most web browsers can read RSS feeds directly, so just view the RSS link directly, and the links will be there to click on. The RSS reader actually extracts the text from RSS feeds, like I need so I can display the data on my LED sign. If you don't care about extracting the individual feed message text, you don't need the reader.

[jrlewis]

Last things first. There is an updated version of the reader that provides better google calendar integration. You can use either the full or basic version of the calendar feed. The basic version gets you something like "Tue Nov 17, 2009 4pm - Kill Moles" for the Item and for the full version you would get "2009-11-17 4:00 PM - Kill Moles". I have also updated the fugly templates that show you the basics of how the driver works. The widgets in red should allow for direct input, but you may need to go in and fix them as several got wiped out and I haven't gotten around to fixing them yet.

Anogee - Get me the exact feeds you are seeing funky characters and I will look at them to see what is up.

Dean - Long term I plan to convert this over using the native xml support at which time it would be ready for prime time. As it is now it is just too ad-hoc in the way it parses the xml feed data. It works well as is, but native xml support would be better.

Sic - While that link you posted is XML, it isn't an RSS feed so the reader wouldn't be of any help. The great thing about XML though is that it does provide structured data so parsing that out would be pretty easy to do, especially with the new XML support. If that is a standard format used by the weather underground a dedicated driver could be written pretty easily.

[anogee]

Thanks so much for supporting this. We really do appreciate it.

First, my troublesome feed, if you will is this:
http://pheedo.msnbc.msn.com/id/3032091/device/rss

Probably just using a strange character set.

Second, and this is with the previous driver, starting yesterday I noticed that I started getting a "Failed to retrieve feed" for Item, and ItemDescription "StatusCode: 200 OK" from my Google Calendar feed. I wonder if they changed anything?

I entered just the base feed manually in a browser (before the ?) and it worked fine, but I don't think it likes some of the parameters. I think either the start-max or the start-min or both are making it fail. I'll have to look into this more. Probably this weekend. They must gave modified something.

[anogee]

Looks like the Google problem fixed itself. Its working again.

[jrlewis]

I haven't seen any issues on my end with the google feed, but I don't really have a full and functioning calendar going. I just have some test entries is all. If you run into that situation where you get a return status of 200 and a "Failed to retrieve feed" message put the driver into high verbose mode and then try and update the feed and check the log. Look for a message that is something like Exception_Moniker.ProcessFeed......
I've been tracking the msnbc feed and haven't seen any non-printable characters yet so will just have to wait a while on that until it rears its head.

[anogee]

With the Google problem, what I did was type the line in a browser, and see what it returned. What I got back was just a screen with my account info on the top, and no other text, so it was a Google problem, not the driver.

On the MSNBC I have to explain the problem a bit more. What happens (I think) is what should be a space instead becomes an unprintable character. If you just look at the text with the field browser you never see the problem because the CQC field browser treats this character as a space, so all looks well. But I send this feed to an LED sign which is not so forgiving. I'd normally just filter this out and convert these characters to spaces, but without using CML, this involves a whole lot of IF statements. I asked Dean for a command to do this, but its probably not on the top of his long list.

Update: I downloaded the Alpha sign protocol and reverse-engineered the character that is likely causing my problem. Its 08 Decimal which is the backspace character, so be on the lookout for the backspace character. That character on my sign puts it into extended character mode, and I think that is what I am seeing.

[jrlewis]

Looking at the FeedTicker in a marquee widget the feed displays fine. If those were ascii characters in that range of non-printable characters it would show up as boxes in the field browser as well as the marquee widget so not sure what you have going on there.

Reply to edit: An ascii character with an ordinal value of 0x08 would show up as a box in the field browser and in any widget displaying the text.

[anogee]

Hmm..The only other thing I can think of is noise between the computer and sign (it uses RS485 because of the 50' distance) but it only occurs with MSNBC and my processing code is the same for all feeds. Don't worry about it. I don't use that feed anymore and have never had the problem since.

[DaveB]

JR;
Did you take any other consideration in looking at the Yahoo RSS Feeds with embedded pictures?

[anogee]

Last things first. There is an updated version of the reader that provides better google calendar integration. You can use either the full or basic version of the calendar feed. The basic version gets you something like "Tue Nov 17, 2009 4pm - Kill Moles" for the Item and for the full version you would get "2009-11-17 4:00 PM - Kill Moles". I have also updated the fugly templates that show you the basics of how the driver works. The widgets in red should allow for direct input, but you may need to go in and fix them as several got wiped out and I haven't gotten around to fixing them yet.


O.K. I'm slow I admit it, so I finally got around to installing this driver update, mainly because ver 0.92 was working fine, and I knew I would have to rewrite my Google Calendar part with these changes. So, it appears the Google calendar feed stopped working for me a week or so, so I thought, hey let me upgrade now, so I did. Anyway, I'm going to have to make a bunch of updates now as expected.

Anyway, previously, to use the Google Calendar, you'd have to build this really long parameter string to tell Google Calendar you want the nearest events returned and in what format, then it took a whole lot of parsing to actually extract the events from what was returned.

So my question, what part was changed? The help file says "better support for Google Calendar" but I'm not really sure what that means since it looks like the driver fields are the same.

[jrlewis]

It has been a while, but I believe the change is in how it displays the entries from the calendar. See the post from 11/17/2009. The method in which you access the google calendar data with the reader won't change. The only way to make that less painful would be to write a driver for full on calendar support. That is on my list of things I would like to do which means I probably will get to it in something like 2020.

[anogee]

What I'm seeing is that "Items" are returned in the following format, sometimes,
Fri Jan 15, 2010 - Debby's Birthday
BUT the date being returned is the same for each event, although the events are correct. In other words EXCEPT for the very last item which seems to have the correct date, and the one before it which says
Sun Jul 4th, 2010<br /><br />Event Status Confirmed...(more Stuff)

So, the first 23 Items all have the same date but the events are correct, the 24th Item has text that says <br /><br />Event Status Confirmed... and the 25th, and last Item has the correct date and text.

If interested, I can shoot you my info so you can see it yourself.

[jrlewis]

Well I thought it worked...evidently not. I don't use googles calendar at all so I would really need to look at your rss feed. If you pm me a link to the feed in question I will take a look.

[froop]

Hey jrlewis,
Only just started playing around with this driver. Works a treat. I haven't got through all the pages of this thread, but has there been suggestion to include a 1-based index for the feed and item?

It'd make it that bit cleaner and easier to display "Feed 1 of 4, Item 1 of 3" instead of "Feed 0 of 4, Item 0 of 3".

[jrlewis]

Unless there is a reason to reserve a zero based index to be used for commands/queries that act on a whole collection I tend to like to stay with standard 0-based collections. I can look at if exposing it as a 1 based index would present any problems. I haven't looked at this in depth in a while so I just can't say off the top of my head. At some point I need to go in and update it with the native XML classes as it currently just uses some custom parsing routines.

[froop]

Unless there is a reason to reserve a zero based index to be used for commands/queries that act on a whole collection I tend to like to stay with standard 0-based collections. I can look at if exposing it as a 1 based index would present any problems.

I tend to agree with you, but from an end-user perspective, 1-based makes more sense. I'm sure that changing the existing fields from 0- to 1- may break a few people's templates, so I was thinking more about adding two new fields, (CurrentFeed, CurrentItem ?) which are basically just FeedIndex+1 and ItemIndex+1. At the moment I'm doing this with a trigger and the variable driver so I can map field text widgets directly.

[jrlewis]

For this particular driver being able to display a number associated with the feed or item doesn't really make much sense to me. It isn't a playlist or a number of movies or something. Displaying the name of a feed and item title makes a lot more sense to me. It never crossed my mind because I never really imagined anyone would really care to display that type of information.

[jrlewis]

Added CurItem and CurFeed for 1 based index of ItemIndex and FeedIndex.

[froop]

Added CurItem and CurFeed for 1 based index of ItemIndex and FeedIndex.

Thanks JR.

[Dean Roddey]

I updated the first post with a 1.0 version of the driver that is fixed for the recent CML compiler fix that now catches Out/InOut parameter issues. This one had a few small issues of that sort. There's nothing else changed, so it will work with older CQC versions still.