Field Generator - Beta version

[Fonceur]

The motivation of this CQC driver, is to provide a way to interact with applications developed outside the constraints of CML, through a TCP connection (Encoding: UTF-8). The flow goes something like this:


SomeApp.exe creates a TCP server on port 9600 and starts listening.
A FieldGenerator instance establishes a TCP connection on port 9600.
Following a successful connection, SomeApp.exe issues the command: "<STX>AppVersion|1.2<ETX>".
FieldGenerator is now "connected to the device" and ready for use.
SomeApp.exe creates various fields in FieldGenerator.
SomeApp.exe sets the field values in FieldGenerator.
A template is created in the interface editor to display the fields from the FieldGenerator instance.
Using the interface viewer, the user gets to see the states of SomeApp.exe.
Writing to the fields of FieldGenerator sends back a message (over TCP) to SomeApp.exe.


The instructions are provided on the Wiki (http://talluscorp.com/mediawiki/index.php?title=Field_Generator_driver).

[Dean Roddey]

Actually you can't use anything other than alphanum, underscore, and hyphen for field names anyway.

[gacevich]

fonceur, what application do you have in mind that caused u to develop this driver?

[Fonceur]

fonceur, what application do you have in mind that caused u to develop this driver?
1) I have developed some addons for BeyondTV, which rely on a .dll to access the BTV API, but recreating all those SOAP calls in CML would be way too painful... While I don't plan on writing that BeyondTV plugin just yet (or for a few months), I figured I might as well get that part done now...

2) For development, I need a driver that will act like a "CQC variable driver", but when a template is re-imported on a different machine won't jump to an existing and unrelated variable driver. This will be a second driver, but will use much of the same logic, just without the TCP part...

3) Tallus LLC (i.e. DavidL) is planning some "Home Automation neutral" applications, which can be used with CQC, MainLobby, HomeSeer, etc., so this driver would be the way in...

[gacevich]

thanks. as a btv user, i was hoping you would say that.

[brotsten]

All three of the BTV users can get excited now.

We are patiently waiting for the opportunity to sneer at all those yappy Sage types! :-D

Brian

[gacevich]

and when the hdpvr becomes stable with btv we will really have something to crow about.

[SomeWhatLost]

just come over to the dark side.... its much better over here... we have sage hd extenders....
they are very nice...

[brotsten]

just come over to the dark side.... its much better over here... we have sage hd extenders....
they are very nice...

No thanks, I tried it. I liked a lot of it's functions, but it's not nearly as intuitive as BTV.

Brian

[SomeWhatLost]

No thanks, I tried it. I liked a lot of it's functions, but it's not nearly as intuitive as BTV.

Brian
true...

but one of these days, you will succumb to the dark side:tounge

[IVB]

From the Sage forums today, a new Sage Server Beta...
We've also got one new piece of fun for those of you with HD extenders (currently only HD200). We've got support now for playing back BluRays that are in a BDMV folder structure. There's no support for menus or interactive features. But it does support multiple audio tracks, chapter selection and later will also support subtitles.

My girl is red hot, your girl ain't doodly squat :-)

[brotsten]

All you sAGE weenies get back over to your own threads. :-)

Brian

[Helheim]

I used to use BTV but sage is so much better... :)

[SomeWhatLost]

I disagree...


but Sage does have the HD extenders... I really like the extenders...

but if the extenders ever go away, I will go back to BTV...


but more important, and on more on topic, hows this driver coming along? and could it be the first step in getting Sonos & CQC to play together?

[Fonceur]

but more important, and on more on topic, hows this driver coming along?
Well, I did update it to support the "SetField" command for StringList, while only the "CreateField" was working for StringList... Not seeing any downloads yet, so there can't be that much interest... ;)

As for applications supporting it, you can expect TZWave.exe and TallusRadio.exe in the short term. They both will be commercial (i.e. $) "home automation neutral" applications, which can be used with either CQC or MainLobby.

- The first one deals with ZWave, DavidL could give more details...

- The second one deals with satellite radio (Sirius/XM/HD) connected through a specific interface/cable... I started TallusRadio.exe yesterday, using a Sirius Demo as foundation, but it's a bit tricky without access to the hardware... ;) I'll be adding the HD and XM support as I go.

and could it be the first step in getting Sonos & CQC to play together?
Well, you still need someone to develop some kind of Sonos.exe application and add some TCP server to communicate with my FieldGenerator, they simply no longer need to know CML themselves and are not restricted by the CML language either...

[SomeWhatLost]

well, I would guess there have been no downloads yet because there is nothing to use it with yet? perhaps similar to the precedent set forth in the case of Chicken Vs. Egg?

I don't have Zwave, so the Zwave thingy doesn't really interest me, but I am curious, what can it do the the current Zwave driver can't?

as for satellite radio, didn't they merge and then go bankrupt?

how bout something useful? like Sonos?? (shameless self serving plug...:-) )

[Fonceur]

well, I would guess there have been no downloads yet because there is nothing to use it with yet?
There's that, but it could have caught some developer's interest... ;)

I don't have Zwave, so the Zwave thingy doesn't really interest me, but I am curious, what can it do the the current Zwave driver can't?
You have to keep in mind that the primary market is MainLobby, where the support is lacking, so adding the CQC support is easy enough to be worth it. Will there be enough differences to be worth it for the end user, I have no clue... ;)

as for satellite radio, didn't they merge and then go bankrupt?
Yes something like that, but as far as I know the hardware can still be used on the other network or something, even though it doesn't quite communicate in the same way with the outside world...

how bout something useful? like Sonos?? (shameless self serving plug...:-) )
Well, I seem to remember seeing some need for (better?) Sonos support on the ML side too, so it might end up somewhere on my "to do" list... ;)

[Brightan]

I think I'm close...
I have my server program running, it is making a connection and sending the Appversion command and CQC shows the Driver State as Connected to Device
Question: Should the driver say something at this point back to my program?

I can create an incoming field (as far as CQC is concerned) and can send it data from my app and it displays on the CQC screen, ya!

But, when I create an outgoing field using
"CreateField|GetDataFromCQC~Int;Write;None;AlwaysWr ite;Normal"
I don't ever receive any data. I can see the field in the CQC editor and I am using a command button to write to the field and it works in that it doesn't give an error, but my program never receives any data
In my command button I'm using Devices::FieldWrite(XBee.GetDataFromCQC, 99)


Also, when you have a minute, I wonder if you could explain the extra parameters in the CreateField command?

thanks.

[Fonceur]

I have my server program running, it is making a connection and sending the Appversion command and CQC shows the Driver State as Connected to Device
Good!

Question: Should the driver say something at this point back to my program?
It doesn't, but it could... ;)

But, when I create an outgoing field using
"CreateField|GetDataFromCQC~Int;Write;None;AlwaysWr ite;Normal"
I don't ever receive any data.
Sorry, I only implemented the Card/Bool/String field change, so Int indeed won't send a reply, I'll fix that in .5... ;)

Also, when you have a minute, I wonder if you could explain the extra parameters in the CreateField command?
Basically, I started from jrlewis' RegisterFields method in Pandora (and his other work), then ran with it... ;) The last parameter is for Field.SetQueuedWrite, which I guess in this context doesn't make all that much sense as you're not going to provide your own method to update that field...

EDIT: Version .5 is up, I also wasn't sending the advertised command (SetField instead of FieldChanged).

[jrlewis]

The details on the format for defining a field are given below. Founc's format is only slightly different than what I use in my drivers since the FldName is included in the specification. Those 6 parameters completely define a field in CQC. Cases where you would use a queued write is where the "device" you are controlling has a very long response time where it isn't practical for the driver to wait for a response. In cases where you set a queued write for a field you are telling cqc you will handle updating the value for that field. Normally with a field write if the FldChanged method returns successfully the fld value will automatically be updated. With a queued write you have to take care of it. Since this driver takes incoming field updates you could use the queued write if the device warrants it.



// FldDef ------------------------------------------------------------------|
// FldName~CQCFldType;CQCFldAccess;FldLimits;WriteSta tus;QueStatus |
// |
// |
// CQCFldType - Fld type as defined by CQCFldTypes |
// |
// CQCFldAccess - Fld access as defined by CQCFldAccess |
// |
// FldLimits - The FLdLimits can have the follwing values: |
// |
// None - no limits |
// Range:a,b - range between a and b |
// LsThan:a - less than a |
// GtThan:a - greater than a |
// Enum: a,b,... - enumerated list |
// RegEx:"RegEx" - any regular expression |
// |
// WriteStatus - The write status can take on the following values: |
// |
// AlwaysWrite - Uses CQCFldDefs SetAlwaysWrite() |
// OnChange - Default |
// ReadOnly - Use for fields that are read only |
// |
// QueStatus - The que status can take on the following values: |
// |
// Normal - Default |
// Queued - Uses CQCFldDefs SetQueuedWrite() |
// ReadOnly - Use for fields that are read only |
// |
// |
//--------------------------------------------------------------------------|

[Brightan]

I'm getting data back now, but it is a little weird
I am getting a constant stream, about 2 per second, of an error message, but the 'E' is missing so I get <STX>rror:None|<0><ETX>
(actually, the first character of all the messages are missing)

I can send data into CQC and now when CQC sends data, I get it
<STX>ieldChanged|GetDataFromCQC~62n<ETX>
(there is an extra 'n' after the value)
But, CQC gives an error that the driver is not currently connected to it's device even though the value was transmitted through

[Fonceur]

(actually, the first character of all the messages are missing)
OK, I thought I was inserting the <STX> before the first character, but apparently I am overwriting it...

But, CQC gives an error that the driver is not currently connected to it's device even though the value was transmitted through
Yes, I need to clean up that part. I haven't started parsing the response yet on the server side, so it's most likely still messy, with things double firing and all. Hold on testing for now, as I get that sorted out. ;)

EDIT: Just reread the definition of "PutAt", and indeed that's not what I needed... ;)

[Brightan]

no problems, I am not trying to pester you, I can provide troubleshooting when you need it as a way of helping (if you can call it that!) When I get a chance I am going to write the interface to the XBee modules and will prove to the nay-sayers that this is a good idea! ;-)

[Fonceur]

Actually, I think that memBuffer.PutCard1At isn't working as the documentation says (supposed to insert/append, which it obviously isn't...), but instead it acts just like the string.PutAt...

[Dean Roddey]

It just writes a value to a particular index. It doesn't move anthing up to make a hole or anything. So, if the docs imply that, they are wrong.

[Fonceur]

It just writes a value to a particular index. It doesn't move anthing up to make a hole or anything. So, if the docs imply that, they are wrong.
Well the documentation is ambiguous. If you compare the wording of string.Insert and string.PutAt, then look at the membuf.PutCard1At, it sounds like it should be doing an insert, when it is indeed doing a put...

[Fonceur]

I can provide troubleshooting when you need it as a way of helping
OK, go try version 0.6.2, this is a much cleaner implementation...

[Dean Roddey]

There is an AppendCard8, at least in the lastest version. I know it was missing and was added at some point. But it probably was added after the last formal release, so no one could use the driver until they upgraded, so probably best not to use it yet.

[Fonceur]

But it probably was added after the last formal release, so no one could use the driver until they upgraded, so probably best not to use it yet.
Yes, for now I am going to use AppendFmt instead.

[bjkiller]

The motivation of this CQC driver, is to provide a way to interact with applications developed outside the constraints of CML, through a TCP connection (Encoding: UTF-8). The flow goes something like this:


SomeApp.exe creates a TCP server on port 9600 and starts listening.
A FieldGenerator instance establishes a TCP connection on port 9600.
Following a successful connection, SomeApp.exe issues the command: "<STX>AppVersion|1.2<ETX>".
FieldGenerator is now "connected to the device" and ready for use.
SomeApp.exe creates various fields in FieldGenerator.
SomeApp.exe sets the field values in FieldGenerator.
A template is created in the interface editor to display the fields from the FieldGenerator instance.
Using the interface viewer, the user gets to see the states of SomeApp.exe.
Writing to the fields of FieldGenerator sends back a message (over TCP) to SomeApp.exe.


The instructions are provided on the Wiki (http://talluscorp.com/mediawiki/index.php?title=Field_Generator_driver).

Fonceur,
May be you have also C++ skeleton for the driver, socket things, communication thread, all parsing things, may be some simple encapsulation basic classes to start from?


Thank you in advance.

[Fonceur]

May be you have also C++ skeleton for the driver, socket things, communication thread, all parsing things, may be some simple encapsulation basic classes to start from?
Not sure how much help that will be as it's in C#, but I'm attaching what I use for TRadio...

The basic idea that I used was to have a blocking thread waiting for a new client, then the client itself has a blocking thread waiting for incoming data. If the length of data is 0, that means the client disconnected, so we wait for a new connection...

[hucker]

A few questions about the Field Generator driver:

1) Is it production ready?
2) Is it likely to be around for future releases of CQC?

I also have a question about the the code that parses the incoming data stream. Do you see occasional errors? It appears that the C# code assumes that the incoming data stream is 'packet' based and not 'stream' based. What I mean by that is that each packet that is received is the same as the packet that was written. As I understand it streams are not required to do that. I *think* it is possible for this code to miss commands occasionally if the stack somehow splits packets. Probably not likely on localhost unless you happen to do some big writes to a stream.

The reason I ask is that I'm used to seeing TCP streams decoded with state machines that operate on a stream of characters rather than a stream of packets IYKWIM.

I'm not a C# expert and I'm just coming up to speed on .NET so forgive me if I'm out to lunch.

Thanks

[Fonceur]

1) Is it production ready?
The testing has been rather limited... I had some initial feedback from Jrlewis and some other developers, but besides my TRadio, I don't think there is any other 3rd party application that made it to testing yet...

2) Is it likely to be around for future releases of CQC?
Sure, so far I've been good at keeping my various projects up to date... ;)

What I mean by that is that each packet that is received is the same as the packet that was written. As I understand it streams are not required to do that.
While the CML part will read between the <STX> and <ETX>, indeed C# does not have such a method. As for my implementation, I do add up partial packets until they are complete (i.e. ends by <ETX>), or split at the <ETX> if multiple messages are found. That's basically the code that I use to parse incoming XML string, representing say 16k songs in SageTV, split over many 8kb buffers...

[Zanthic]

I'm just getting back into this after a number of months away so this might be a dumb question. How can my server app query the Field Generator to find out the previously defined fields? You might think that my app should already know this because it is the one that created the fields, but I am thinking of a situation where my app has shut down and restarted but the Field Generator driver is still running. I could store the fields on my side but then there is the case where the field Generator might have reset. I thought maybe the QueryAllFields command might return the information I want but I guess not.

[Zanthic]

Also, would it be possible to have the FieldCreate command overwrite an existing field with new settings?

[Fonceur]

Sorry for the late response. things are busy and I haven't touched it in a while, so it's getting fuzzy what it actually does anymore and how.. ;)

The QueryAllFields will return all the values, I guess that you would also like to know the parameters used to set up the fields, right?

To modify a field, you could delete it (DeleteField|<FieldName>) and recreate it... The process will lose all the currently stored values though.

[Zanthic]

I'll put my requests on hold for now, I have my initial program working and I'll wait and see whether it is really necessary to retrieve the field names or not. And, I'll see about using the deletefield as you mentioned.