Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Early MQTT support
#11
I decided to go ahead and tackle field writes. I've got the infrastructure basically worked out and have tested it with one type of field (boolean.) So I just need to add a bit for the other types and test that out. I'll hopefully get a drop up tomorrow.

So one thing to be aware of, if you want the driver to be able to get values when it first comes up, then any published values has to be published in 'retain' mode so that the sever retains the values. Otherwise, the driver won't see anything until the next time the source publishes that value.

I've added a flag to the config (for writable fields) that lets you say whether you want them to be written in retain mode or not. It defaults to true if not set explicitly since you mostly would.

If it's a public server, it may not exactly blast them out all at once when the driver connects and subscribes, so as to keep down overhead. Not sure really, I'm just going by the mosquitto test server, which is very slow to send them out. But it's a freebie and gets beaten on hard with fairly light resources (relative to the load.) Presumably a local server would immediately send them.

I was worried that I'd have to do the Z-Wave style thing and just queue up field writes and return without knowing if they worked or not. But I worked it out so that the driver can wait for the I/O thread to send him back a success or failure indicator, while he continues to process other incoming events.
Dean Roddey
Explorans limites defectum
Reply
#12
OK, 5.3.939 is posted. This one has a much improved MQTT driver. The basics are:

1. Supports field writes now
2. Supports client side certificates if the server needs it (AWS apparently does)
3. Supports ALPN (application level protocol negotiation) if the MQTT server needs it (AWS apaprently does.)
4. The CheckMQTTCfg program now has a /TestConn option. If you pass that it will also do a simple connection to the MQTT server to see if the connection is good. This could save you a lot of messing about.
5. Supports a Generic field type for stuff that doesn't fall cleanly into one of the semantic field types.
6. Supports more useful field <-> MQTT value mapping options
7. More configuration options to support the above stuff

Configuration

Here is a new example config file:

Code:
<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE MQTTCfg PUBLIC "urn:charmedquark.com:CQC-MQTTCfg.DTD" "CQCMQTTCfg.DTD">
<MQTTCfg Version="1" MQTTPort="1883" MQTTAddr="test.mosquitto.org"
        MsgTrace="Yes" VerboseMode="Yes">

   <Fields>
       <LightSwitch Topic="CQCDev/Light1" BaseName="Light1" Access="RW">
           <PLFmt Type="Card" Bytes="1"/>
           <BoolNumMap FalseOut="0" TrueOut="0xFF">
               <BoolNumVal TarVal="False" MinSrc="0" MaxSrc="64"/>
               <BoolNumVal TarVal="True" MinSrc="0xF0" MaxSrc="0xFF"/>
           </BoolNumMap>
       </LightSwitch>

       <Dimmer Topic="CQCDev/Dimmer1" BaseName="Dimmer1" Access="RW" MinVal="0" MaxVal="100">
           <PLFmt Type="Card" Bytes="1"/>
       </Dimmer>

       <Generic Topic="CQCDev/Status" BaseName="SystemStatus" Access="R" FldType="String"
                Limits="Enum: Startup, Ready, Shutdown">
           <PLFmt Type="Text"/>
           <TextMap>
               <TextVal TarVal="Startup" Dir="In">"Loading", "Initializing"</TextVal>
               <TextVal TarVal="Ready" Dir="In">"Ready1", "Idle"</TextVal>
               <TextVal TarVal="Shutdown" Dir="In">"Cleanup", "Stopping", "Stopped"</TextVal>
           </TextMap>
       </Generic>

       <TempSensor Topic="CQCDev/TempSensor1" BaseName="Temp1" FldType="Int"
                   MinVal="0" MaxVal="100">
           <PLFmt Type="Text"/>
           <ScaleRangeMap MQMin="0" MQMax="2000" FldMin="0" FldMax="100"/>
       </TempSensor>

       <Generic Topic="CQCDev/HouseMode" BaseName="HouseMode" FldType="String"
                   Limits="Enum: Night, Day, Party, Quiet" Access="RW">
           <PLFmt Type="Text"/>
           <EnumMap>
               <EnumVal FldVal="Night" MQTTVal="off"/>
               <EnumVal FldVal="Day" MQTTVal="normal"/>
               <EnumVal FldVal="Party" MQTTVal="open"/>
               <EnumVal FldVal="Quiet" MQTTVal="low"/>
           </EnumMap>
       </Generic>

   </Fields>

</MQTTCfg>

Value Maps

There's a very different scheme for mapping values between MQTT and the field values from the previous preview. If you don't declare a map, then a 'passthrough' map will be installed by default. It just passes the values directly as is. So, if the MQTT values and the CQC values are the same, you don't need a map.

Otherwise, you need a map to map between the two sides, well possibly one way only if the field is read only or write only. The map types currently are:
  • BoolNumMap - To map a boolean field to numbers. Often it may just be False to 0 and true to 1, but it might be more complex than that. Each entry indicates whether its for True or False and provides MinSrc/MaxSrc values, to define a range of values for that map entry. If it's just a single value set both to the same thing. For field writes, it's going to take the first value of the first True/False entry that matches the written field value. If you want a specific value to be written for true or false, add entries for those values first.
  • EnumValMap - For enumerated string fields where there is a one to one mapping between MQTT values and field values.
  • MRangeMap - A multi-range map where multiple MQTT values map to one field value, and vice versa. So you indicate a direction (In, Out, InOut), a target value and the min/max range of values that map to that.  See an example below.
  • NumMap - For a small set of one to one numeric value mappings. See an example below.
  • ScaleRangeMap - When the MQTT and field have a contiguous range of values but they are different. This will map between the two scales. So if the field is 0 to 100 and the MQTT value is 0 to 500, you provide the two ranges and the values are just mapped between them
  • TextMap - When one MQTT or field value maps to more potentially more than one value on the other side. In this case, the source values are in the form of a quoted comma list in the body of the map element.
Others can and I'm sure will be added, but this is a good starting point. For the ones that aren't represented in the example. Here is an MRangeMap:

Code:
<MRangeMap>
   <MRangeVal Dir="In" TarVal="1" MinVal="0" MaxVal="10"/>
   <MRangeVal Dir="In" TarVal="2" MinVal="11" MaxVal="25"/>
   <MRangeVal Dir="In" TarVal="3" MinVal="26" MaxVal="50"/>
</MRangeMap>

So it maps three MQTT value ranges to three CQC field values. When a value comes in, the list is searched till it hits the first In or InOut one where the MQTT value falls into the range indicated, and it takes the target value as the field value. And the opposite when the field is written to, it finds the first Out or InOut slot that the field value falls into and sends that target value to MQTT.

Here is a NumMap sample:

Code:
<NumMap>
   <NumMapVal Dir="InOut" TarVal="0" SrcVal="0"/>
   <NumMapVal Dir="InOut" TarVal="1" SrcVal="0xFF"/>
   <NumMapVal Dir="InOut" TarVal="5" SrcVal="0xFF"/>
</NumMap>

So basically the same as above but there's just a single source value instead of a range.

Secure Connections

If you need to do secure connections set Secure="Yes" in the main MQTTCfg element, and set the appropriate secure port for the target server. If it requires ALPN you can set the PerfProtocol="" value. Here is an example of just the main element:

Code:
<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE MQTTCfg PUBLIC "urn:charmedquark.com:CQC-MQTTCfg.DTD" "CQCMQTTCfg.DTD">
<MQTTCfg Version="1" MQTTPort="8883" MQTTAddr="8fkfjk88af-ats.iot.us-east-1.amazonaws.com"
        MsgTrace="Yes" VerboseMode="Yes" PrefProtocol="x-amzn-mqtt-ca" Secure="Yes">

If you need to support client side certificates, most likely all you need to do is add the client side cert into the Windows certificate system (in the personal certs area for the local machine I would think). If you need to you can force it using the same syntax as the CQC web server uses, such as:

Code:
<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE MQTTCfg PUBLIC "urn:charmedquark.com:CQC-MQTTCfg.DTD" "CQCMQTTCfg.DTD">
<MQTTCfg Version="1" MQTTPort="8883" MQTTAddr="8fkfjk88af-ats.iot.us-east-1.amazonaws.com"
        MsgTrace="Yes" VerboseMode="Yes" PrefProtocol="x-amzn-mqtt-ca" Secure="Yes"
        CertInfo="MStore:My,MyClientCert">

So CertInfo="" indicates the certifivate. MStore means the local machine store, and My means the personal certs area. MyClientCert would be the name that the certificate is for.


Generic Fields

There is now a Generic field that lets you set up other types of fields beyond the semantic types directly supported. You have to provide the access, field type, and any field limits that should be applied to the field. In this case the BaseName is actually the full field name, since there's no semantic bits or V2 bits to add to it.

Retain Mode

The default for all writable fields is Retain="Yes", so you don't see it any of the example fields. This tells the target MQTT server to retain the value. Without this, any values written are dropped once they are delivered and if the driver starts back up, it won't get a value for that field until the thing publishing the values publishes a new one.

If you do need to prevent retention, add Retain="No" to the field.

PayLoad Format

The payload format stuff is basically the same as before. For each field you can set a payload format that tells the driver how to get the value out of the incoming msg and how to build the payload value for an outgoing msg in the case of a field write. The values are:
  • Type - Card, Int or Text. Floating point values are assumed to always be formatted as text
  • Bytes - If Card/Int (binary) then the number of bytes that the value is encoded in
  • Offset - The offset within the payload to look for the value, defaults to zero if not set
  • Endian - Either Little or Big, and defaults to Big since MQTT is by default big endian. Only valid for Card/Int
  • DecDigits - If a float field, which is assumed to be encoded in text, this is the number of decimal digits. This is primarily used for outgoing field writes, to set the right number of digits when the value is formatted back to text. Defaults to zero.
  • Precis - If the field is float, the payload may still be Card/Int but just multiplied up to get rid of the trailing decimal digits. This tells the driver how many powers of ten to divide the value by after it converts back from text to float. Defaults to zero.

Note that the Offset only works for incoming msgs (MQTT -> Field.) If the offset isn't zero that means there's other info in the payload. The driver wouldn't have any way to know what that info should be.
Dean Roddey
Explorans limites defectum
Reply
#13
Finally got around to getting this connected.

I use MQQT Fx as a test client where I can subscribe and publish data successfully. Confirmed with MQTTBox as well.

I am using the default config file just for testing at the moment.

Using Light1 I am able to turn it off from the test client with a 0 but cant turn it on with 256, 0xff or FF.

FF is close but it changes the field to ???? instead of True.

While the field is in this state you cant write to it - so the read/write only fields are stuck until the device updates them - too bad if say you reload the driver or CQC and the fields are ???? - you cant use the field until your device phones home to get it operational (which hopefully it is doing)

If I try and write true to the field from the CQC admin interface it does change to true (in CQC) but the client get the following message

*** PAYLOAD IS NOT VALID JSON DATA ***

Illegal character ((CTRL-CHAR, code 1)): only regular white space (\r, \n, \t) is allowed between tokens
at [Source: java.io.StringReader@5b009fd4; line: 1, column: 2]

THis is confirmed in MQTTBox as there is only a square to represent a non printable character.

I also don't understand the dimmer1 - sending a 0 gives a 48, sending 100 results in 49 and 50 results in 53?

From what I can tell it should be just 0 to 100 (FF gives 70).

Again, writing a value from CQC yields the same error as above
Mykel Koblenz
Illawarra Smart Home
Reply
#14
That error would have to come from presumably the device or the MQTT server, since it's a Java thing. We have no way currently to support a JSON type msg, so you wouldn't be able to do any field writes to such a field if they require something like that.

Which default config file are we talking about? The one I posted above? If so, make sure that the thing you are using understands it's a hex value you are sending. So most likely it needs 0xFF, not just FF. And make sure it understands that the value is a single byte.

On the dimmer, I'm not sure what that is. Can you read back the value using that test client? I.e. just a sanity check to see what is getting set?
Dean Roddey
Explorans limites defectum
Reply
#15
Its not the server generating the error. I asked the client to show the json parsed data and none existed.

I had the client displaying plain text and it was not displaying anything because you are sending hex 01 and 00 for the Light1 field.

Not sure how many devices can send hex like that - Blue Iris sends text and so does the client (have not worked out how to send hex if it can do it).

That would mean then that if I want to send text 1 and 0 for true false the mapping needs to change - is that correct?
If so how do I do that - I had a go and cant figure out the requirements.

Your config file is kind of misleading the way I see it.

<BoolNumMap FalseOut="0" TrueOut="0xFF">
<BoolNumVal TarVal="False" MinSrc="0" MaxSrc="64"/>
<BoolNumVal TarVal="True" MinSrc="0xF0" MaxSrc="0xFF"/>
</BoolNumMap>

I read this as "0" (ie a text zero of 0x30 is false and not 0x00 - similarly True is "0xF0" to "0xFF" in text - the quote make it misleading.

But what is recorded by the client does not match at all - True is 0x01 and not 0xFF.

I'm confused so far - not much is working for me yet - but still learning.
Mykel Koblenz
Illawarra Smart Home
Reply
#16
Further testing has raised another question

Working on the Light1 field only at the moment. Configuration is as follows now

<LightSwitch Topic="CQCDev/Light1" BaseName="Light1" Access="RW">
<PLFmt Type="Card" Bytes="1"/>
<BoolNumMap FalseOut="0x30" TrueOut="0x31">
<BoolNumVal TarVal="False" MinSrc="0x30" MaxSrc="0x30"/>
<BoolNumVal TarVal="True" MinSrc="0x31" MaxSrc="0x31"/>
</BoolNumMap>
</LightSwitch>

The way I read it - a MQTT value of 0x30 will set the field to false and 0x31 to true - this will allow a text 0 and 1 to drive the field value.

I also read it that when I write a value to the fiel it will output a 0x30 or 0x31 (false or true) but the test client is still getting a 0x00 and 0x01 - why?
Mykel Koblenz
Illawarra Smart Home
Reply
#17
The field is actually text. 0x30 and 0x31 are just ASCII '0' and '1' characters. The test client is probably showing you the ASCII formatted values since the payload is text format. Also, the above will make it send binary content, when the value really maybe is defined to be text.

I guess I need to do a BoolTextMap that is specialized for this very common scenario, so False<->'0' and True<->'1' . That'll save work as well since it's probably a very common scenario; and, since it's a pre-fab map all you'll have to do is <BooTextMap/>.

I'll do that for the next drop.
Dean Roddey
Explorans limites defectum
Reply
#18
(4 hours ago)Dean Roddey Wrote: The field is actually text. 0x30 and 0x31 are just ASCII '0' and '1' characters. The test client is probably showing you the ASCII formatted values since the payload is text format. Also, the above will make it send binary content, when the value really maybe is defined to be text.

Yes, the fields are text - thats why I set them to 0x30 and 0x31, but the test client is set to display hex format and is reporting it is receiving 00 and 01 and not 30 and 31 - so I think the driver is not converting the output properly

There is probably going to be a lot of text mapping needed. Even a CardTextMap or IntTextMap for scenarios where a string may be wanted as a number (say four states identified as strings, but used in CQC as int's 1 - "Off", 2 - "Standby", 3 - "On", 4 - "Unknown")

There may be times where you don't want to enumerate them as strings (your HouseMode field is a string to string enumeration) but as integers (we do it now for our SCADA systems) and vice-versa and int/card enumerates to a string.

I'll keep bashing at it to learn more about the driver - so far its going well. With BlueIris introducing MQTT as a way to control and send notifications I think a MQQTBI configuration is going to be very easy to do once the string mapping is taken care of (no documentation on sending hex so we need the strings to work)
Mykel Koblenz
Illawarra Smart Home
Reply
#19
Text and binary aren't really interchangeable in MQTT. Text is sent as UTF-8 encoded text with a leading two digit length. So if something is expecting text, and you send binary data, it won't work right except just by luck in some cases perhaps.

The driver can be told the format of the payload and adapt but the MQTT server will store binary as binary and text as text and will deliver it as received which will confuse any subscribers if they think it's one and get the other (and there's no way for them to know it's wrong, a serious weakness in MQTT) they can only try to interpret as they believe it should be.
Dean Roddey
Explorans limites defectum
Reply


Possibly Related Threads...
Thread Author Replies Views Last Post
  Does the auto-gen template system support Roku? ghurty 3 269 12-31-2018, 06:14 PM
Last Post: Dean Roddey
  What CQC compatible sound system can support 14 rooms? ghurty 14 2,262 08-05-2018, 07:18 AM
Last Post: zra
  I found NVRs that support WebRTC ghurty 3 796 05-25-2018, 01:07 PM
Last Post: Dean Roddey
  New Amazon Echo Support Dean Roddey 591 149,696 02-21-2018, 12:56 PM
Last Post: Dean Roddey
  Does GC-100 Driver Support Tri-Port Cable? Jnetto 18 4,822 01-29-2018, 08:16 AM
Last Post: RichardU
  Support for Bryant Evolution Connex agarden 21 7,703 10-24-2017, 05:17 PM
Last Post: jkmonroe
  Support for Honeywell Ademco Vista agarden 14 5,351 07-06-2017, 07:13 PM
Last Post: agarden
  V2 Somfy API Support? agarden 19 7,623 06-29-2017, 09:18 AM
Last Post: Dean Roddey
  Spotify Support? agarden 1 1,466 04-29-2017, 09:46 AM
Last Post: Dean Roddey
  Group Support for Philips HUE agarden 11 4,066 04-28-2017, 10:23 PM
Last Post: agarden

Forum Jump:


Users browsing this thread: 2 Guest(s)