Charmed Quark Systems, Ltd. - Support Forums and Community
Early MQTT support - Printable Version

+- Charmed Quark Systems, Ltd. - Support Forums and Community (
+-- Forum: General Discussion (
+--- Forum: CQC Support (
+--- Thread: Early MQTT support (/showthread.php?tid=10784)

Pages: 1 2

Early MQTT support - Dean Roddey - 02-25-2019

As of 5.3.937, some early support for MQTT has been added, if that's something you are interested in. There is a new driver which acts as an MQTT client (which means it can talk to some MQTT server you want it to talk to and access data that has been published to that server.) That can be local or remote, as you wish.

If you aren't familiar with MQTT, it's a simple 'messaging middle-ware' scheme, which means it can take messages from sources, and distribute those to consumers who ask to see them. The published messages are 'topics', and consumers subscribe to these topics. When the publisher of a topic updates the message value, the server will try to send that new value to all of the subscribers.

In the context of CQC, this means that any device or system that publishes data to an MQTT server can be indirectly gotten to from within CQC. So it makes it possible to get various more obscure bits and bobs into CQC without having to have specific drivers for them. Various types of gadgets out there can publish to an MQTT server.

For now, the support is pretty simple. The big problem, a with Z-Wave only worse, is that there's just this random information available, and how does the driver know what info is there, what type of field to store that info in, and how to pull the data out of the MQTT server and convert it to the field data type (there are no standards for data format in MQTT.)

Here is a sample configuration file, which you would put into [cqc]\CQCData\Server\Data\MQTT\Config.xml. Eventually there will be a client side driver config tab, but we need to get much further along in solidifying the configuration format. As mentioned that's far and away the hard part.

<?xml version="1.0" encoding="US-ASCII"?>
<MQTTCfg Version="1" MQTTPort="1883" MQTTAddr=""
         MsgTrace="Yes" VerboseMode="Yes">

        <LightSwitch Topic="CQCDev/Power1" BaseName="KitchenFan" Access="RW">
            <PLFmt Type="Text">
                    <TextVal MQVal="off"   FldVal="False" Out="Yes"/>
                    <TextVal MQVal="on"    FldVal="True"  Out="Yes"/>
                    <TextVal MQVal="allon" FldVal="True"/>

        <Dimmer Topic="CQCDev/Light2" BaseName="KitchenTable" Access="RW"
                MinVal="0" MaxVal="100">
            <PLFmt Type="Card" Bytes="2" Endian="Little"/>

        <MotionSensor Topic="CQCDev/Motion1" BaseName="HallMot">
            <PLFmt Type="Card" Bytes="1"/>

        <MotionSensor Topic="CQCDev/Motion2" BaseName="LRMot">
            <PLFmt Type="Card" Bytes="1"/>

        <TempSensor Topic="CQCDev/Temp1" BaseName="LivingRoom" Type="Int"
                    MinVal="-10" MaxVal="48">
            <PLFmt Type="Text"/>

        <TempSensor Topic="CQCDev/Temp2" BaseName="Kitchen" Type="Float"
                    MinVal="30" MaxVal="115">
            <PLFmt Type="Text" Precis="1"/>



The header allows you indicate the address and port to use when contacting the MQTT server, and some other options.

  • MsgTrace - The driver supports a trace file, which will be important for debugging issues in the field. You can enable it via the file. Currently that's the only way to do so.
  • VerboseMode - You can force the driver into verbose mode via the file if you want to make sure it stays in verbose mode even across reloads for a while. Obviously remove this eventually.
  • MQTTAddr - The host name/address of the MQTT server to connect to.
  • MQTTPort - The port to connect to.
  • UserName/Password - MQTT server can support login. If so, set these two to appropriate values. If either is set, it is passed to the server in the MQTT connection process.
  • Secure - Set to Yes if you want to make a secure connection to the MQTT server. Obviously it has to support that, and the appropriate security certificates must be in place.  The port to use will differ based on secure or non-secure connection type. The defaults are 1883 for non-secure and 8883 for secure.
  • SecPrincipal - When doing secure connections you have to provide a 'security principal', which is basically the DNS name for which the security certificate is issued. Normally it is the same as MQTTAddr so you don't have to provide this. But, if it's different, indicate that here. 


In this XML file there is a Fields element. Under there you will define one or more fields. They are indicated by way of their semantic field types (LightSwitch, Dimmer, MotionSensor, etc....) The supported types are all demonstrated in a sample config file below. For each one you provide some basic information:

  • Topic - This is the MQTT topic path for the value that you want to store in this field. It obviously has to be something that makes sense for the type of field you choose.
  • BaseName - The is similar to other drivers where you provide the basic part of the name, which may be 'decorated' in various ways, such as for V2 compatibility. The only has to be unique enough to prevent duplicate field names (i.e. you can have a motion sensor and light named Kitchen, since their ultimate field names will be MOT#Kitchen and LGHT#Sw_Kitchen, assuming they are V2 compatible.) The V2 compatible fields will be used if the field configuration allows for such.
  • Access - The field access, which is R, W, or RW. Obviously needs to make sense for the topic .
  • MinVal/MaxVal - If the field is numeric, you can assign min/max values which will be used to set a range limit.
  • Type - For a temp sensor (and possibly for others later) where the type is not defined strictly by the semantic type, you can indicate a type (currently Int or Float) that the field should have, usually driven by the values the topic provides.

Currently we don't have any sort of 'enumerated value' field support. That's obviously something we'll need to support moving forward. And of course more semantic field types. The ones in the example are all that are supported so far.

Payload Format

Within each field is a 'payload format' element,  which you use to tell the driver how to interpret the MQTT payload. A payload can be either a text string, or some binary value. A lot of them are text strings apparently, but they could be binary.

The simplest ones are the two temp sensors. In this case, it just says the format is text, which implies that the values in the text are literally just the sensor value. So 50 or -2 or 45.20 or whatever as a text value. The driver just converts to binary. 

Note the second one has a 'Precis' value, which means precision. In that case it's assumed the formatted value is 10x times the actual value. So 45.2 shows up at 452. The driver will divide by 10 (one digit of precision) to get the final value.

For a boolean field that comes in as text, you have to provide a map. The first one is a light switch (obviously boolean) and it provides a TextMap element. Each value in the text map maps between an MQTT value and a field value (which in this case must be True or False.) You can have multiple values that map to each. In this example both on and allon map to true.

Though it's not used yet (no field writes are supported yet) you have to mark the ones that should be used when the field is written to, and CQC has to convert that true/false value to a topic value to send out. In this case the off and on values are marked as Out="Yes" to indicate these are the values to use.

For binary payloads, you can indicate the type (Card or Int, so unsigned or signed), the number of bytes in the value, the offset into the payload where the value is, and the 'endianness' of it. If you don't say explicitly it defaults to big endian since MQTT itself is big endian and probably most programs will follow that lead when they publish binary values. But, if they don't, you can indicate little endian, as the KitchenTable dimmer one does.

In most cases it will probably just be the value and nothing else. But, if that's not true, you can indicate an Offset value, e.g. Offset=10, which indicates that the value to extract is ten bytes into the payload data.

There is no means currently to map a range of MQTT values to a different range of field values. That's something that will obviously be provided moving forward. Currently, the value that comes in from MQTT must be the actual value for numeric fields.

Config Validation

Because the config file is kind of complicated, and to avoid a slog of edit, reload, check the logs cycles, if you open a CQC Command Prompt, you can run a little program called CheckMQTTCfg and it will check the file and report any errors. If it passes that it at least syntactically correct and as semantically correct as the program can figure out, though I'm sure there are more checks I can do. We'll discover that as we move forward.

But it should avoid a LOT of busy work getting the config file right. As I said, I don't want to do heavy docs until we get closer to the real thing. In the meantime, just ask if you have questions.

Trace File

If you enable the trace file, it will be in the same directory as you created above for the config file. It will be called Trace.Txt. It may have zero size, but should still have content in it. You can just do:

type Trace.txt

to see the output. You can't open it in an editor, since it is a live file. But you can do:

type Trace.Txt > MyTrace.Txt

or whatever and redirect the output to another file that you can open in an editor. Mostly it's for my use, but serious MQTT users will obviously get to understand what it's telling them.

In Conclusion and Thusly and Henceforth So

This is early stuff, but in so far as the limitations go, just getting info into CQC for a fairly small set of field types, it should be pretty useful already.  So give it a shot and see how it works for you.

One thing I'm considering, in order to avoid the config file getting out of hand, because the possible things you'd need to tell it are almost endless, is to maybe just turn it into a CML class that the driver will load and run. Or, to do a Razor/Blazor style thing, with CML snippets embedded into the file. Mostly that would take over the things like the text map and to do things like map between MQTT values and field values.

That might be the only way to keep it from turning into something like the Z-Wave driver, which took endless years of work to get it flexible enough to handle most of the things it needs to. Of course if we were just exposing raw values that would be one thing, but we need to have that strong mapping between CQC's well defined types and MQTTs completely undefined topic formats. Else we lose a lot of the value of doing this.

RE: Early MQTT support - Dean Roddey - 03-02-2019

Anybody get a chance to play with this guy yet?

RE: Early MQTT support - znelbok - 03-04-2019

I have not had any time to myself at all. My server has not even been logged into for about two months due to other commitments - I am trying hard to get a night to load up the new version, deal with the issues and then set up the testing

RE: Early MQTT support - Dean Roddey - 03-10-2019

So I've been going back and looking at this again in terms of how to deal with the mapping of in and out values. I'm not going to go with the CML snippets thing. I'm sticking to XML and just setting up a much more flexible, open ended scheme.

RE: Early MQTT support - batwater - 03-11-2019

Ditto what Mykel said, should have time in the next week or so...

RE: Early MQTT support - Dean Roddey - 03-12-2019

Thing started solidifying fairly well today wrt to the in/out value mapping stuff. I've got it simplified but more flexible.

RE: Early MQTT support - Dean Roddey - 03-13-2019

OK, progress is being made. I got distracted for a while to day about the AWS MQTT server. Apparently it requires support for ALPN (Application Level Protocol Negotiation) which is part of the secure sockets stuff. So I had to go figure how how that works, and everything to do with the Win32 SChannel stuff (which is what I wrap to support secure sockets) is some sort of state secret pretty much. I got that figured out, though I've not connected to the AWS server. Bryan was messing with that and gave me his URL to try but there's some other configuration issues we need to work out.

* Strangely, Hue announced that they will now require SNI support as of I guess it was April 30th. That's also something that is part of the secure sockets stuff. I looked and I am already doing what is necessary for that. So in theory that should just keep working.

Anyhoo, I got back to the config and value mapping stuff. Here is a new example config file that shows some of the new scheme:

<?xml version="1.0" encoding="US-ASCII"?>
<MQTTCfg Version="1" MQTTPort="1883" MQTTAddr=""
         MsgTrace="Yes" VerboseMode="Yes">

        <Generic Topic="CQCDev/Status" BaseName="SystemStatus" Access="R" FldType="String"
                 Limits="Enum: Startup, Ready, Shutdown">
            <PLFmt Type="Text"/>
                <TextVal TarVal="Startup" Dir="In">"Loading", "Initializing"</TextVal>
                <TextVal TarVal="Ready" Dir="In">"Ready1", "Idle"</TextVal>
                <TextVal TarVal="Shutdown" Dir="In">"Cleanup", "Stopping", "Stopped"</TextVal>

    <TempSensor Topic="CQCDev/TempSensor1" BaseName="Temp1" FldType="Int"
                MinVal="0" MaxVal="100">
            <PLFmt Type="Text"/>
            <ScaleRangeMap MQMin="0" MQMax="2000" FldMin="0" FldMax="100"/>



So there is a new Generic type for stuff that doesn't fall into one of the support standard semantic field types. You have to indicate the field type and any limits yourself. And BaseName is actually the whole field name in that case.

And you can see two of the new mapping schemes. The first one is a generic that's an enumerated string type field. It has values Startup, Ready, and Shutdown. It sets up a text map that will map various MQTT values to those field values.

A lot of the new maps are going to have that sort of scheme. So a target value (which is the field value for Dir="In" or "InOut" and/or the MQTT value for Dir="Out" or "InOut") and any source values that can map to that target value. So here Loading and Initializing map to Startup. Ready1 and and Idle map to Ready, and Cleanup, Stopping and Stopped map to Shutdown.

This is a read-only field so there are no Out mappings. But they would do the opposite. It wouldn't make a lot of sense to use InOut here, you'd really need separate In and Out mappings if this was a read/write field.

The next one a temp sensor. The field has a range of 0 to 100. The sensor has a range of 0 to 2000. It uses a ScaleRangeMap to just map between the two ranges, so that's pretty easy to do. This type is inherently two way (in and out) depending on the field's access. TEmp sensors are always read only so it will only be used to map values coming in from MQTT to the field's values.

I'll get more mapping schemes done pretty quickly now that I have the infrastructure set up to make it pretty easy to do.

RE: Early MQTT support - Dean Roddey - 03-14-2019

And apparently AWS requires client side certificates as well, which I hadn't added support for since IMO they are sort of useless. But I've made the changes that I think are required to support them and the config file has a new optional CertInfo="" value. The format is the same as for the CQC web server.

RE: Early MQTT support - Dean Roddey - 03-15-2019

So, because of all of the changes to the secure channel stuff, I went back to insure that it still worked for the server side in secure mode with certificates. I had removed the old ones because they didn't include the alt name stuff that is mostly required and had never generated any new ones. So I set out to do that last night and test it, and it turned into the most ridiculous flailing about. I just now got it working again. And there was indeed a little glitch I had introduced, but of course I assumed it to was something about the certificate generation process (which is way too complicated) so I was trying all kinds of stuff, and a lot of the inf out there is out of date as usual.


RE: Early MQTT support - Dean Roddey - 03-17-2019

I think I have enough stuff back under the new value mapping regime to get a drop out. Still no field writes, but it's now reasonable to add it. But I'll probably get a drop out first and then work on that part.