Charmed Quark Systems, Ltd. - Support Forums and Community
MQTT support discussion - Printable Version

+- Charmed Quark Systems, Ltd. - Support Forums and Community (
+-- Forum: General Discussion (
+--- Forum: Beta Discussions (
+--- Thread: MQTT support discussion (/showthread.php?tid=10774)

Pages: 1 2 3

MQTT support discussion - Dean Roddey - 01-26-2019

OK, so I have a program that is connecting to the test Eclipse MQTT server. It can connect, publish, subscribe, disconnect, etc... Of course I'm just publishing a simple string topic, then subscribing to it, and updating it, which gets me a publish msg back.

So all of the basic plumbing is there. This is just a little command line program. The point of this is to use it for us to work out a viable scheme for some info files, which have to describe data types of payloads and what values are in it and how to extract them and all that. Presumably something like the Z-Wave device info files, but these would be topic info files I guess. So on a per-topic basis how do we know what values are in it, what fields and field types and such they correspond to and so forth.

Then there would have to be a configuration file for you to configure yours to subscribe to these topics, for each one provide any non-fixed information for field names, ranges, etc... 

That will not be a trivial matter. So I don't want to really make this a big production until we work that out. So this little command line program will let us do that. It will have commands you can type to see the state of things, see fields it's defined, reload the config, etc...

Once we have it pretty well worked out, then this little test program will become a new optional service you can install, and I'll do a driver that talks to that service. The driver will just self-configure, basically like the logic server works now.

But doing it first like this means it can be small and quick to make changes. I can just post a zip file with the new stuff for you to try without having to do a new release, etc...

So, the immediate issue, is how do we know how to get the data out of the topics we publish. I have yet to see a single real world example of a what format any MQTT enabled device publishes its data in.

RE: MQTT support discussion - znelbok - 01-28-2019

My understanding is that there is no format. The data is a string and can take any value. I have not spent any real time with this other than playing with a few things.

Some devices live relays just send "on" or "off" on the desired topic.

Have a look as Tasmota - software for the popular Sonoff equipment.

C-Bus lighting MQTT implementation

RE: MQTT support discussion - Dean Roddey - 01-28-2019

They all have some 'format', even if it's off/on. It might be JSON data or whatever. So that needs to be figured out, and a way come up with to encode in a 'device info' file what topics are available, what data is in them, how to access that data, the form of that data, what type of CQC field is represents, and all that.

We don't want to have to hard code in support for every device. Like with the Z-Wave driver, we need this sort of mechanism to be able to support devices via some sort of device description that the server can load up and use to know what fields to create and how to get data out of the incoming msgs and into those fields.

RE: MQTT support discussion - Dean Roddey - 01-30-2019

I've gone back and broken out some core MQTT stuff to a separate library so that it could be used in other things if needed in the future, and reworked the (eventually to be) service to be a lot more like what it will need to ultimately be, with everything async and the ability to queue msgs for ongoing ack or retry attempts and such.

I need to get it back to where it was before in the new guise and then I can start to really work out how to deal with actual data and such.

RE: MQTT support discussion - Dean Roddey - 02-04-2019

OK, some days of brain burning head scratching and reorganizing of the code to get it really set up right before jumping into the deep end. There are a lot of ways to skin all these types of cats, so I've been bouncing around kind of crazily, but I'm honing in on what I think is a good setup. At least in terms of code organization. Haven't really delved too deeply into the issue of configuration yet.

RE: MQTT support discussion - Dean Roddey - 02-09-2019

OK, things are solidifying. I realized I was being too client-centric. I'm thinking about subsequently taking this guy up to a MQTT server, and then to maybe also offer it as a standalone product for non-CQC customers. So I went back and reworked it to get rid of that. There's still plenty of extra stuff that a server has to do. But, within the core infrastructure I have it's now good for either client or server side.

And I've now created a little test program that lets me publish and subscribe so that I can set up test data on the test server against which I can then test the MQTT client. I did that because I decided to get rid of the testing configuration of the MQTT client service and just write it as it will be from the start. It will be a lot cleaner that way. So I needed some sort of separate test rig. And I also need something to do basic testing of the interface for the CQC driver that will expose the data, and any administrative interface it may have. It'll be easier to initially test that stuff via a little test program as well.

Anyhoo, I've been using that guy to test out and tweak the msg formatting and parsing stuff, since it's a far simpler environment to test in, and there were plenty of glitches. That stuff is looking good now. I'm publishing and subscribing and seeing reports come back to me when I make changes to the published topics, and it's all reasonably convenient to work with. So it's looking pretty good as such things go.

Now I need to take the next step in the MQTT client service and start to work out the 'device info' type stuff that will be used to translate from MQTT stuff to driver fields. Then I can start testing some more realistic scenarios, where the service thinks it's subscribing to real devices, creating fields, etc...

RE: MQTT support discussion - znelbok - 02-10-2019

I like using the MQTT fx Client for testing. Its been the easiest one to use so far for me.

Following with interest. Integrating a MQTT into CQC makes a lot of sense, but I think it also needs to have an option to work as a client only as well for those that may be using a dedicated MQTT server

RE: MQTT support discussion - Dean Roddey - 02-10-2019

In this case, I'm going the other way, testing-wise. I have a client that I want to test, and there's a server to test against, I just need a pseudo device to talk to the test client, not to my stuff (which only talks to the test server.)

RE: MQTT support discussion - Dean Roddey - 02-10-2019

So, looking at some documents of things that use MQTT, unless I'm missing something obvious the topic paths are horribly defined. It's like no one ever bothered to consider that anyone but their stuff would be attached to a given MQTT server. There's nothing remotely like a well defined naming hierarchy designed to keep each organization or manufacturer's devices from happening to use the same format for topic paths.

Sonoff uses:


That's utterly generic. IBM's stuff using:


That's only very slightly better. I mean didn't it occur to any of these people that something like:


might make a LOT more sense? The schemes they have chosen, paritularly the Sonoff one could so easily be used by someone else and cause all kinds of problems.

RE: MQTT support discussion - Dean Roddey - 02-10-2019

OK, so for a first cut I realized that I don't need to get too galactic about the configuration file. Later we can add more 'device oriented' stuff but initially it can just be field based. So a sample config file might look like:

<?xml version="1.0" encoding="US-ASCII"?>
<MQTTCfg UserName="Bubba" Password="Elmo" Version="1" MQTTPort="1883">

       <LightSwitch Topic="CQCMQTTTest/Light1" BaseName="Light1" Access="RW"/>
       <TempSensor Topic="CQCMQTTTest/Temp1" BaseName="Temp1" Type="Int"
                            MinValue="10" MaxValue="100"/>


So it's one topic for one field, driven by semantic field type. For the initial drop I've done boolean and light switches, dimmers, temp sensors, and motion sensors. That'll be a good test set. Each type requires some specific set of information. If the configured info indicates that it can be V2 compatible, it will be set up as a V2 field.

So I have that info being parsed correctly, so I can move forward now to using that info to set up subscriptions and storing incoming data and such.