5.5 Beta Discussions Thread

[Dean Roddey]

5.4 is out, so any beta types discussions now should be in the context of the 5.5 release. Though of course some of the things that might come up here may be addressed in some 5.4.x followup releases. More to follow, just getting the new threads created and the old ones closed and unstuck.

[Dean Roddey]

It doesn't seem too likely that any 5.4.0 issues are going to come up immediately and require a followup release. So I'm taking my chances at having to work on two streams at once and going ahead with some early 5.4 beta stuff. Some of this early stuff could be 5.4.1 stuff, and some might end up in such. But, for now it's beta bits. Because the changes are so small, it's essentially 5.4. The biggest change was just to remove the old RIVA system. That's not likely to introduce any issues, but it's by far the biggest change in the 5.4.900 drop I just put up.

[kblagron]

The two things I would like to see added:

1) A search engine for the Help System
2) ability to perform a Load Overlay command in the Timeouts pulldown .vs. just a LoadNewTemplate.  Although this works fine now, if there is ever a communications issue between the IV computer and the master server, the timeout becomes disabled.

[Dean Roddey]

Something I'm considering doing here soon, which falls into the 'needed infrastructure improvement' category, is to deal with our support of Unicode. The whole issue of Unicode is a mess. Currently we use UTF-16 on Windows since that's the native format. But it sometimes take two of these to represent a character. That makes it possible that you can split a character in the middle if you aren't careful, and in theory you can never really just index a string because the Nth UTF-16 code point may be in the middle of a character.

Some folks argue for UTF-8, but it's just so complicated. I use it for persisted text, where it's great. No endian issues and compact. But for an in-memory representation, it's brutal. It's variable length, so just to get the length of a string you have to traverse the whole thing and count the characters. Characters can't be just native types anymore, they'd have to be a little structure of some sort with a byte count. A lot of people act like it's all simple and why would you ever want to get the 3rd character from a string. But it's utterly common to do such things.

The other option is UTF-32, which uses 4 bytes. That's the native Linux format. It wastes some bytes, but it allows us to handle all Unicode characters as fundamental values, index strings in a natural way, calculate the size of a string from the byte size of the buffer, do calculations on byte offsets and such.

But, that doesn't deal with 'graphemes'. A lot of things you see displayed aren't individual code points, but a combination of two that make up one displayed 'glyph'. So, even with UTF-32 you might split a grapheme. That doesn't create invalid Unicode though, which is important. It just means that something that was intended to be displayed as two bits together get one bit going to one string and one bit going to another. I'm not sure that's worth worrying about relative to the other advantages of still having actual characters and indexable strings, particularly given that there is zero means of algorithmically determining the boundaries of graphemes anyway. You can only depend on them not having any of the usual separation characters in them (spaces, new lines, etc...) In terms of text processing in general, I think that treating each UTF-32 value as a character is reasonable.

A lot of people say you have to UTF-8 these days and act like indexing a string or having to find the length of the string or get some character near the middle or some such are some kind of unusual operations. But it clearly those are common, and a lot of things you would commonly do in text manipulation become a lot harder and much higher overhead.

Of course UTF-16 is still the native Windows format, so Windows would necessarily become the worst case scenario in that all text would have to get converted to native on each use. Or, the string class would maintain a separate native format pointer and any time the internal format is change a flag will be set indicating that any use of the native format requires it be re-generated. That would then though mean that a string would have to hold both a UTF-32 and UTF-16 form of the string, which is a lot given that most of the characters are actually 1 byte. But it's the only thing that would ensure good performance. Even then it could hurt because if you change one character in the internal format, the native format has to be regenerated. In some cases that would be every time, such as updating display text every time the user types another character.

If we did UTF-8 we'd have to do something similar since the conversion is even higher overhead in that case, being variable length. In that case we'd pay the price of the complexity of UTF-8 and still need to keep around the UTF-16 (Windows) or UTF-32 (Linux) format for passing to system APIs, or convert from UTF-8 to native every time text is passed to the system. So it either wouldn't save much space and would introduce a lot of complexity, or it would save a lot of space but introduce a lot of complexity and overhead.

Of course not using the native wide character format means that, even down in the operating system wrapper layer, we can't use any native text processing helper stuff. We need to move it up to our own code, which is potentially something that will be difficult. But, otherwise, we have to convert to native, use some native processing, then convert back. That would be sort of stupidly heavy for most operations.

Anyhoo, I'm sort of typing out loud. It's all quite heavy. But something needs to be done. The path of least resistance is to get something that works for Linux and Windows and that drastically reduces the chances on Windows that we split characters when dealing with non-western languages. That would argue for UTF-32. But, maybe it's time to just suck it up and do UTF-8 and be done with it and accept the pain. I dunno.

[kblagron]

Another thing that I think has been echoed a few times on the last beta discussion - Is how the Align buttons now works within the IE.  I know you said it's based on Z-order, which isn't easily determined visually.  The old way of selecting the first widget, and then shift selecting the remaining widgets and allowing the align to go left, right, etc. is a much easier way to do it.   I can usually get that one to finally align left and right, but have more or less given up on how to do that if you want to evenly space widgets.  That one is frustrating, and I usually end up doing it by eye because I can't get it to work right.

[bryanb]

Dean,

I agree that the align buttons are a high priority for me too. Currently, I'm just aligning everything by eye instead of using the align buttons.

Also, you said you could create a way to close an popup or popout by clicking outside the popup or popout. That way is would operate more like a drop down box in Windows.

Bryan

[Dean Roddey]

So I looked at that some today and at least reminded myself why it changed the way it did. The ultimate problem is that, right now, when you ask to align, I have no way of knowing if the selected widgets were selected manually in a particular order, or lasso selected.

Lasso selection was always a problem because it had no idea which one you really wanted to treat as the anchor. So I updated it so that it sorted things in the direction of alignment and used the x-most one as the anchor.

Once I'd done that, I realized that now there was no way to also support the 'first selected' scheme, and too much work had been done to change at that point. The reason being that, to make both of those work, the selection scheme will have to change such that I can remember how things were selected, and therefore know which of the things to do.

So, anyhoo, now I know again what the problem is I can think about dealing with it.

[Dean Roddey]

OK, it's a bit of a hack and I'd like to have done something more fundamental, but I think I have this taken care of now, so that it works correctly with lasso and manual selection. If you cut widgets and paste them back in, this distinction is lost. It only knows which kind of selection you did when you originally selected them.

So this should be happy in the next drop. I've been doing some consolidation to get rid of one of the few areas of redundancy, and I want to complete that before I do the next drop, to get it all out at once. Basically I'm creating a 'CQC Server Framework' that standardizes all of the activities involved in getting one of the back end servers initialized and cleaned up, leaving each server to only deal with things that are particular to it. That also makes it a lot easier to add some new functionality and be sure it's implemented across all the CQC server programs.

[Dean Roddey]

5.4.901 is posted. This one has a significant internal plumbing improvement, but also restores the old way that the IV editor chose the anchor widget for alignment operations. No, if you lasso, it's the one furthest in the direction of alignment. If you select them one at a time, it's like it used to be with the first on selected being the anchor.

The plumbing changers were to factor out common functionality that are in all of the CQC servers for start up and shutdown, and put them in a common base framework class, which makes sure things always happen in the same order and handles almost all of that work itself, but allowing each server to customize where needed.

Though that work seems to be doing fine, and is ultimately a significant simplification, it is a substantial change and it's possible you might see an issue of some sort.

[Dean Roddey]

I just posted 5.4.902, which takes care of a long hiding bug in the Z-Wave driver related to secure devices, and a problem in the old (non QS) Homeworks client side driver, and an issue I introduced in the previous drop where actions running inside the web server (Echo stuff) can fail due to lack of rights.