Thinking about the next step in RIVA

[Dean Roddey]

So, I almost hate to even bring this up, because I'm worried everyone will think it's a great idea and we'll have to do it, but it makes a lot of sense what I've been thinking about. Of course the devil is in the details and maybe it'll get a lot uglier than what I think it will, but here is the broad outline. This is mostly for the RIVA developers, but others may read and comment on it if they have the stomach for such things. Some of this is just me brain dumping to make sure it actually makes sense by having to type it out loud.


The big issue of course is that people want to have native type scrolling. Of course they also want it in the regular IV and this subject addresses that as well. If I add that sort of support to the IV, I REALLY don't want to have to special case it all away for RIVA clients. That will add a lot of extra complexity of it's own, and getting rid of that complexity offsets some of the complexity of trying to let the RIVA clients do it as well.

The problem with supporting native scrolling is that it requires that the RIVA clients have information they don't have. In order to do a scroll of, say, a track list, it's pretty much necessary to draw the (yet) offscreen content into a bitmap, and scroll that bitmap into the visible area of the widget. This is generally also required because the stuff being scrolled has to be composited onto the underlying content, because it's usually stuff that has a full or partially transparent background.

The way the RIVA system works currently, the RIVA clients cannot do that because they don't know anything about what is not currently visible. They can only be told to draw what is visible, and therefore is being drawn. Those drawing commands are passed to the RIVA clients and they draw them locally. They can't anticipate information to the left or right (or below/above) that might be scrolled to and be prepared to scroll it. Even if we tried some way to provide them with that info, it would never work to try to scroll by them sending scroll commands to the server and the server sending them all those drawing commands back to repeatedly redraw the content (as required to simulate one layer sliding over another.)

Part I - Layers

So, what I've been thinking about is that the RIVA clients be extended to support layers. They still don't know about individual widgets and they don't need to. But instead of just one flat drawing surface that the RIVA server sends them drawing commands for, they would be able to support multiple layers (the main template bgn being just one of them), and the server could send them commands to draw on any given layer. Each layer would have a unique id and the RIVA client would be told about them, and how large they should be.

Each widget that has to support scrolling (and only a small set of them do) would then maintain its own memory bitmap, bigger than the actual widget, and it would draw content into that bitmap any time it wanted to, even if it's not visible. The widget would maintain a 'viewport', which is graphics speak for what area of the bitmap is currently visible with the rectange of the widget. This is how the IV itself would implement scrollable stuff. More stuff would be there than is actually visible, so you would just be moving the widget viewport around within the larger memory image of the widget.

For the RIVA clients, information about each of these memory bitmaps would be passed to them, and they would just treat them as layers. They wouldn't care any more about what's in them (widget-wise) than they currently do about the single big image. As mentioned each one would have a unique id so that drawning commands could target output to a specific layer. That output could be outside of the visible area. The RIVA client would be kept up to date as to the viewport position on each layer. So any time any content is drawn into the layer that intersections the viewport, they would need to draw that visible bit that changed.

The layers would also be given a z-order, and the clients would be responsible for composting the layers. I.e. when it's time to draw info to the screen, because some drawing commands intersected a layer's viewport area, or because the user closed a dialog box or something like that and uncovered part of the template, the client would just build up the actual content of the affected area by blitting in (using alpha based blitting) any content of each layer in z-rorder that intersects that update area. Then it finally blits the resulting composited image to the screen.

The main template would be the lowest level layer and would always be be composited in. After that, only those layers whose viewport intersects the update area would be updated. Usually it's only one. It would be very unusual to have two scroll widgets overlaying each other.

Now, to get to the real meat of the situation, scrolling would then become a local operation. When the user scrolls, the client would just keep the server informed as to where the viewport for the layer is moving. The server doesn't need to do anything but update his information for the viewport location. That way the client is always in sync with the client as to where the view port lies on any given layer. But the client would be free to move around within the available layer content locally all he wants. The server would need to know the viewpoint origin in order to correctly translate mouse/finger clicks to know what part of the actual widget content was hit.

So, as an example, the layer for a toolbar would contain all of the text areas of the tool bar. They would be predrawn into the memory bitmap of the layer. The RIVA client doesn't know it's a toolbar. The RIVA server just sends him commands to draw all of the content of the tool bar, even that not visible. The RIVA client can then just move it around all he wants. And that's all the user cares about as well, to be able to move around within the available content. When the user clicks on the screen, the RIVA client sends that to the server. The server finds the visible widget with the highest z-order that intersects that point. It subtracts the widget origin from the click point to get the relative position within the widget. It then uses the viewport position to know where within the larger virtual content of the widget the user actually clicked.


Part II - Flies in the Ointment

The complex issue will be those widgets where the whole potentially visible content cannot be put into a memory based virtual layer all at once. For most of them that won't be a problem, but for some it will be. We can't use up 50MB of memory just to scroll through a list, at least not on smaller devices.

Basically the widgets that will need to support scrolling are:

1. ToolBar - Not a problem
2. Horz/Vert Lists - Mostly not a problem, but will have to dynamically decide if they can contain the whole contents or not. Generally they can, but some might have too many items to do that. This includes the generic list browsers, the media category browser and other widgets that derive from the basic h/v list browsers.
3. Cover Art Browser - Generally never could be assumed to be able to contain all content.
4. Media Item Brower - Though it would be nice to say that no one would ever load enough tracks to become a problem, clearly they will. So this guy would have to assume it can't load all content at once.
5. Dynamic and Static Text - These shouldn't be an issue. If someone decided to load some pathologically large amount of text into one, they could just toss the excess. In any realistic scenario about the max text they will ever display is a movie description, or maybe a page or two of text gotten from a web site or some such thing. I'd be tempted to just say don't make these be scrollable, but ultimately they would need to be.

Though there are other potential scenarios, such as overlays that contains more content than they make visible, these would not have to be done immediately. And, once the RIVA clients implemented support for the general concept more scrollable things can be introduced and they wouldn't know the difference or care. So it's not an all or nothing up front gamble.

But, anyway, for those that are a problem, it gets a lot more complicated. They will have to build up content off screen but not in the same sort of natural and easy way the others can. They will like have to support multiple virtual pages, not just a single layer. The server can send the clients content for any of the pages as required. The pages would likely just be numbered (so the unique layer id plus a page number) and sized to the visible area of the widget. As the client indicates its moving the view port around, the server would have to send it content for pages currently off screen in the direction of travel. That would also mean that the pages would have to be in a circular buffer, such that the one on the far end can be brought around to the far end of the other side. It's the old thing of picking up the track behind the train and running it forward to put it in front of the train.

[Dean Roddey]

Obviously this makes things a lot more complex than the more basic scenario I was talking about above, at least for the RIVA clients. The regular IV would already be doing this in order to support native scrolling itself. As with the simpler scenarios, it would just be exposing this stuff to the RIVA clients. But it would obviously make for a good bit more complexity for the RIVA clients than the simpler scenario where the layer contains all of the widget's contents. There would have to be more commands sent to support things like, move the leftmost page to the right end and it should now become page number X, so that both sides stay in sync. It would then send drawing commands for page X as the user is moving towards it.

Clearly, as happens with even fully native apps, if you scroll faster than data can be obtained there will have to be pauses as the data is gotten. The RIVA clients will have to have some rules about not scrolling past a page that they have available (which will be obvious since they will remain numbered based on what actual position in the list of total possible pages the client is at. I.e. if we kept 2 pages on either side, and the user was on page 8, the pages would be numbered 6, 7, 8, 9, and 10. If the user scrolled right two pages, the client would realize he is on the last page. He would have already sent the scroll command to the server, which would send him new drawing commands for the new page and then a command to move page 6 over to the 11 slot, then the client and scroll again. Most likely we'd keep more than two, but it can't be 50 either. That would eat up entirely too much memory. So there will always be possible delays.

The IV would clearly be updated to do background downloading of information in order to try to stay ahead of the game on these special case scrollable widgets, and the fgn thread would be building up new pages while the user is viewing the current one, and those drawing commands would be going to the RIVA client to keep pages ready for scrolling.


Part III - Image Ids

One thing that will be necessary to make the performance acceptable for media cover art browsing is the ability for the clients to cache media images long term, not just on a per-application run basis. To support that, the media repositories will be udpated to provide a unique id for images which will remain the same as long as the image is the same, or as reasonably so as can be guaranteed for a given repository type. But most of them would almost never get it wrong. When the RIVA client (or regular IV) asks for a media image, it will get this unique id. And it's asked to draw a media image, it will also get the id. It can check its cache for the image and use the local one if it has it. If not it can query for the image.

There would be no way to get rid of cached images that are no longer in use, so the RIVA clients should have some means of tracking last access of images cached, so it can throw away the least recently used one if it hits some sort of storage limit it imposes on itself or that is imposed on it. All the images for a given repository would soon enough get cached. And in fact we could possibly provide a means for the RIVA clients to let the user to ask to pre-cache images locally for a given repository. Maybe one or two new ones get added now and again but that wouldn't be much of a problem and they would just be pulled down normally as they are encountered during browsing.

The unique id will not be specific to a given driver moniker, only to the image in a way that would only change if the image itself changed. So if you change repo driver monikers, that wouldn't invalidate all those cached images. The ids would be universally unique. Since they are, there'd be no need to try to keep them separated by repo or anything. It could just be a flat cache, or whatever works in terms of best performance.

For the actual IVs themselves, it could also provide the means to implement a fast flip scroll type of thing, since it could pre-cache images on disk. And then it could download all the basic text metadata required, since that wouldn't take up much space. And just keep the whole list in memory. See below for a possible scenario for the RIVA clients as well. A background process could run ahead through the data, making sure all images were in the cache and grabbing any not available yet so that they'll be there in time. Or we could just have some template level config that could say to the IV, "I will be accessing images from these repositories, please download them" and at the IV level that data could be scanned in the background upon IV startup. So by the time you got to the media screen in normal usage it will have pulled down.


More Advanced Concepts

Another possible option is that one of the layers is marked as a 'client media browser' layer associated with repository moniker XYZ. The RIVA client could then just query raw media repo data and display it any way he wants. The server wouldn't ever draw anything to the layer. Since the layer would indicate the view port and the actual origin of the view port on the overall template, the client would know where to display the info. Once the user makes a choice the client would send the selection to the server, and the server would act like it was made normally and do whatever it would have done based on that selection for the target widget that owned that layer.

But that would be really iffy in terms of style because the RIVA client couldn't know how to fit its output into the overall style of the template. It could of course just display cover art and that's all, and it could send both selection and currently show selection info to the server which could react by updating other widgets to show metadata just as it would in the normal scenario. I.e. if the client wanted to do a flip page thing, where only one image was displayed at a time, it could send a currently displayed cookie to the server which would act as it would if it was doing a flip page display itself and send out a selection event that could update other widgets with the metadata. The clients would have to do this on a delayed basis so that flipping really fast through the images wouldn't inundate the server with selection updates. I.e. only do it after the user stops for some time, like half a second or something.

Since the RIVA client would only ever display images, they'd just be displayed over whatever background is there, so it would probably work out easily enough, I dunno. Anyway, this is a possibility that could be explored and might be simpler in the end than the above scenario of the server trying to keep in sync with the server via pre-drawing off screen pages.

HOWEVER, that still doesn't get us out of the woods wrt to other widget types that have the 'too much data' problem. So though it might be something folks would want to do for its own sake, it wouldn't get rid of the requirement to have some sort of way of dealing with these other widget types. And I don't think that they would be amenable to this type of treatement.



Anyhoo...

I know that's a lot to bite off. It would be a significant change. And it would be a non-backwards compatible change. It would be stepping forward and not looking back. Maybe it's more than you guys really care to deal with. But the thing is, I need to move forward in the IV before too long to support this stuff, and trying to let the RIVA clients stay they way they are while the IV moves forward will be pretty tricky. So, from my perspective, it's probably the path of least resistance to allow the RIVA clients to move forward as well, if you guys are willing to take this on. It probably won't be any more work for me, and maybe less ultimately.

If you guys don't want to take this on, then I'd just move forward with the IV supporting it and I'd have to disable this functionality on the RIVA server so the RIVA clients never saw any consequences of it. In the RIVA server case I'd just draw directly to the main graphics device as I do now and not to any memory image. Since the RIVA clients couldn't send any commands to invoke the fancy scrolling, it would remain page oriented as it is now, driven though buttons sending scroll comamands.

But, I throw this out there as something to be discussed and possibly implemented. It would be a hellaciously smaller undertaking than anything approaching a full bore native IV, but would get rid of the single biggest issue people have with the RIVA system currently.

[Dean Roddey]

OK, here's another, more radical but possibly much better, idea to throw out there....

Ultimately, the RIVA system is a battle between two things:

• More local control and flexibility
  
• Keeping RIVA clients simple and able to support most new widget types and changes to the interface system without modification.
  

That's what it basically comes down to. The current system is maxed out towards #2. It's very decoupled from the semantics of the interface system, and is purely a remote display and input system, but it allows for very little local control by clients, and that includes local interaction techniques. It also, BTW, means that the clients are very limited in what they can sort of cache locally and avoid the server resending. The server resends almost everything, very redundantly, because the clients have no semantic context for what they receiving. The only exception really is images.

#2 is very important because we cannot get into a situation where every time a new widget gets added, or even a tweak to the appearance or functionality of one, the RIVA clients have to be updated. It's only because we've maintained a high level of abstraction in the RIVA clients that things have gone as smoothly as they have. I can put out new releases mostly without concern, and they aren't constantly chasing me and only getting a new one out about the time I've in turned moved another step forward.

So, if we could find a way to maintain the required level of #2, but increase #1, that would be optimal. And, if it could greatly increase the reliability and efficiency of the RIVA clients, that would be even better. There might be a way.


The general issue is that RIVA clients know nothing about widgets. And, as a rule, they cannot know much about them. That would violate rule #2. We want to keep the RIVA clients as pure display mechanism. So, what if we told the RIVA clients just enough for them to independently draw the widgets? That might sound like a huge #2 (pun intended) but it doesn't have to be.

The thing is, we already have a means by which we can tell RIVA clients how to draw things. There's nothing to say that we have to send this information over and over again. What if the server basically sent the RIVA clients this information:

• Area of a widget
  
• A unique identifier for it
  
• A set of named values, which are either text or image paths, and their initial values.
  
• A set of states the widget can be in, and the initial state.
  
• The drawing commands (in the existing RIVA format) to draw each state. They will have references to text and images, not the actual text and images. Those references will be to the named values mentioned above.
  
• A couple flags like if it's transparent background, whether it's visible or not, whether it's clickable or not.
  

Now, the RIVA clients are able to completely redraw locally. They have all the information they need to redraw widgets, without having to have any actual understanding of what the widgets are, or having any local understanding of how any given widget type is to be drawn.

The RIVA server doesn't have to communicate to the clients at the graphical device output level anymore, it can talk directly to them from the RIVA extended interface viewer class. It doesn't have to continuously send the RIVA client drawing commands. It just has to send it things like:

• Change the state of widget X, to this state. The client updates his current state for that widget and can now just uses the drawing commands for the new state instead of the previous one.
  
• Hide or show a widget (client just sets or clears the visible flag on the widget.)
  
• Update the named value for this widget. So, if the text for a widget changes, the RIVA server doesn't have to send all the drawing commands, it just says update the named text value Foo with this new text. The client updates that named value and now it just redraws locally.
  
• Sometimes user logic may change the image or text, or the opacity of an image. In that case, the server may need to resend the drawing commands for that widget, but only reaction to that change. The client just stores the new state drawing commands and uses those from there on out.
  
• Remove widgets X through X+Y from the list (all the widgets in a given overlay or popup will be in a contiguous range just naturally, so removing a popup or replacing an overlay just involves the client removing that contiguous range of widgets from his list.
  

It doesn't make the RIVA clients any more complex really, and probably makes them less so, because they aren't drinking from a firehose anymore. They accept the same sort of fast chunk of data on the load of a template, to get the initial data, not really much worse than it does now. And after that it's fairly steady state. If there's a clock widget or animation widget it'll be sending state or text changes, but those would currently be causing many messages in the form of drawing commands, so it comes out far better.


And, finally, to get to the real meat of the argument, it offers a way out of the conundrum mentioned in the previous post about list type widgets. We can special case list widgets so that they also include:

• An indicator that they are a list widget, horizontal or vertical or grid.
  
• The size/position of the slots.
  
• And therefore the drawing commands are not for the widget itself, lists never have any actual visual content themselves, but for the individual slots of the list. This one set of drawing commands, offset in the x and y, can be used to draw the list items into any of the slots that need filling. There will be states here as well, generally for the pressed and unpressed state of a slot, with separate drawing commands for each.
  

Now the client has all the info he needs to totally draw lists locally in any way that makes sense to support the type of scrolling supported. The client can be sent all the data for the list and store it, associated with the widget. It would be in the form of a set of X named pairs, one for each value to be drawn in a given slot, usual one or two, maybe three, so it's not a huge deal. Possibly more can be sent so that the server can send it new drawing commands if the user changes sort order or what value is to be displayed. Then the client will just choose another of the named values to display.


So, ultimately, the RIVA clients would do this:

• Accept chunks of widgets and drawing commands and such on the load of a new template or overlay.
  
• Respond to changes in state, drawing commands, or visiblity.
  
• Redraw widgets affected by the above.
  
• Provide input as before. They wouldn't send scrolling commands to the server, they would do local scrolling on their own. They would only tell the server where the scroll ended up, i.e. what's the topmost item currently in the list. The server knows how many slots are visible and therefore how many items might be visible. This lets the server know which item is hit when a click type command is sent from the client. But clients can scroll lists any way they want, they only restriction is that they ultimately snap to the slot positions by the end of the scroll.
  

Redrawing is not very difficult. Any area affected, either by commands from the server or due to local windows popping up and uncovering areas that need to be redraw just require that the client go through the list of widgets from 0 to the end of the list. If one intersects the area and isn't marked invisible, draw it, using the drawing commands for the currenty set state. At the end, blit the affected area from the resulting in memory image to the screen. That's it.

The clients would provide all their own clipping of output for the most part. All drawing commands are clipped to the invalidated area (the area of the widget modified or of the uncovered area.) There may still be short term clipping commands within the drawing commands of a single widget. But there wouldn't be any of the concerns that we currently have where the server is sending all these bracketed set/clear commands for clipping, fonts, colors, etc... and if any command is missed it can all go unbalanced and fall over. Each widget would be self-contained for drawing.


It would also easily allow for things like scrollable text widgets, where if the text doesn't fit all at once in the widget area it can be scrolled to see the rest, mostly for things like movie descriptions. The client already has all the text, so it can scroll that text locally all it wants. Since these would be non-interactive (clickable flag is not set), the client never sends any click when these are hit, so doesn't matter that he scrolls it locally. The server doesn't care.



Obviously this is a fairly basic analysis, but I think it would work. The core function of the client would just be the redrawing loop, which is a very constrained chunk of code really. It would grab intersecting widgets, pass them to a drawing method which would look at the state, grab the commands for that state, and draw them, looking up indirectly referenced text and images as it hits those types of commands. There would be a few special cases to deal with.

Even if it turned out that there one or two widgets that just won't work out, it would still be worth doing. We already have a few anyway. And of course it fixes the issue of web browser widgets since we can special case the web browser widget and the client can fill that area with a browser window. We could always leave out the worst special cases first time through and concentrate on everything else.


Does any of that make sense?

[Dean Roddey]

Oh, and it also avoids all the issues of hiding or exposing scrolling support to the RIVA clients and how to do it, which is a big issue otherwise. I can do any sort of scrolling I want in my own IV, because the the RIVA clients will never see any of that, and RIVA clients can do any sort of scrolling they want. They never send the server any scrolling commands, so it never has to care what they do. And it means no need to create big bitmaps hold offscreen content so as to feed it to RIVA clients. The IV and the RIVA clients can implement large bitmaps or offscreen page pre-creation in the background, or on the fly creation of new content as required or desired.

[Dean Roddey]

Widget types that I can think off immediately that would be issues, or require special casing are:

1. The web widget of course. But it's a fairly simple special case. Just put the web browser window in the widget's area.
2. Marquee text. Having to constantly update the drawing commands to achieve offset would be silly. It would be easy enough to special case this one by just allowing the client to provide marquee scrolling using the smooth scroll features of the client. Of course in the short term they could just be treated like non-marquee text to get going.
3. Graph widget. An ever changing list of data points to draw as a graph. Wouldn't want to continuously re-send them, so a special case here to be able to add points to the point list, and just let the client draw them efficiently. Clients could initially just ignore these to get going.
4. And of course lists. The clients would have more work to do on these, but it's not that bad. They just need to scroll the list contents around using local features, clipping output to the widget area. We might not even need them to inform the server of current scroll area. They could easily enough report which one they hit.

I think that's generally it. Even if we initially ignore all these, we'd be no worse off in terms of widgets available to RIVA clients, and would have a vastly better system in place. The number of bytes being flung to the clients would go down drastically, and their local responsiveness would go up drastically as well.

It also means that the whole issue of how to get referenced images (and in this new scheme lists of info required) would be much easier. Currently, they get drawing commands and have to draw now, and if they don't have info they need they have to try to get it quick. In the new scheme, they control drawing so they can buffer up the info and then draw. We could support background downloading of images and lists for them much more easily in this scheme.

And that could even include incremental display, which is important for phones and such. I.e. they can start requesting just enough info to get the list oriented widgets displayed, then continue to get the rest in the background. They'd have a full list of all images referenced of course immediately upon getting the new widget definitions, so they could start checking their cache and downloading uncached stuff immediately as they are preparing to draw.

[Dean Roddey]

Fonceur and Cmd Vimes, any thoughts on scenario two above? It could be a way to really take the RIVA clients to the next level and make them less complex and and more robust all at the same time. It would be vastly more work for me than for you guys probably, but I'm willing to take it on if you guys are.

[brianmount]

I will take a look at all this in a few weeks. My current employment is coming to an end next week, and one of the first orders of business is to release the more-or-less complete app we have written using the XML Gateway protocol. Once that's done, I'll hopefully have time to think about next steps.