Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Class: MediaRepository
#1
General Description

This thread is for discussion of the device class for the Media Repository class. This is one of those where a device class effectively defines all of the functionality of a driver, though of course there can be other stuff implemented by such drivers beyond this class definition, or in some cases implement other device classes.

Media repositories have two fundamental jobs. They load up and cache media metadata from some source (files on disk, exported info from media management program, etc...) They then serve up that metadata to clients via a fairly extensive set of backdoor commands. These commands are understood by the standard metadata oriented media widgets, and that is how they display and allow you to interact with metadata.

There are some important issues that all media repo drivers must deal with, mostly related to loading data, and their current reported state.

Data Loading

Clients need to know if repositories are ready to interact with, which means that they have loaded the required metadata and can now serve that data up correctly. They all have a load status field that indicates their current state, and clients will use this to know if it is safe to interact with them.

The 'load status' field has these values:
  • Initializing - The repo is currently in the process of starting up, and of course is not ready for use.
  • Loading - The repo is doing an initial load, so it is up and running but has not yet successfully loaded its data. So the client shouldn't try to use it yet, but probably can successfully wait for it become ready without having to block for too long.
  • Ready - The repo has loaded repository contents and is ready to use.
  • Failed - The repo has failed to load, and is not going to successfully do so without some sort of change in circumstances and most likely some sort of user intervention.
Repositories CAN support being asked to reload their contents, and most will. However, note that the above states do not cycle through again. All repositories MUST be able to load into a temporary media database, in an offline way, while leaving any current data accessible. Therefore the repo MUST stay in Ready state if it is already ready. If the offline load succeeds, then the new data will be swapped in quickly, with no interruption of access to the clients.

This scheme ultimately provides a much better experience for the user. If a reload fails, it doesn't leave the repo incapacitated. The last successfully loaded contents remain in place and available. And it also means that data remains available during the actual reload process, which can take a bit of time in some cases. It does of course mean more memory usage since for a short while the database will be in memory twice, but the usage is small by modern standards.

Note that there is also a more general 'Status' field, which is opened ended and is used by the repo driver to reflect its current state. This field MUST cycle through on each reload, and can be used to indicate actual loading status.

The basic strategy will be:

- On driver initialization, set it to initializing
- Once the driver starts its first load attempt, set it to loading
- Once the first load is complete and data is available, set it to ready and leave it there. If reloads fail, just log information about the failed loads.
- Use the Status field, as opposed to the LoadStatus field, to indicate current loading status on each reload.


Caching Support

In order to be practical, media data must be cached by clients, so that they don't have to continually come back to the repo driver to get it. As of CQC version 4.4.902, this is accomplished by way of the 'Client Service', which is a service that runs on all machines, including clients, and which will over time provide various services for local clients. Currently one of them is to download and cache media repo data and serve it up to clients on that host.

All media repository drivers MUST implement a few simple strategies to allow the client service to cache data as effectively as possible. There are currently two caching strategies involved:
  • Metadata - The metadata itself (the in-memory database that each repository maintains) is one set of data that the client service downloads and caches. To support this all repositories MUST provide a 'database serial number' that changes any time the database changes. Where at all possible this serial number SHOULD be the same if the driver is stopped and restarted, so that the client service doesn't think it needs to re-download the data.
  • Cover Art - Cover art is even more of a burden, and all possible steps must be taken to avoid downloading it more than is required. All media repo drivers MUST generate a 'persistent id' for images that will always be the same for a given image, even across a restart of the driver. This way the client service can use this persistent id to know if it already has the image locally. Worst case, the driver must read in the image and hash it, if the metadata source doesn't provide some unique id that it already has in place.

For the overall metadata database, if the driver reads in an XML file, for instance, it can just read the whole file in and hash the contents. This is guaranteed to only change the serial number if the file changes. If it doesn't have such a single source, it can load the database, then flatten the data and hash the resulting memory buffer. Actually, given that the repo driver probably doesn't use many of the fields exported by the source program, the latter might be preferable since it would only change if the actual data used changed. If the metadata source provides a unique id, then that can be used of course, as long as it is guaranteed to change if the metadata changes.

These values are just strings, so they can be hashes of data, they can be numeric values formatted to text, concatenations of pieces of info provided by the metadata source, or whatever is useful. As long as they only change when the actual data changes, they will serve their purpose. If doing a concatenation of information, and that information could be long, it would be better to hash it, so as to keep the id short and faster to compare.


Fields Provided
[INDENT]The public field interface of media repositories is quite simple, despite the fact that the internals are very complicated. The fields provided by this device class have pre-determined names, and these MUST be implemented as indicated here. They are all prefixed by the device class prefix in the form:

MREPO#fieldname

where MREPO# indicates it is a field of this device class, and fieldname meets the general requirements of CQC field names. There will never be multi-unit considerations for this type of device class.
  • DBSerialNum. This field MUST be a string field. Any time the database is modified such that it could invalidate existing information, this MUST be updated to uniquely reflect that loaded version of the database.
  • LoadStatus. This field MUST be an enumerated, read-only field that indicates the status of the repository. The enumerated values MUST be Initializing, Loading, Ready, and Error. See the above discussion for how this field MUST be used.
  • ReloadDB. This field MUST be a write only Boolean field. Any value written to it will tell the repository driver to reload data from its metadata source. Where possible it should just kick off an asynchronous process that will load the new data while leaving the driver in a ready state in the meantime, and then just switch over to the new data without ever leaving the Ready state. If the driver can determine that the actual data hasn't changed, it should maintain the same value in DBSerialNum.
  • Status. This field MUST be a string field, but the contents are open ended and purely for human consumption, to provide status primarily during the loading process. Providing information about the current state of the loading process is often helpful for in the field diagnosis of issues. In many cases it will be status text gotten from the actual metadata source, just being passed through into this field. Note that this field DOES recycle on every reload of the database, unlike the LoadStatus field.
  • TitleCnt. This field MUST be a read only, unsigned value that is set to zero initially before the first good load. One data is successfully loaded, it is just updated when new data is swapped in after a successful reload. It MUST NOT be reset to zero on each reload. Leave it reflecting the number of titles in the currently available data.
[/INDENT]


Multi-Unit Considerations

[INDENT]This device class does not expect there to ever be multiple repositories within one controllable device, server, etc... at least not any that are considered to be of a piece. There may well be separate media repositories, but they will have separate driver instances.[/INDENT]

Backdoor Commands/Queries

[INDENT]There are many backdoor commands defined for media repositories, but they are not currently publically defined.[/INDENT]
Dean Roddey
Explorans limites defectum
Reply
#2
(reserved for expansion)
Dean Roddey
Explorans limites defectum
Reply
#3
(reserved for expansion 2)
Dean Roddey
Explorans limites defectum
Reply
#4
I hope that as a part of this you will finally expose MediaRepository code in CML. I would love to just use XBMC as my media repo and dump MyMovies. It is a lot more flexible, updates automatically, and has support for things like TVShows on top of movies.
Wuench
My Home Theater/Automation Website

[THREAD=5957]BlueGlass CQC Config[/THREAD]
[THREAD=10624]Wuench's CQC Drivers[/THREAD]
Reply
#5
Possibly so. So far having do so would have been a disaster because it's continued to change quite a bit, and we'd have been locked in badly probably. It will require adding a lot of support functionality in CML to insure that things get done right as well.
Dean Roddey
Explorans limites defectum
Reply
#6
wuench Wrote:I hope that as a part of this you will finally expose MediaRepository code in CML. I would love to just use XBMC as my media repo and dump MyMovies. It is a lot more flexible, updates automatically, and has support for things like TVShows on top of movies.
Yes!! I am looking into using XBMC also for all. Currently I am trying to get it setup to record my tv shows from my HDHomerun tuners.
tia, Ron

My HT equipment I want to control by CQC (some day hopefully)
Yamaha CX-A5100, Dune HD pro 4k, Dune HD Pro 4k plus, ISY 994i, LG 86" 4k FP, and a projector in the future
Reply
#7
I took a whack at a definition for repositories. I hadn't planned on doing this stuff yet. But, because of the new ability for interfaces to dynamically resize and to take advantage of the inertial dragging stuff, both being introduced for 4.5, it's necessary to get rid of the paged scheme used by the cover art browser.

That requires an ability to cache info and art more efficiently, and that requires changes in the repo drivers that I'll need to make. It'll also in many ways simplify the CAB because it won't have to be page oriented anymore. And it'll get rid of the need for the repo drivers to maintain query cursors, which is a lot of complexity that can be tossed on the server side.
Dean Roddey
Explorans limites defectum
Reply
#8
Updated to reflect the latest thinking, driven by the current work I'm doing on the repos. The primary change is that all of them now load into a temporary database. That means that, once there is a successful load, they are always available. If a reload fails, it doesn't affect the currently loaded contents, which stay available. And it also means that the currently loaded contents are available during reloads.

This works out much better I think. It means that the new LoadStatus and the TitleCnt fields don't change during a reload. Once a good load is done, the LoadStatus field stays at Ready. And the TitleCnt field only gets changed at the end of a reload, when the new data is swapped in. It doesn't get reset to zero and count up during the load.

There is a generic Status field, that each repo can use to display whatever they want, and which does track current load activity, and is more for spelunking. It's not something that will be used by client logic.
Dean Roddey
Explorans limites defectum
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Class: Security Dean Roddey 34 35,422 01-02-2019, 12:26 PM
Last Post: Dean Roddey
  Class: Weather Dean Roddey 6 6,688 10-11-2018, 11:09 AM
Last Post: Dean Roddey
  Class: Thermostat Dean Roddey 17 26,481 01-29-2016, 10:15 AM
Last Post: Dean Roddey
  Class: SceneCtrl Dean Roddey 20 20,839 03-31-2015, 12:38 PM
Last Post: Dean Roddey
  Class: Irrigation Dean Roddey 8 9,532 01-28-2015, 12:09 PM
Last Post: Dean Roddey
  Class: NowPlaying Dean Roddey 8 9,785 09-23-2014, 02:01 PM
Last Post: Dean Roddey
  Class: Switcher Dean Roddey 9 10,626 08-20-2014, 08:17 AM
Last Post: Dean Roddey
  Class: Projector Dean Roddey 29 27,010 08-11-2014, 08:56 AM
Last Post: Dean Roddey
  Class: Lock Dean Roddey 5 7,192 07-31-2014, 01:24 PM
Last Post: Dean Roddey
  Class: ContactClosure Dean Roddey 7 8,742 07-31-2014, 10:14 AM
Last Post: Dean Roddey

Forum Jump:


Users browsing this thread: 1 Guest(s)