Another point wrt services: for maintenance and maintainability point of view creating a huge object model for the entire controller isn't really helpful, it doesn't convey the module boundaries of the system very well. We need to structure the controller into services so they become isolated units that can be updated individually, and whose dependencies to other services is clearly defined through service configuration. That's why you need to be thinking in terms of services, not an object graph which gives a monolithic view to the system. You kind of did that with the registry by recognizing it's a singleton (ie. a service) but then you just went and said its a service that does pretty much everything that was left over from the object model to do (so just another monolith ;-)

Getting real implementations in place is the best way to learn what this is about and what the specific needs of HA controller will be from a middleware standpoint. This is a JBoss application from our eyes and as CB commented privately not a very a complicated one either.

In KNX, they talk at length about the lookup tables and logical tags and bla bla bla, it kind of puts me to sleep. Clearly a device is a singleton and can be named hierarchically roomA.lampA etc by a user-mapper or better yet by discovery of these devices. In KNX I read about exporting interfaces and declaring services that the device offers. Self describing devices could easily be mapped by an end user as every time you pop in a lamp switch something tells you about it. Adressing seems like a one-time event at setup and during maintenance, yet seems to be a huge sticking point for a lot of people.

As to typing a system. I agree that typing the Controller may be premature, and as you said we will learn greatly from a implementation first! Datapoints emitting events and registering to events is not too hard a programming model to offer to our users. We don't need the flexibility of a JBoss, because we are an application of JBoss. We don't need to think generic but rather applied. Focused on HA. Some folks have been talking about Domain Specific languages in our forums for example which I found very interesting in a AO way. We may find ourselves developing another tag-driven language very soon (like a SEAM but smaller) with tags that mean something even to an installer.

I think part of the point is that if the object domain is that simple, typing it (both system and user) once and for all would provide a stable, type-safe, dummy proof API to develop against. A Good Thing (tm). Everyone is looking for that it seems.

The only level of flexibility I'm talking about here is:

1) being able to choose what components go into the Controller based on the profile I've created in the Online Manager. I don't want Crestron protocol implementations if I'm not using Crestron.

2) Being able to update protocol implementations without bringing the whole controller down, or being able to manage multiple protocol implementations in case of firmware implementation differences, for example.

Juha Lindfors wrote on Aug 09, 2008 15:29:
The only level of flexibility I'm talking about here is: 1) being able to choose what components go into the Controller based on the profile I've created in the Online Manager. I don't want Crestron protocol implementations if I'm not using Crestron.

Right, absolutely. What is the packaging btw, have you settled on that? OSGi? MC? I see Ales Justin lurking around :)

2) Being able to update protocol implementations without bringing the whole controller down, or being able to manage multiple protocol implementations in case of firmware implementation differences, for example.

Right, absolutely, dynamic profile management is a must. Do make the discussions you are having public as I think they justify a lot of what we are doing.

Marc Fleury wrote on Aug 09, 2008 16:52:
Right, absolutely. What is the packaging btw, have you settled on that? OSGi? MC? I see Ales Justin lurking around :)

MC beans at the moment. We need to update first to latest AS libraries to go OSGi, and from Ales I understood AS services are not OSGi yet. Once AS is OSGi, I will be reasonably confident that the implementation more or less works ;-)

Right now it would look like OSGi brings a more fine-grained classloading control over the existing classloader repository (or so I understood from Ales -- Ales if you're there please jump in).

Juha Lindfors wrote on Aug 09, 2008 18:11:

Marc Fleury wrote on Aug 09, 2008 16:52:
Right, absolutely. What is the packaging btw, have you settled on that? OSGi? MC? I see Ales Justin lurking around :)

MC beans at the moment. We need to update first to latest AS libraries to go OSGi, and from Ales I understood AS services are not OSGi yet. Once AS is OSGi, I will be reasonably confident that the implementation more or less works ;-) Right now it would look like OSGi brings a more fine-grained classloading control over the existing classloader repository (or so I understood from Ales -- Ales if you're there please jump in).

Better late then never. :-)

Two initial things I see/care about OSGi is fine-grained classloading and service registry. Classloading is mostly in place, just few edge cases to polish. The service registry is something that is not that crucial for us atm, so it's on standby. Also due to the fact you can already get similar functionality through MC, hence our OSGi service registry is gonna be nothing more than facade on top of MC API.

In order to apply fine-grained classloading, you need extra information. It can be either OSGi Manifest.mf or our custom jboss-classloading.xml. All other pieces are already configured in deployers.xml. The main problem here is, that this is not documented anywhere yet, more than Adrian's initial forum post. But this is on my (or Adrian's) top todo list.

I've started a MC demos project. Classloading will be one of the subjects I also mention. Or you can already look at my demo I did for this year's CommunityOne conference: http://anonsvn.jboss.org/repos/jbossas/projects/demos/trunk/osgi/

For more info on classloading ping me on MC user forum. ;-)

Hello Ales,

Thanks for getting back to us.

Just copying the response on IM for everyone's benefit here.

At the moment it looks like our classloading needs are pretty ordinary for hot-deployment, from circa 2003. Trying to constrain custom types within individual services to enable runtime type evolution if necessary. Thinking about going all serialized between services since performance won't be an issue, using xPL like string format. So nothing OSGi specific, just plain old Java class definition and loading. Once we know what we are doing, can build compiled types on top of that if necessary.

Will ping you for more MC related details once we get to a point where we need to add developer-friendly features to the core packaging (regarding custom deployers and annotations etc that we emailed about earlier).

Wade Wassenberg wrote on Aug 09, 2008 16:07:

2) Translators are stand-alone services.

Why a service? In my model, they are already stand-alone. But they aren't a service, they don't need to be a service.

Protocol implementations and protocol translation implementations are the parts that are likely to change over the life time of the controller. That's why they should be clearly separated from the routing implementation (which is likely to change less often). Separation in a sense that allows a class implementation to change at runtime requires them to be deployed as stand-alone units with individual classloaders. Hence a service. This could be an OSGi service or JBoss service (or something else that defines runtime classloading semantics).

If you don't like that, then the assumption with not creating services are:

We create a set of core classes that never change or if they change require a JVM restart. If these assumptions are acceptable then service deployment (OSGi, JBoss or otherwise) and classloading becomes irrelevant and frankly we can cut down our runtime considerably. Assumption would be that protocol implementations and translations rarely require updates. Any changes to the sofware would be done purely via bootstrap classloader or are unique package additions via dynamic classloading that are only loaded once. Users these days are conditioned to accept restarts when they update so that assumption might make this a viable approach.

I don't like it because I think as engineers we should be able to do better :-)

The other road is to build the system without specific types. Think of everything typed as Strings. String implementation won't change (without changing the JVM first, anyway). This is essentially serialization. In that case we don't need an object model, we need a serialization specification. The performance would be worse but in our case is hardly an issue.

Wade Wassenberg wrote on Aug 09, 2008 16:07:

What needs to be defined for protocol handlers is how they locate the routing service which is either via the existing middleware kernel registry or JNDI depending on how we want to set things up (or even injected at some point if somebody feels that brings added value).

See, in my model, this doesn't have to be defined as it's a singleton in the JVM, and they all share it. It's protocol agnostic. Thus, in my model, this becomes far less complicated.

You are misunderstanding. A routing service is a single instance in both cases (as are all services). A lookup name from either JNDI or bean registry (or reference injection) is essentially relying on a specific instance of a class.

Wade Wassenberg wrote on Aug 09, 2008 16:07:
Overall, I really don't think we're all that far off from one another. I just think I didn't do as good of a job, I could've better explained the model so that you could see just how close it is to your vision. (which is impressive in that we haven't really talked about it all that much).

As I said, the elements are all the same:

• Registry (done)
• Scheduler (done)
• Protocol Handlers (X10, KNX, INSTEON, Crestron, ...)
• Protocol Translator (need more than one protocol first)
• Router (needs protocol handlers and translators to really be needed first)
• Address Table (needs all the above first before really needed)

And I am definitely assuming a transitive chain of translators from which can be immediately deduced if (X->Y) even if no direct translator exists.

My issue with the object model is not the elements it implies (apart from some naming) but that it doesn't immediately show me where the service boundaries are which is something that is needed should we want to update the system at runtime.

Once we agree on the services (see the list above) we can think about what types the services expose. Types are important since the references to those types will limit our ability to update parts of the system at runtime.

Juha Lindfors wrote on Aug 09, 2008 18:04:
Protocol implementations and protocol translation implementations are the parts that are likely to change over the life time of the controller. That's why they should be clearly separated from the routing implementation (which is likely to change less often). Separation in a sense that allows a class implementation to change at runtime requires them to be deployed as stand-alone units with individual classloaders. Hence a service. This could be an OSGi service or JBoss service (or something else that defines runtime classloading semantics). If you don't like that, then the assumption with not creating services are: We create a set of core classes that never change or if they change require a JVM restart. If these assumptions are acceptable then service deployment (OSGi, JBoss or otherwise) and classloading becomes irrelevant and frankly we can cut down our runtime considerably. Assumption would be that protocol implementations and translations rarely require updates. Any changes to the sofware would be done purely via bootstrap classloader or are unique package additions via dynamic classloading that are only loaded once. Users these days are conditioned to accept restarts when they update so that assumption might make this a viable approach. I don't like it because I think as engineers we should be able to do better :-)

Yep and that's what all of the arrogant a-holes at microsoft think too, and look how crappy their products are... there is such thing as trying to be too fancy. BTW I'm not trying to insult you by comparing you to a Microsoftie, but I also don't want to try to get too cutsie. Besides, how often do you see these things changing? If things change daily, and you are going to update the core on a daily basis, then yes, hot-loading would be a good solution, but if you see this thing updating once every 6 months... then who cares if you have to restart the system (unless there are state issues related to restarting the system... this is a problem we're going to have to discuss at some point in the future).

The other road is to build the system without specific types. Think of everything typed as Strings. String implementation won't change (without changing the JVM first, anyway). This is essentially serialization. In that case we don't need an object model, we need a serialization specification. The performance would be worse but in our case is hardly an issue.

I'm going to pretend that you didn't just say this... First off, everything isn't a string, when I'm talking down a serial port, I'm dealing with bits and bytes, not strings. Secondly, your argument here doesn't hold water because even if we use pre-defined objects with standardized inheritance, then it isn't gonna change the basic structure of an Object without changing the JVM.

Maybe I don't get the point you were trying to make here, but this makes no sense to me at all.

You are misunderstanding. A routing service is a single instance in both cases (as are all services). A lookup name from either JNDI or bean registry (or reference injection) is essentially relying on a specific instance of a class.

Yeah, but why do I need to complicate things by looking it up in JNDI, when if it's loaded by the bootstrap classloader, I can call the static accessor method on the singleton to get the instance?

I'm getting mixed signals from you here, in some cases I feel like you think I'm making this too complicated, but then in other instances, it seems to be you that is making it more complicated than it needs to be. Am I wrong?

As I said, the elements are all the same:

• Registry (done)
• Scheduler (done)
• Protocol Handlers (X10, KNX, INSTEON, Crestron, ...)
• Protocol Translator (need more than one protocol first)
• Router (needs protocol handlers and translators to really be needed first)
• Address Table (needs all the above first before really needed)

See in my model, I consider the network-wire protocol between say an iPhone and the Controller Server to also be a Protocol and thus require it's own Protocol Handler. It may be that this protocol is the base protocol such that in my model most of the translators are written so that this is either the source or the destination protocol... but my model is flexible to this possibility. My model is also flexible to the possibility that the KNX protocol is extensible enough to serve as the base protocol for the system... if we do it my way, we don't pigeonhole ourselves into one de-facto base native protocol (all are treated equal).

And I am definitely assuming a transitive chain of translators from which can be immediately deduced if (X->Y) even if no direct translator exists.

I'm glad we agree on this point... I think we wouldn't be doing ourselves any favors not to take this approach.

My issue with the object model is not the elements it implies (apart from some naming) but that it doesn't immediately show me where the service boundaries are which is something that is needed should we want to update the system at runtime.

Oh, where to begin here... Ok, first, this is an object model not a process diagram. Second, this shouldn't have any boundaries... it's intended to be more abstract than that... the boundaries may be different depending on which protocol you are dealing with.

As far as the naming... that's why this is a living document... it's just my first draft... let's come up with some better names, shall we :)

Once we agree on the services (see the list above) we can think about what types the services expose. Types are important since the references to those types will limit our ability to update parts of the system at runtime.

See, this is where we disagree, my opinion here is that if we establish the interfaces, then we aren't boxing ourselves in... apparently you think more inside-out, whereas I think more outside-in.

limit our ability to update parts of the system at runtime.

This, actually, doesn't have to be the case... especially if everything is interface-driven (which is how I'd prefer it anyway). All you have to do is play some shell-games with some classloaders, just like Tomcat, and all of the other major servlet-containers do it. Which you are going to have to do anyway, it's just a matter, again, of where do you shift the responsibility to... I'd like to isolate this complexity out of the core design and handle it from the outside, that way it becomes an implementation detail and not a design-detail, but that's just me.

Wade, add a little Tequila, a Pill, whatever :)

This is not a life or death decision, relax. Typing not typing, big deal. We can restart the machine, not restart the machine, who cares. Juha said easier if we type and don't go for hot-deploy fine by me.

Relax, we got a long way to go and real discussions to have.

Marc Fleury wrote on Aug 09, 2008 20:57:
Wade, add a little Tequila, a Pill, whatever :) This is not a life or death decision, relax. Typing not typing, big deal. We can restart the machine, not restart the machine, who cares. Juha said easier if we type and don't go for hot-deploy fine by me. Relax, we got a long way to go and real discussions to have.

I thought I made myself very clear in the beginning... I'm a perfectionist! :P

If I gave any tone that I was considering it life or death... that wasn't my intention at all. I just like to do things right the first time rather than trying to hit a moving target. :)

Wade Wassenberg wrote on Aug 09, 2008 20:36:
First off, everything isn't a string, when I'm talking down a serial port, I'm dealing with bits and bytes, not strings.

I was talking about serializing references to avoid type cast problems. It has nothing at all to do with your serial port...

Wade Wassenberg wrote on Aug 09, 2008 20:36:
Maybe I don't get the point you were trying to make here, but this makes no sense to me at all.

Ok, well I tried :-)

Wade Wassenberg wrote on Aug 09, 2008 20:36:
See in my model, I consider the network-wire protocol between say an iPhone and the Controller Server to also be a Protocol and thus require it's own Protocol Handler.

Well yes, but it's kinda special because the protocol is HTTP and the protocol handler is a servlet. What it needs is a translator from HTTP payload to native control protocol.

Wade Wassenberg wrote on Aug 09, 2008 20:36:
All you have to do is play some shell-games with some classloaders, just like Tomcat, and all of the other major servlet-containers do it.

:-)

Wade Wassenberg wrote on Aug 09, 2008 20:36:
Which you are going to have to do anyway, it's just a matter, again, of where do you shift the responsibility to... I'd like to isolate this complexity out of the core design and handle it from the outside, that way it becomes an implementation detail and not a design-detail, but that's just me.

Yes, the implementation detail is to create and package things as services and control which types you import/export at runtime.

Ok, some other time. Have a look at OSGi if you get a chance. Tequilas are on me.

Juha Lindfors wrote on Aug 09, 2008 09:28:
1) It's incorrectly named, this looks more like a (Control) Protocol Handler object model

That may be... I guess my definition of Controller and your definition of Controller must differ... I was viewing this purely from a standpoint of being the bridge between protocols.

(although there are some services mixed in) for specific protocol implementations.

A concrete example here might be helpful.

What we call the controller is the Java middleware, which is a service based architecture so it doesn't really map into this object containment model, think of a bus based architecture instead.

That's exactly what I'm thinking about. I've reduced everything to an Asynchronous event that contains an address and a command... now maybe this needs to be expanded a bit to contain two addresses, a source address and a destination address.

2) Translators are stand-alone services.

Why a service? In my model, they are already stand-alone. But they aren't a service, they don't need to be a service.

There's going to be a routing service in the controller which will eventually decide the need for a protocol translation depending on the destination of an event or command and lookup and invoke the appropriate protocol translator service.

Exactly, and this is what the Registrar in my model does. In my model, it's a singleton, and it's the routing service. It's going to require more helper classes, and some mechanisms for determining how to route things. But my assumption was going to be that we would have some sort of xml file or database that is used to define each Device (hence the Device object), and part of that Device definition would be a system-wide standardized device name (however you all want to name them, that's fine), and so the registrar would know how to look up a device to determine which translator to use.

What needs to be defined for protocol handlers is how they locate the routing service which is either via the existing middleware kernel registry or JNDI depending on how we want to set things up (or even injected at some point if somebody feels that brings added value).

See, in my model, this doesn't have to be defined as it's a singleton in the JVM, and they all share it. It's protocol agnostic. Thus, in my model, this becomes far less complicated.

3) For the same reason as in #2 the protocol agnostic Event, Command and Address interfaces are part of the routing service API, something that is not shown here.

No, that's exactly what I'm showing here (are you sure you are looking at the same model?). There are base, protocol-agnostic interfaces that unify all Commands, Addresses, and Events. However, the concrete implementations of these interfaces will be protocol-specific.

4) A protocol handler has relatively simple input and output interfaces

Unless you want to treat certain things in the system as being synchronous (which I don't think for the problem we're solving, that this is a very good idea), then everything can be treated as an asynchronous event consisting of an address and a command. Therefore, you should drop the concept of input and output, and more generally refer to it as an event... an event can represent input or output (as you are describing), but it can also represent no-operations or other types of things besides just input/output. It's a much better approach to an asynchronous system.

: input is a command or event pushed by the routing service: this can be exposed via generalized Command/Event interface of the routing service API

Yes, this is how I prefer it... I prefer it to be exposed as a generalized Command/Event interface of the routing service API (maybe this would be a better name for my model... routing service API)

or directly as a protocol specific implementations of them

In my model, it's even allowable for an event to be represented by concrete implementations of the interfaces that aren't specific to the eventual destination protocol, just as long as a translator is provided that the registrar can then use translate.

(since the router is already aware of the wire protocol requirements of a registered protocol handler and can get the native format via translators).

In my model, the router which is the registrar is NOT aware of any wire protocol requirements (it's protocol agnostic afterall)... all wire protocol requirements are the responsibility of the translators (which is what the second half of your system agrees with).

The output is also an Event/Command that is pushed to router to handle (see #2). Again since the router is aware of the wire format of a registered protocol handler we could allow the protocol handler implementation to push out its native format and have the protocol translators turn it into the router's Command/Event API; this would reduce to implementation contract for various protocol handlers which might be a good thing, we want to keep the contract for getting new protocols as minimal as possible.

You're assuming that the registrar isn't smart enough to chain two or more translators. If I provide an X10->O.R.Protocol translator and I provide an O.R.Protocol->INSTEON translator, then if the router sees that it is receiving an X10 Event object, and it needs to route it to INSTEON, It looks in it's registry of translators and sees that it can translate X10->O.R.Protocol, and then translate that new Event from O.R.Protocol to INSTEON. However, it'd have been my assumption that in a lot of cases, all of the registered protocols would receive a router-translated, protocol-self-satisfactory event to each of their controllers, that then they could decide whether or not needed to be propagated into their own little network.

Now, if you weren't assuming that the registrar wasn't smart enough to chain translators... well then you aren't solving the problem (read: you aren't simplifying anything), you're merely relocating it into a different part of the architecture.

5) Registrar is not needed as an additional public API.

Admittedly, I was a bit worried about naming it this because I knew someone was going to compare it to JNDI... it's meant to be the router and Device registry... I can split it into two separate objects. And by Device registry, I mean that it is the API that is used to obtain a Device object. In my model, the Device object (or more specifically concrete implementations of the Device Interface) shouldn't be constructed on a once-per-use basis, they should exist in the Device Registry and should be looked-up... now if the Device Registry interfaces with JNDI under the covers... that's just an implementation detail.

If the router or other service implementations need to register listeners on an individual protocol handler then that should be part of the protocol handler's API facade (see #4 about having two interfaces for input and output).

I'm not sure what you are getting at here, but I think we're on the same page. And I don't think my model implies otherwise.

6) Same with any kind of timed scheduling, these are existing stand-alone services. There are several scheduler implementations that we can plug into the kernel and use them to push events to routing service as needed. Same applies for instance to a workflow wanting to schedule events.

Yes, I was also worried about this as well... I just wanted to show an example of an impelementation of a Macro... but I think it is probably just going to confuse people, so I probably need to remove it...

That's why I wanted to discuss this :)

Overall, I really don't think we're all that far off from one another. I just think I didn't do as good of a job, I could've better explained the model so that you could see just how close it is to your vision. (which is impressive in that we haven't really talked about it all that much).