Concept of a flexible but simple smart network

[axillent]

Doing my staff and doing mysensors, thinking about others I came intp conclusion that smart network required to be more abstract, it's basics should be in logic disconnected from hardware realization. From a long long thinking I made a concept I'm currently working on. I will be glad to share and discuss it to get different views.

Abstracting requires to split all into separate levels starting from application and dropping down up to hardware. It can be done in different ways. I will describe here my way.

First of all the message. It very common to have here a header with RX & TX ids and message type. But it should have nothing about underling like CRC sum. The underling is coming on lower level. This allows to write an application which does not depends on hardware.

So to start I present a layering diagram

Layers are listed here from left to right:

• Super Application. This is top-top abstract layer as a cover of regular application. Example are the Bridge (interconnects two networks) and Router (interconnects with routing 2+ networks)

• Application. This is a regular application required by any smart device. It allows you have a network connected device. It is so common that it can be used for whatever you want and you can plug so called Features into it to extend its functionality without re-programm the applicariton. See Features bellow.

• Application Feature. This is a plug-and-play extensions to Application. Examples are:

• basic1 allows the device to auto assign ID or to have ID assigned manually by network command
• basic2 contains support for "red" button and indicating LED. LED represents Application status: Normal, NOIDUssigned, WAITCOMMAND. Button allows enter/leave WAITCOMMAND status. This status allows device to receive messages with special WAITCOMMAND broadcast address regardless its own ID. This one is similar as Inclusion status for ZWave devices.
• callback. regular application allows you to read network messages by ::RX method. callback Feature allows you to define per message type callback functions
• broadcast. regular application address device by its direct ID. broadcast Feature allows you to define a list of special broadcasts for device to have an alternative addressing. For example you can put RTC clock inside your network and any device can request clock source without knowing the source ID

• Dispatcher. This Feature organize needed logistic for incoming and outgoing messages. Example:

• P2P peer to peer network. All outgoing messages are send immediately to hardware
• M&S master and slave. Outgoing messages from slaves will be send only on session start request from Master. This for example can be used on half-duplex RS485 networks

• Translator. This one is very close to hardware. It translate abstract standard message from-to hardware layer. For example it can put the message into cover with CRC checksum if underling interface requires this. Other example - text console. All messages are translated to text for manual monitoring and also there is a possibility to send message

• Driver. This one deals with hardware directly or using specified hardware library. Examples:

• NRF24
• RS485
• UART

[axillent]

An Example of simple device, application configuration:

// --- Smartlets
typedef Message::Message<20>		msg_type;
// --- Options

// w3p driver
typedef HandlerPortD<ExtIntMode::FallingRising>	pinchange_drv;
typedef PortD<6, PinMode::Interrupt>		w3p_rx;
typedef PortD<5, PinMode::Inverse | PinMode::OutputPushPull> w3p_tx;
typedef Timer2<mcu, 0, 6, 1000>			w3p_timer;
typedef Driver::W3P<w3p_rx, pinchange_drv, w3p_tx, w3p_timer>	drv;
// translator
typedef Translator::W3P<msg_type>		transl;
// dispatcher
typedef Dispatcher::P2P<msg_type, transl, drv>	disp;
// application
typedef Feature::Basic1<msg_type, disp, ep_id>	basic1;
typedef Feature::CallBackByType<Message::Type::LightScene, msg_scene_cbk>	scene_callback;
typedef Feature::CallBack<msg_type, scene_callback, basic1>	callback;
typedef Feature::Basic2<msg_type, disp, button, led1, callback>	basic2;
typedef Application::Application<msg_type, disp, basic2>	appl;

And simple main, this one supports 2 channel relay functions and 2 NTC sensors

int main()
{
  mcu::Init();

  ep_id::SetDefault(Message::ID(THIS_NET_SEGMENT_ID));
  ep_key::Set();

  appl::Init();

  adc::Start();
  timer::Start();
  mcu::InterruptEnable();

  var.timer	 = 10;
  var.timer1 = 60;
  var.timer2 = 60;

  struct {
	int	temperature1, temperature2;
  } prev;

  while(1) {

    appl::Loop();

	if( !var.timer ) {
	  var.timer = 10;

	  if( !var.timer1 || abs_int(prev.temperature1 - var.temperature1) > 10 ) {
		Message::Value	msg(100, var.temperature1, Message::Value::TemperatureInt16);

		if( appl::Tx(Message::ID(Message::ID::Value), msg.Type(), msg, msg.Size(), false) == Message::State::TxOK ) {
			// message is to be send
			prev.temperature1 = var.temperature1;
			var.timer1 = 60;
		}
	  }

	  if( !var.timer2 || abs_int(prev.temperature2 - var.temperature2) > 10 ) {
		Message::Value	msg(101, var.temperature2, Message::Value::TemperatureInt16 );

		if( appl::Tx(Message::ID(Message::ID::Value), msg.Type(), msg, msg.Size(), false) == Message::State::TxOK ) {
			// message is to be send
			prev.temperature2 = var.temperature2;
			var.timer2 = 60;
		}
	  }

	}

    mcu::InterruptWait();
  }
}

call back:

bool msg_scene_cbk(const uint8_t& msg) {

  const Message::Relay& data = *(const Message::Relay*)&(&msg)[sizeof(Message::Header)];

  switch( data.channel ) {
  case 0:
	relay1.Set(data.relay);
	led2.Run(1, 5);
	return true;
  case 1:
	relay2.Set(data.relay);
	led2.Run(2, 5);
	return true;
  }

  return false;
}

[gohan]

That resembles pretty much computers' network stack, am I wrong?

[axillent]

in details it is very different but by the idea it looks like TCP/IP stack
Also by addressing, I assume 16 bit global address allows you to address any devices on the same network or other network through bridge or router. And there is no any requirement to have a controller. Devices are able to communicated directly. Controller is a device able to manage network config.uration by assigning ID's (very similar to DHCP). But. ID can be assigned manually and it is also possible to run ID-less configuration by using local or global broadcasts. For example wall switch can be configured to send Scene ID to LightScene brodcast address, and any lighting devices can receive this by brodcast and be activated based on scene ID instead of network address.
Just to illustrate idea in more details I have other picture

[axillent]

based on this concept I make a text based console which is actually Super Application, application one is 3 wire physical interface with manchester encoding, application 2 is UART console connected to VT220 terminal. Super Application bridges to networks together. By adding Broadcast with accept to All messages broadcast I allow Bridge to pass all the messages between two networks

// --- bridge application
typedef Application::Bridge<msg_type, w3p_appl, uart_appl>						bridge_appl;


//------------------------------------------------------------------------------
// Variables
//------------------------------------------------------------------------------

//------------------------------------------------------------------------------
// Main
//------------------------------------------------------------------------------
int main()
{

  mcu::Init();

  ep_id::SetDefault(Message::ID(THIS_NET_SEGMENT_ID));
  ep_key::Set();

  bridge_appl::Init();

  timer::Start();
  led2.Set();

  mcu::InterruptEnable();

  uart_transl::SetID(ep_id::Get());
  uart_transl::Print("\n\n\x1b[31m--- Console v1.0 started\x1b[30m");

  while(1) {

	bridge_appl::Loop();
    mcu::InterruptWait();

  }

}

All is plug & play. Etc. it requires NO modification of basic library to create this console. A typical session bellow:

[axillent]

Not that important. But there is no stick to Arduino. It is multiplatform based thing-set. Atmel + STM8 + STM32 are supported. ESP is comming

[gohan]

If you can find a way to make a "router" between different mysensors networks, you could make many people happy

[dbemowsk]

@gohan Wouldn't that be handled in the bridges that he has shown in this image?

Maybe I am thinking of this wrong, but I would think of it in the sense of a MySensors gateway being the bridge or router that would translate the communication from the local computer network to the different types of devices. Instead of connecting that gateway to a controller like Vera, Domoticz or OpenHAB, it would connect to the IP network. It is basically converting the data from one transmission media such as nRF24L01 or RFM69, over to ethernet.

[gohan]

you could use bridges but you would need to have unique IDs throughout all the networks if you want nodes to talk to each other across networks, if I understood right.

[dbemowsk]

@gohan Wouldn't you need that anyway? How else would you address a node?

[axillent]

yes, to address devices across different network segments you need a global ID
but there is a trick... Please see the first picture. It is a layer called Translator.
it is possible to translate non-global ID to global and back and this will allow to create a bridge

[dbemowsk]

@axillent Isn't that in a sense a broad definition of what a bridge does? Translating information between two different types of networks.

[axillent]

@dbemowsk each layer does it's job
bridge just creates a bridge
will it do a translation between depends on translation layer
translation job can be about translation of address or translation of data form or even both
all this depends on the goal but brings wide flexibility

[axillent]

some more details about addressing

global adress is 16 bit number. 12 most significant bits are defining network segment number while 4 less significant bits are providing nodeID

section 0xFE00-0xFEFF is reserved for local broadcasting, 0xFF00-0xFFFF - for global broadcasts. Etc. 256 broadcasts of each type

segments and differenciation between local and global broadcasts are needed for efficient support of bridging and routing.

bridging is a simplified version of routing. bridging is connecting network segments on different physycal inrterfaces while routing can assume to bypass many physical interfaces between (routing throuth).
one of the bridge example is a link between wired interface and MQTT.
one of the routing example is a link between 2 wired interfaces across public Internet

each network segment can have 15 nodes. nodeID = 0 is a special local segment broadcast addressing all devices in segment.

each physical interface can have several segments, etc. there is no limitation to have 15 nodes on physical interface. you just add as many segments to interface as you need to have devices

[alowhum]

Is encryption built in?

[Zeph]

What is W3P? Is this some sort of CANBUS?

[kimot]

I think your addressing concept do not respect hardware - microcontrollers and radio modules we are using.
Some hardware ( RFM69, serial in 9bit mode, etc. ) can filter incoming messages itself and do not disturb controller when
receive different address then node own ID or broadcast.
For example with RFM69 you can store node ID and broadcast address in radiomodule and it will receive only messages with this specific addresses.
But your address schema uses 4 lower bits for node ID ( 0-16 ) and lower 8 bits in all combination for broadcast too.
Your node must receive all 16 bits to distinguish - if it is node addressed message or broadcast call.
So you cannot use addressing features of our hardware.
I think, better is divide 16 bits to 8 bit node ID and 8 bits network ID.
Network filtering and routing then make gateway and distribute messages to connected nodes or other networks.

[axillent]

@zeph yes and no. it is very very simple hardware, similar to LocoNet. allows to transfer a few kbit/sec on single wire (w3p - 3 wire VCC + Signal + GND) using 1 npn transistor + 4 resistors at absolute minimum

[axillent]

@kimot the main idea of all my project is to manage network abstraction layer regardless hardware
only translation + driver layers are hardware related, but them allow to transfer logical layer data throuth any physical hardware
my 16bit addressing could be translated to addresses used by your hardware to keep handle communication based on hardware advantages.

I just launched a network with translation throuth Ethernet/TCP with application server running on orangepi
Working pretty well. On logical level it is always 16bit addressing with broadcasting support.
developing device you should not take care how message will reach another device
this is handled while developing appropriate translations/drivers on bridge side.

bridge - a device connecting 2 or more different physycal layers.

it is also allows to crate a building blocks.
For example I'm running ESP8266 with TCP interface to network on one side and UART&JSON interface on other side
attaching MCU to ESP by UART a can connect any kind of network to my network

[axillent]

just to illustrate how translation is working
you do not need direct logic match between abstract layer and the layer of used interface

here is how translation is working between my network and MQTT broker:

bellow is initialization of 2 different nodes attached to 2 different interfaces, one to TCP/IP network, other to MQTT broker
it is a code from linux application server

//-----------------------------------------------------------------------------------------
// Nodes
//-----------------------------------------------------------------------------------------
// --- inet
typedef Smartlets::Common::RouteTableFile									RouteTable;
typedef STAVRP::Linux::TCPDrv<0>											TCPDrv;
typedef Smartlets::Interface::Translator::INet<Message, TCPDrv, RouteTable>	TranslINet;
typedef Smartlets::Interface::P2P<Message, TranslINet, 16, 16>			InterfaceINet;
typedef Smartlets::Node::NodeBasic<Message, InterfaceINet, ApplInet>		NodeINet;

// --- mqtt
typedef Smartlets::Interface::Translator::Mqtt<Message>				TranslMqtt;
typedef Smartlets::Interface::P2P<Message, TranslMqtt, 16, 16>			InterfaceMqtt;
typedef Smartlets::Node::NodeBasic<Message, InterfaceMqtt, ApplMqtt>		NodeMqtt;

Inside Translator:Mqtt the following is used for example inside TX method (sending message):

	static bool TX(const Message& msg) {

		char key[128];

		if( msg.header.tx.IsBroadcast() )
			sprintf(key, "/%s/bc/o/%d/%d/%d/%d", data.swi_domain.c_str(), msg.header.tx.GetBroadcastID(), msg.header.rx.GetNetSegment(), msg.header.rx.GetNetDevID(), msg.header.type.type);
		else
			sprintf(key, "/%s/m/o/%d/%d/%d/%d/%d", data.swi_domain.c_str(), msg.header.tx.GetNetSegment(), msg.header.tx.GetNetDevID(), msg.header.rx.GetNetSegment(), msg.header.rx.GetNetDevID(), msg.header.type.type);

		std::string	payload = json_ext::FromMessage(msg);
		data.drv.TXPut(STAVRP::Linux::MqttEntry(key, payload));

		return true;
	}```

and this is how it is finally bridged between:

while(1) {

	//---------------------------------------------------------------------------
	// loop inet
	//---------------------------------------------------------------------------
	if( NodeINet::Loop() ) {

		Message msg = NodeINet::RX();

		Logger::root() << log4cpp::Priority::DEBUG << "inet-> tx=" << msg.header.tx.id;

		NodeINet::CommitRX(false);

		// --- routing
		if( NodeINet::TX(msg) ) {
			Logger::root() << log4cpp::Priority::DEBUG << "inet->inet TX OK";
			// delayed queue
		} else {
			Logger::root() << log4cpp::Priority::ERROR << "inet->inet TX failed";
		}

		if( NodeMqtt::TX(msg) ) {
			Logger::root() << log4cpp::Priority::DEBUG << "inet->mqtt TX OK";
		} else {
			Logger::root() << log4cpp::Priority::ERROR << "inet->mqtt TX failed";
		}

	}

	//---------------------------------------------------------------------------
	// loop mqtt
	//---------------------------------------------------------------------------
	if( NodeMqtt::Loop() ) {

		Message msg = NodeMqtt::RX();

		Logger::root() << log4cpp::Priority::DEBUG << "mqtt-> tx=" << msg.header.tx.id;

		NodeMqtt::CommitRX(false);

		if( NodeINet::TX(msg) ) Logger::root() << log4cpp::Priority::DEBUG << "mqtt->inet TX OK";
		else Logger::root() << log4cpp::Priority::ERROR << "mqtt->inet TX failed";

	}

	sleep(1);

}```

[axillent]

made a few steps forward

launched ESP8266 node & bridge and STM32 bridge and TCP/IP router based on linux (orangepi)
bellow is my working example
list of supported platforms:

• STM8S/IAR
• STM32F0/Keil - draft
• ATMEL/Atmel studio - draft
• Arduino ESP8266
• Arduino STM32
• Linux/g++

i;m working on 2 libs. first is to handle platform dependant things. second is platform independant smart devices network
regardless platform and regardless communication hardware the framework allows to transfer messages using RX/TX notation with 16 bit global addressing.
Addressing is organized as 4096 segments where 4094 are 15 node segments each and 2 segments are for broadcadsting

framework allows to connect potentially absolutelly different communication hardware
currently i'm running:

• simple one wire (1 wire signal + 2 wire power) network
• wifi based nodes
• twisted pair connected ethernet devices

bridge between different types of network is organized on TCP/IP application server running currently on linux
etc. potentially allows to connect network segments regardless physicall location, only internet connection is required

why i'm doing this? mostly for fan but also to implement 2 principles:

• application layer to be independant from hardware and transport. message format is universal and is the same regardless platform and communication hardware. Addressing is also universal. Application should not take care about intermediate specific, this is handled by other components
• avoid a central unit. each device can communicate with each device. the only central component is application server. but is is very simple, robust and reliable. failover is supported

My example:

[axillent]

for esp8266 i made an universal bridge application
it can be used for both:

• single node, esp8266 is running a final application. on the photo above my thermostats to measure room temperature, report it and to send ON/OFF commands to heaters

• bridge. can be used to connect other type networks wirelessly to application server. UART is used for this + json translation. currently this is implemented to connect heating controllers located on each floor

main and failover servers can be configured
also each node is configured to handle from one to a few network segments. this allows simple routing from application server back to node
esp8266 is using ArduinoOTA and zero hardcording.
after fresh programming esp creates AP and smartphone can be used to preconfigure a node
any updates are keeping configuration except major updates there stored structure can be affected

application server is also support MQTT translation
etc. it looks like a routing between my network and MQTT server
RX/TX adresing are mapped to MQTT topic while message is translated into JSON:

[axillent]

also from last
byteorder handling is build in
library allows to choose any order
I select bigendian because this is natural to my major MCU - STM8
on little endian platforms (like STM32, ESP8266 and ARM64) data is automatically translated

[axillent]

just published my libraries
anyonw is welsome on review and comments https://github.com/axillent :

• list item stavrp C++ multiplafrom suport library, platforms are at different level of support

• list item smartletsp C++ smart devices network

• list item swilib and integration of above two libraries with arduino framework

[axillent]

Linux Router https://github.com/axillent/swi_appserver