Can local sensors log data?

[BulldogLowell]

so if you don't get an ack on the gw.send() you could try to store the data and a unix timestamp to an SD card (or EEPROM circular buffer). between regular intervals, retransmit the old data with the timestamp.

[Jan Gatzke]

The problem will be to make the controller save the data with the correct timestamps. The controllers I know do not support this.

[BulldogLowell]

@Jan-Gatzke

you wouldn't use the controller to save the time... use the MySensors:

gw.requestTime(receiveTime);

function along with the Time.h library... save the data along with an epoch timestamp to an SD card when you can't get an ack.

re-transmit the data to your server with both the data (e.g. temp) and the timestamp once you get a successful transmission.

easy-peasey lemon squeezie.

[Jan Gatzke]

The data is useless till saved in the controllers database. And when the controller saves it, it uses the current time as time stamp, right?

[BulldogLowell]

@Jan-Gatzke

no, you are posting a data point and a time (i.e. data set) to the database/server... But I'm confused by your use of the word "controller" so perhaps I am not understanding your point.

[Jan Gatzke]

The serial API does not cover the transmission of time stamps, or am I missing something? And still the controller will not care for the time stamp, even if it ever reaches the controller. So how should this help to fix the problem with the broken charts? The controller has to save the data with the correct time stamp to fix this. And all controllers I know get a sensor's value and the it along with the current time.

[BulldogLowell]

@Jan-Gatzke said:

The serial API does not cover the transmission of time stamps,

sure it can

an unsigned long of 32bytes can hold a unix time stamp, or you could break it into four bytes, transmit it and reassemble it server-side.

now()

returns an unsigned long...

[hek]

@Jan-Gatzke, you are correct. The protocol does not cover back-dated data. And it's a quite complicated thing (to do right). A few examples:

1. What should we do with data collected before any time was received from controller?
2. Sleeping. If you sensor sleeps it requires some hacks to step up millis().
3. Handling of timezones? Should data always be stored in UTC?
4. DST?
5. How should the data be handled by the controller? Should it take actions on it or just do logging. I.e. you have a rule to turn on light when detecting motion in a room. The communication gets delayed from the motion sensor. When controller finally replays it a few hours later the light turns on.

We're doing a similar thing (at my dayjob) and it requires quite some logic to handle replaying of data and time in intermittently disconnected devices.

[Jan Gatzke]

Ok, this way you could transmit the time stamp. But this would need a seperate message, right? And the controller would have to convert it back to a unix time stamp. I still see no easy way to get the result the thread opener wants.

@hek

Sorry, missed your answer. I think point 1 could be easy fixes by using an RTC. This could fix point 2, too. I do not think timezones are an issue because mySensors installations across multiple timezones are not the most common thing. :) I am still thinking, that point 5 will be the show stopper. Most controllers like openHAB receive a value for a sensor and save this value together with the time stamp the received the data. Changing the time stamp or back-date a value would need changes to the controller.

[diggs]

I was also thinking an RTC would be needed and perhaps an little sd card for extra data storage at the sensor node. The issue still would seem to be getting the domoticz controller to properly manage the missed data when it does eventually come through. In the case of domoticz, it seems to just use the last read data and plot that. Might also start a thread on the domoticz site and see what thoughts they have on that side as well.

Thanks for all the input

[BulldogLowell]

@diggs

FYI, using the Time.h library does not require RTC. You would merely sync the time regularly to adjust for any clock creep in the microprocessor.

That being said, if your sensor is not in communication for a very long time, or a power outage creates a long period when Time cannot be sync'ed well you will have an issue. So adding an RTC would be helpful but not salient in solving your issue of backdating data.

Since I don't know domoticz, I have to bail out here... I'm watching this thread with interest, however.

[trollkarlen]

Is this solved yet ?

The sensor clock and GW clock dont need to be synced, the node just need to back track how long ago the sample was taken.

So and example "123:23.7" in this 123 seconds ago the temp was 23.7 and default is 0.
I need this also to save power and use bulk sending, so sample one hour and then send that hour of data in a chunk.

/T

[tbowmo]

@trollkarlen

If you sleep the arduino you'll loose track of time, as it will switch off timers etc.

So for a lowpower node, there is no timekeeping..

[Alexander Voronin]

@tbowmo What about watchdog? I used it as power-on source in my custom project (not mysensors, raw GCC and AVR) and got about 10sec power-down cycles at 3.3V (should be 8sec on 5V). Yes, it is voltage dependent, but may be used as "interval counter" if you have some logic to recalculate times from (approx. fixed) intervals.

AFAIR, ESP8266 also has DeepSleep mode, but haven't IRQ handler for safe handling interrupt (only hardware reset on DeepSleep completion)

[tbowmo]

@Alexander-Voronin

Yes, you can use the watchdog, and wake up periodically, but it can never be used for time critical things. For example, my sensebenders, I have them sleeping 1 minute. But it varies between 50 and 70 seconds.. So can't be used to keep track of real time..