Gas Meter Reading Using a Magnetometer

[dpcr]

@gohan I had to use a magnetometer because my meter did not have a magnet in it. I tried a reed switch but it never actuated and my meter is outside. So I was stuck using a magnetometer which I mounted in a water proof plastic box. The issue I had was getting the sine wave to create a pulse that was usable for MySensors. I did look at using an inverse sine, or as someone else mentioned, use an exponential moving average of the data but they both failed. When the gas stops flowing, a pilot light as an example as seen in the attached picture, there is little to no gas flowing so the angle doesn't change much. It looks like no gas is flowing when it actually is. I decided to go with dividing the wave or meter movement into separate divisions to create a more accurate way of measuring the gas flow in all flow situations.

I haven't had a chance to test the accuracy of my math against the gas meter but I will at some point. The issue I'm currently having is that the bellows inside of the meter that the magnetometer is reading do not move at the same rate at all times. This causes the FLOW output to change when I know the gas is flowing at a steady rate. The VOLUME doesn't seem to be effected too much. So what I did was to take a moving average of the last three FLOW readings. This helps a little.

The hardware and ideas were done by me but the coding was done by my 24 year old son. I always enjoy a challenge. Please feel free to comment.

[gohan]

Ok, now it makes sense. If you plot the data with the pilot using the gas, don't you still see a sine wave but with, of course, a longer period?

[Yveaux]

@gohan I assume you do. By interpreting the sine and thereby increasing the resolution you might be able to see there's gas flowing, even if the rate is very low.

[gohan]

good luck with the math then, I am calling myself out! :sweat_smile:
Back in high school I was quite good at math, but now... :D

[dpcr]

@gohan Yes you do, but what about the times when there is gas flowing. If I remember correctly there was about 20 or 25 minutes for full sine wave when only the pilot was running. That SEND_FREQUENCY (1500000) was too large when the gas was actually flowing to get an accurate measurement. I just wanted to get an accurate reading of the gas flow at all flow rates. Am I missing something?

Here are some screenshots from Domoticz for the past week:
)

[ThomasDr]

@gohan Hello,
My thought is simpler.
The original sensor is too expensive.
With a Reedcontact I must find the right position.
The magnetometer I could simply attach somewhere in the proximity.

regards
ThomasD

[gohan]

@ThomasDr
I see your point, but you pay the simplicity with a more complex coding ;-)

[dpcr]

@ThomasDr Sorry not to get back to you when I saw your post, your project looks interesting. Does the HMC5983 magnetometer output something similar to the HMC5883L? I just used the HMC5883L magnetometer because I found it useful, I didn't run across it during my research.

[dpcr]

Ran into a problem. The magnetism value changes when the outside air temperature changes which causes many incorrect readings. So my initial sketch that finds the TOP and BOTTOM of the wave is having problems so I'm in the process of trying to find another way to get some useful data out of the magnetometer. Any ideas?

[gohan]

If you find a correlation between temperature and change of magnetic field, maybe you could add a thermistore and use it to compute a correction coefficient.
Or maybe even a second magnetometer with a fixed magnet to use as reference 🤔

[dpcr]

@gohan Thanks, that was what I thought initially but the changes in the magnetic forces were all over the place but did seem to coincide with the outdoor temperature. They were relatively small in comparison to the total but enough the cause problems. It was almost like the meter itself was doing this?? Since we rely so heavily on the Top and Bottom magnetic readings to not only find the top and the bottom of the "wave" but to also calculate the intervals, we made some updates to the sketch. It now calculates a new top and bottom on its own. It has been running for 12 hours and so far so good. For now we are writing the top and bottom values to EEPROM, but once I get some memory we'll store the values there.

I did have a question that I can't seem to find a answer to - what length I2C bus can be used with out pull up resistors? The current bus being used is about 2 meters without pull up resistors and there doesn't seem to be too much noise compared to when it was only several inches . I would like to make it a little longer if possible.

[gohan]

Have you noticed if the magnetic forces are really temperature affected or if it is the actual rotating counter that could have other magnets (like one each group of colored digits???) ?
About the I2C bus I think I read it could reach 5-6 meters (of course depending on external factors)

[dpcr]

@gohan We have had large outside temperature changes around here in the last month. Two weeks ago it was 20 DegF now it is 53 DegF. The bottom magnetic reading when it was 20 degrees outside was around 380 and things seemed to work OK. Now the bottom is 495 and problems started when it warmed up. The top doesn't seem to change as much. Not sure if this has anything to do with it, but our meter (Rockwell or Sensus does have what they call a "temperature compensation element". I assume it adjusts itself to make the meter more accurate due to temperature changes. Thanks for the info on the I2C bus.

[markcame]

Dear All,
I've tried to make a gas meter sensor using the magnetometer because I've started only few weeks ago approaching mysensor setup so be patient with me...

I've now two problems , the first is regarding the accuracy of the measure, it seems that after I've count the number of cycles Vs the gas measured of the meter and found the value of vpp (I've used the scketch posted by dpcr) afther a while the values measured drift from the real
The second problem is that I'm not able to measure the flow... it gave me values very high (1249225088.00 l/min)
Someone have ideas what's happen ?

Thanks a lot

[dpcr]

@markcame Updated sketch:

It sounds like your pulse count is not correct, this could probably be due to your top and bottom values being incorrect.

One hardware change was to add some memory because the EEPROM can only handle so many cycles. I used Adafruit I2C FRAM but you can use whatever you like. This sketch uses Adafruit I2C FRAM, what I had.

There have been many changes since the last sketch. We found that the magnetism values were changing due to changing outside air temperatures (the top and bottom values are very important). When the arduino is turned on for the very first time (top and bottom values are not in memory) it will run for a period of 2 minutes where it will find the top and bottom values, so the gas has to be flowing during that time (turn the stove on). It then stores those values in memory. Going forward we calculate the top and bottom values on the fly to keep them updated as the magnetism values change with the outside air temperature. If not it would go through a 'auto program' every time there was a reset. The gas has to be flowing during the 'auto program' and if its summer and no gas is flowing your top and bottom values will be incorrect. During testing I powered the arduino with the serial connector to get the serial data. It is currently powered with a separate 3.3 v wall wart.

We removed several bugs and updated the way it calculates pulses and put a smoothing on the FLOW output.The smoothing help keep the FLOW values going down to 0 but doesn't stop it at all times.

I uploaded this to the Arduino to first verify the correct vector. I could have use a total vector calculation (square root of x2+y2+z2) but this would have just amplified some of the noise. Using a chosen vector worked fine but took a little more work. After finding the correct vector (sorry - first build and mount the magnetometer on the meter), download the sketch below.

/*This version is different in several ways.
 * first is is using some memory (Adafruit I2C FRAM) memory to store the top and bottom. This is used to keep the top and bottom
 * values after a restart. I guess we could have put them on the controller but can't remember why we didn't.
 * second it will only calculate the top and bottom if the values are not in memory.
 * third it auto calculates the top and bottom values as it is running. 
 * the only settings that should be changed are vpp, SEND_FREQUENCY, and metric. However I am currently using
 * Domoticz as my controller and it doesn't seem to accept English units well so for not I'm using Metric.
 */



#define MY_DEBUG
//#define MY_DEBUG_VERBOSE
#define MY_RADIO_NRF24

#include <MySensors.h>                  
#include <Wire.h>                       //I2C communications library
#include <Adafruit_FRAM_I2C.h>          //Adafruit FRAM memory library

#define CHILD_ID 1                      //ID of the sensor child
#define SLEEP_MODE false                //prevent sensor from sleeping
#define address 0x1E                    //0011110b, I2C 7bit address of HMC5883 magnetometer

int top = 0;                            //highest magnetic field registered from meter (Ga)Initialize low if using autoDetectMaxMin
int bottom = 0;                         //lowest magnetic field registered from meter (Ga) Initialize high if using autoDetectMaxMin
int tol = 50;
unsigned long SEND_FREQUENCY = 30000;   // Minimum time between send (in milliseconds). We don't want to spam the gateway.

bool metric = true;                     //sets units to Metric or English TODO: move to void setup()
bool pcReceived = false;                //whether or not the gw has sent us a pulse count
bool autoDetect = false;                //true if the program is auto detecting Top and Bottom
bool rising = true;                     //whether or not a pulse has been triggered
bool safe = false;                      //whether or not it is safe to switch directions
unsigned long pulsecount = 0;           //total number of pulses measured ever
unsigned long oldPulseCount = 0;        //old total
double vpp = metric ? 0.160891193181 : 0.00568124219857;//Volume of gas per pulse
unsigned long lastSend = 0;             //time since last transmission - msec
double volume = 0;                      //Cumulative amount of gas measured
const int len = 3;                      //number of flow rate measurements to save
double flow [len];                      //array of previous gas flow rate measurements
double avgFlow = 0;                     //average of all elements in flow array
double oldAvgFlow = 0;                  //previous average flow
int y = 0;                              //magnetic field reading
int oldy = 0;                           //previous magnetic field reading
int newTop = -9000;                         //potential new Top
int newBottom = 9000;                  //potential new Bottom
int totDividers = 10;                   //Number of dividers
int increment = 0;                      //space b/w dividers
int counter = 0;                        //used to count pulses over periods longer than SEND_FREQUENCY
int topAddr = 0;                        //address of TOP in FRAM
int bottomAddr = topAddr + 2;           //address of BOTTOM in FRAM

MyMessage flowMsg(CHILD_ID,V_FLOW);
MyMessage volumeMsg(CHILD_ID,V_VOLUME);
MyMessage lastCounterMsg(CHILD_ID,V_VAR1);

Adafruit_FRAM_I2C fram = Adafruit_FRAM_I2C();

void setup(){
  //Initialize Serial, I2C, and FRAM communications
  Serial.begin(9600);
  Wire.begin();
  fram.begin();

  // Fetch last known pulse count value from gw
  request(CHILD_ID, V_VAR1);
  
  //Put the HMC5883 IC into the correct operating mode
  Wire.beginTransmission(address); //open communication with HMC5883
  Wire.write(0x02); //select mode register
  Wire.write(0x00); //continuous measurement mode
  Wire.endTransmission();

  //get Top and Bottom from FRAM. addresses are hard-coded seeing as there are only 2
  newTop = readInt(topAddr);
  newBottom = readInt(bottomAddr);
  updateBounds();
  
  //WARNING: MAKE SURE GAS IS RUNNING ON FIRST RUNNING OF THIS PROGRAM!!!
  if(top == 0 && bottom == 0){    
    autoDetect = true;
    newTop = -9000;
    newBottom = 9000;
    
    //determine max and min magnetic field strength over a few minutes
    Serial.println("FRAM has been cleared. Auto-detecting max and min magnetic field reading.");
    Serial.println("WARNING: MAKE SURE GAS IS RUNNING!!");
    lastSend = millis();
    
    while(millis() - lastSend < 120000){
      y = readMag();
      detectMaxMin();

      //display details
      Serial.print("y: ");
      Serial.print(y);
      Serial.print("  Top: ");
      Serial.print(newTop);
      Serial.print("  Bottom: ");
      Serial.print(newBottom);
      unsigned long remainingTime = 120000 + lastSend - millis();
      Serial.print("  Time remaining: ");
      Serial.print(remainingTime / 60000);
      Serial.print(":");
      remainingTime = (remainingTime % 60000) / 1000;
      if(remainingTime >= 10){
        Serial.println(remainingTime);
      }
      else{
        Serial.print("0");
        Serial.println(remainingTime);
      }
    }

    updateBounds();
    autoDetect = false;
  }

  y = readMag();
  oldy = readMag();
  while(abs(y - oldy) < increment / 2){ //wait until difference b/w y and oldy is greater than half an increment
    y = readMag();
  }
  rising = (y > oldy);
  Serial.println(rising ? "Magnetic field is rising" : "Magnetic field is falling");
}

void presentation()
{
    // Send the sketch version information to the gateway and Controller
    sendSketchInfo("Gas Meter", "0.6 (2/24/17)");

    // Register this device as Gas sensor
    present(CHILD_ID, S_GAS);
}

void loop(){
  if (!pcReceived) {
    //Last Pulsecount not yet received from controller, request it again
    request(CHILD_ID, V_VAR1);
    return;
  }
  
  //detecting magnetic pulses - variable boundary method
  while(millis() - lastSend < SEND_FREQUENCY){
    //check if the signal has significantly increased/decreased
    if(abs(oldy - y) > increment){
      pulsecount ++;
      //increment or decrement oldy by one increment based on direction
      oldy += rising ? increment : -1 * increment;     
      safe = false;             //reset safe now that oldy has updated     
    }
    //check if the signal has recently switched directions
    else if(safe){                  //first make sure y has moved a significant distance from oldy
      if((rising && y <= oldy) || (!rising && y >= oldy)){
        pulsecount ++;              //add one extra pulse
        rising = !rising;           //update direction
        safe = false;
      }
    }
    
    //take another reading
    y = readMag();
    //check if y has moved a significant distance from oldy
    if(abs(y - oldy) > tol / 2){
      safe = true;
    }
    
    //update newTop and newBottom
    detectMaxMin();                
  }
  
  //shift all flow array elements to the right by 1, ignore last element
  for(int idx = len - 1; idx > 0; idx--){
    flow[idx] = flow[idx - 1];
  }
  //calculate newest flow reading and store it as first element in flow array
  flow[0] = (double)(pulsecount - oldPulseCount) * (double)vpp * 60000.0 / (double)SEND_FREQUENCY;
  //display flow array state
  Serial.print("Flow Array State: [");
  for(int idx = 0; idx < len - 1; idx++){
    Serial.print(flow[idx]);
    Serial.print("|");
  }
  Serial.print(flow[len - 1]);
  Serial.println("]");
  //calculate average flow
  avgFlow = 0;                                //reset avgFlow
  for(int idx = 0; idx < len; idx++){         //calculate weighted sum of all elements in flow array
    avgFlow += (flow[idx] * (len - idx));
  }
  avgFlow /= (len * (len + 1) / 2);           //divide by triangle number of elements to get linear weighted average
  Serial.print("Average flow: ");             //display average flow
  Serial.println(avgFlow);
  //send flow message if avgFlow has changed
  if(avgFlow != oldAvgFlow){
    oldAvgFlow = avgFlow;
    send(flowMsg.set(avgFlow, 2));
  }

  //send updated cumulative pulse count and volume data, if necessary
  if(pulsecount != oldPulseCount){    
    //calculate volume
    volume = (double)pulsecount * (double)vpp / (metric ? 1000.0 : 1);

    //send pulse count and volume data to gw
    send(lastCounterMsg.set(pulsecount));
    send(volumeMsg.set(volume, 3));

    counter += (pulsecount - oldPulseCount);      //update counter
    if(counter >= ((totDividers + 1) * 2)){
      updateBounds();                 //update bounds if at least 1 cycle has been read
      counter = 0;                    //reset counter
    }

    oldPulseCount = pulsecount;              //update old total
  }

  lastSend = millis();
  
}

void receive(const MyMessage &message)
{
  if (message.type==V_VAR1) {
    unsigned long gwPulseCount=message.getULong();
    pulsecount = gwPulseCount;
    oldPulseCount = pulsecount;
    Serial.print("Received last pulse count from gw:");
    Serial.println(pulsecount);
    pcReceived = true;
    lastSend = millis();
    //set magnetic field starting point
    oldy = readMag();
    y = readMag();
  }
}

void updateBounds(){
  if(((top + tol) != newTop) && ((bottom - tol) != newBottom)){   //check if anything has actually changed
    //lock in Top and Bottom
    top = newTop - tol;
    bottom = newBottom + tol;
    
    //recalculate increment to match new top and bottom
    increment = (top - bottom) / totDividers;
  
    //reset newTop and newBottom
    newTop = -9000;
    newBottom = 9000;
  
    //store updated Top and Bottom in FRAM
    writeInt(topAddr,top);
    writeInt(bottomAddr,bottom);

    //reset newTop and newBottom
    newTop = -9000;
    newBottom = 9000;
  
    //display new bounds
    Serial.println("NEW BOUNDARIES SET:");
    Serial.print("Top = ");
    Serial.println(top);
    Serial.print("Bottom = ");
    Serial.println(bottom);
    Serial.print("Increment = ");
    Serial.println(increment);
  }
}

void detectMaxMin(){
  if(y > newTop){
        newTop = y;                    //update newTop if new max has been detected
      }
  else if(y < newBottom){
    newBottom = y;                     //update newBottom if new min has been detected
  }
}

void writeInt(int addr, int val){       //write an int value to memory
  byte b = highByte(val);
  fram.write8(addr,b);
  b = lowByte(val);
  fram.write8(addr + 1,b);
}

int readInt(int addr){                  //read an int value from memory
  int result = 0;
  result += (int)fram.read8(addr);
  result = result << 8;
  result += (int)fram.read8(addr + 1);
  return result;
}

int readMag(){
  int x = 0, z = 0;
  
  //Tell the HMC5883 where to begin reading data
  Wire.beginTransmission(address);
  Wire.write(0x03); //select register 3, X MSB register - was called Wire.send but the compiler had an error and said to rename to to Wire.write
  Wire.endTransmission();

  //Read data from each axis, 2 registers per axis
  Wire.requestFrom(address, 6);
  if(6<=Wire.available()){
    x = Wire.read()<<8; //X msb
    x |= Wire.read(); //X lsb
    z = Wire.read()<<8; //Z msb
    z |= Wire.read(); //Z lsb
    y = Wire.read()<<8; //Y msb
    y |= Wire.read(); //Y lsb
  }

  if(!autoDetect){
    //show real-time magnetic field, pulse count, and pulse count total
    Serial.print("y: ");
    Serial.print(y);
    Serial.print(rising ? "  Rising, " : "  Falling, ");
    Serial.print("next pulse at: ");
    Serial.print(rising ? oldy + increment : oldy - increment);
    Serial.print("  Current Number of Pulses: ");
    Serial.print(pulsecount - oldPulseCount);
    Serial.print("  Last Total Pulse Count Sent to GW: ");
    Serial.println(oldPulseCount);
  }
  
  return y;
 
}

I still have not verified the vpp with the meter but I know it's close. When the water heater is on it Domoticz says I'm flowing enough gas for 30,500 BTU/H. The water heater has an input rating of 28,000 BTU/H.

Hope this helps, please let me know if you have any questions. I will try to add some pics.

[markcame]

@dpcr I've tried the sketch but seems something is wrong, I've modified it for using my controller to store the top and bottom values and this function very well I'm able to check on the serial that the values are correctly retrieved and stored, but during the running after some cycles start counting pulse at every reading of magnetic field

[markcame]

I've try to analyze the program on the old sketch everything are good except the flow rate, I've found an error when you perform the shift of the element on an index this is the reason because the number reported wrong to the gateway.

I want to use the new sketch but i have two question, the first is that i didn't understand the pulse count

while(millis() - lastSend < SEND_FREQUENCY){
   //check if the signal has significantly increased/decreased
   if(abs(oldy - y) > increment){
     pulsecount ++;
     //increment or decrement oldy by oInsert Code Herene increment based on direction
     oldy += rising ? increment : -1 * increment;     
     safe = false;             //reset safe now that oldy has updated     
   }
   //check if the signal has recently switched directions
   else if(safe){                  //first make sure y has moved a significant distance from oldy
     if((rising && y <= oldy) || (!rising && y >= oldy)){
       pulsecount ++;              //add one extra pulse
       rising = !rising;           //update direction
       safe = false;
     }
   }
   
   //take another reading
   y = readMag();Insert Code Here
   //check if y has moved a significant distance from oldy
   if(abs(y - oldy) > tol / 2){Insert Code Here
     safe = true;Insert Code Here
   }
   
   //update newTop and newBottom
   detectMaxMin();                
 }

and the second is about the initialization of variables retrieved from the FRAM , i i have understand to start the "calibration" you need to put 50 and -50 on the memory otherwise the right condition was not triggered right ?
i didn't use the FRAM but I've coded to retrieve form the controller

 //get Top and Bottom from controller***********************
 Serial.println("Ask for top value @ controller");
 //Get Last top
 request(CHILD_ID, V_VAR2);
    
 Serial.println("Ask for bottom value @ controller");
 //Get Last bottom
 request(CHILD_ID, V_VAR3);
 
 wait(2000, 2, V_VAR1);
 wait(2000, 2, V_VAR2);
 wait(2000, 2, V_VAR3);
 Serial.println("Variables Succesfully Retrived from GW");
 //************************************************************  

[dpcr]

@markcame Not sure if I understand you. It still sounds like your not finding your top and bottom values. It works and still is working on my set up. The reason the top and bottom values are so important is the sketch takes those values and divides the difference into sections (totDividers). Those calculated values must be in the range of what the magnetometer is reading. If they don't the pulse counts will be all over the place or none at all. What are your top and bottom values and what are you top and bottom values received from the magnetometer?

[markcame]

@dpcr They find corect top and bottom, and start to pulse but after first recalculation of bounds they statrs to count a pulse for every readings.

[dpcr]

@markcame what is the "next pulse at" reading? This happened to me as well when I was testing it but we changed a few this and it worked OK.