Gas Meter Reading Using a Magnetometer

[dpcr]

Yes, a major change in magnetic field readings while the sensor is powered down could result in a large number of pulses being triggered if Top and Bottom are drastically different than what the sensor has stored. It is best to reset Top and Bottom to 0 in that case, forcing Auto-Detect to run and get accurate values for Top and Bottom.

In future versions, I plan to add a feature that lets users choose to reset Top and Bottom from the Serial Monitor. If it's been a while since the sensor was last running or if operating conditions have changed (the sensor was relocated or something similar), users can use that feature to ensure that the program begins with proper values for Top and Bottom. I also plan on publishing two versions of the sketch: one that stores Top, Bottom, and Pulse Count in Fram, and one that stores those values on the controller.

As far as counting pulses goes, the old version and the new version aren't too different. They both divide the space between Top and Bottom into a number of divisions, and trigger pulses once a division is passed. The math they use to do it is slightly different. The old version calculates what number the magnetic field reading needs to rise/fall to and compares that to the most recent reading. The new version checks if the distance between the magnetic field reading that last triggered a pulse and the most recent reading is equal to or greater than the distance between divisions. The old version had problems when the magnetic field was switching from rising to falling (probably because I didn't continuously update Top and Bottom), but the new version feels more elegant. There's probably not much of a difference at this point, though.

I did have a problem where it seemed like the sensor wasn't printing anything to the serial monitor, but that was due to bad program flow that occasionally made it impossible to update lastSend. I've never had a problem that prevented the sensor from sending data to the Gateway.

[dpcr]

@dpcr New Sketch: This one is version 1.0 and works well with the hardware I'm using. It has an interactive start up menu in the serial monitor when starting which allows one to choose several options and var1 (variable used to store the pulse count on the controller) was removed and the pulse count is now stored in FRAM. It's been running for several months with no problem.

/*
 * Created by Sean Creel <creels15@gmail.com>
 * 
 * Based on these projects by Henrik Ekblad and Korneliusz Jarzębski:
 * https://www.mysensors.org/build/pulse_water
 * https://github.com/jarzebski/Arduino-HMC5883L
 * 
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * version 2 as published by the Free Software Foundation.
 * 
 * DESCRIPTION:
 * Uses an I2C triple-axis magnetometer to measure a home's gas usage by
 * detecting small fluxuations in the local magnetic field caused by the
 * meter's bellows pumping in and out. Requires users to determine how 
 * many cycles their meter completes per unit volume of gas flow.
 */



#define MY_DEBUG                        //enables debugging of MySensors messages
#define MY_RADIO_NRF24                  //lets MySensors know what radio you're using

#include <MySensors.h>         
#include <Wire.h>                       //I2C communications library
#include <Adafruit_FRAM_I2C.h>          //Adafruit FRAM memory library

#define CHILD_ID 1                      //ID of the sensor child
#define SLEEP_MODE false                //prevent sensor from sleeping
#define address 0x1E                    //0011110b, I2C 7bit address of HMC5883 magnetometer
#define topAddr 4                       //address of Top in FRAM
#define bottomAddr 6                    //address of Bottom in FRAM
#define pulseAddr 0                     //address of pulseCount in FRAM

//USER-DEFINED CONSTANTS
#define METRIC true                     //specifies the units of measurement
#define AXIS "Y"                        //specifies which axes of the magnetometer to read from (multiple axes permitted, e.g. "ZX" or "YXZ")
#define CYCLES_PER_CUFT 8               //cycles of the meter's bellows per cubic foot of gas (varies by  meter model)
#define NUM_DIVISIONS 10                //desired number of subdivisions per cycle. higher => more accuracy, lower => more tolerant of noise
#define ERR 0.00221513400975639000      //used to fix discrepancies between theoretical and actual volume per pulse (vpp)
#define LEN 3                           //number of flow rate measurements to save for moving average. set to 1 if you want to send raw flow data instead
#define SEND_FREQUENCY 30000            //Minimum time between messages to the GW (msec)
#define TOL 50                          //margin of error to account for noise coming from the sensor

bool autoDetect = false;                //true if the program is auto detecting Top and Bottom
bool rising = true;                     //whether or not a pulse has been triggered
bool safe = false;                      //whether or not it is safe to switch directions
unsigned long pulseCount = 0;           //total number of pulses measured ever
unsigned long oldPulseCount = 0;        //old total
double vpp = ERR + (METRIC ? 28.3168 : 1.0) / (CYCLES_PER_CUFT * 2 * (NUM_DIVISIONS + 1));//Volume of gas per pulse ft^3/L
unsigned long lastSend = 0;             //time since last transmission - msec
double flow [LEN];                      //array of previous gas flow rate measurements
double avgFlow = 0;                     //average of all elements in flow array
double oldAvgFlow = 0;                  //previous average flow
int b = 0;                              //magnetic field reading
int oldB = 0;                           //previous magnetic field reading
int top = 0;                            //highest magnetic field registered from meter (Ga)Initialize low if using autoDetectMaxMin
int bottom = 0;                         //lowest magnetic field registered from meter (Ga) Initialize high if using autoDetectMaxMin
int newTop = -9000;                     //potential new Top
int newBottom = 9000;                   //potential new Bottom
int increment = 0;                      //space b/w dividers
int counter = 0;                        //used to count pulses over periods longer than SEND_FREQUENCY

MyMessage flowMsg(CHILD_ID,V_FLOW);
MyMessage volumeMsg(CHILD_ID,V_VOLUME);

Adafruit_FRAM_I2C fram = Adafruit_FRAM_I2C();

void setup(){
  //Initialize Serial, I2C, and FRAM communications
  Serial.begin(9600);
  Wire.begin();
  fram.begin();
  
  //Put the HMC5883 IC into the correct operating mode
  Wire.beginTransmission(address); //open communication with HMC5883
  Wire.write(0x02); //select mode register
  Wire.write(0x00); //continuous measurement mode
  Wire.endTransmission();

  //Prompt user for permission to clear all or part of FRAM
  Serial.println("Do you wish to clear all or part of long-term memory?");
  Serial.println("Please enter a number 0-3:");
  Serial.println("0: (default) Clear nothing");
  Serial.println("1: Clear Top and Bottom");
  Serial.println("2: Clear Pulse Count");
  Serial.println("3: Clear Top, Bottom, and Pulse Count");
  int choice = 0;
  lastSend = millis();
  while(Serial.available() == 0 && millis() - lastSend < 30000); //wait 30s max for user input
  choice += Serial.parseInt();
  if(choice == 1){
    clearFram(4,8);
    Serial.println("Top and Bottom values reset");
  }
  else if(choice == 2){
    clearFram(0,4);
    Serial.println("Pulse Count reset");
  }
  else if(choice == 3){
    clearFram(0,8);
    Serial.println("All stored values reset");
  }

  //get pulseCount from FRAM
  pulseCount = readUL(pulseAddr);
  oldPulseCount = pulseCount;
  //get Top and Bottom from FRAM
  newTop = readInt(topAddr);
  newBottom = readInt(bottomAddr);
  updateBounds();
  
  //WARNING: MAKE SURE GAS IS RUNNING ON FIRST RUNNING OF THIS PROGRAM!!!
  if(top == 0 && bottom == 0){    
    autoDetect = true;
    newTop = -9000;
    newBottom = 9000;
    
    //determine max and min magnetic field strength over a few minutes
    Serial.println("FRAM has been cleared. Auto-detecting max and min magnetic field reading.");
    Serial.println("WARNING: MAKE SURE GAS IS RUNNING!!");
    lastSend = millis();
    
    while(millis() - lastSend < 120000){
      readMag();
      detectMaxMin();

      //display details
      Serial.print("Magnetic Field: ");
      Serial.print(b);
      Serial.print("  Top: ");
      Serial.print(newTop);
      Serial.print("  Bottom: ");
      Serial.print(newBottom);
      unsigned long remainingTime = 120000 + lastSend - millis();
      Serial.print("  Time remaining: ");
      Serial.print(remainingTime / 60000);
      Serial.print(":");
      remainingTime = (remainingTime % 60000) / 1000;
      if(remainingTime >= 10){
        Serial.println(remainingTime);
      }
      else{
        Serial.print("0");
        Serial.println(remainingTime);
      }
    }

    updateBounds();
    autoDetect = false;
  }

  readMag();
  oldB = b;
  while(abs(b - oldB) < increment / 2){ //wait until difference b/w b and oldB is greater than half an increment
    readMag();
  }
  rising = (b > oldB);
  Serial.println(rising ? "Magnetic field is rising" : "Magnetic field is falling");
  Serial.print("Volume Per Pulse = ");
  Serial.print(vpp,16);
  Serial.println(METRIC ? " L/pulse" : "cuft/pulse");
  lastSend = millis();
}

void presentation()
{
    // Send the sketch version information to the gateway and Controller
    sendSketchInfo("Gas Meter", "1.0 (4/4/17)");

    // Register this device as Gas sensor
    present(CHILD_ID, S_GAS);
}

void loop(){  
  //detecting magnetic pulses - variable boundary method
  while(millis() - lastSend < SEND_FREQUENCY){
    //check if the signal has significantly increased/decreased
    if(abs(oldB - b) > increment){
      pulseCount ++;
      //increment or decrement oldB by one increment based on direction
      oldB += rising ? increment : -1 * increment;     
      safe = false;             //reset safe now that oldB has updated     
    }
    //check if the signal has recently switched directions
    else if(safe){                  //first make sure b has moved a significant distance from oldB
      if((rising && b <= oldB) || (!rising && b >= oldB)){
        pulseCount ++;              //add one extra pulse
        rising = !rising;           //update direction
        safe = false;
      }
    }
    
    //take another reading
    readMag();
    //check if b has moved a significant distance from oldB
    if(abs(b - oldB) > TOL / 2){
      safe = true;
    }
    
    //update newTop and newBottom
    detectMaxMin();                
  }
  
  //shift all flow array elements to the right by 1, ignore last element
  for(int idx = LEN - 1; idx > 0; idx--){
    flow[idx] = flow[idx - 1];
  }
  //calculate newest flow reading and store it as first element in flow array
  flow[0] = (double)(pulseCount - oldPulseCount) * vpp * 60000.0 / (double)SEND_FREQUENCY;
  //display flow array state
  Serial.print("Flow Array State: [");
  for(int idx = 0; idx < LEN - 1; idx++){
    Serial.print(flow[idx]);
    Serial.print("|");
  }
  Serial.print(flow[LEN - 1]);
  Serial.println("]");
  //calculate average flow
  avgFlow = 0;                                //reset avgFlow
  for(int idx = 0; idx < LEN; idx++){         //calculate weighted sum of all elements in flow array
    avgFlow += (flow[idx] * (LEN - idx));
  }
  avgFlow /= (LEN * (LEN + 1) / 2);           //divide by triangle number of elements to get linear weighted average
  Serial.print("Average flow: ");             //display average flow
  Serial.println(avgFlow);
  //send flow message if avgFlow has changed
  if(avgFlow != oldAvgFlow){
    oldAvgFlow = avgFlow;
    send(flowMsg.set(avgFlow, 2));
  }

  //send updated volume data if necessary
  if(pulseCount != oldPulseCount){

    //send volume data to gw
    send(volumeMsg.set(((double)pulseCount * vpp / (METRIC ? 1000.0 : 1)), 3));

    //store updated pulse count total in FRAM
    writeUL(pulseAddr,pulseCount);

    counter += (pulseCount - oldPulseCount);      //update counter
    if(counter >= ((NUM_DIVISIONS + 1) * 2)){
      updateBounds();                 //update bounds if at least 1 cycle has been read
      counter = 0;                    //reset counter
    }

    oldPulseCount = pulseCount;              //update old pulse count total
  }

  lastSend = millis();
  
}

void updateBounds(){
  if((top != newTop) && (bottom != newBottom)){   //check if anything has actually changed
    //lock in Top and Bottom
    top = newTop;
    bottom = newBottom;
    
    //recalculate increment to match new top and bottom
    increment = (top - bottom - (2 * TOL)) / NUM_DIVISIONS;
  
    //store updated Top and Bottom in FRAM
    writeInt(topAddr,top);
    writeInt(bottomAddr,bottom);

    //reset newTop and newBottom
    newTop = -9000;
    newBottom = 9000;
  
    //display new bounds
    Serial.println("NEW BOUNDARIES SET:");
    Serial.print("Top = ");
    Serial.println(top);
    Serial.print("Bottom = ");
    Serial.println(bottom);
    Serial.print("Increment = ");
    Serial.println(increment);
  }
}

void detectMaxMin(){
  if(b > newTop){
        newTop = b;                    //update newTop if new max has been detected
      }
  else if(b < newBottom){
    newBottom = b;                     //update newBottom if new min has been detected
  }
}

void writeInt(int addr, int val){       //write an int value to FRAM
  byte b = highByte(val);
  fram.write8(addr,b);
  b = lowByte(val);
  fram.write8(addr + 1,b);
}

void writeUL(int addr, unsigned long val){  //write an unsigned long falue to FRAM
  byte b = B00000000;
  for(int idx = 0; idx < sizeof(unsigned long); idx++){
    b = val >> 8 * (sizeof(unsigned long) - 1 - idx);
    fram.write8(addr + idx,b);
  }
}

unsigned long readUL(int addr){         //read an unsigned long value from FRAM
  unsigned long result = 0;
  for(int idx = 0; idx < sizeof(unsigned long); idx++){
    result = result << 8;
    result += (unsigned long)fram.read8(addr + idx);
  }
  return result;
}

int readInt(int addr){                  //read an int value from FRAM
  int result = 0;
  result += (int)fram.read8(addr);
  result = result << 8;
  result += (int)fram.read8(addr + 1);
  return result;
}

void clearFram(int first, int last){
  for(int addr = first; addr < last; addr++){
    fram.write8(addr,byte(0));
  }
}
void readMag(){
  static int x = 0, y = 0, z = 0;
  
  //Tell the HMC5883 where to begin reading data
  Wire.beginTransmission(address);
  Wire.write(0x03); //select register 3, X MSB register - was called Wire.send but the compiler had an error and said to rename to to Wire.write
  Wire.endTransmission();

  //Read data from each axis, 2 registers per axis
  Wire.requestFrom(address, 6);
  if(6<=Wire.available()){
    x = Wire.read()<<8; //X msb
    x |= Wire.read(); //X lsb
    z = Wire.read()<<8; //Z msb
    z |= Wire.read(); //Z lsb
    y = Wire.read()<<8; //Y msb
    y |= Wire.read(); //Y lsb
  }

  if(String(AXIS).equalsIgnoreCase("X")){
    b = x;
  }
  else if(String(AXIS).equalsIgnoreCase("Y")){
    b = y;
  }
  else if(String(AXIS).equalsIgnoreCase("Z")){
    b = z;
  }
  else if(String(AXIS).equalsIgnoreCase("XY") || String(AXIS).equalsIgnoreCase("YX")){
    b = x * x + y * y;
  }
  else if(String(AXIS).equalsIgnoreCase("XZ") || String(AXIS).equalsIgnoreCase("ZX")){
    b = x * x + z * z;
  }
  else if(String(AXIS).equalsIgnoreCase("YZ") || String(AXIS).equalsIgnoreCase("ZY")){
    b = y * y + z * z;
  }
  else if(String(AXIS).equalsIgnoreCase("XYZ") || String(AXIS).equalsIgnoreCase("XZY")
          || String(AXIS).equalsIgnoreCase("YXZ") || String(AXIS).equalsIgnoreCase("YZX")
          || String(AXIS).equalsIgnoreCase("ZYX") || String(AXIS).equalsIgnoreCase("ZXY")){
    b = x * x + y * y + z * z;
  }
  else {
    b = 0;
  }

  if(!autoDetect){
    //show real-time magnetic field, pulse count, and pulse count total
    Serial.print("Magnetic Field: ");
    Serial.print(b);
    Serial.print(rising ? "  Rising, " : "  Falling, ");
    Serial.print("next pulse at: ");
    Serial.print(rising ? oldB + increment : oldB - increment);
    Serial.print("  Current Number of Pulses: ");
    Serial.print(pulseCount - oldPulseCount);
    Serial.print("  Last Total Pulse Count Sent to GW: ");
    Serial.println(oldPulseCount);
  } 
}```

[dynamite]

@dpcr Nice adjustment of your Original code. Based on your Original version I have started coding as well as I like the simpleness of just adding a sensor to the meter without taking care of the direct position.
My current code can be found at github via the following link:

https://github.com/rspaargaren/Read_Gasmeter/blob/Workinprogress/Read_Gasmeter.ino

Basic differences are:

• I have removed the fram reference but this could be added again if you use it.
• I have added a different setup sequence for fixing the top and bottom. In my case it waits until four changes in direction have occured. So no direct gas flow have to flow after a restart.
• The number of pulses between a change in direction is fixed independent of a change in top or bottom. So in my case every half a cycle will give ten pulses and a full cycle 20 pulses.
• The results are submitted to the gateway after a intervall but also after half a cycle.
• The top and bottom is checked after each cycle. So in case of major difference the interval is changed.
• During normal run, the reading of the y value is executed within the loop. So basically the arduino has time to do some other stuff as well in between the readings.

The code is currently working but requires some clean up. Also I would like to make an option to store the values at the gateway and maybe change the hardware to an ESP version so I can have multiple ways of output. Like MQTT.
This is the current output in Grafana:

[dpcr]

@dynamite I just scanned quickly over your code. So there is no need to store the top and bottom values? What do you do on a power failure? I look forward to hearing from you more.

Do you have any pics of your hardware?

[dynamite]

@dpcr At the moment I do not store the top and bottom value permanently. In case of a power failure I will "loose" to the max two rounds of the wheel on pulses as it uses that to calibrate the top and bottom. That could be reduced to potentially only one cycle. Which is in my case 0,01m3 to 0,02 m3 of gas indication.
After the power failure the process starts in the calibration loop which is ended only after the required cycles have registered. So this could last 30 sec in case you are having a shower but can last a week if you are away...

I will post a picture of my setup later.

Now I am thinking of it. If you do not care about the accuracy that much you do not need the top and bottom values at all. You just need to detect the rice and fall. That still gives you double the accuracy than a reed contact or Hall effect. The sketch for that will be very simple and after power down you can continue most probably without loss of pulses.

[dpcr]

@dynamite Great idea (top and bottom). We were kicking that around when we made this but wanted to err on the side of missing little to no gas flow. We have a pilot light on our water heater the draws around .1 lpm and I want to capture that as well.

I agree with you on the accuracy of using a magnetometer over a hall or reed switch, but they (hall or reed) are much easy to program. Too bad there isn't a library for converting a sine wave to pulses, don't forget that sometimes it doesn't look like a sine wave.

I'm curious to see how you mounted your magnetometer to your gas meter, mine seems a little too bulky and obtrusive. I did manage to buy another magnetometer and dip it in some stuff called PlastiDip. Its used for insulating the handles of tools. I just haven't switched them out yet to see how well it works.

[dynamite]

@dpcr Hi the reason that it is sometimes not a sine wave is the reason why I used a fixed number of pulses between the top of bottom. So in case the flow is very low as for your pilot light it will detect every puls. In case you have a very heavy flow pulses can be skipped as long as the change is detected then in my case 10 pulses are submitted to the gate way. I was detecting in your original sketch sometimes extra pulses were submitted as the sine was little bit rising or the opposite dropping.

The code for just detecting rise and fall can be quite straigh foreward. I might make a second version like that...just for the record.

My Setup is quite straight forward. A small box with the arduino pro mini and radio. A cable to the magneto sensor. In my case water proof is not required. I have applied the sensor by Velcro strap so I can remove the setup in case of "Maintenance".

[Yveaux]

@dynamite I seem to have the same meter; the BK-G4 :

The 'official' place to stick a reed sensor is a hole on the bottom-right, as you can see in the picture.
I had a spare reed switch from a Senseo coffee machine, which perfectly fits the hole and works perfect!

Maybe if you stick the compass sensor here, the signal will be more consistent/repeatable, as it should be more or less the same with each revolution.

[dpcr]

@dynamite I noticed some pulses missing as well. It looks like the newer sketch catches them. I also checked the volume used with our gas suppliers volume used and it was pretty close. We added an error correction variable to try to be more accurate. When I get a chance I'm going to try your sketch, it looks interesting.

[dynamite]

@Yveaux Hi thanks for the suggestion. Basically the position I am using now gives me a very accurate signal. It is more the small changes in the Sine curve which could result in miss reading. But I have added some filtering by appying a running average.

I have tried other position just to see the results. Basically if you stick it to close to the magnet used, like the official reed position it will go in overload and magnetic reading is maximized for a period.

[dynamite]

@dpcr Hi i am looking forward for your experience, as far as I have experienced now, and it took me quite some debugging I did not miss a pulse. :-)

@Yveaux @dpcr My next testing is to apply this sensor for the watermeter.

[gohan]

For water meters I saw people using those lines trackers that are used on small robots

[dynamite]

@gohan Yes i have Seen them as Well that would be the easy way out ;-)

I am just curious wether i am picking up a workable signal when i stick this sensor to it. If so it would make installation quite simple.

[Yveaux]

@dynamite said in Gas Meter Reading Using a Magnetometer:

I am just curious wether i am picking up a workable signal when i stick this sensor to it

Yes, works like a charm! At least for my sensor

[dynamite]

@Yveaux nice! and what code are you using? or are you referring to putting the ir sensor to the meter?

[dpcr]

@dynamite Just checking in, how is your set up working? We just had some major problems with too many extra pulses. It looks like something you experienced earlier, we added some smoothing to the readMag output similar to what you have in your sketch and it seems to working fine. Any updates?

[jumping]

Hello,
I have a BK-G4L gas meter and I want to build a Mysensor node and measure the gas flow: what is the best solution, a reed switch or a magnetometer?
and for a magnetometer based sensor, do I need HMC5883 or HMC5983?
thanks

[dynamite]

@jumping Hi I have used the magnetometer. (GY-282 HMC5983)

I these are the +/- of both the systems:

• More easy to locate on your gas meter
• Higher accuracy of output (more pulses per cycle)

• Code is more complex

Read Sensor

• Simple code

• Exact installation from position
• 1 puls per round

[dynamite]

@dpcr Hi, I need to recalibrate to see if the reading is still OK. Somewhere during a total reset I have lost track. But it is on my to do list to get everything up and running again.
As far as I could recollect an overshoot had no influence on my code as the number of pulses for a total cycle is consistent within my code.

On the Y reading from the sensor I do have a running average smoothing.
Basically every cycle is splitted into rising and falling part. Each splitted into a fixed number of steps. Based on the current reading of y the number of pulses is calculated within that step. So the amount of pulses within a rise or fall can never be more than the number of steps within that cycle.

I am thinking of making the code more simple and only give a pulse on the changes and maybe one in the middle. That would give me about 4 pulses per cycle.

[jumping]

@dynamite thanks for your answer! do you use a adafruit FRAM in your node?