Thursday, May 10, 2012

It's Not The Heat...

Last time around, I wrote up how rain information is sent from the Davis Weather Station Integrated Sensor Suite (ISS) to the console sitting on my kitchen counter.  That left me with one more piece of weather data to figure out: humidity.  Evil humidity.

I had been puzzling over humidity for weeks before I decided to switch to rain.  The data seemed to make sense for a while and then I would get a datapoint that made no sense at all.  I was pretty certain that humidity was sent in message 0xa0 from the station, and I was also pretty certain that the data would be contained in just one byte.  Sounded reasonable because humidity is displayed on the console to the nearest percent.

Humidity was stumping me until I got serious last weekend and wrote a Python script to collect STRMON data continuously while my DIY datalogger dutifully captured the console's interpretation of the data.  Then I could make some plots and figure out what was going on.  Like this:
Click To Embiggen
It was quickly apparent in looking at this data that two bytes were involved in encoding humidity.  You can see as the humidity drops, Byte 3 cycles through a couple times while there are some discrete steps in Byte 4.  The weird thing was that Byte 4 isn't flat as in a high byte, low byte kind of thing.  There is jitter on it that I didn't expect.

Puzzled, I thought some fresh eyes on the subject might help.  So I posted this plot and a link to the data I collected on this wxforum post.  It was user Kurgan for figured it out for me within eight hours of my post.  He da man!  As he wrote in his post:
If you take byte 4 and convert it to binary, you will see that the values is has assumed in the test, which are 24,25,26,27 then 40,41,42,43 and then 56,57,58, 59 are made of two parts: the first four bits are fixed (the big steps), and the last four are variable (the jitter).

So:

 24 25 26 and 27 have the first four bits set at 0001
 40 41 42 and 43 have the first four bits set at 0010
 56 57 58 and 59 have the first four bits set at 0011

So, you have to add the first four bits of bit 4 to byte 3 to get a 12-bit number. Try to plot this 12-bit number and compare it to the hum graph.

The last four bits of byte 4 are something else, or are random noise. I don't know.
I thought he had it wrong the first time I wrote up this post but forgot that the values in Byte 4 as written above had been converted to decimal.  Convert them back to hex for his explanation to make sense.  Next step: make a pretty plot.
Now That's Pretty
That is bang on.  The calculated value works out to the humidity displayed on the console * 10, so a calculation of 750 = 75% RH.  The lines don't sync perfectly because the timing is a bit off and the logger logs in intervals of 1 minute vs irregular intervals on STRMON.

As I wrote in a post that followed, there are other mysteries are buried in the data.  For example, here is data from message 0x90 (144 decimal). 
39.2    144   16   1   47
90.5    144   16   1   63
141.7   144   16   1   79
193     144   16   0   95
244.2   144   16   1   111
295.5   144   16   0   127
346.7   144   16   2   143
398     144   16   3   159
449.2   144   13   3   18
500.5   144   13   3   34
The columns are elapsed time in seconds, message number, byte3, byte4, and byte 5.  I deleted the wind and crc data.  The numbers in byte 5 jump up and down mostly in step sizes of 16 before resetting and then starting over again at some different value.  Looks like a counter of some kind.  And why is byte 4 in this same message just bouncing between 0 and 3?  Byte 4 in the rain message (0xe0, 224 decimal) does this too.  I'm also told that the ISS sends a reading of its battery voltage to the console  That is also hidden away somewhere in all this data I've collected as well.  And then of course there is data from the sensors I don't have: solar, leaf wetness, and soil moisture.

But I don't care too much.  I'm getting all the weather related data my ISS can send me.  Figuring out other stuff down the road would just be a bonus.  Now it should be a relatively straightforward matter of writing some code for the Alternative Davis Weather Station Receiver and having its values exactly match the values displayed by the console.  Cool.

If only I had more time to actually do this...

6 comments:

  1. Very interesting...

    ReplyDelete
  2. If you want more info on the humidity sensor they use. Here is a link to the datasheet
    http://www.sensirion.com/fileadmin/user_upload/customers/sensirion/Dokumente/Humidity/Sensirion_Humidity_SHT1x_Datasheet_V5.pdf

    ReplyDelete
    Replies
    1. Welcome to February, 2011.
      http://madscientistlabs.blogspot.ca/2011/02/busy-weekend-coming-up.html

      Not that this would have helped me anyway as the ISS sends processed data rather than raw. There are also something like three different sensors that Davis has used in this console over time.

      http://www.wxforum.net/index.php?topic=15273.msg147997#msg147997

      Delete
  3. Hi DeKay,
    I'm working on a Si1000 based replacement for a Davis Vantage Vue console and I'm getting some interesting differences in the protocol that my ISS is sending compared to what you and others have kindly documented.

    I had a weather station running nicely till some vandals smashed it, so instead of buying a new console I decided to build my own. I'm the author of the 'SiK' firmware for the 3DR telemetry radios, so I thought I'd use my existing radio hw (based on a HM-TRP Si1000 module) rather than getting a CC1101 board.

    Using the CC1101 register settings from your code I've created an equivalent set of settings for the Si1000, and I setup the frequency hopping pattern you used. With that I'm successfully getting packets from my ISS, with no loss. If you are curious, my code is in the davis-wip branch here:
    http://github.com/tridge/SiK
    (the master branch is for the normal 2-way telemetry radio mode of these radios).

    The data I'm getting consists of 8 byte packets. The lead byte clearly plays the same role as what you see, with the high nibble being the packet type, and the low nibble containing the transmitter ID. The hopping pattern is the same regardless of what transmitter ID I set on the ISS.

    The 2nd byte is the wind speed, but is in a weird format. The low nibble seems to be 14-speed, and the high nibble is the high bits of the speed. So the byte is 0xE at zero wind speed. Quite bizarre. I'm planning on doing a wind speed calibration this afternoon by holding the ISS out a car window. Unfortuately I can't do the usual trick of feeding my console fake input to do the calibration, due to the thorough smashing the vandals gave the console.

    The 3rd byte is the wind direction, just as you saw. That is the easy one.

    The temperature has stumped me so far. I'm not even sure it is packet type 8. I get type 8 packets every 54 seconds, and the ISS manual claims temperature is every 10s, so it makes me suspect temperature is type 1, 10 or 7, which are all sent every 10s.

    The 16 bit checksum is in bytes 7 and 8, same as you found, but the checksum always comes out as 0xb5b8 when calculated over the 8 bytes, instead of the expected zero. I wonder if Davis use the checksum as a type of protocol version number? They could have one console handling different types of ISS by looking at how the checksum comes out.

    I'll keep poking at it, but I thought you might like to know that there seems to be different protocol varients out there. I've put a log from last night up here:
    http://uav.tridgell.net/DavisISS/
    The format is 10 protocol bytes, followed by a timestamp in seconds since 1/1/1970 UTC. The reason for 10 protocol bytes is that I had read elsewhere that the protocol had 10 bytes, so I save the first 10 bytes off the FIFO. The last 2 bytes are constant 0xc2 0x16, and thus aren't interesting.
    This is the radio I'm using:
    http://store.diydrones.com/3DR_Radio_USB_915_Mhz_Ground_module_p/br-3drusb915.htm
    although any Si1000 radio, including some of the very cheap HM-TRP modules would do.
    Cheers, Tridge

    ReplyDelete
    Replies
    1. So I was about to ask you a few really basic questions. Then I saw your nick at the bottom of your comment and that rang a bell. Then I saw your full name at the top of this comment and realized that this comment was written by a Free Software Reverse Engineering God.

      I AM NOT WORTHY!

      Anyhoo, your findings so far are bizarre. I can't understand why the hopping pattern would not change when you change the ISS transmitter ID. I'm assuming you somehow reset the ISS after doing so? It also seems very odd to me that Davis would change up basic stuff like the checksum and even the basic message IDs.

      I am now watching your work on Github but I also don't get over there too often. Please keep me posted here or, maybe better yet, start a thread on http://www.wxforum.net/index.php?board=59.0. Either way, good luck!

      Delete
  4. Hi again,
    I was going to do a wxforum posting but registration was closed for maintenance at the time.
    Just in case anyone else runs across these comments in the future and wants to follow up, here is the wxforum thread that I've now started after registration came back online:
    http://www.wxforum.net/index.php?topic=18718.0
    Cheers, Tridge

    ReplyDelete