Friday, January 3, 2014

More Than One Way to Skin a Cat

I know what you're thinking: "DeKay hasn't posted in a while.  Maybe he's dead?"  Fear not.  For while the frequency of my posting has dropped, I remain very much alive.  I've just been a little busy at work.
Thankfully the holiday season rolled around and I was able to take a couple weeks off. I took that time to relax and catch up on all the things normal people like to do: enjoy some good food, visit with friends, and hack on microcontrollers with integrated RF transceivers.
This bad boy is the Moteino from LowPowerLab. It is an Arduino clone with a 3.3V Atmega 328p that integrates a HopeRF RFM69 RF module. And let me tell you, this little module rocks. They are small, cheap, and readily available. Felix from LowPowerLab is getting tons of range out of the high power variant of these little guys.  They are easily programmed over a SPI bus so you don't need special hardware like you do with TI's CC1110, and cheap versions operating in the 900 MHz band are easy to get unlike TI's CC1101.  And recently, interoperability with the RFM12B popularized by the JeeNode has been achieved.

Now the RFM12B deserves a bit of discussion.  Like the RFM69 module, it is small, cheap, and easily programmed.  But it sucks in a number of ways.
A small FIFO means that you better have something real time or close to real time listening to the module. Otherwise its buffer overflows and you lose data. Something like an Arduino works well because it isn't doing other stuff like running an operating system. But worse than this is the hardcoded sync bytes. This pretty much limits the RFM12B to talk to other RFM12B's. And that ain't cool.

The RFM69 fixes all these problems and adds some nice features while it is at it:
  • it has a 66 byte FIFO, a fully programmable sync pattern up to eight bytes in length, and a variable length preamble
  • address checking and CRC validation is optional
  • it supports OOK, FSK, or GFSK
  • it can do AES encryption
  • other stuff
In other words, it looks like it can do what other transceiver modules can do and more.

Intrigued, I picked up a Moteino with integrated RFM69W module and a couple standalone modules while I was at it.  Then I got busy (see above).  Then I got some free time (see above).  Then I started hacking (see below).

Devoted readers of this blog will know that I am strangely compulsed to receive wireless transmissions from my Davis weather station using all kinds of strange hardware platforms.  So why stop now?  I've been looking for a platform like this that is available and easy to use so that more people can take a crack with this stuff (it isn't like you can walk into a Toys-R-Us and pick up an IM-ME anymore).  I also don't want to be glued to the Arduino for everything.  While it is great for running as a remote node, I like the idea of something like my BeagleBone Black connected directly to a transceiver module running some home monitoring and control software (HouseMon?) with Internet connectivity.

But, baby steps.  What I decided to start with was my RFM69 equipped Moteino and the LowPowerLab RFM69 Arduino library.  Because once you've got the wireless protocol figured out, hard hard can it be?  Turns out, harder than I thought.  But after slamming my head against a brick wall a time or two, I got this.
Success!
This is the data from my Davis weather station out in my yard and received on my RFM69-equipped Moteino.  The first field is the channel number, the data is the eight data bytes sent by the ISS on every transmission (see here for what those bytes mean), RSSI is the Received Signal Strength Indication, and the CRC is calculated from the first six bytes in the data packet. Every packet is sent with a CRC in the seventh and eigth bytes, so those two values will agree with the calculated CRC if the packet is good.  I've got some problem somewhere where the first packet I receive is bad, as is any packet on Channel 42.  But those are details.  I'm off to a decent start here and I'm sure I can sort those issues out in time.

Want to check it out?  My DavisRFM69 library is on GitHub and the data above was collected using the ISSRx sketch in the Examples folder.  The approach that I've taken is to subclass my driver from the LowPowerLab one, overwriting just a couple of methods that are specific to this application (namely the initialization and the interrupt handler).  That reduces duplication and lets me automatically pick up any improvements Felix incorporates into his main library.  I have made a couple minor tweaks to his main library for some register definitions that I've issued a pull request for.  Hopefully these will be incorporated into his main library soon.  In the meantime, you'll want to use my fork of the library, so everything will compile cleanly.  This and more is all in the README.

This work can be taken a lot further:
  • do a full emulation of the Davis indoor console for use by weather software like WeeWx and Cumulus
  • hang an RFM69 directly off a BeagleBone Black and write a driver to work under node.js
  • why just receive?
Pull requests gladly accepted  :-)

I'm glad to now have this stuff under my belt.  I could have taken the entire break to just sit in front of the TV eating fruitcake.  That would be easy because My Lovely Wife makes a rocking fruitcake.  But what I really find satisfying is that sense of accomplishment when you set a challenge for yourself and make it happen.  This was a good way to kick off 2014, and the year is just getting started.

And don't get me wrong: I had lots of fruitcake.

27 comments:

  1. As always a very interesting article! I tried to load and compile your RFM69 sketch but "SPIFlash.h" seems to be missing and therefore compilation breaks with error "ISSRx:31: error: 'SPIFlash' does not name a type". As I am a Ardiuno freshman, this is certainly my beginners fault. However a hint how to get around this would be very helpful.

    ReplyDelete
    Replies
    1. Sorry for spoiling this article, I just found out that it needs the SPIflash Library from LowPowerLab (https://github.com/LowPowerLab/SPIFlash) also included. Now it compiles fine and I have to wait for the HW to arrive to give it a practical shot.

      Delete
    2. Seems we can all use these
      https://www.kickstarter.com/projects/1754284174/ardurf-wireless-arduino-done-right
      Martin

      Delete
  2. In "DavisRFM69.cpp" I can see the frequency table for US. I would love to add the one for Europe here. From this thread (http://www.wxforum.net/index.php?topic=18718.25) it looks to me like Europe ISS does hopping with just 5 frequencies:
    Index: 00, FREQ2: 3A, FREQ1: 19, FREQ0: 45 --> 868066725Hz
    Index: 01, FREQ2: 3A, FREQ1: 1D, FREQ0: 45 --> 868297125Hz
    Index: 02, FREQ2: 3A, FREQ1: 21, FREQ0: 45 --> 868527525Hz
    Index: 03, FREQ2: 3A, FREQ1: 1B, FREQ0: 45 --> 868181925Hz
    Index: 04, FREQ2: 3A, FREQ1: 1F, FREQ0: 45 --> 868412325Hz

    Any idea how these bytes need to be recomputed to fit into the DavisRM69 frequency table schema? As numbers above are for Si1000 I guess they will not work for RFM69 without some recalculation.

    ReplyDelete
  3. I tried some math with the frequency values given in your example.
    1) oscillator frequency of RFM69 is: Fxosc = 32 MHz
    2) available frequency step size of RFM69 are: 61.0351563 Hz (32 MHz / 2^19)
    3) programmed frequence deviation for Davis is: Fdev = 4760 Hz
    4) frequence for index 0 packets is expected to be 902.5 MHz
    5) when I do the math on the index 0 entry in the frequency table of your code "{0xE1, 0x98, 0x71}" it results to: 0xE19871 * 61.0351563 Hz = 902.381.897 Hz
    But that is quite a large deviation from the expected value, even when I add Fdev.

    Any idea what is wrong in my math? Unless I understand this it will be hard to do the trick for the Europe frequencies ;-)

    ReplyDelete
    Replies
    1. Thanks for your interest. Your math is actually correct. Here is where you are going wrong...
      3) Fdev is not a part of this calculation. It is needed for figuring out the modulation, but not the carrier frequency
      4) I'm not expecting Index 0 packets at 902.5 MHz. Check out the post http://madscientistlabs.blogspot.ca/2011/03/davis-frequency-hopping-sequence.html and look at the embedded spreadsheet. My example was for channel 1, not channel 0, and the sniffed RF value was 902.381925 MHz. Although that spreadsheet does have an "RF nominal" column, that doesn't mean anything today. I thought at the channels at the time would be evenly spaced, but now I know for a fact that they are not.

      I thought later that I code up the European frequencies in here too. If you beat me to it, send me a pull request. I'm trying to knock off another item from my todo list right now. My little spreadsheet that figures this all out says the European frequency array should be...

      [{0xD9, 0x04, 0x45},
      {0xD9, 0x13, 0x04},
      {0xD9, 0x21, 0xC2},
      {0xD9, 0x0B, 0xA4},
      {0xD9, 0x1A, 0x63}]

      Delete
    2. Thanks for helping me understand. To be honest, I was not aware that more data comes when I do scroll horizontal ;-)

      When I take your frequency data above it results according to my math to what is listed below (deviation in Hz to expected values as listed above) in brackets:
      51: 868066711 (14)
      52: 868297119 (6)
      53: 868527466 (59)
      54: 868181885 (40)
      55: 868412292 (33)
      As all these deviations are below the Fstep of abt 61Hz these look to be the correct values.

      For the moment I don't pull, as I don't have the HW yet, which I need to produce real results and not just noise like now. When adding the lines, you might have a look at the US frequencies as well, as there seems to be missing one. I had expected 51 value triples (#0 - #50), but only #0 - #49 there.

      Delete
    3. Count again.
      https://github.com/dekay/DavisRFM69/blob/master/DavisRFM69.h

      Array is on rows 31 - 81. (81 - 31) + 1 = 51

      Delete
    4. This stuff is great. I'm considering buying the barebones VP2 without its ugly and (for me at least) useless console. I'd need the console just to be able to hook the it up to the web anyway. I'm also in Europe so I'm gonna need the 868 MHz version too. If this seems doable I'll prolly buy the VP2 and test out your solution and integrate it with WeeWX or something similar (haven't decided on the SW yet).

      Delete
    5. Try (0xE3, 0xD8, 0x90) for the first hop and see if it gets rid of the problem with the first packet.

      Delete
    6. Thanks for the input. I'll give it a shot this weekend when I get more time on my hands.

      Delete
    7. rdsman: it didn't help. I tried going 30 kHz in the other direction too and that didn't help either. Strange...

      Delete
    8. How about moving Hop 0 to the end and start at Hop 1 and see if it changes anything.

      Delete
    9. Found the problem with the very first packet. I don't know why, but the chip was reporting that the FifoNotEmpty and FifoLevel bits in the ReqIrqFlags2 register after powerup and reset for reasons I don't understand. This register can be cleared by writing to the FifoOverrun bit. Did this in my init routine and my first packet problem went away.

      I was also seeing packet errors during normal reception and that was caused by the LowPowerLabs code being overly chatty to the radio. Fixed that too.

      Now my reception seems to be working perfectly. Yes!

      Delete
  4. I'll give that a try but I suspect the problem will always be the first frequency. It occurred to me late last night that I am not programming the frequency registers when I initialize all the rest of them, but do it a little later. Unfortunately I had very little time to play around with this stuff last weekend. Gotta leave some time this weekend!

    BTW, are you able to put up your source where you were trying to emulate the console? There is a thread on wxforum where you said you were making progress on this and then ran out of RAM. The Moteino has double the RAM, so it might be doable. I noticed that your source files for this stuff are no longer on your website.

    ReplyDelete
  5. It will be back soon. I just haven't had the time to clean it up.

    ReplyDelete
    Replies
    1. The zip file is back:

      http://www.raydees.com/Downloads.html

      Delete
    2. Hi RDSman
      I was using version .5 and it was working OK.
      However I just downloaded new ver .8 and it seems to be broken somehow, it hangs/restarts after receiving 3 to 6 packets...

      Delete
  6. I've bought a VP2 ISS EU version (without console), and currently trying your code for reception on a Moteino/868. I've replaced the FRF array with your calculated one above for 868 MHz and adjusted the hop count, but I'm getting only garbage so far:

    0 - Data: 37 13 60 4 21 11 44 18 RSSI: -104dBm CRC: BF90
    1 - Data: DE 88 E8 94 84 E4 16 80 RSSI: -107dBm CRC: CBCC
    2 - Data: 59 77 3D 6F 1B FD 37 47 RSSI: -109dBm CRC: 12A
    3 - Data: 7F E6 F9 20 7E 6B A7 E3 RSSI: -111dBm CRC: 6A71
    4 - Data: 75 40 0 14 0 4 1A 92 RSSI: -106dBm CRC: C8D2

    Something's missing for this frequency to work. Since I don't have the console I'll need to get it working. I'll ultimately end up emulating a Davis console for alternative weather apps. Do you have any suggestions or directions to go? Anything comes to your mind I'm more than willing to test on the ISS.

    ReplyDelete
  7. Well, I have some moderate success. I modified the code to only hop when the crc is right. The band seems to be crowded here. I get consistent readings but I have many interemittent gaps and I had to move the ISS near the receiver (they're now about 50cm/20" apart only, very unpractical). It seems that some receiver parameters are not optimal for reception. How would one go optimzing them? I tried to shift frequencies a little but it didn't really help.

    ReplyDelete
    Replies
    1. Alright, it turned out that the noise picked up from the PC was the problem. I ran the Moteino from a battery and made it so every successful reception blinked the led. It was blinking every 2.5 secs continuously, so far soo good. Moving to phase 2. Thank you for your great lib and all the info you gathered, without this it wouldn't have been possible :]

      Delete
    2. Wow! I never actually expected anyone to actually use this :-)

      Your find is interesting. I have been getting intermittent lost packets and maybe my PC is the cause? I'll have to try battery power sometime.

      I am actually well along the road to emulating the Davis console and hope to have something that works with Cumulus over the weekend. In the meantime, get onto Ebay and order up a DHT22, BMP085 or BMP180, and a DS3231. Stay tuned...

      Delete
    3. Cool. I actually already have a couple of DHT22's and a BMP085. I plan to build a few sensor nodes similar to the EmonTH (or actually buy a few).

      But first, I need to finish my quest on the internet-connected ISS but I'm stuck again. Looked high and low but can't figure out the formula to convert the raw temp data on my metric unit. I commented on it here too: http://www.wxforum.net/index.php?topic=18718.msg208650#msg208650

      You might have an idea...

      Delete
  8. Hi
    https://www.kickstarter.com/projects/1754284174/ardurf-wireless-arduino-done-right seems to be something we can use.
    Martin

    ReplyDelete
  9. Hi DeKay,

    I read your blog and I must say: very cool project. I'm looking for a new project for my sparetime. That is the reason why I ordered a Monteino board. So before I start to experiment I have a question to your DavisRFM69 Lib. Is the lib working with the european version (886MHz) of the Davis Vantage Pro2?

    I would be glad, if you would say yes.

    Best regards
    Martin

    ReplyDelete
  10. For those who find it hard to get a moteino, here's an easy to get alternative for Europe:
    https://www.canique.com/mk1 - almost identical to the moteino!

    ReplyDelete
  11. Dear DeKay,

    I was wondering if your library can be used to receive the ISS data that is currently connecting to a Vantage Pro (pretty old , I know :-) ) The visibility of the screen of my Console is getting really bad now and I would love to capture the ISS packets and send that to a MQTT server for further processing.

    Appreciate your feedback..

    ReplyDelete