Showing posts with label Davis VP2. Show all posts
Showing posts with label Davis VP2. Show all posts

Monday, February 17, 2014

Build Your Own Davis Weather Station Console!!!

This Davis weather station hacking stuff is addictive like crack cocaine is addictive.  I always seem to be taking another hit from the pipe.  I've figured out how to get into the Davis console's serial port (after which the two of us had a complete and frank discussion COMPLETELY IN UPPERCASE).  I then built my own Data Logger so I wouldn't have to buy the ridiculously expensive Davis dongle.  Then, after I figured out the wireless transmissions from the outdoor sensor suite, I made my own ISS receiver with a Pretty Pink Pager.  But of course, I can quit any time I want.  Me and Amy Winehouse.
"They tried to make me go to rehab I said No, No, No"  -  Bad Decision

Famous dead drug-addled musicians aside (and how long a list is that?  Don't they know that only Keith Richards is Immortal?), the path I was to go down became clear after Davis pulled a ridiculously stupid stunt in the infamous Firmware 3.0 release they started shipping in new consoles a while back.  The new firmware read 64 bytes from a fab-written, read-only section on the data logger's memory chip.  It took this data, did some algorithmic juggling, and then did a check against another 64 byte write-once block on the data logger put there by Davis.  If the console didn't like what it saw, it declared that the logger was invalid and refused to function further.

This was an obvious and misguided attempt to shut down a few hackers from building and selling third-party data loggers for a fraction of the price Davis charged.  Problem #1 was that people with an older, authentic Davis logger discovered that their pricey little dongle stopped working after a firmware "upgrade".  Problem #2 was it also disabled the serial interface connection I had discovered.  And that pissed me off.

Unfortunately for Davis, the console does not run a 2 GHz Core i7 processor capable of state of the art encryption and decryption algorithms.  It has a little Atmel Atmega processor running at 1 MHz or something like that.  It's abilities are somewhat more... restricted.  Like something that could be cracked by a 256 byte lookup table and a trivial bit of code. Hats off to Watson on WXForum for figuring this out. His most excellent hack is posted below for posterity.  Break out your Arduino and start cutting and pasting if you want to calculate a valid security code for a DIY logger.

u8 const GreenDot_Table[256] =
0x00, 0x04, 0x08, 0x0C, 0x10, 0x14, 0x18, 0x1C, 0x21, 0x25, 0x29, 0x2D, 0x31, 0x35, 0x39, 0x3D,
0x46, 0x42, 0x4E, 0x4A, 0x56, 0x52, 0x5E, 0x5A, 0x67, 0x63, 0x6F, 0x6B, 0x77, 0x73, 0x7F, 0x7B, 
0x8C, 0x88, 0x84, 0x80, 0x9C, 0x98, 0x94, 0x90, 0xAD, 0xA9, 0xA5, 0xA1, 0xBD, 0xB9, 0xB5, 0xB1, 
0xCA, 0xCE, 0xC2, 0xC6, 0xDA, 0xDE, 0xD2, 0xD6, 0xEB, 0xEF, 0xE3, 0xE7, 0xFB, 0xFF, 0xF3, 0xF7,
0x18, 0x1C, 0x10, 0x14, 0x08, 0x0C, 0x00, 0x04, 0x39, 0x3D, 0x31, 0x35, 0x29, 0x2D, 0x21, 0x25,
0x5E, 0x5A, 0x56, 0x52, 0x4E, 0x4A, 0x46, 0x42, 0x7F, 0x7B, 0x77, 0x73, 0x6F, 0x6B, 0x67, 0x63,
0x94, 0x90, 0x9C, 0x98, 0x84, 0x80, 0x8C, 0x88, 0xB5, 0xB1, 0xBD, 0xB9, 0xA5, 0xA1, 0xAD, 0xA9,
0xD2, 0xD6, 0xDA, 0xDE, 0xC2, 0xC6, 0xCA, 0xCE, 0xF3, 0xF7, 0xFB, 0xFF, 0xE3, 0xE7, 0xEB, 0xEF,
0x31, 0x35, 0x39, 0x3D, 0x21, 0x25, 0x29, 0x2D, 0x10, 0x14, 0x18, 0x1C, 0x00, 0x04, 0x08, 0x0C,
0x77, 0x73, 0x7F, 0x7B, 0x67, 0x63, 0x6F, 0x6B, 0x56, 0x52, 0x5E, 0x5A, 0x46, 0x42, 0x4E, 0x4A,
0xBD, 0xB9, 0xB5, 0xB1, 0xAD, 0xA9, 0xA5, 0xA1, 0x9C, 0x98, 0x94, 0x90, 0x8C, 0x88, 0x84, 0x80,
0xFB, 0xFF, 0xF3, 0xF7, 0xEB, 0xEF, 0xE3, 0xE7, 0xDA, 0xDE, 0xD2, 0xD6, 0xCA, 0xCE, 0xC2, 0xC6,
0x29, 0x2D, 0x21, 0x25, 0x39, 0x3D, 0x31, 0x35, 0x08, 0x0C, 0x00, 0x04, 0x18, 0x1C, 0x10, 0x14,
0x6F, 0x6B, 0x67, 0x63, 0x7F, 0x7B, 0x77, 0x73, 0x4E, 0x4A, 0x46, 0x42, 0x5E, 0x5A, 0x56, 0x52,
0xA5, 0xA1, 0xAD, 0xA9, 0xB5, 0xB1, 0xBD, 0xB9, 0x84, 0x80, 0x8C, 0x88, 0x94, 0x90, 0x9C, 0x98,
0xE3, 0xE7, 0xEB, 0xEF, 0xF3, 0xF7, 0xFB, 0xFF, 0xC2, 0xC6, 0xCA, 0xCE, 0xD2, 0xD6, 0xDA, 0xDE

u8 const Adesto_Factory_Programmed[64] = 
  // put here AT45DB011D Security Register from byte 64 to 127

u8 OneTime_User_Programmable[64];

OneTime_User_Programmable[0] = 0x00; // or whatever you want
OneTime_User_Programmable[1] = 0x00; // or whatever you want
OneTime_User_Programmable[2] = 0x00; // or whatever you want

u8 n, value;

for(n=3;n<64 br="" n="">{
 value = (u8)(Adesto_Factory_Programmed[n] + n);
 OneTime_User_Programmable[n] = GreenDot_Table[value];

For more on this whole sad saga, you'll want to read this masterwork by torkelmj, especially if you enjoy reading stuff dripping with spite and vitriol.  Like I do.  Or skip that and just buy a security-coded third party logger from WXForum awesomedude Belfryboy.  Or order up his PCB on oshpark and build your own.  There are options now, and options are good.

Now one might think that everything is just peachy now that DIY loggers are possible once again.  Not me.  I was pretty grumpy that Davis had tried to screw their customers over even if they did release subsequent firmware updates that tried to undo some of the damage.  Once burned, twice shy, so they say.  I started thinking of a Plan B.  I want to build my own console.
Gratuitous Picture of My Dog
My IM-ME thing was a neat hack, but that is about as far as it goes.  IM-ME's aren't actually rolling off the production lines in millions of units per month these days, and their ability to hook up to other little do-dads would not be easily done.  Cheap transceiver boards based off of the popular CC1101 chip that work in the 915 MHz band are hard as hell to come by.  Ask me how I know.  The Seeedstudio RFBee comes tantalizingly close, but others who have tried to go this route run out of RAM and Flash on the limited processor found on this unit before getting too far.

I needed another way and that led me to the Moteino with its awesome little RFM69B transceiver module.  I got it picking up Davis ISS transmissions earlier this year.  Since then I have been coding away in my spare time and have been pulling together the various bits of hardware that I need to get something going.  Something like this.
Beautiful, Isn't It?

Some cool stuff has sprung up from this very breadboard.  My serial connection to the console grew up there on the left side.  The DIY logger design is in the middle.  The stuff on the right half starting from the left is the humidity sensor, the pressure sensor module, the Moteino itself, and finally my USB to Serial Adapter for the project I'm writing about today.

Now I know what you are saying to yourself: "The title of the blog is "Build Your Own Davis Weather Station Console!!!" but that doesn't look like a console at all!  Where is the keyboard?  Where is the display?" Well, I guess I'm not going to get anything past you today.  There isn't a keyboard or a display... unless you want to add one.  The hardware and software here is all open for you to add on what you'd like.  Me, I'm thinking of hooking this bad boy to a serial port on my Beaglebone running Openhab and having all my weather data available from anything in my house that can bring up a web browser.  Very doable, and no display or keyboard necessary for that kind of application.

But why call it a "Console"?  Because I couldn't think of a better name.  And I did indeed try.  It isn't a "Receiver" because it does more than receive given that it has its own sensors hanging off of it.  "Indoor Unit" sounds stupid.  "Thingamahooie" was tempting, I must admit, but didn't quite convey the application well enough.  And hey, the guy that builds the hardware, develops the code, and writes the blog gets to call what he comes up with whatever the hell he wants.  I'm going with "Console", and you'll just have to deal with it.

But I, as I so often do, digress.  Back to business.  On the hardware side of things, here is what I'm working with in more detail:
  • Moteino with an RFM69 module on it.  Pick the 915 MHz RFM69W (standard transmit power) or RFM69HW (high transmit power) at your option.  I have the serial version but it probably would have made sense to pick up the USB variant so I wouldn't need to tie up my USB-Serial converter all the time.  You probably want to add on the male headers option (why even think about it for the one dollar price?) and the flash memory option for four dollars.  I'm not using that yet, but it might come in handy for data logging at some point.  And if you re-dedicate the Moteino to something else later, this memory can be used for wireless remote programming.  Cool.
  • a DHT22 Temperature and Humidity Sensor for less than five bucks.
  • a BMP085 Temperature and Pressure Sensor for less than six bucks.  But it looks like that module is now much cheaper thanks to the compatible BMP180 sensors now out there for under three bucks.  The two are compatible, and I'd probably go with the newer BMP180 myself just because it is smaller.
  • a (yet to arrive and be connected) DS3231 Real Time Clock and Memory Module for less than three bucks (I felt bad for getting something at this price with free shipping so I bought two).  Note that this module also has a 24C32 EEPROM on it that is bound to come in handy.  Don't be dumb and order a DS1307 clock module instead: that one doesn't work well with the 3.3V Moteino and it is horribly inaccurate.  What good is a clock chip that can't keep time?
Arduino is all about little software libraries, and these are the ones I'm working with right now.
  • the LowPowerLab RFM69 library.  Felix has merged the changes I've needed so no need to use my fork of his code any longer.  I don't know if I'll be using this in the long term or come up with my own that is better suited for this application.  Another library that I'm including right now but not really using yet is his SPIFlash library.
  • the JeeLabs DHT22 code.  I have extracted what I need from the JeeLib code and included it into my own library.  JCW's stuff is a leaner and meaner version of some of the other ones floating around on the net.
  • the Adafruit BMP085 library.  I used this as is.  I did look at using the JeeLabs code but it was a little too tied up with JCW's PortsLib for me to bother untangling at the time.
  • the Arduino Serial Command library to parse command input.  This is a really great little library that lets you register callbacks for each separate command and also does command line argument parsing.  I needed to make a few tweaks as noted below, but nothing major.  My changes have been pushed but not merged at this time, so use my fork here.
  • my own DavisRFM69 library, of course.  I've got to be doing something to make all this stuff work together.
Enough chitchat.  Let's cut to the chase, shall we?  Take a look at this thing actually doing something.
Success - I Never Tire of It
This is SandaySoft's Cumulus software talking to my homebrew weather station console.  The outdoor readings are those from my ISS in the front yard.  The indoor readings are from the sensors I have hanging off the Moteino.  On the left you can see the Serial Port Monitor Software recording the RS-232 messages going back and forth.  Cumulus has no idea that it isn't talking to a real Davis console.

Thanks to the Davis Serial Protocol Manual, getting this working wasn't too terribly difficult.   I knew Cumulus used the Davis SDK to talk to the console so the Cumulus author might not be aware of every single command being sent to the station and back.  I also knew that the LOOP command was used to get the data out of the station so I implemented that and some other useful commands on the Moteino.  Then I hooked my real console up and watched some packets go by and found a few more I needed to implement (namely BARDATA, HILOWS, and RXCHECK).  I had a few hiccups of course.  For example, Davis says that a command should always be terminated with '\n' (0x0a) like this...
 except when they don't, like this...
And that is the fun with emulation: you have to emulate not only the functionality, but the bugs as well.  This one caused me to tweak the serial port library to accept multiple command terminators.  Another oddity is that the console itself goes to sleep all the time and is woken by simply sending a lone '\n' that the console is supposed to acknowledge.  The serial library I'm using hadn't actually accounted for that situation.  Now it does.

Want to play along?  Here is what you need to do.   From within your Arduino's libraries directory...
  • Get my version of the SerialCommand library that supports empty commands and a second line terminator.  And of course, you'll want to get a copy of my DavisRFM69 library.  The pared down DHT22 library is tucked away inside the latter.
git clone SerialCommand
git clone
git clone Adafruit_BMP085
git clone
git clone
Hardware hookup is like this.

DHT22 Pin 1 -> Moteino 3.3V Out
DHT22 Pin 2 -> Moteino D4 with 10K Pullup resistor to Moteino 3.3V Out
DHT22 Pin 3 -> Not connected
DHT22 Pin 4 -> Moteino GND

BMP085 Module Pin VCC -> Moteino 3.3V Out
BMP085 Module Pin SDA -> Moteino SDA
BMP085 Module Pin SCL -> Moteino SCL
BMP085 Module Pin GND -> Moteino GND
Once you've got all of this hooked up, you'll find a "VP2.ino" sketch in the DavisRFM69 library's Examples directory.  Open that up in the Arduino IDE and select Control-U to compile and download it to the Moteino (be sure to tell the Arduino IDE that you are using an Arduino Uno).  Open up the serial monitor, set the baud rate to 19200 and "Newline" as the line ending.  You should see... absolutely nothing.  Nothing that is, until you type in some (uppercase) commands like "LOOP 1", "TEST", or "STRMON".  The Moteino will respond exactly the same way as a real console would.  Hook it up to Cumulus (with the "Use Data Logger" option unchecked) and revel in a job well done.

I'm off to a decent start here but there is still lots to do.  Here are some things on my list.

  • integrate the real time clock functionality
  • put in the frequencies for consoles in Europe and other regions
  • add some more useful commands like PUTRAIN so you can tell this thing how much rain you've had this year that it can keep a running count of afterward
  • add in some kind of EEPROM functionality so it retains settings like total rainfall between power downs.  Conveniently, there is an EEPROM on my yet to arrive three dollar clock module just begging to be used.
  • make the reception more resilient.  There are some odd things going on here: I get a spurious interrupt on startup (setup issue?) and when I move the board around (need a pulldown on the interrupt line?)  Reception is also poor during the day.  Might be that I just need to adjust the clear channel RSSI value a bit to accommodate more background noise.
  • integrate the real time clock functionality (daily / weekly / monthly / year highs and lows)
  • etc
Anyone that has done much coding on the Arduino gets to know memory limits.  The Moteino has double the RAM and EEPROM than the standard Arduino UNO, so I've got a whopping 2K of RAM and 32K of Flash to play with.  So far I have used roughly half of each, so I've got some headroom.  I haven't done much to try and streamline any of the code besides a few const arrays here and strings stuffed in the F() macro there, so I am sure there is room for improvement if things get really tight.

With this project, I think I've hit the goals I set out to achieve: come up with something functional, inexpensive, and with a low barrier to entry so that other people can dig in.  Have a new feature you'd like to see?  Send me a pull request on Github.  Want to develop a plug in circuit board (kind of like this) with all the dodads attached to lower the barrier to entry even further while making a few bucks on the side?  Be my guest.

I'll wrap this up by saying that the Davis guys make good stuff.  The ISS is very well put together and I love that the thing is solar powered with battery backup.  I've had mine for years and it has been operating trouble free on Battery #1.  The display on the indoor console is nice, big, and easy to read and it also just sips on battery.  I'm guessing that 99.9% of Davis' customers are going to continue buying standard Davis stuff off the shelf despite whatever I come up with here.  Anything I play around with  is ony going to be picked up by that other 0.1% of enthusiasts.  And it is those enthusiasts that are not only going to continue buying Davis products but will recommend them to their friends as long as they are kept happy.  If Davis pulls another stupid Firmware 3.0 thing, they'll do more damage to themselves than I ever did.  I hope they continue documenting their communication protocol and maintaining accessible expansion ports in their current and future products.  That would be a win-win for everybody.

With that in mind, it seems as good a time as any to unveil the new profile picture.
Be Nice

Friday, January 3, 2014

More Than One Way to Skin a Cat

I know what you're thinking: "DeKay hasn't posted in a while.  Maybe he's dead?"  Fear not.  For while the frequency of my posting has dropped, I remain very much alive.  I've just been a little busy at work.
Thankfully the holiday season rolled around and I was able to take a couple weeks off. I took that time to relax and catch up on all the things normal people like to do: enjoy some good food, visit with friends, and hack on microcontrollers with integrated RF transceivers.
This bad boy is the Moteino from LowPowerLab. It is an Arduino clone with a 3.3V Atmega 328p that integrates a HopeRF RFM69 RF module. And let me tell you, this little module rocks. They are small, cheap, and readily available. Felix from LowPowerLab is getting tons of range out of the high power variant of these little guys.  They are easily programmed over a SPI bus so you don't need special hardware like you do with TI's CC1110, and cheap versions operating in the 900 MHz band are easy to get unlike TI's CC1101.  And recently, interoperability with the RFM12B popularized by the JeeNode has been achieved.

Now the RFM12B deserves a bit of discussion.  Like the RFM69 module, it is small, cheap, and easily programmed.  But it sucks in a number of ways.
A small FIFO means that you better have something real time or close to real time listening to the module. Otherwise its buffer overflows and you lose data. Something like an Arduino works well because it isn't doing other stuff like running an operating system. But worse than this is the hardcoded sync bytes. This pretty much limits the RFM12B to talk to other RFM12B's. And that ain't cool.

The RFM69 fixes all these problems and adds some nice features while it is at it:
  • it has a 66 byte FIFO, a fully programmable sync pattern up to eight bytes in length, and a variable length preamble
  • address checking and CRC validation is optional
  • it supports OOK, FSK, or GFSK
  • it can do AES encryption
  • other stuff
In other words, it looks like it can do what other transceiver modules can do and more.

Intrigued, I picked up a Moteino with integrated RFM69W module and a couple standalone modules while I was at it.  Then I got busy (see above).  Then I got some free time (see above).  Then I started hacking (see below).

Devoted readers of this blog will know that I am strangely compulsed to receive wireless transmissions from my Davis weather station using all kinds of strange hardware platforms.  So why stop now?  I've been looking for a platform like this that is available and easy to use so that more people can take a crack with this stuff (it isn't like you can walk into a Toys-R-Us and pick up an IM-ME anymore).  I also don't want to be glued to the Arduino for everything.  While it is great for running as a remote node, I like the idea of something like my BeagleBone Black connected directly to a transceiver module running some home monitoring and control software (HouseMon?) with Internet connectivity.

But, baby steps.  What I decided to start with was my RFM69 equipped Moteino and the LowPowerLab RFM69 Arduino library.  Because once you've got the wireless protocol figured out, hard hard can it be?  Turns out, harder than I thought.  But after slamming my head against a brick wall a time or two, I got this.
This is the data from my Davis weather station out in my yard and received on my RFM69-equipped Moteino.  The first field is the channel number, the data is the eight data bytes sent by the ISS on every transmission (see here for what those bytes mean), RSSI is the Received Signal Strength Indication, and the CRC is calculated from the first six bytes in the data packet. Every packet is sent with a CRC in the seventh and eigth bytes, so those two values will agree with the calculated CRC if the packet is good.  I've got some problem somewhere where the first packet I receive is bad, as is any packet on Channel 42.  But those are details.  I'm off to a decent start here and I'm sure I can sort those issues out in time.

Want to check it out?  My DavisRFM69 library is on GitHub and the data above was collected using the ISSRx sketch in the Examples folder.  The approach that I've taken is to subclass my driver from the LowPowerLab one, overwriting just a couple of methods that are specific to this application (namely the initialization and the interrupt handler).  That reduces duplication and lets me automatically pick up any improvements Felix incorporates into his main library.  I have made a couple minor tweaks to his main library for some register definitions that I've issued a pull request for.  Hopefully these will be incorporated into his main library soon.  In the meantime, you'll want to use my fork of the library, so everything will compile cleanly.  This and more is all in the README.

This work can be taken a lot further:
  • do a full emulation of the Davis indoor console for use by weather software like WeeWx and Cumulus
  • hang an RFM69 directly off a BeagleBone Black and write a driver to work under node.js
  • why just receive?
Pull requests gladly accepted  :-)

I'm glad to now have this stuff under my belt.  I could have taken the entire break to just sit in front of the TV eating fruitcake.  That would be easy because My Lovely Wife makes a rocking fruitcake.  But what I really find satisfying is that sense of accomplishment when you set a challenge for yourself and make it happen.  This was a good way to kick off 2014, and the year is just getting started.

And don't get me wrong: I had lots of fruitcake.

Sunday, April 21, 2013

Dead End

It is springtime here at Mad Scientist Labs, though you would never know it.  Mother Nature has decided to totally screw us out of our spring.  Snow still covers most of our yard.  I took Buddy for a walk today wearing an old parka and nearly froze to death.

"That sucks", you must be thinking.  You would be thinking wrong.  The weather sucking outside lets me play around inside with little gadgets and gizmos.  The focus of my efforts the last couple of weeks has been trying to get my Stellaris LM4F120 (bought for the princely sum of $4.99 with free shipping) to play nicely with a little RF1100SE CC1101-based board I picked up at Deal Extreme.  The goal here, of course, is to pick up the signal from my Davis Weather Station's Outdoor Sensor Suite (ISS).  In other words, I want to get the Stellaris to tell me how cold it is outside without me having to walk the dog to find out.
Stellaris On Left, RF1100SE On Right, BMP085 Module Hanging In Mid-Air In Middle
"Should be a piece of cake", thought I.  Long ago, I had figured out the mysteries of the ISS wireless transmissions, going so far as to get my Pretty Pink Pager to work as a portable weather station.  Turns out it wasn't so simple.  It might not even be possible.

I've burned a couple of weeks on this trying to get it to work, and when it works, it barely works.  I was seeing all kinds of strange things, and I didn't know if it was my unfamiliarity with the Stellaris board or with the poorly documented RF1100SE CC1101-based module.  At times I would get a bunch of garbage.  At times I would get nothing at all.  But now I think I've come to a conclusion: if you are thinking of picking up one of these RF1100SE boards to work at anything other than 433 MHz, don't to it.  Simple as that.  And here's why.

This picture is taken from the CC1101 datasheet for using the module at 433 MHz.
Recommended CC1101 Schematic for 433 MHz Operation
And this picture is taken from the same datasheet for using the module at 915 MHz, the frequency the ISS operates at.
Recommended CC1101 Schematic for 915 MHz Operation
The two circuits are very different in the connection between the CC1101 and the antenna.  The 433 MHz recommended circuit has four inductors between the chip and the antenna, while the 915 MHz recommends either six or seven inductors between the chip and the antenna.

Now, here is a picture of my module taken from the DX website.
RF1100SE Module With Inductors Circled
Count the inductors circled in red and it becomes pretty obvious pretty fast that these RF1100SE modules are designed for 433 MHz operation.  I feared that this might be the case, and I hoped that getting a proper length antenna for 915 MHz operation might be enough.  It is not.

This is the output from my Stellaris board talking SPI to the RF1100SE with a 915 MHz antenna attached.
This Sucks
This is what I get after tweaking the register settings to within an inch of their life.  You can see all kinds of things here.
  • My frequency error (a two's complement value shown on the very right" is about as low as you can get, so I'm dialed right on the signal.
  • My RSSI value is terribly low: I'm barely seeing the signal above the background noise.
  • As tweaked up as I am, I'm still seeing a lot of bit errors.  Those packets starting with 68 and 78 are corrupted - all of the packets from this ISS should end with zero.
  • You can't see it here, but many times the radio misses the packet altogether.
In the process of banging my head against a wall, I went looking for other people using this RF1100SE module at 915 MHz and found none.  People have tried but I haven't come across anybody reporting success.  And this is why.  The CC1101 needs a very different circuit between the 433 and 868 / 915 MHz bands.  This board is designed for the one, and getting it to work at the other isn't going to happen if you expect any kind of performance out of it.

What to do, what to do?  Not quite sure yet.  Any kind of CC11xx module that doesn't already incorporate some kind of processor on it (8051, Atmega) and is cheap seems to be designed for 433 MHz.  What I'm looking for is just the RF chip itself that I can drive directly with the Stellaris board without having to fuss with two different toolchains, processors, etc.  I might have to consider a non-CC11xx type chip instead, like something HopeRF based (like this one, or maybe this new one out soon).

Anyway, this problem was just consuming me for a while.  It wasn't the "not working" part, it was the "why isn't it working" part.  I gave up several times trying to get this working, only to keep going back to it when I realized I couldn't stop thinking about the problem.  Now I know what's wrong.  I haven't been defeated.  I just realize that victory with this module doesn't look to be possible, and I'm OK with that.  You can't do what you can't do.

Mad Scientist Labs: wasting weekends so that you don't have to.

Saturday, January 5, 2013

2013: The Year That Will Be

In the novel Flowers For Algernon by Daniel Keyes, a man named Charlie Gordon with an IQ of 68 is selected to have an operation to increase his intelligence following a similar surgery on a laboratory mouse.  The surgery works but the results turn out to be only temporary.  The story is told through the diary that Charlie keeps as his intelligence shoots up, only to later fade away.

This was one of those books that I was assigned to read in grade school, and it is one of the few that really stuck with me.  It has played on my mind more and more in the last few years.  I am at the age where my mind and body are not quite where they used to be.  I have worn glasses since I was around 10 years old but recently had to go to progressive lenses.  I sometimes have trouble putting a sentence together because the words just don't come (oddly enough, this seems to be restricted to nouns: persons, places or things).  My memory is terrible.

I am stronger than I have ever been but I realize that that won't last forever.  At some point I will go from gaining strength, to maintaining it, and then to losing it.  Aches and pains now stick with me longer.  I think I might have a touch or arthritis in a joint or two.

What to do?  Well...
“The path I choose through the maze makes me what I am. I am not only a thing, but also a way of being--one of many ways--and knowing the paths I have followed and the ones left to take will help me understand what I am becoming.”
The key in the quote above is "The path I choose...".  Every morning when you wake up gives you the opportunity to make a new and better choice.  I think I let a few too many years just kind of slide by.  A friend of mine likes to say that "Youth is wasted on the young."  Indeed.  But no more.
"I'm like a man who's been half-asleep all his life, trying to find out what he was like before he woke up."
I now regularly challenge myself both physically and mentally.  I want to use my time to reach my goals and get things done.  I love the internet for letting me expand my horizons into areas I never knew existed or thought I'd be able to do.  For example, I was never much of a cook but thanks to Sous Vide and a little motivation, I can whip up a pork tenderloin or turkey breast that is better than any restaurant in town.  And with a few simple techniques, I can serve that with a slice of bread so good it would make you cry...
... Like This Blue Cheese & Toasted Sesame Seed Bread
Now more than ever, I feel at the top of my game.
“I'm living at a peak of clarity and beauty I never knew existed. Every part of me is attuned to the work. I soak it up into my pores during the day, and at night—in the moments before I pass off into sleep—ideas explode into my head like fireworks. There is no greater joy than the burst of solution to a problem. Incredible that anything could happen to take away this bubbling energy, the zest that fills everything I do. It's as if all the knowledge I've soaked in during the past months has coalesced and lifted me to a peak of light and understanding. This is beauty, love, and truth all rolled into one. This is joy.”
Accomplishment.  That is now where it is at for me.  I want to learn new things, make things work, and get things done.  My biggest problem now is finding the time for all of the things I want to do.  I have to force myself to focus instead of taking off on a dozen different tangents.  I realize that I'll never get good at anything new if I spread myself too thin.  I also have some upcoming obligations that will dig into that time even further.

In my last post, I reviewed the goals I set for myself in 2012.  I worked hard to try and hit those goals, and the ones I missed weren't for lack of effort.  The list itself was an excellent motivator.  Given that, it is time to set myself a new list for 2013.  Note first, however, that these are not resolutions along the lines of "I will quit smoking next week."  I wrote last year how resolutions of that type just show that you suck.  If there is something you want to do, and you can do it right now, then why aren't you doing it right now?

Working Out

I set a stronger squat as one of my goals for 2012 and this is a very worthwhile goal to carry into 2013.  It lags well behind my deadlift that I am quite happy with.  The squat is one of those standard strength markers that people measure themselves by.  Like I realized last time, I must continue to build up my bottom position strength to achieve some decent gains here.  I'll carry over the same goal that I set for myself last year, which was...

Goal #1: Squat 280lbs for a single rep below parallel.  285lbs - 290lbs would be even better.

I posted this video last year, and it is no less awesome this year.

 I get mistaken for Tom Platz all the time

I'm happy with my Military Press but I have a fairly crappy Bench Press.  I'll continue to work on the latter, but really it is the lift I care the least about.  It can also be pretty tough on my shoulders, so I'd rather maintain good shoulder health instead of trying to wring a few extra pounds out on this one.


I mentioned last year that work isn't really something I talk about on this blog so I will continue not to do so for the most part.  However, it is worth mentioning that I am often called to put in substantial amounts of extra effort as required.  This leaves me with less time to play around with the stuff I talk about here.  Hopefully things won't be too crazy this year in that respect.


One of the things I wanted to get done last year was some basic home monitoring set up.  If I head off for a weekend in the wintertime, it would be nice to know that my furnace is still running without having the neighbor to check in on it daily.  I have myself a couple of JeeNodes that I have been playing around with but I haven't had enough time to really dig into them and get them talking back to, say, my router which could then push the data out to someplace like Cosm (nee Pachube).  So this goal remains the same as I set it last year...

Goal #2: Get some basic home monitoring going by 2013.

For some time I have also been playing around with my Davis weather station.  I wrote a little while ago that Davis is actively working to close off this platform and prevent me from getting at my own data.  I wasn't very happy about this.  Their actions have made me even more determined to implement my own alternative receiver.  My current platform is a Pretty Pink Pager, and while the cool factor in this approach is off the charts, it isn't a solution that lends itself to widespread use by others.  I want to take an approach that is more accessible.

I'm mulling over a couple of different options for an alternative receiver.  Right now I'm liking PanStamps.  These are Arduinos with an on-board radio receiver chip compatible with that used in the Davis console.  They also seem to have a very nice low level library that would let me work with the radio chip at the level I need.  Another consideration is using a Ciseco Eve Alpha.  This has the benefit of being multi-radio capable but they aren't calling it "Alpha" for nothing.  Either way...

Goal #3: Reverse engineer the Davis frequency hopping algorithm and implement it on an independent platform.

and while I'm at it...

Goal #4: Get a skeleton driver for the Davis console into MESS.  This one won't make much sense unless you've read my earlier post on the subject.

I consider #4 to be like a KickStarter-like "stretch" goal.  It would be great to get this done, but it will take a lot of work to make it happen.  The AVR emulator in MESS right now will need some enhancements to work with the specific chip used in the console, and I'll have to implement a sufficient level of functionality within the emulation to make it worthwhile submitting.  However, doing so would enable a lot of other people to dig into the console and figure out what makes it tick.  It would also be fun to be part of the MESS project itself.

Wrapping Up

That should do it in terms of goals but I won't be restricting myself to just those.  There are still a lot of other things I want to get done this year, such as
  • I have a bunch more landscaping I want to get done around here.  There is an area around our walk-out deck that needs to get finished up that is going to take a ton of work.
  • I want to grow more of our own food out here.  We have a great garden that we get a lot of wonderful fresh vegetables from.  Last year we planted some grapes and kiwis and are looking forward to their fruit this year.  This year I'm thinking strawberries and more haskaps at a minimum.  Deer, rabbits, and other critters remain our nemesis.
  • I'd like to learn a little OpenGL and maybe play around with more emulation stuff.
These things lead me to the same goal I set myself last year.  I met this goal, but it is a goal that can never be considered complete.

Goal #5: Keep this blog going, share the adventure, and enjoy the ride.
“Strange about learning; the farther I go the more I see that I never knew even existed. A short while ago I foolishly thought I could learn everything - all the knowledge in the world. Now I hope only to be able to know of its existence, and to understand one grain of it. Is there time?”
And I also want to re-read this book.

Tuesday, January 1, 2013

2012: The Year That Was

Today is the first day of 2013.  I sit at my kitchen table in front of my laptop with a compile of MESS going in the background.  Like many people today, I sit here finding myself thinking back over the last 365 days.

Certainly, the biggest thing that stands out for me was the passing of our dear dog, Abby.  She was a constant and faithful companion for her eleven and a half years, and was a far better dog than we could have hoped for.  I think about her all the time.
Abby:  January 2001 - August 2012
We now have Buddy in our lives.  Buddy is quite a different dog than Abby was, and certainly a lot more will be written about him in the days and weeks ahead.  He has scared us with some health issues and he has entertained us with his puppy-dog antics.  I look at him each day and wonder what kind of dog he'll turn out to be.  The one thing we learned from Abby is that this is pretty much impossible to predict.
Impossible, Except To Say That He Is Going To Be Trouble
Anyhoo, I sat down and set some goals for myself in 2012 around this time last year.  This was actually a very useful thing to do.  I like to cross things off my TODO list, and I found that putting this list out there was a pretty strong motivator in getting me off my butt.  I need to mull this over a bit still, but I certainly want a new list going in to 2013.  In the meantime, why not take a look back while waiting for the compile to finish to see how I've done in 2012?

Working Out

Goal #1 Hit: "Deadlift 405lbs for a single rep. That's four wheels a side. This goal should be doable. I just have to do it."

It wasn't long after I wrote this that I wrote some more about how much it sucked to have written that in the first place.  In a nutshell, why was I talking about something I could do rather than just going and doing it?  So I went and did it, and made it with pounds to spare.  The year went by and my deadlift continued to improve, culminating with this bad boy.
435 lbs, Baby

Goal #2 Missed (barely): "Squat 280lbs for a single rep below parallel.  285lbs - 290lbs would be even better, as this would be 2x bodyweight."

This was a case of "close, but no cigar".  I hit a 275 lb squat in May, a weight that was indeed 2x my bodyweight at the time.  But 275 lbs is not 280 lbs, so I did not reach this goal.  It wasn't for lack of trying though.  I hit that 275 after several runs of Sheiko, an extremely high volume routine designed to bring your numbers up.

This is a worthy goal to keep shooting for.  Unfortunately, a long stint in Mexico has caused my strength to drop off sharply and my squat seems to have been hit the hardest.  It is coming up again since my return, but a second trip is on the horizon and back down I'll go.  Such is life.


Goal #3 Missed: "Get some basic home monitoring going by Christmas 2012."

I put some considerable time into this one but didn't hit it either.  I ran into a couple of snags.  First was the aforementioned  long stint in Mexico.  Second was that my WRT-54G router decided to start acting flaky as I sat down to implement some of this stuff.  The idea was to hack the router so I could plug a Jeenode into it to act as a receiver to other remote Jeenodes around the house.  I added the serial port and SD card mods, but my attempts to get Optware installed on the thing were randomly failing and I couldn't figure out why.  I had suspected bugs in the custom firmware and kept trying various options to see if that would make the problem go away.  Then the router fell over altogether so I bought the mighty Asus RT-NT16 to replace it.  That router now runs Tomato by Shibby with Optware installed just fine.

Again, this is a worthy goal to keep pursuing in 2013.

Goal #4 - Hit: "Get a standalone receiver picking up the transmissions from a Davis outdoor sensor suite by the end of 2012."

I hit this baby back in April and was pretty psyched at having done so.  After that I had to spend a bunch of time figuring out what those transmissions actually carried.  First I sorted out rain and followed that up with outdoor humidity.  The deciphering of all the weather related parameters is now laid bare on github.
This Took Forever To Get Right
I want to keep plugging away at this stuff going into 2013.  I wrote last time about how Davis' recent antics were making me pretty grumpy.  It is my data, and I don't want people trying to take it away from me.

Goal #5 - Hit: "Keep this blog going, share the adventure, and enjoy the ride."

I put up 25 posts in the past year.  Not incredibly prolific, but far from abandoned.  I wrote some stuff that made people laugh, and I wrote some stuff that made people cry.  The goal of this blog was never just to put something out there.  If it wasn't going to be interesting or entertaining, it wouldn't be worth the effort.  Hope you think that it has been.

Have a good 2013 from all of us here at Mad Scientist Labs.  Enjoy the ride

Friday, December 28, 2012

Making A MESS

I loved video games growing up.  Pacman and Donkey Kong reigned supreme with the general populace, but for me it was Defender that grabbed me hardest and wouldn't let go.  I remember once having a single quarter burning a hole in my pocket.  I jumped on my 10-speed and biked the couple of miles to the nearest arcade where that classic Williams' game awaited me.  I dropped in my quarter and suddenly became the pilot of that ship.  With its frantic gameplay and excellent graphics, I could have played that game for hours.
My First Love
Unfortunately for me, the game was fiendishly difficult and all I had was one quarter and no skill.  It was only a matter of moments before my butt was back on my bike seat and I was headed home.  I got a lot of exercise that way, one quarter at a time.

Today our computers have processors easily 1000x more powerful than the Motorola 6809 running at 1 MHz at the heart of Defender.  All of that processing power has been put to work in projects like MAME (Multiple Arcade Machine Emulator) and MESS (Multi Emulator Super System) to deliver cycle-accurate emulations of the processors in these old arcade wonders.  These projects do more than just the emulation of the processors: many games have perfect sound and video emulations as well.  Dig up the original ROMs that were in these games, and you aren't playing just some reasonable facsimile of the game.  You are playing THE game.

There are many things I love about the MAME and MESS projects.
  • They are intended as a preservation project rather than an emulation project.  The emulation is an accurate representation of the hardware, sans a few things like emulating transistor level stuff like programmable logic chips and a couple other exceptions.
  • They are open source.  As a matter of fact, MAME and MESS joined forces a short while ago and now their code lives in a common repository.  Want to know how this stuff works?  Grab a copy, poke around, and try compiling it yourself.  It isn't hard.
  • One look at their SVN Repository shows the project is very active.  There are additions and fixes going in all the time.
  • The scope is enormous.  There are around one hundred processors and a similar number of sound chips emulated.  Thousands of games and old computer platforms are emulated.  Thousands.
Now let's change gears.  Something like three years ago, I bought myself a Davis weather station.  The undocumented expansion port at the back of the thing nagged at me.  With some knowledge of electronics and some helpful documentation from the manufacturer, I was first able to figure out how to interface to the weather station console with nothing more than a commonly available USB to LVTTL serial adapter.  A while later I sorted out how to build my own datalogger to store weather data for periods when the console wasn't connected to a computer.  These two projects were cheap and easy to put together.  They also faithfully duplicated the expensive equivalent sold by the manufacturer.

Other people then expanded on this work with projects of their own.  One person developed a way to connect the console directly to a WRT54G router and upload weather data without a computer in the middle.  Another developed a wireless interface with commonly available XBee modules.  One or two enterprising folk actually started selling low cost third-party designs to what could only have been a fairly limited audience of enthusiasts.  This gave the enthusiast crowd a low cost option for an interface to their console for their own data while the mainstream crowd could always buy a console interface off-the-shelf if they wanted one.  Everybody seemed happy.  Or so we thought.

Apparently, Davis is putting the hammer down.  There is a new version of the console firmware that prevents the DIY interfaces from working.  This is a pretty wrong-headed move if you think about this for a bit.  They are pissing off their biggest supporters by deliberately breaking something that would amount to a small fraction of their sales of these units.  These big supporters are the same people who recommend these systems to other people.  These supporters also buy a lot of Davis hardware in the first place.  So maybe, just maybe, Davis could lose money rather than make money by doing something like this.  Also consider that what Davis can do to differentiate their logger from a DIY version is very limited.  Efforts are already underway to see if the new logger can be reverse-engineered.  Given that the SPI interface between the logger and the console is easily sniffed, I don't expect this reverse engineering effort to take very long or be difficult to emulate.  Then we'll all be back to where we started, but with incompatible hardware versions floating around.  Brilliant.

This situation doesn't make me very happy.  Not happy at all.  In fact, it makes me kind of angry.  Angry enough to wonder if there isn't something I can do about it.

Maybe there is.  Followers of this blog will know that I've built my own receiver using a child's toy and can decode all of the weather related information transmitted from the ISS (Integrated Sensor Suite) outdoor unit.  There was still some work to do though: the ISS transmits a Frequency Hopped Spread Spectrum signal, and I was able to determine the hop sequence so I could write some code to follow its transmissions continually.  However, I only figured the hop sequence out for one of the eight possible transmitter IDs, and only for the version sold in North America.  Figuring out the hop sequence for the other IDs and the other regions would be a lot of work that I didn't feel like doing.

I then considered reverse engineering the firmware by disassembling it.  This is not a lot of laughs either.  The disassembler doesn't know code from data so you've got to tell it which is which, meaning that you need to figure that part out first.  Then you aren't able to actually run the code and get an understanding of how it works.

That made me think about emulation.  I had thought about writing my own CPU emulator for the Atmel ATMEGA128 processor running the console.  I had done this kind of thing before, actually.  I wrote an 8086 emulator in x86 assembly language for the old Retrocade arcade game emulator that brought Q-Bert to the party.  I learned a ton in doing so, but one of the biggest things I learned is that writing a CPU emulator is HARD.  Very hard.  The initial coding isn't so bad.  It is debugging it that is a killer.  A very simple error in the emulator can totally screw the emulation up, and often these errors are very difficult to find.

I didn't want to spend a ton of time writing my own emulator so I started poking around.  Atmel's own AVR Studio looked promising, but it was more suitable for code development than reverse engineering.  I also couldn't see how I'd be able to hook in functionality to the processor emulator for things like the console's keyboard.  Then it occurred to me that MAME/MESS might have a CPU emulator I could mess with.  And what do you know?  Looks like a lot of work has gone into an Atmel 8-bit CPU emulator that had potential for use in reverse engineering how the Davis code works.  I checked the SVN Repository which also showed that there were a bunch of recent fixes going into this emulator since the last official release.  And finally, I was impressed by the statement at the top of the code stating that this emulator could faithfully run code written for the "demo scene".  These demos (check it out) are written to push a platform to its absolute limits using every trick in the book.  The fact that demo code could work on this emulator spoke well of its accuracy.

Check it Out

Looking at the avr code showed me that the emulator itself ties in all kinds of MAME / MESS dependencies for loading firmware, enabling a debugger, etc.  I would have to work around all of that if I wanted to use this CPU emulator in a project of my own.  Then I realized that I was looking at things backwards.  Why consider this a project of my own?  Why not see if I could make this a part of MAME / MESS itself?  I had a week off between Christmas and New Year, so why not see how far I could get?
Click To Embiggen
What you are looking at is MESS running the actual Davis firmware in its debug mode.  The window in the upper left shows a disassembly of the code at the current program counter location as well as other good stuff like CPU register values.  The right side shows a view into the firmware, RAM, and EEPROM memory spaces.  I am now able to single step through the code and see exactly what it is doing.

Getting this far wasn't all smiles and sunshine.  MAME / MESS undergoes constant churn, so an accurate tutorial on how to add a new driver to the code base doesn't seem to exist.  And while the AVR emulator seems to be quite accurate, that doesn't necessarily mean it is complete.  Last night I had to code up the emulation of the ELPM (Extended Load Program) instruction.  That ended up being pretty straightforward based on the existing infrastructure, but it was just another hoop to jump through.  Never mind that there also seem to be some compiler bug triggered by MAME that had me doing nothing but segfault when I started out.

How far will I get with this?  We'll see.  It is certainly a powerful tool for deep-diving into the belly of the console to figure out how it works.  Working against me is that I have some stuff coming up that will severely limit how much time I can spend on this stuff.

What will I do with what I find out?  That is a topic for a future post assuming all goes well.  But it is cool.

Thursday, May 10, 2012

It's Not The Heat...

Last time around, I wrote up how rain information is sent from the Davis Weather Station Integrated Sensor Suite (ISS) to the console sitting on my kitchen counter.  That left me with one more piece of weather data to figure out: humidity.  Evil humidity.

I had been puzzling over humidity for weeks before I decided to switch to rain.  The data seemed to make sense for a while and then I would get a datapoint that made no sense at all.  I was pretty certain that humidity was sent in message 0xa0 from the station, and I was also pretty certain that the data would be contained in just one byte.  Sounded reasonable because humidity is displayed on the console to the nearest percent.

Humidity was stumping me until I got serious last weekend and wrote a Python script to collect STRMON data continuously while my DIY datalogger dutifully captured the console's interpretation of the data.  Then I could make some plots and figure out what was going on.  Like this:
Click To Embiggen
It was quickly apparent in looking at this data that two bytes were involved in encoding humidity.  You can see as the humidity drops, Byte 3 cycles through a couple times while there are some discrete steps in Byte 4.  The weird thing was that Byte 4 isn't flat as in a high byte, low byte kind of thing.  There is jitter on it that I didn't expect.

Puzzled, I thought some fresh eyes on the subject might help.  So I posted this plot and a link to the data I collected on this wxforum post.  It was user Kurgan for figured it out for me within eight hours of my post.  He da man!  As he wrote in his post:
If you take byte 4 and convert it to binary, you will see that the values is has assumed in the test, which are 24,25,26,27 then 40,41,42,43 and then 56,57,58, 59 are made of two parts: the first four bits are fixed (the big steps), and the last four are variable (the jitter).


 24 25 26 and 27 have the first four bits set at 0001
 40 41 42 and 43 have the first four bits set at 0010
 56 57 58 and 59 have the first four bits set at 0011

So, you have to add the first four bits of bit 4 to byte 3 to get a 12-bit number. Try to plot this 12-bit number and compare it to the hum graph.

The last four bits of byte 4 are something else, or are random noise. I don't know.
I thought he had it wrong the first time I wrote up this post but forgot that the values in Byte 4 as written above had been converted to decimal.  Convert them back to hex for his explanation to make sense.  Next step: make a pretty plot.
Now That's Pretty
That is bang on.  The calculated value works out to the humidity displayed on the console * 10, so a calculation of 750 = 75% RH.  The lines don't sync perfectly because the timing is a bit off and the logger logs in intervals of 1 minute vs irregular intervals on STRMON.

As I wrote in a post that followed, there are other mysteries are buried in the data.  For example, here is data from message 0x90 (144 decimal). 
39.2    144   16   1   47
90.5    144   16   1   63
141.7   144   16   1   79
193     144   16   0   95
244.2   144   16   1   111
295.5   144   16   0   127
346.7   144   16   2   143
398     144   16   3   159
449.2   144   13   3   18
500.5   144   13   3   34
The columns are elapsed time in seconds, message number, byte3, byte4, and byte 5.  I deleted the wind and crc data.  The numbers in byte 5 jump up and down mostly in step sizes of 16 before resetting and then starting over again at some different value.  Looks like a counter of some kind.  And why is byte 4 in this same message just bouncing between 0 and 3?  Byte 4 in the rain message (0xe0, 224 decimal) does this too.  I'm also told that the ISS sends a reading of its battery voltage to the console  That is also hidden away somewhere in all this data I've collected as well.  And then of course there is data from the sensors I don't have: solar, leaf wetness, and soil moisture.

But I don't care too much.  I'm getting all the weather related data my ISS can send me.  Figuring out other stuff down the road would just be a bonus.  Now it should be a relatively straightforward matter of writing some code for the Alternative Davis Weather Station Receiver and having its values exactly match the values displayed by the console.  Cool.

If only I had more time to actually do this...

Sunday, May 6, 2012

Here Comes The Rain

Today's weather at Mad Scientist Labs: Cloudy and Cold.  Wet and Miserable.  Rain, rain, and more rain.  In a word: perfect.
 This Takes Me Back...
As a (hopefully) interesting aside, The Cult used to go by the name of the Death Cult, and before that the Southern Death Cult.  I guess there is something to be said for brevity.  There's also something to be said for picking a name palatable to all those moms who buy music for their kids.  Music was on "records" or "LPs" back in those days.  And I walked to school uphill both ways.

With weather this lousy, going outside and taking care of various chores out there just isn't going to happen.  This gives me a perfect excuse to take it easy after a very busy week and do some stuff I enjoy doing.  What might this be, you ask?

Well, moments ago I pulled a good looking loaf of Jim Lahey's "Pane all’Olive" out of the oven.  This is a fancy way of saying No Knead Olive Bread.  How'd it turn out?
Damn Good, That's How.  And I Don't Like Olives.
I've been a big fan of his No Knead technique lately, since the taste to effort ratio is astronomically high.  Here is a link for this bread and two others that you might want to try.

I've also spent some time today reverse-engineering the STRMON data spit out from my weather station.  I had all of the weather-related parameters sorted out now with the exception of rain and humidity.  It was raining this morning, so that gave me the chance to figure rain out.  And figure it out I did.
Yay Me
In fact, I wrote up a bit of Python code to help me figure this out.  It grabs the STRMON data, timestamps it as the number of seconds since the script started, converts everything to decimal so the math is easier, and then writes the data as a CSV file.  I don't show the checksum bytes in the last two bytes of every message because that would just pointlessly make the file bigger.
The rain was coming down pretty good and I watched this stuff scroll across the screen.  When the console added another hundredth of rain, I wanted to see the message that caused it to happen.  Let's cut the irrelevant crap out of the above.
Message 244 is 0xe0.  I had suspected before that this message held rain, but I expected to see it in the second last field above.  What I thought would happen was the console would send a "1" any time the rain bucket tipped.  But then I saw this field bounce around between 1 and 3 randomly.  It turns out I was off by one byte.  Rain is actually in the byte before (40... 41).  What the ISS does is send a running total of rain between 0 and 255.  In the example above, it would continue to send 41 until the rain bucket tipped again.  Then byte 3 (counting from 0) in message 244 (0xe0) would change to 42.

This is smarter than what I was thinking.  My method could have had the console miss a count of rain every now and then if there were interference.  But by sending a running total, this can't really happen.  It is up to the console to track the changes in this byte.

I haven't figured out what those values bouncing around between 1 and 3 are.  I'm suspecting they are just noise from a sensor that I don't have and isn't connected.  Or something like that.  I plotted this field against the rain plot above and there was zero correlation.  In other words, I don't care.

One thing I discovered along the way and that I do care about.  In the example above where the STRMON 0xe0 message changes from 40 to 41, the change doesn't show up immediately on the display.  In fact, it didn't show up until the next 0xe0 message came along ten seconds later.  Interesting, and good to know as I continue my hunt for humidity.

"I'm an excellent driver."

Sunday, April 8, 2012

Achievement Unlocked: IM-ME Weather Station Receiver!

Ever have days like this?

I try to make it a habit not to talk about my job on this blog, but let's just say that the last three weeks have been pretty challenging.  Something came up late this week that made it look like a bomb would go off and blow my weekend all to hell.  I was certain this was going to happen and had resigned myself to busting ass this Easter long weekend instead of taking it easy.

But the bomb didn't go off.

Now, not only do I get to enjoy a long weekend, but I've also tacked on an extra day to give myself four days off.  That is three days longer than last weekend and four days longer than the weekend before.  To make things even better, the weather outside has been crappy.  That gave me the excuse I needed to do whatever the hell I felt like this weekend.  Manna from heaven, baby.
Manna Can Be What You Want It To Be
All this is good because I had a few things on my list that I wanted to get through.  The first item on the agenda was to eat a whole hell of a lot of cheesecake.  I've been trying to lose a few pounds over the last couple of months, and I've kept the idea of eating a cheesecake in mind the whole time as a reward.  My Lovely Wife doubled the recipe, so my cheesecake is the right half.
Before - My Half Is On The Right
After - Mission Accomplished
Now I'm no Cheesecake Master, but I did what I set out to do and packed this bad boy away in an evening.  And I was some full.  The detailed stats on this gluttony:
Weight: 1139 grams (2.5 lbs)
Calories: 3394 grams (not counting various toppings)
Fat: 188 grams
Carbohydrates: 332.6 grams
Protein: 84.3 grams
What I'm quite amazed by is that I stepped on the scale the next morning and my weight was exactly what it was the day before, despite all this food and despite pounding back a ton of water.  Yay me.  My Lovely Wife got the half on the left and did a bang up job of it too.  Let's just say that there isn't any of this taking up space in the fridge one day later.

The second item I had on my list for the weekend was making progress on using my Pretty Pink Pager as an alternative Davis Weather Station Console.  This was Goal #4 from the list of items I hoped to accomplish this year.  You might recall from my last blog post that  I had made some progress in decoding the data stream coming in from the Davis outdoor wireless Integrated Sensor Suite, and before that I had been able to sense the power coming from the ISS.  But it was still a pretty big step to go from what I had to getting real data.  And faithful readers of my blog (ha ha ha) might have noticed that I haven't posted for almost a month.  "Perhaps he fell into a cheesecake induced diabetic coma and died?", you may have wondered.

Well, wonder no more.  You see, the wheels of progress here at Mad Scientist Labs might grind slowly, but they grind fine.  I have been chipping away at this stuff for a long time and  finally have something worth showing.  Take a look!
Click To Embiggen
This photo shows my IM-ME perched on top of my laptop.  The STRMON output from my console is chugging away in a terminal in the background.  Note how the eight data bytes on the IM-ME display match the eight data bytes on the screen directly above.

Now I think that is pretty cool.

I first proposed that this might be possible back in January of last year, saying "It's so crazy, it just might work".  And you know what?  It works very well indeed.  Let's take a closer look.
This Took Forever To Get Right
Now here is what you are looking at:
  • Freq is pretty obvious.  Chan is the corresponding channel number from 0 - 50.
  • Data is the eight data bytes received from the ISS.  And guess what?  They come in Least Significant Bit first, and I have to reverse the bit order in my software.
  • CRC indicates if the check is OK or not.
  • LQI is the Link Quality Indicator.  The radio is kind enough to automatically collect this information for you and stick it onto the end of the packet if you ask it nicely.  It is an estimate of how easily the radio demodulated the last 64 symbols immediately following the sync word.  I don't have a good feel yet if "50" shown here is good or not.  About all I know is this is a 7-bit number, so I'm kind of in the middle of the range here.  It might get better if I clean up the offset (below).
  • RSSI is the Received Signal Strength Indicator.  Again, the radio sticks this onto the end of the packet for you.  The NOW value shows the current value and updates in a blur.  You can see that when the carrier came along, it was at a booming level well above that of the background noise.
  • OFFSET is an indication of how far you are off from the true center frequency of the signal.  It is actually the value in the FREQEST register that I grab after receiving the packet and setting the radio to idle.  This value is two's complement with a resolution of (Fref / 2^14), and that translates to around -13 kHz for the offset I have now.  I need to tune this in a bit, and I suspect that will improve the LQI number above.
I hit some funny problems as I went along.  Here are a few of the things I tripped over, keeping in mind that I don't do embedded firmware development for a living.  Or at all, for that matter.
  • I could not get interrupts to fire for the life of me.  I was triggering what should have been one from my code and... nothing.  Only accidentally did I trip over the CC1110's Global Interrupt Enable bit that is disabled by default.  I enabled that and interrupts starting working fine.
  • Davis got the bit order of the two sync bytes backwards from the manufacturer recommended 0xd3 0x91.  I had to tell the CC1110 to sync on 0xcb 0x89.
  • My RSSI reading was locking up at unpredictable times.  Turns out this is actually intended behavior.  I had my sync word detection set up to be very lax and the IM-ME was syncing on random noise.  As it says in the datasheet...
"In RX mode, the RSSI value can be read continuously from the RSSI status register until the demodulator detects a sync word (when sync word detection is enabled). At that point the RSSI readout value is frozen until the next time the chip enters the RX state."
I'm really happy to have gotten this far.  There was a lot of stuff I had to dig in to and reverse engineer to get to this point.  There is still lots to do, but I believe the remaining bits are relatively straightforward by comparison.  On my list:
  • Get this thing frequency hopping.  I just dwell on Channel 0 right now, but I know the sequence of hopping frequencies, and I have oodles of time to get from one channel to the next.  I don't know yet how the radio calculates what frequency to go to next, but I do have the raw frequencies, and that will do in a pinch.
  • Correct out the offset frequency.  TI is kind enough to publish a design note on how to do exactly this.
  • Actually check the CRC.  It always says OK right now.
  • Decode what all these data values mean and show them on a non-debug display.  I've gotten some help here and have also figured out a few things in a look at this a while back, so I think I can get a few of the basic things displayed quite easily.
  • Gracefully adapt between the 27 MHz oscillator in my IM-ME and the 26 MHz oscillator many other people have.  Some of the register settings are tied to the oscillator frequency.
  • Clean up the code.  It is a dog's breakfast right now.
I have just uploaded my code to github with a little "git push origin packetrx".  You can find the code here.  It compiles with sdcc via the magic incantation "make -f Makefile.pocketwx" on the packetrx branch.  If you happen to have an IM-ME and a GoodFET, you can then run "make -f Makefile.pocketwx install".  Like I said, the code is pretty ugly and disorganized right now, but you might find it useful if you are looking at doing a bit of fixed length packet reception stuff on an IM-ME or any other CC1110 based device.  There isn't much stuff out there on this right now that I could find.

Now I know what you are wondering.  The number of people with an IM-ME, the means to program it, and a Davis wireless weather station to listen in on are Quantity 1: me.  But let's think about this a little more.  Fine outfits like Archer Trading Post will sell you standalone ISS outdoor units that we now know we can build a compatible receiver around.  But you can't walk into a store to buy an IM-ME anymore and it isn't expandable.  That is where Ciseco comes in.  They will sell you a CC1110 based XRF module in an XBee form factor for the low low price of £9.90 and dirt cheap shipping. And yes, I already have one.  In fact, I might be the only person to ever order just one, as they are meant to be used in pairs.
XRF Module: TI's Development Kit is 41.3x More Expensive at $649
This all opens the door to building a pretty capable DIY, open source weather station console based on high quality commercial components to gather up the data.  Davis sells really good outdoor hardware for this, but their indoor stuff is limited, low powered, expensive, and doesn't feature the kind of connectivity to hook into something like Pachube very easily.  A little module like this brings a lot of capability to the table.  This is my long term goal and won't happen anytime soon, especially with summer coming up and having to shift my time to various landscaping projects.  But I do have a lot of fun playing around with it, and maybe things will move along faster if some other hardcore Davis weather fanatics decide to jump on board.

Before heading out, I would be remiss if I didn't thank a couple folks for helping me get this far.  It was Travis Goodspeed who first mentioned that the CC1110 in the IM-ME should be compatible with the CC1021 in the console.  Travis also designed the GoodFET, a board he literally gives away to further people digging in to stuff like this.  It is a must have for IM-ME hacking.  Then there is Joby Taffey, who wrote the PinkOS operation system for the IM-ME.  I wouldn't have stood a chance without this code to get me rolling.  And of course there is Michael Ossman who wrote the IM-ME based spectrum analyzer based on Dave's pioneering work.  I'd have been screwed without all the previous work from these guys because, as this picture shows, I'm not much of a programmer.
RIP Dennis M. Ritchie