Thursday, January 1, 2015

Wifi on the BeagleBone Black with Systemd - An Addendum

So I go to all the trouble to write up a nice long blog post on how to get wifi going on the BeagleBone Black and what happens?  The 'Bone bites me in the ass, reinforcing the latter part of our Love / Hate relationship.  Here is the story, and the fix.

I posted my writeup to the BeagleBoard Google Group and one hour later, my wireless connection started acting up again.  I was always getting some DUP packets with my ZD1211 wifi adapter e.g.

64 bytes from ord31s22-in-f4.1e100.net (216.58.216.228): icmp_seq=3 ttl=50 time=71.4 ms
64 bytes from ord31s22-in-f4.1e100.net (216.58.216.228): icmp_seq=4 ttl=50 time=70.5 ms
64 bytes from ord31s22-in-f4.1e100.net (216.58.216.228): icmp_seq=4 ttl=50 time=75.6 ms (DUP!)
64 bytes from ord31s22-in-f4.1e100.net (216.58.216.228): icmp_seq=4 ttl=50 time=76.2 ms (DUP!)

that I couldn't figure out but otherwise things seemed to be working OK.  Then the ZD1211 stick just decided to drop the connection, take its ball, and go home.

I was pissed, but I also have one of those brains where the subconscious continues to crank away on the problem no matter now much you want to drive over your 'Bone with a car.  I was lying awake in bed when I remembered a post about someone else's problem being solved by changing the performance governor the 'Bone uses to set its clock speed.  The 'Bone is set up to go into "ondemand" mode when it isn't that busy, and it drops its clock speed all the way down to 300 MHz from its 1 GHz maximum when it does so.  Could changing this fix the problem?  Only one way to find out, but first I had to figure out how.

The Jessie snapshot I'm using on the 'Bone does not (yet) use the newer cpupower utility that the cool kids like Arch Linux are using these days.  Instead, Jessie still uses the older cpufreq-utils stuff.  So all you need to do to set the 'Bone to 1 GHz (and keep it at 1 GHz, dammit) is:
  1. Edit /etc/default/cpufrequtils (you might need to create it if it doesn’t exist).
  2. Specify the governor with the GOVERNOR variable by putting this line in the file:

    GOVERNOR="performance"
Then reboot the 'Bone with
root@beaglebone:~#  systemctl reboot
Once it has rebooted, check that the change is in effect by using the cpufreq-info command and verifying that the 'Bone is hanging out at 1000 MHz for 100% of the time.
root@beaglebone:~# cpufreq-info
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
  driver: generic_cpu0
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency: 300 us.
  hardware limits: 300 MHz - 1000 MHz
  available frequency steps: 300 MHz, 600 MHz, 720 MHz, 800 MHz, 1000 MHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance
  current policy: frequency should be within 300 MHz and 1000 MHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 1000 MHz (asserted by call to hardware).
  cpufreq stats: 300 MHz:0.00%, 600 MHz:0.00%, 720 MHz:0.00%, 800 MHz:0.00%, 1000 MHz:100.00%
The wifi on my 'Bone using the TP-Link TL-WN722N dongle has been rock solid since I did this a day or two ago.  No DUPs, no lockups.  I wasn't so lucky with my ZD1211 based dongle (the firmware for which will be appearing in future Jessie snapshots BTW), and I've decided not to pursue this any further.  There are some old bugs on this issue so it is unlikely that this is something I can do much about.  And I don't really care, because at least I have a setup now that works and is solid.

If you are interested, all the details can be found on this post to the BeagleBoard Google Group.  The explanation for using the "ondemand" governor is so that the 'Bone can be powered by USB.  But this doesn't make sense to me.  Nothing prevents the 'Bone in "ondemand" to clock up to 1 GHz for an extended period of time, and if USB won't cut this, you're screwed anyway.  And is it better to have mysterious failures like this happening to people and getting them so frustrated that they just want to run over their 'Bone with a car?  After all, there are other people besides me that burned off a bunch of hours thanks to this issue.

Want some take-aways to wifi on the Beaglebone Black?  Here are some take-aways.
  • Use an Atheros-based wifi dongle like the TP-Link TL-WN722N
  • Set up your wireless connection using NetworkManager and, if you are working from the command line, its nmtui interface
  • Set your CPU governor to "performance" or live to regret it
  • Don't power your 'Bone from USB.  Get yourself an external supply rated for 5 Volts at 2 Amps.  I use this one from the good folks at Adafruit.
Now, just like my last post, I'll toss in a few other tidbits that I've learned along the way.
Compiling your own kernel or wanting to use the SGX libraries?  Make sure that you have lzop installed.  I did not, and couldn't figure out why the kernel build wasn't producing a zImage and putting it into the deploy directory as noted in these instructions.  And if you are going to be doing that, might as well have ccache installed as well.  On Arch Linux:
root@beaglebone:~#  pacman -Sy lzop ccache
Want to turn off screen blanking not only for X-Windows, but for serial / SSH terminal sessions and the raw framebuffer (so that your OGLES2ChameleonMan demo doesn't blank after ten minutes)?  Edit /boot/uEnv.txt and add consoleblank to your cmdline, like so:
cmdline=quiet consoleblank=0
Did you flash your 'Bone with something containing the LXQT desktop or something similar?  What if you don't want LXQT to start on boot?  This awesome post shows to turn off LXQT in a running system:
root@beaglebone:~# systemctl isolate multi-user.target
and this starts it up again:
root@beaglebone:~# systemctl start graphical.target
To avoid starting up a graphical desktop at boot run:
root@beaglebone:~# systemctl set-default multi-user.target
and change back with:
root@beaglebone:~# systemctl set-default graphical.target
That's enough knowledge bombs for one post. See you next time here at Mad Scientist Labs.

Monday, December 29, 2014

Wifi on the BeagleBone Black with Systemd

EDIT 2015-01-01: What I thought was the last word in getting wifi going on the Beaglebone Black was missing one key piece of information.  Read this addendum, or you're screwed.

Unlike most of my posts on this blog, the title of this post is neither clever nor funny.  I'm feeling serious today.  Why?  I have a Rev B BeagleBone Black (BBB) and I have a love / hate relationship with it.  And I suspect I'm not the only one.
It's Not Me - It's You
I love that it is a hell of a lot more powerful than the Raspberry Pi in all aspects except probably for its graphics: it has a faster processor, more RAM, onboard flash, more I/O, and the incredibly powerful PRUs.  The new Raspberry Pi B+ certainly helps level the field with more I/O than the original B and four sweet USB ports, but for raw power the award still goes to the 'Bone.

On the other hand, I hate that the 'Bone can be such a bastard to get working to its full potential.  One problem is the Imagination Technologies SGX530 graphics core.  This company is open-source hostile, and their closed-source graphics binary blob works only with certain kernels.  All attempts to reverse-engineer an SGX driver have unfortunately failed.  The Raspberry Pi has both more powerful graphics hardware and Broadcom is (amazingly) supporting the development of an open source driver.  Incidentally, the upcoming Beagleboard-X15 will also have an SGX graphics core in it, and they will not be getting my money for this reason alone. Huge win for the Pi here.

The other problem I've had is with getting wifi going.  One dongle was crashing the kernel.  I wasn't the only one having problems, either.  I spent a ridiculous amount of time on this in the fall with no luck.  I thought the best way to fix the problem was to just drive over the 'Bone with my car and get a Pi.

But I'm stubborn.  I've got some time over the Christmas break and I thought I'd give it one more shot.  Amazingly, I seem to have something working using the Jessie snapshot I got from here named BBB-eMMC-flasher-debian-jessie-lxqt-armhf-2014-12-19-2gb.img.xz and running kernel 3.14.26-ti-r42.  I also learned a bit about systemd along the way.  Systemd is fairly new to Debian, and that fact makes a lot of the wireless tutorials out there obsolete.  So how did I get wifi working on the Beaglebone Black with systemd?  Read on, McDuff...

First of all, get the latest Jessie snapshot from the link above and flash it to the eMMC (flash) memory on the 'Bone. I'm not going to go into the details of doing that, because lots of other people have. That is what Google is for.

OK, so you've flashed the 'Bone with the Jessie image and want to get wireless networking going. I'm going under the assumption that you have console access to the 'Bone, either via a USB to Serial connection or SSH over a hard-wired connection. No fancy Graphical User Interface stuff for me.

Next thing to know is that a lot of wifi dongles need some kind of firmware loaded into them.  My TP-Link TL-WN722N already has its firmware on the image linked above (from the ath9k_htc package) so no need to download anything in this case just yet.  Other dongles like those based on the ZD1211 or RaLink chipsets will need you to install a firmware package with the old "apt-get update; apt-get install... " dance.  Here, the dmesg command and good ole' Google are your friend, helping you to figure out what you need.

Now you've got a 'Bone with a dongle plugged in and its firmware loading up.  After that, the thing that trips people up is that recent kernels "soft block" (disable) things like wifi and bluetooth by default. A program called rfkill is necessary to enable them. rfkill is part of the image above (and because I politely asked Robert C. Nelson to put it in) so you should be good to go without having to download something over a hardwired ethernet connection first.

Let's kick things off by checking to see if the wifi is blocked, and enable it if it is.
root@beaglebone:~# rfkill list wifi
0: phy0: Wireless LAN
Soft blocked: yes
Hard blocked: no
root@beaglebone:~# rfkill unblock wifi
root@beaglebone:~# rfkill list wifi
0: phy0: Wireless LAN
Soft blocked: no
Hard blocked: no
Good start. "Soft blocked" changed from "yes" to "no" after we unblocked it.

Next thing we need to do is bring the wifi interface up using the ip command.  Note a couple things.  First, Blogger hates me.  The square brackets below are supposed to be angle brackets, but the Blogger editor gets confused when you switch back and forth between "Compose" mode and "HTML" mode (and yes, I tried coding in the HTML entities and no, that didn't work).

Before getting into that, how do we know the name of our wireless interface?  Assuming your dongle is plugged in, do...
root@beaglebone:~# ip link
1: lo: [LOOPBACK,UP,LOWER_UP] mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: [NO-CARRIER,BROADCAST,MULTICAST,UP] mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
3: can0: [NOARP,ECHO] mtu 16 qdisc noop state DOWN mode DEFAULT group default qlen 10
link/can
4: can1: [NOARP,ECHO] mtu 16 qdisc noop state DOWN mode DEFAULT group default qlen 10
link/can
5: wlan1: [BROADCAST,MULTICAST] mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
There we go.  My dongle is wlan1 and I'll be using that in the examples that follow.  I'll also strip out other interfaces in the output going forward.  Numbers I'd rather you didn't see have been replaced by "xx".

Now let's bring the wlan1 interface up and check its status.
root@beaglebone:~# ip link set wlan1 up
root@beaglebone:~# ip link
5: wlan1: [ NO-CARRIER,BROADCAST,MULTICAST,UP ] mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
See how the status changes from [ BROADCAST,MULTICAST ] to [ NO-CARRIER,BROADCAST, MULTICAST,UP ] once the link is up?

The next thing we'll do is associate the dongle to a wireless router.  This example is for the connection  to my router that is wide open.  If yours is not open or not WEP encrypted, you'll need to do something different like bringing wpa_supplicant to the party (or just skip ahead to the automatic stuff below).  My router connection is named Abby_Slow, so you'll have to substitute the name of your access point below.  The really important thing to note in the following is that older command line tools like iwconfig have been deprecated and will probably not work anymore.  The new tool in town is called iw.  We will first use iw query the status of the link, and then make a connection.
root@beaglebone:~# iw wlan1 link
Not connected.
root@beaglebone:~# iw wlan1 connect Abby_Slow
root@beaglebone:~# iw wlan1 link
Connected to xx:xx:xx:xx:xx:xx (on wlan1)
SSID: Abby_Slow
freq: 2437
RX: 10694 bytes (75 packets)
TX: 2029 bytes (13 packets)
signal: -72 dBm
tx bitrate: 15.0 MBit/s MCS 0 40MHz short GI

bss flags:      short-preamble short-slot-time
dtim period:    0
beacon int:     100
So now we have a link, but that isn't going to get us as far as we need to go...
root@beaglebone:~# ping www.google.com
ping: unknown host www.google.com
We need to use something called dhclient to get a DHCP address, like so...
root@beaglebone:~# dhclient wlan1
root@beaglebone:~# ping www.google.com
PING www.google.com (216.58.216.228) 56(84) bytes of data.
64 bytes from ord31s22-in-f4.1e100.net (216.58.216.228): icmp_seq=1 ttl=50 time=75.2 ms
^C
--- www.google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 75.251/75.251/75.251/0.000 ms
root@beaglebone:~#
The sweet, sweet ping of success.  There is just one more thing before we go automatic on this problem.  Some dongles have issues going into power saving mode.  Let's not take our chances and disable this.  Inspired by this post, create a file called /etc/udev/rules.d/wifi_power_save.rules and add this one long line to it (no line breaks no matter what it looks like below, please).
ACTION=="add", SUBSYSTEM=="net", KERNEL=="wlan*" RUN+="/usr/sbin/iw dev %k set power_save off"
Here is how to check it the next time you boot the board:
root@beaglebone:~# iwconfig wlan1
wlan1 IEEE 802.11bg ESSID:"Abby_Slow"
Mode:Managed Frequency:2.437 GHz Access Point: xx:xx:xx:xx:xx:xx
Bit Rate=5.5 Mb/s Tx-Power=20 dBm
Retry short limit:7 RTS thr:off Fragment thr:off
Encryption key:off
Power Management:off
Link Quality=20/100 Signal level=20/100
Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
Tx excessive retries:59 Invalid misc:48 Missed beacon:0
Now obviously you are not going to want to go through this whole process every time you boot the board.  This stuff needs to be automated.  But how?  One thing you could do is trawl the interweb and try to understand how systemd-networkd works.  Then, working from the sparse set of examples on the web, you could manually create some text files against all odds that whatever you come up with is actually going to work.  You could do that, because that is what I tried to do to start with.  I created a 20-dhcp.network file in /etc/systemd/network that should have automatically worked but didn't.  Be smart.  Don't do that.  Do this:
root@beaglebone:~# apt-get update
root@beaglebone:~# apt-get install network-manager
root@beaglebone:~# nmtui
nmtui is the NetworkManager Text User Interface bundled into the NetworkManager package and it is very easy to use.  Ugly, but easy to use.

Ugly, But Works Beautifully
You'll hear a lot more about nmcli, the NetworkManager Command Line Interface.  Don't even try using that unless you are a NetworkManager God.  Stick with nmtui and you'll be good to go in no time.  You can use it to setup connections to open and encrypted access points that will automatically connect up when the board boots.  No fuss, no muss.  It worked perfectly for me.

Until this started to happen...
root@beaglebone:~# [ 2282.574398] INFO: task wpa_supplicant:485 blocked for more than 120 seconds.
[ 2282.581864] Tainted: G O 3.14.26-ti-r42 #1
[ 2282.592586] "echo 0 } /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Then the 'Bone locked up solid.

FML.

I tried once or twice more and the dongle kept locking up hard after a short while, taking the 'Bone along with it.  This is where I started to think that maybe I've got a bad TP-Link dongle.  Fortunately for me, I have another dongle, this one based on a Zydas ZD1211 chipset that is also well supported in Linux.  It needs some firmware that currently isn't in the default image, so I rebooted the 'Bone and did the following before the board decided to crash again.
root@beaglebone:~# apt-get install zd1211-firmware
I plugged in the Zydas dongle and then rebooted the board so the kernel could detect it and load its firmware (might have worked if I hot-plugged it, might not have).  Then I ran nmtui again, set up the dongle, and the network has been working well since then.  But you better believe I'm keeping my fingers crossed.

Incidentally, some people have reported that some dongles work poorly when plugged directly into the 'Bone.  A popular fix seems to be plugging it in via a USB extension cable or powered hub to get a little separation between the two of them.  If you've followed this guide and still can't get things working, try giving that a shot.

This would be a good place to wrap things up with my usual "now go out and do something" line.  But heck, while I'm at this, I might as well throw in some other useful things I picked up along the way to get it all into one place.  If I don't put this stuff here, I'll just be looking for it myself later anyway.
How do you change the name of the dongle's device interface???  See here on the Archlinux wiki.

How do you check out NetworkManager's status?
root@beaglebone:~# systemctl status NetworkManager
How can you see what systemd is putting into its journal for NetworkManager?  (Tip: use the left and right arrow keys to move horizontally a half screen at a time to see long lines of text!)
root@beaglebone:~# journalctl -u NetworkManager
How do I reboot the 'Bone?
root@beaglebone:~# systemctl reboot
How do I power it off?
root@beaglebone:~# systemctl poweroff
Anyhoo, there you have it: a reasonably comprehensive and up-to-date tutorial on how to get wireless networking going on one of the more powerful little ARM boards out there.  Included at no extra charge was a bunch of related systemd stuff that was previously scattered all over the net.  Now get out there and make that 'Bone do something.

Friday, December 26, 2014

No Pressure - I Got This

Earlier this year, My Lovely Wife and I enjoyed a visit from My Evil Brother-In-Law, who also just so happens to work for Environment Canada.  Naturally he was interested in my Davis Weather Station and some of the hacking I've done on it.  But he was more interested in its barometric pressure reading of somewhere around mid-800 kPa.  That is a really low number that could mean only one of two things:
  1. My barometer was not working.
  2. We were in the eye of a hurricane.
The weather seemed pretty nice out, but I know from what little TV I watch that the weather can be deceptively nice in the eye of a hurricane, only to worsen when the eye moves on and you're subjected to the hurricane's nastier bits.  So we waited a while and the weather stubbornly continued to be nice.  Broken barometer it was.  Later my suspicions were confirmed when the barometric pressure would just show as a series of dashes.  No need to take cover under the stairs, for sure.
Broken Barometer Badness
Fast forward a little bit and My Evil Brother-In-Law decides to take up the invitation from My Lovely Wife and myself to visit us for Christmas.  My barometer is still broken, but we've offered our invitation a couple months in advance.  I've got time.  I would fix my barometer before the weatherman's arrival.

The first part of the process was figuring out how the hell I was going to do this.  Everything else on the station was working well, so I figured it was just a dead sensor.  But exactly what sensor are we talking about?  I figured this much out a while ago and documented it in this post.
The part is a Measurement Specialties 5534CM.  Applications include "Weather control systems" according to its product page (man, I'd love to be able to control the weather). Here is its product page, here is where you can buy it, and here is some Arduino code to drive it!
The cool thing about this part is that it has a digital interface rather than an analog one I had originally guessed at.  It also is factory calibrated.  Davis was smart and kept things simple: the amount of A/D they have to do in the console is exactly zero.  But there seems to be a price to pay for this simplicity: $27.27 for a single piece, to be exact.  Good thing they are cheaper by the 1,000.
But there was a problem.  This picture was taken in my old console that has since failed me.  I got a new one and U84 above is not populated.  U85 is.
What is U85?
Good question.  What is U85???  I figured after a bit of research that it must be a minified version of the 5534CM: the 5540CM.  Here is where you can buy it.
You can, but...
Note I said you "can buy it" there, but I didn't say you "should buy it" there.  The price of this little bastard is $32 plus shipping that was going to put me out over $40 total.  Looking good in front of My Evil Brother-In-Law was one thing, but $40 is another.  What to do?  Well, Digikey is a well known name in the electronics components biz, but so is Shenzhen Kesun Electronics Co., Ltd.  Actually, no they aren't.  But their price and their shipping was about a third that of Digikey's.  Here is where you should buy it.  Take my money!  All $12 of it.
The Price is Definitely Right
The only price to be paid for not paying much of a price is time: I ordered this on November 12th and got it December 19th.  It was obviously on the slowest of the slow boats from China, but I do feel Canada Post was at least somewhat to blame.  The good folks at Shenzhen Kesun Electronics Co., Ltd. shipped within two days of my order and gave me a tracking number to boot.  However, that tracking number was never updated again by Canada Post.  Even after I received it, the order showed that it was still inbound from China.  I think my little sensor somehow got lost in the Christmas parcel rush.

So I had the What part of the equation covered.  Equally important is the How.  How do you get the old part off and the new one on?  Any time I had ever tried to take a multi-leaded part off a board like this with just a soldering iron and desoldering wick (a fine copper braid that soaks up solder like a sponge), it always ended in heartache.  You never get all of the solder, and you can't get enough heat around all the part at once to keep the solder molten to have the whole thing come off.  In the end, I would end up just wrecking what I was trying to remove.

That's why I got me a hot air rework station.  Which one?  This one.
Dave Jones Has Many Fine Traits.  Brevity is not One of Them 

Actually, mine is a Yihua 858D.  Same difference.  There are a lot of combo units that combine a soldering station with hot air rework, but I already have a decent iron.  This unit is small, it is quiet, the temperature is well regulated, it comes up to temperature very fast, and it has a neat cooldown function.  Best of all it is cheap.  Better of all, it works great.  I'm not sure how I lived without it.  I got it from here for $79.99 with Free Shipping and a spare heater element.  Nice.  This is my new favorite toy: you can take my hot air gun from my cold dead hands.

I got the What.  I got the How.  The Who is me.  The Where is here.  The When is now.  I got a two week Christmas holiday.  Time to get busy.  Open the case.  Set up on something that could take an accidental blast of heat (in this case, my stove), and put some aluminum foil around the surrounding parts to keep the heat off them to prevent their inadvertant removal.
Preparing for Battle
I set my trusty hot air gun to 340°F and held the nozzle just over the part.  The solder melted after maybe ten seconds and the part was off.  No fuss, no muss.  I freaking love this thing.

The next part of the operation did not go smoothly.  At all.  The plan was to use my soldering iron to solder the new part into place.  I took the new sensor out of its package, applied a bit of flux to the circuit board with a flux pen (always use flux when soldering), and placed the part on the board.  The first problem I had was the part did not want to line up well on the board: the part had some solder on each of its eight pads on its underside.  The circuit board also had a bit of solder on the eight copper pads receiving the part.  Placing the part on the board invariably caused the part to slip out of place.  My first attempts to solder the part in place were met with total failure: the part would be misaligned, I'd get a solder bridge, I'd get a missing connection... and all of my attempts had the part seemingly sitting on top of the board rather than being fastened to it.

A good night's sleep made the answer clear.  As so many instruction manuals say, installation is the reverse of removal.  I was using my soldering iron to attach the new part when I should really have been using the hot air station.  I had been leery to try this because I thought the heat might damage the new part.  But thinking about it some more, these boards are usually built using some kind of wave soldering machine that basically heats the whole board with all parts attached in an oven brought to temperatures above that of the solder's melting point.  That is basically what the hot air station does.

So, I laid down the aluminum foil in the area again and hand placed the part as best I could after having put a little liquid flux in the area.  I set the fan speed of the air gun about halfway and again set the temperature to 340°F.  I slowly lowered the nozzle of the gun directly over the part, trying hard not to blow the part off to one side.

What happened next was beautiful.  The solder around the part melted and the resulting surface tension moved the part perfectly into position.  The part also visibly pulled itself toward the circuit board and was no longer just kind of sitting on top of it.  Huzzah!

I let the area cool down a bit and then used a Q-tip and some isopropyl alchohol to clean the residual flux away.  I put it all together and...
Victory Is Mine!!!
The console display is 1020 hPa, or 102.0 kPa, a value that checked out nicely with the value reported from the nearest airport.  Better yet, the repair passes muster with My Evil Brother-In-Law.  Now all I've got to do is keep him on a steady diet of good food and good booze for the rest of his visit.  It's all good.

And by the way, Merry Christmas and a Happy New Year from all of us here at Mad Scientist Labs.

Wednesday, June 18, 2014

I Want To See Violet

One of my earliest memories is standing in front of My Wonderful Mom as a young lad proclaiming how much I loved the purple and white pants I was wearing.  They were so cool.  They were the best.  There was only one problem: they weren't purple.  They were blue.

It probably came as no great surprise to my parents that I was colorblind (the politically correct name for which these days is "color vision deficient", but I have been and always will be colorblind).  My uncle on my mom's side is red-green colorblind (and how he made a successful living as a farmer is beyond me).  Color blindness is genetic and passed along from your mom.  I have her to thank for my colorblindness, my puny musculature, and 50% of the other things I can blame on my DNA.  Thanks Mom.

But I was young and able to bounce back.  I was accepting of the fact that many of the professions I might otherwise aspire to probably would not work out in the end: interior designer ("What have you done to my beautiful house!?!?"), bomb disposal expert ("Cut the green wire."), and last but not least, professional telephone cable splicer.  I was the butt of these jokes and many others through life.
I Have Nightmares Like This
My chosen profession of Engineering is bad enough.  Geology labs back in University were horrible.  I remember having to identify different types of rocks in an exam, and one of the keys in doing so is going by color.  Worse was the infamous resistor color code. Carbon film resistors are marked with a series of colored bands to indicate their value and tolerance.  To make matters worse, my difficulty in resolving colors is compounded when the colored bit I'm looking at is small.  To this day, I have never bothered to memorize this fundamental code: there's no point in me doing so. 
Pointless
When people think of color-blindness, they think of the whole rods and cones thing, but there is more to it than that.  The best explanation I've come across is this one:
Colour vision is based on an opponent process [1]. There are three kinds of colour receptors called L, M and S (for Long, Medium and Short) and they are receptive to colours centred around the red, green, and blue parts of the visible light spectrum respectively.

Here is a good graph showing how sensitive each receptor is to different wavelengths of light: 
But we don't perceive colour directly from these receptors. Instead, our visual systems combine the output of these receptors to form three channels: black vs. white (L+M+S), red vs. green (L-M) and blue vs. yellow (S-(M+L)). This is why there is no such colour as a reddish-green or a bluish-yellow; our visual systems are not able to perceive these (except under exceptional laboratory conditions). As you can see in the graph, the L, M and S receptors aren't just receptive to pure red, green and blue but to a range of wavelengths centred near these colours, and they all overlap to some degree. But the L (green) and M (red) receptors overlap quite a bit.
In red-green colour blindness, either the L or M are shifted so they're even closer. This means that the red vs. green (L-M) channel is no longer able to distinguish between the two, so this channel is always close to zero and the person no longer sees either red or green just something muddy in between. Blue-yellow colour blindness is similar, but in this case it's the S channel that's shifted towards the L and M. In this case it overlaps with both and this means there's no longer any colours that activate the S without also activating L and M (and vice versa), making it impossible to distinguish blue from yellow.

I stumbled on this when skimming through YCombinator on a lazy Sunday afternoon.  The link above pointed to an article on BoingBoing called "Color for the Colorblind".  I read it and was fascinated.  The article describes sunglasses from a company called EnChroma that promises to correct colorblindness while you are wearing them.  Wow.

First you take a test on their website to determine what kind of colorblindness you have and how bad it is.  The test results told me I was a "strong duotan", and I had no idea initially what that meant.  A little research told me that they were telling me what I already knew: I was red-green colorblind, and I had it bad.
Damned If I Can See Anything but a Bunch of Dots
Then I got to this video on the Press section of their site.  At around 45 seconds into it, I started to cry.
...there is always an "aha" moment.  When they'll turn and look at something and say... "Is that flower... violet?" And they've never seen violet before...
Neither had I.

I pulled myself together, went downstairs, and asked My Lovely Wife to bring up first this video called No Such Thing as Color.  It starts with a guy looking at different houses and trying to guess what color they are.  Thing is, I can't tell if he is getting them right or wrong.  I'm guessing he gets a bunch of them wrong, or he wouldn't be colorblind.

 Skip The Last Half - It Gets Pretty Lame

Then I asked My Lovely Wife to watch the EnChroma video.  At around 45 seconds in, I started crying again.  I think she thought I was laughing until she turned around and saw the tears in my eyes.  I desparately wanted to see violet too.

I put in my order to EnChroma later that night for a pair of EnChroma Cx Receptors.  These are designed to fit over top of my prescription glasses, without which I see nothing but a blur.  The three to five business day wait for shipping turned into three weeks.  Seems a lot of other people had read that same BoingBoing article and wanted to see violet as well.  Oh well.  I had been colorblind all my life.  I could wait a few more weeks.

As I waited for my order to arrive, I didn't tell anyone besides My Lovely Wife, My Wonderful Mom, and My Cool Sister.  I had to tell somebody, but I didn't want to spread the news around too much; I wasn't sure if these things were going to work in the first place.  I did drop the odd hint though, like changing my Facebook profile picture.
I kept a close eye on my UPS tracking number as my precious shipment wound it's way up from California to my home in the Great White North.  When the happy day arrived, I went to the courier to pick them up and got a little more than I bargained for, to the tune of...
  • $20.77 in customs fees
  • $21.81 in taxes
  • $64.60 in UPS brokerage fees (!).  No wonder there have been several class action lawsuits filed against UPS for shipments into Canada.  This is exhorbitant.
Now none of these are Enchroma's fault.  Fees and taxes are inescapable, but I do wish they would offer to ship by US Postal Service to minimize shipping costs to non-US residents.

Anyhoo, they say the glasses work best on a bright sunny day.  I got home late on Friday so decided to wait until Saturday to try them out.  The weather on Saturday was crap so I grudgingly pushed the first try to Sunday.  The crappy weather continued, and it wasn't until we got a small break in the crappiness on Tuesday (with the forecast for extreme crappiness in the days following) that I decided to finally give them a shot.

I wanted to try them first in our backyard.  It has a big flowerbed in the back with several different plants in bloom.  Our backyard also faces a river with farmers fields beyond that, making for quite a sight, colorblind or not.  I also had my wife but on her blouse that she tells me is purple.  I wanted to see violet, after all.

I took a long look around before putting on the EnChromas.  I wanted to get a good sense of "before".  My Lovely Wife got the camera out, started recording, and I put the glasses on for the first time.

The partly-cloudy sky transformed itself into something from an alien world.  I didn't really know if what I was looking at now was accurate or just... different.  Whatever I was looking at had far more "pop" than what I had noticed before.  I just stood there and stared for a while, repeating that it "looked different".  I noticed my mouth was hanging open.

I turned around and our pink stucco house exploded into my view with a vibrancy I could not have imagined.  What I thought had been dull and faded was far from it.  Then I looked down and the chickweed in our lawn blazed in front of my eyes like it had been electrified.  I was taken aback.  I tried to describe it to My Lovely Wife as the recording went on but my voice started to crack and I felt the emotion washing over me like that day from weeks ago.  I am probably the first person ever to be overcome by emotion from a weedy lawn.

I looked over at My Lovely Wife and could see that she was getting choked up as well, her violet blouse shining before my eyes.

I had gotten what I was after.  I signaled to her to stop the recording so we could see how it turned out.  I wanted to share the experience I'd just had with others but, alas, there were technical difficulties and these words will just have to do.

I kept the sunglasses on and took a walk around the yard.  The brown stain on our deck was no longer brown, it was brown, with a richness I had not seen before.  I then noticed how browns stood out so much more from greens than they did before.  A small dead pine tree some distance away that would have been almost invisible to me before in the grassy background was easy to spot.  And was our green truck ever green!  A world that had been hidden from me was starting to reveal itself.

I looked down at myself and the burgundy shirt I was wearing lit up before my eyes.  "WOW!", I shouted.  "LOOK AT MY SHIRT!!!"  My Lovely Wife had always liked the shirt more than I did.  To me, it seemed a bit dull.  I was wrong.  I kept staring back at it thinking I had made some kind of mistake.

It was time to go back into the house.  I kept the glasses on but didn't expect much wearing them inside.  As I thought, I didn't see any real improvement, and the folks at EnChroma caution you about this: they work best out in the open with bright natural light.  They came up empty in early evening on a cloudy day.

I am looking forward to the coming days and weeks.  Apparently, they become more effective as your brain adapts to a new way of seeing the world when the sunglasses are on.  The crappy weather might slow that process down, unfortunately.  On the other hand, I'm really hoping for a nice rainbow sometime.  People with this eyewear rave about rainbows (and fortunately for me, the price on these things has come down since that article as well).

I got into this wanting to see violet, but so far it is the greens that are blowing me away.  The new growth on the tips of spruce trees stand out like the lights on a Christmas tree.  I find myself distracted by the rain soaked, green grass growing brightly in the ditches on the drive in to work.  And I will always remember the first time I saw that chickweed and really saw green.  It is only too bad that these sunglasses aren't a cure for colorblindness.  They only do their magic while being worn.  Once I take them off, I return to my muted, drab world.  But to me, something is a hell of a lot better than nothing.

Now my new Facebook profile picture should make sense to everyone.

Sunday, June 15, 2014

Diablo Bread, The Bread of the Devil

I first blogged about bread around three years ago.  I look back at that post discussing my tips and tricks for Tartine bread and it makes me cringe.  What I thought was a good loaf of bread back then is far below what I'm able to come up with these days.  What has made me a better baker?  The same way you get to Carnegie Hall.  Practice.  I make a loaf of bread pretty much every weekend.
Tartine Bread Try #43
And it isn't just Tartine.  I find I am making that recipe ever month or so, while the other weekly bakes will sometimes be a No Knead bread of some kind or another based on Jim Lahey's technique outlined here.  A post back in May 2012 gave a link to another excerpt from "My Bread" that has more recipes you should check out as well.

Now with familiarity comes the courage to strike out on one's own.  I've been on quite a Sriracha kick lately and I got to thinking, what about a Sriracha No Knead Bread?  Put that term into Google and what do you get?

No results found for "Sriracha No Knead Bread".

Not acceptable.  Time to blaze a trail.

I call this bread Diablo Bread, The Bread of the Devil.  The Sriracha in the recipe is amped up by copious amounts of crushed red pepper flakes.  The orange-red color of the baked result is a warning that this is not a bread to be messed with.

Before starting, watch this video from the man himself to get an idea what you'll be doing.

No-Knead Bread From the Master

My recipe is along these lines, but my method is somewhat different from the second rise onward.  I also use a round Lodge Combo Cooker rather than the oval shaped Dutch Oven Jim uses.  You'll have to tweak my procedure a bit if you are using the latter.  But I'll tell ya, the Lodge is pretty great for baking bread.  And it is an awesome abuse of Amazon's free shipping policy.
Lodge LCC3 Combo Cooker
So without further adieu...
Diablo Bread, The Bread of the Devil
Inspired by Jim Lahey's No Knead Breads 
Ingredients
400 grams unbleached all purpose flour
7 grams table salt
½ teaspoon quick rise instant yeast
1 ½ tablespoons crushed red pepper flakes (Yes.  That is a lot.  Not a typo)
260 grams cool water (55 - 65 °F)
60 grams (4 tablespoons) Sriracha sauce (Don't wimp out!  Not a drop less!)
Wheat bran for dusting
Tools
Cooking spray or oil
Parchment paper
Tin pie plate
Dough scraper
Plastic wrap
Lodge LCC3 Combo Cooker or cast iron Dutch Oven
Serrated knife
Oven mitts
An ice cube
 Procedure
  1.  Mix the flour. salt, yeast, and pepper flakes together in a medium sized bowl.  Add in the water and Sriracha sauce.  Mix using a wooden spoon or your hand until all of the flour is incorporated and the dough is sticky.  This should only take 30 seconds or so.  Add more water if the dough seems too dry.
  2. Transfer the dough to another bowl lightly oiled or sprayed with cooking spray.  This will help prevent the dough from sticking when you dump it out after the first rise, and this in turn helps to prevent the dough from deflating.  The dough will increase in volume probably by a factor of four at least, so be sure to use a big enough bowl.  Use a bowl that is red on the inside to amp up the viciousness that is this dough.
  3. Cover the bowl with plastic wrap so the surface of the dough doesn't dry out during the bulk ferment (a fancy-schmancy baker's term for first rise).  Let it rise at a temperature around 70°F in a spot out of the sun for 18 hours.  Or you could be like me (everybody should be like me) and put the bowl in a water bath maintained by your home-built Sous Vide rig at a rock-solid 21.2°C.  You'll have to shorten the rise or use a little less yeast if rising at temperatures that are a lot warmer, but try not to get into this pickle in the first place.  The long rise time contributes to the flavor of the loaf.
  4. While the dough is rising, cut a circular piece of parchment paper to fit the Combo Cooker using the pan as a template.  Cut it wide enough to stick up above the edges of the cooker.  You'll be putting this paper in the pie plate for the dough's second and final rise.
  5. After the 18 hour bulk ferment (fancy-schmancy!), generously flour your work surface.  The circular piece of parchment paper should be sitting on top of your pie plate.
  6. Gingerly tip the bowl containing the risen dough over the floured surface, taking care not to deflate the dough.  You don't have to have your dog looking out the window when you do this, but it helps.
  7. I usually end up with a round blob of dough that I want to shape into a ball.  To do so, I first use a dough scraper (actually a plastic drywall knife, 'cause that's how I roll) or my hands to gently fold two opposite ends of the blob toward the center to make kind of a rectangle.  Then I use my hands and gently grab one long end of the rectangle and roll it up towards the other to get my ball of dough.  Place the ball of dough seam side down in the center of the parchment paper that is sitting on the pie plate.




  8. Dust the top of the dough with some wheat bran.  Cover the whole thing with a clean tea towel or plastic wrap to keep the top of the dough from drying out.  I use a shower cap stolen from a hotel - the loose elastic does a great job of getting around the pie plate without pulling the plastic too tight.
  9. Place the pie plate containing the dough in a nice warm spot for the second and final rise (around 75 - 80°F).  I put it in the oven with the light on if the house is cool.  I let this rise go for two hours.  It should look something like this when it is done.  Note the little dimples on top.  I was giving this dough the poke test: the idea is that you poke the bread and the dough should hold the indentation when it has properly proofed.  Don't go longer than you need to here, or the dough will be over-proofed and rise poorly in the oven.
  10. Half an hour before the end of the final rise, put the Combo Cooker into your oven with the rack set in the middle.  Set the oven to 475°F.  You want to be using the Combo Cooker with the shallow frypan part as the base and the deep cooker part as the lid.  It is beyond obvious that if you are doing the final rise in the oven with the light bulb on, that you should take the dough out first.
  11. After the half hour preheat at 475°F, take the cover off your risen dough and place it next to your oven.  Have your ice cube and serrated knife standing by.  You are going to work quickly here.  Quickly and extremely carefully.
  12. Put on your oven mitts and take the smoking hot Combo Cooker out of your oven and on to the top of your stove.  Take off your oven mitts, pick up your pie plate and gently set it about halfway into the pan of the Combo Cooker.  Slide the parchment paper with the dough sitting on top out of the pie plate and onto the pan, using your fingers on the parchment paper to maneuver it into place.
  13. Once the dough is centered on the pan, use a serrated knife to cut a slash or two onto the top of the dough.  The dough will rise in the oven and expand where you make these cuts.
  14. Working really fast, put an ice cube between the pan and the parchment paper under the dough.  The idea is to add extra steam inside of your cooking vessel and prevent a crust from prematurely forming on the bread that would inhibit its rise.
  15. Get you oven mitts back on, flip the deep part of the Combo Cooker upside down, and use it to cover the pan.  You'll hear the ice cube sizzling away even with the cover on.  Put the Combo Cooker into your hot oven.  Keep the oven temperature at 475°F.  Let the bread cook covered for 30 minutes.
  16. After 30 minutes, take the cover off the bread and gasp at its beauty.  You'll want it to go back into the oven until the top is a nice dark brown.  This usually takes only an extra five to ten minutes.  Keep a close eye on it so it doesn't burn.  I usually set the lid of the Combo Cooker under the pan to serve as a heat shield for the bottom of the bread to prevent it from getting too dark.  If you have a thermometer, you'll want to see around 205°F in the center of the loaf when it is done.
  17. Once the loaf is a nice dark brown, take it out of the pan and set it on a wire rack to cool for at least an hour.  You'll be rewarded by a lovely crackling sound as the bread cools if everything went well.
Notes
  • If you don't have a kitchen scale to measure out these quantities, go buy one.  I'll wait.  Every  baker should have a scale.
  • The usual No-Knead recipe calls up ¼ teaspoon of yeast instead of the ½ teaspoon I use here.  It seems the spices kick the hell out of the yeast so extra is required.
  • A little less salt is used in this recipe vs. the usual because of the salt in the Sriracha.  The liquid volumes were adjusted accordingly as well.
This bread goes great with pretty much everything.  It is a match made in heaven with chili.  Or pop it into the toaster oven with some sliced cheddar on top.  Me?  I often eat it plain without even butter or jam.  I just close my eyes and savor every bite.  If you are thinking there is something that this bread won't go with, you are wrong.

This bread is a favorite of mine, and not just because I invented it.  The Sriracha and crushed red pepper flakes are great in combination but are not overwhelming - the bread naturally moderates the heat and you are left with some pretty fantastic flavor that can stand up well with anything you dare pair it up against.
Go Make Some.  Now.

Monday, February 17, 2014

Build Your Own Davis Weather Station Console!!!

This Davis weather station hacking stuff is addictive like crack cocaine is addictive.  I always seem to be taking another hit from the pipe.  I've figured out how to get into the Davis console's serial port (after which the two of us had a complete and frank discussion COMPLETELY IN UPPERCASE).  I then built my own Data Logger so I wouldn't have to buy the ridiculously expensive Davis dongle.  Then, after I figured out the wireless transmissions from the outdoor sensor suite, I made my own ISS receiver with a Pretty Pink Pager.  But of course, I can quit any time I want.  Me and Amy Winehouse.
"They tried to make me go to rehab I said No, No, No"  -  Bad Decision

Famous dead drug-addled musicians aside (and how long a list is that?  Don't they know that only Keith Richards is Immortal?), the path I was to go down became clear after Davis pulled a ridiculously stupid stunt in the infamous Firmware 3.0 release they started shipping in new consoles a while back.  The new firmware read 64 bytes from a fab-written, read-only section on the data logger's memory chip.  It took this data, did some algorithmic juggling, and then did a check against another 64 byte write-once block on the data logger put there by Davis.  If the console didn't like what it saw, it declared that the logger was invalid and refused to function further.

This was an obvious and misguided attempt to shut down a few hackers from building and selling third-party data loggers for a fraction of the price Davis charged.  Problem #1 was that people with an older, authentic Davis logger discovered that their pricey little dongle stopped working after a firmware "upgrade".  Problem #2 was it also disabled the serial interface connection I had discovered.  And that pissed me off.

Unfortunately for Davis, the console does not run a 2 GHz Core i7 processor capable of state of the art encryption and decryption algorithms.  It has a little Atmel Atmega processor running at 1 MHz or something like that.  It's abilities are somewhat more... restricted.  Like something that could be cracked by a 256 byte lookup table and a trivial bit of code. Hats off to Watson on WXForum for figuring this out. His most excellent hack is posted below for posterity.  Break out your Arduino and start cutting and pasting if you want to calculate a valid security code for a DIY logger.

u8 const GreenDot_Table[256] =
{
0x00, 0x04, 0x08, 0x0C, 0x10, 0x14, 0x18, 0x1C, 0x21, 0x25, 0x29, 0x2D, 0x31, 0x35, 0x39, 0x3D,
0x46, 0x42, 0x4E, 0x4A, 0x56, 0x52, 0x5E, 0x5A, 0x67, 0x63, 0x6F, 0x6B, 0x77, 0x73, 0x7F, 0x7B, 
0x8C, 0x88, 0x84, 0x80, 0x9C, 0x98, 0x94, 0x90, 0xAD, 0xA9, 0xA5, 0xA1, 0xBD, 0xB9, 0xB5, 0xB1, 
0xCA, 0xCE, 0xC2, 0xC6, 0xDA, 0xDE, 0xD2, 0xD6, 0xEB, 0xEF, 0xE3, 0xE7, 0xFB, 0xFF, 0xF3, 0xF7,
0x18, 0x1C, 0x10, 0x14, 0x08, 0x0C, 0x00, 0x04, 0x39, 0x3D, 0x31, 0x35, 0x29, 0x2D, 0x21, 0x25,
0x5E, 0x5A, 0x56, 0x52, 0x4E, 0x4A, 0x46, 0x42, 0x7F, 0x7B, 0x77, 0x73, 0x6F, 0x6B, 0x67, 0x63,
0x94, 0x90, 0x9C, 0x98, 0x84, 0x80, 0x8C, 0x88, 0xB5, 0xB1, 0xBD, 0xB9, 0xA5, 0xA1, 0xAD, 0xA9,
0xD2, 0xD6, 0xDA, 0xDE, 0xC2, 0xC6, 0xCA, 0xCE, 0xF3, 0xF7, 0xFB, 0xFF, 0xE3, 0xE7, 0xEB, 0xEF,
0x31, 0x35, 0x39, 0x3D, 0x21, 0x25, 0x29, 0x2D, 0x10, 0x14, 0x18, 0x1C, 0x00, 0x04, 0x08, 0x0C,
0x77, 0x73, 0x7F, 0x7B, 0x67, 0x63, 0x6F, 0x6B, 0x56, 0x52, 0x5E, 0x5A, 0x46, 0x42, 0x4E, 0x4A,
0xBD, 0xB9, 0xB5, 0xB1, 0xAD, 0xA9, 0xA5, 0xA1, 0x9C, 0x98, 0x94, 0x90, 0x8C, 0x88, 0x84, 0x80,
0xFB, 0xFF, 0xF3, 0xF7, 0xEB, 0xEF, 0xE3, 0xE7, 0xDA, 0xDE, 0xD2, 0xD6, 0xCA, 0xCE, 0xC2, 0xC6,
0x29, 0x2D, 0x21, 0x25, 0x39, 0x3D, 0x31, 0x35, 0x08, 0x0C, 0x00, 0x04, 0x18, 0x1C, 0x10, 0x14,
0x6F, 0x6B, 0x67, 0x63, 0x7F, 0x7B, 0x77, 0x73, 0x4E, 0x4A, 0x46, 0x42, 0x5E, 0x5A, 0x56, 0x52,
0xA5, 0xA1, 0xAD, 0xA9, 0xB5, 0xB1, 0xBD, 0xB9, 0x84, 0x80, 0x8C, 0x88, 0x94, 0x90, 0x9C, 0x98,
0xE3, 0xE7, 0xEB, 0xEF, 0xF3, 0xF7, 0xFB, 0xFF, 0xC2, 0xC6, 0xCA, 0xCE, 0xD2, 0xD6, 0xDA, 0xDE
};

u8 const Adesto_Factory_Programmed[64] = 
{
  // put here AT45DB011D Security Register from byte 64 to 127
};

u8 OneTime_User_Programmable[64];

OneTime_User_Programmable[0] = 0x00; // or whatever you want
OneTime_User_Programmable[1] = 0x00; // or whatever you want
OneTime_User_Programmable[2] = 0x00; // or whatever you want

u8 n, value;

for(n=3;n<64 br="" n="">{
 value = (u8)(Adesto_Factory_Programmed[n] + n);
 OneTime_User_Programmable[n] = GreenDot_Table[value];
}

For more on this whole sad saga, you'll want to read this masterwork by torkelmj, especially if you enjoy reading stuff dripping with spite and vitriol.  Like I do.  Or skip that and just buy a security-coded third party logger from WXForum awesomedude Belfryboy.  Or order up his PCB on oshpark and build your own.  There are options now, and options are good.

Now one might think that everything is just peachy now that DIY loggers are possible once again.  Not me.  I was pretty grumpy that Davis had tried to screw their customers over even if they did release subsequent firmware updates that tried to undo some of the damage.  Once burned, twice shy, so they say.  I started thinking of a Plan B.  I want to build my own console.
Gratuitous Picture of My Dog
My IM-ME thing was a neat hack, but that is about as far as it goes.  IM-ME's aren't actually rolling off the production lines in millions of units per month these days, and their ability to hook up to other little do-dads would not be easily done.  Cheap transceiver boards based off of the popular CC1101 chip that work in the 915 MHz band are hard as hell to come by.  Ask me how I know.  The Seeedstudio RFBee comes tantalizingly close, but others who have tried to go this route run out of RAM and Flash on the limited processor found on this unit before getting too far.

I needed another way and that led me to the Moteino with its awesome little RFM69B transceiver module.  I got it picking up Davis ISS transmissions earlier this year.  Since then I have been coding away in my spare time and have been pulling together the various bits of hardware that I need to get something going.  Something like this.
Beautiful, Isn't It?

Some cool stuff has sprung up from this very breadboard.  My serial connection to the console grew up there on the left side.  The DIY logger design is in the middle.  The stuff on the right half starting from the left is the humidity sensor, the pressure sensor module, the Moteino itself, and finally my USB to Serial Adapter for the project I'm writing about today.

Now I know what you are saying to yourself: "The title of the blog is "Build Your Own Davis Weather Station Console!!!" but that doesn't look like a console at all!  Where is the keyboard?  Where is the display?" Well, I guess I'm not going to get anything past you today.  There isn't a keyboard or a display... unless you want to add one.  The hardware and software here is all open for you to add on what you'd like.  Me, I'm thinking of hooking this bad boy to a serial port on my Beaglebone running Openhab and having all my weather data available from anything in my house that can bring up a web browser.  Very doable, and no display or keyboard necessary for that kind of application.

But why call it a "Console"?  Because I couldn't think of a better name.  And I did indeed try.  It isn't a "Receiver" because it does more than receive given that it has its own sensors hanging off of it.  "Indoor Unit" sounds stupid.  "Thingamahooie" was tempting, I must admit, but didn't quite convey the application well enough.  And hey, the guy that builds the hardware, develops the code, and writes the blog gets to call what he comes up with whatever the hell he wants.  I'm going with "Console", and you'll just have to deal with it.

But I, as I so often do, digress.  Back to business.  On the hardware side of things, here is what I'm working with in more detail:
  • Moteino with an RFM69 module on it.  Pick the 915 MHz RFM69W (standard transmit power) or RFM69HW (high transmit power) at your option.  I have the serial version but it probably would have made sense to pick up the USB variant so I wouldn't need to tie up my USB-Serial converter all the time.  You probably want to add on the male headers option (why even think about it for the one dollar price?) and the flash memory option for four dollars.  I'm not using that yet, but it might come in handy for data logging at some point.  And if you re-dedicate the Moteino to something else later, this memory can be used for wireless remote programming.  Cool.
  • a DHT22 Temperature and Humidity Sensor for less than five bucks.
  • a BMP085 Temperature and Pressure Sensor for less than six bucks.  But it looks like that module is now much cheaper thanks to the compatible BMP180 sensors now out there for under three bucks.  The two are compatible, and I'd probably go with the newer BMP180 myself just because it is smaller.
  • a (yet to arrive and be connected) DS3231 Real Time Clock and Memory Module for less than three bucks (I felt bad for getting something at this price with free shipping so I bought two).  Note that this module also has a 24C32 EEPROM on it that is bound to come in handy.  Don't be dumb and order a DS1307 clock module instead: that one doesn't work well with the 3.3V Moteino and it is horribly inaccurate.  What good is a clock chip that can't keep time?
Arduino is all about little software libraries, and these are the ones I'm working with right now.
  • the LowPowerLab RFM69 library.  Felix has merged the changes I've needed so no need to use my fork of his code any longer.  I don't know if I'll be using this in the long term or come up with my own that is better suited for this application.  Another library that I'm including right now but not really using yet is his SPIFlash library.
  • the JeeLabs DHT22 code.  I have extracted what I need from the JeeLib code and included it into my own library.  JCW's stuff is a leaner and meaner version of some of the other ones floating around on the net.
  • the Adafruit BMP085 library.  I used this as is.  I did look at using the JeeLabs code but it was a little too tied up with JCW's PortsLib for me to bother untangling at the time.
  • the Arduino Serial Command library to parse command input.  This is a really great little library that lets you register callbacks for each separate command and also does command line argument parsing.  I needed to make a few tweaks as noted below, but nothing major.  My changes have been pushed but not merged at this time, so use my fork here.
  • my own DavisRFM69 library, of course.  I've got to be doing something to make all this stuff work together.
Enough chitchat.  Let's cut to the chase, shall we?  Take a look at this thing actually doing something.
Success - I Never Tire of It
This is SandaySoft's Cumulus software talking to my homebrew weather station console.  The outdoor readings are those from my ISS in the front yard.  The indoor readings are from the sensors I have hanging off the Moteino.  On the left you can see the Serial Port Monitor Software recording the RS-232 messages going back and forth.  Cumulus has no idea that it isn't talking to a real Davis console.

Thanks to the Davis Serial Protocol Manual, getting this working wasn't too terribly difficult.   I knew Cumulus used the Davis SDK to talk to the console so the Cumulus author might not be aware of every single command being sent to the station and back.  I also knew that the LOOP command was used to get the data out of the station so I implemented that and some other useful commands on the Moteino.  Then I hooked my real console up and watched some packets go by and found a few more I needed to implement (namely BARDATA, HILOWS, and RXCHECK).  I had a few hiccups of course.  For example, Davis says that a command should always be terminated with '\n' (0x0a) like this...
 except when they don't, like this...
And that is the fun with emulation: you have to emulate not only the functionality, but the bugs as well.  This one caused me to tweak the serial port library to accept multiple command terminators.  Another oddity is that the console itself goes to sleep all the time and is woken by simply sending a lone '\n' that the console is supposed to acknowledge.  The serial library I'm using hadn't actually accounted for that situation.  Now it does.

Want to play along?  Here is what you need to do.   From within your Arduino's libraries directory...
  • Get my version of the SerialCommand library that supports empty commands and a second line terminator.  And of course, you'll want to get a copy of my DavisRFM69 library.  The pared down DHT22 library is tucked away inside the latter.
git clone https://github.com/dekay/Arduino-SerialCommand.git SerialCommand
git clone https://github.com/dekay/DavisRFM69.git
git clone https://github.com/adafruit/Adafruit-BMP085-Library.git Adafruit_BMP085
git clone https://github.com/LowPowerLab/RFM69.git
git clone https://github.com/LowPowerLab/SPIFlash.git
Hardware hookup is like this.

DHT22 Pin 1 -> Moteino 3.3V Out
DHT22 Pin 2 -> Moteino D4 with 10K Pullup resistor to Moteino 3.3V Out
DHT22 Pin 3 -> Not connected
DHT22 Pin 4 -> Moteino GND

BMP085 Module Pin VCC -> Moteino 3.3V Out
BMP085 Module Pin SDA -> Moteino SDA
BMP085 Module Pin SCL -> Moteino SCL
BMP085 Module Pin GND -> Moteino GND
Once you've got all of this hooked up, you'll find a "VP2.ino" sketch in the DavisRFM69 library's Examples directory.  Open that up in the Arduino IDE and select Control-U to compile and download it to the Moteino (be sure to tell the Arduino IDE that you are using an Arduino Uno).  Open up the serial monitor, set the baud rate to 19200 and "Newline" as the line ending.  You should see... absolutely nothing.  Nothing that is, until you type in some (uppercase) commands like "LOOP 1", "TEST", or "STRMON".  The Moteino will respond exactly the same way as a real console would.  Hook it up to Cumulus (with the "Use Data Logger" option unchecked) and revel in a job well done.

I'm off to a decent start here but there is still lots to do.  Here are some things on my list.

  • integrate the real time clock functionality
  • put in the frequencies for consoles in Europe and other regions
  • add some more useful commands like PUTRAIN so you can tell this thing how much rain you've had this year that it can keep a running count of afterward
  • add in some kind of EEPROM functionality so it retains settings like total rainfall between power downs.  Conveniently, there is an EEPROM on my yet to arrive three dollar clock module just begging to be used.
  • make the reception more resilient.  There are some odd things going on here: I get a spurious interrupt on startup (setup issue?) and when I move the board around (need a pulldown on the interrupt line?)  Reception is also poor during the day.  Might be that I just need to adjust the clear channel RSSI value a bit to accommodate more background noise.
  • integrate the real time clock functionality (daily / weekly / monthly / year highs and lows)
  • etc
Anyone that has done much coding on the Arduino gets to know memory limits.  The Moteino has double the RAM and EEPROM than the standard Arduino UNO, so I've got a whopping 2K of RAM and 32K of Flash to play with.  So far I have used roughly half of each, so I've got some headroom.  I haven't done much to try and streamline any of the code besides a few const arrays here and strings stuffed in the F() macro there, so I am sure there is room for improvement if things get really tight.

With this project, I think I've hit the goals I set out to achieve: come up with something functional, inexpensive, and with a low barrier to entry so that other people can dig in.  Have a new feature you'd like to see?  Send me a pull request on Github.  Want to develop a plug in circuit board (kind of like this) with all the dodads attached to lower the barrier to entry even further while making a few bucks on the side?  Be my guest.

I'll wrap this up by saying that the Davis guys make good stuff.  The ISS is very well put together and I love that the thing is solar powered with battery backup.  I've had mine for years and it has been operating trouble free on Battery #1.  The display on the indoor console is nice, big, and easy to read and it also just sips on battery.  I'm guessing that 99.9% of Davis' customers are going to continue buying standard Davis stuff off the shelf despite whatever I come up with here.  Anything I play around with  is ony going to be picked up by that other 0.1% of enthusiasts.  And it is those enthusiasts that are not only going to continue buying Davis products but will recommend them to their friends as long as they are kept happy.  If Davis pulls another stupid Firmware 3.0 thing, they'll do more damage to themselves than I ever did.  I hope they continue documenting their communication protocol and maintaining accessible expansion ports in their current and future products.  That would be a win-win for everybody.

With that in mind, it seems as good a time as any to unveil the new profile picture.
Be Nice