Monday, February 17, 2014

Build Your Own Davis Weather Station Console!!!

This Davis weather station hacking stuff is addictive like crack cocaine is addictive.  I always seem to be taking another hit from the pipe.  I've figured out how to get into the Davis console's serial port (after which the two of us had a complete and frank discussion COMPLETELY IN UPPERCASE).  I then built my own Data Logger so I wouldn't have to buy the ridiculously expensive Davis dongle.  Then, after I figured out the wireless transmissions from the outdoor sensor suite, I made my own ISS receiver with a Pretty Pink Pager.  But of course, I can quit any time I want.  Me and Amy Winehouse.
"They tried to make me go to rehab I said No, No, No"  -  Bad Decision

Famous dead drug-addled musicians aside (and how long a list is that?  Don't they know that only Keith Richards is Immortal?), the path I was to go down became clear after Davis pulled a ridiculously stupid stunt in the infamous Firmware 3.0 release they started shipping in new consoles a while back.  The new firmware read 64 bytes from a fab-written, read-only section on the data logger's memory chip.  It took this data, did some algorithmic juggling, and then did a check against another 64 byte write-once block on the data logger put there by Davis.  If the console didn't like what it saw, it declared that the logger was invalid and refused to function further.

This was an obvious and misguided attempt to shut down a few hackers from building and selling third-party data loggers for a fraction of the price Davis charged.  Problem #1 was that people with an older, authentic Davis logger discovered that their pricey little dongle stopped working after a firmware "upgrade".  Problem #2 was it also disabled the serial interface connection I had discovered.  And that pissed me off.

Unfortunately for Davis, the console does not run a 2 GHz Core i7 processor capable of state of the art encryption and decryption algorithms.  It has a little Atmel Atmega processor running at 1 MHz or something like that.  It's abilities are somewhat more... restricted.  Like something that could be cracked by a 256 byte lookup table and a trivial bit of code. Hats off to Watson on WXForum for figuring this out. His most excellent hack is posted below for posterity.  Break out your Arduino and start cutting and pasting if you want to calculate a valid security code for a DIY logger.

u8 const GreenDot_Table[256] =
{
0x00, 0x04, 0x08, 0x0C, 0x10, 0x14, 0x18, 0x1C, 0x21, 0x25, 0x29, 0x2D, 0x31, 0x35, 0x39, 0x3D,
0x46, 0x42, 0x4E, 0x4A, 0x56, 0x52, 0x5E, 0x5A, 0x67, 0x63, 0x6F, 0x6B, 0x77, 0x73, 0x7F, 0x7B, 
0x8C, 0x88, 0x84, 0x80, 0x9C, 0x98, 0x94, 0x90, 0xAD, 0xA9, 0xA5, 0xA1, 0xBD, 0xB9, 0xB5, 0xB1, 
0xCA, 0xCE, 0xC2, 0xC6, 0xDA, 0xDE, 0xD2, 0xD6, 0xEB, 0xEF, 0xE3, 0xE7, 0xFB, 0xFF, 0xF3, 0xF7,
0x18, 0x1C, 0x10, 0x14, 0x08, 0x0C, 0x00, 0x04, 0x39, 0x3D, 0x31, 0x35, 0x29, 0x2D, 0x21, 0x25,
0x5E, 0x5A, 0x56, 0x52, 0x4E, 0x4A, 0x46, 0x42, 0x7F, 0x7B, 0x77, 0x73, 0x6F, 0x6B, 0x67, 0x63,
0x94, 0x90, 0x9C, 0x98, 0x84, 0x80, 0x8C, 0x88, 0xB5, 0xB1, 0xBD, 0xB9, 0xA5, 0xA1, 0xAD, 0xA9,
0xD2, 0xD6, 0xDA, 0xDE, 0xC2, 0xC6, 0xCA, 0xCE, 0xF3, 0xF7, 0xFB, 0xFF, 0xE3, 0xE7, 0xEB, 0xEF,
0x31, 0x35, 0x39, 0x3D, 0x21, 0x25, 0x29, 0x2D, 0x10, 0x14, 0x18, 0x1C, 0x00, 0x04, 0x08, 0x0C,
0x77, 0x73, 0x7F, 0x7B, 0x67, 0x63, 0x6F, 0x6B, 0x56, 0x52, 0x5E, 0x5A, 0x46, 0x42, 0x4E, 0x4A,
0xBD, 0xB9, 0xB5, 0xB1, 0xAD, 0xA9, 0xA5, 0xA1, 0x9C, 0x98, 0x94, 0x90, 0x8C, 0x88, 0x84, 0x80,
0xFB, 0xFF, 0xF3, 0xF7, 0xEB, 0xEF, 0xE3, 0xE7, 0xDA, 0xDE, 0xD2, 0xD6, 0xCA, 0xCE, 0xC2, 0xC6,
0x29, 0x2D, 0x21, 0x25, 0x39, 0x3D, 0x31, 0x35, 0x08, 0x0C, 0x00, 0x04, 0x18, 0x1C, 0x10, 0x14,
0x6F, 0x6B, 0x67, 0x63, 0x7F, 0x7B, 0x77, 0x73, 0x4E, 0x4A, 0x46, 0x42, 0x5E, 0x5A, 0x56, 0x52,
0xA5, 0xA1, 0xAD, 0xA9, 0xB5, 0xB1, 0xBD, 0xB9, 0x84, 0x80, 0x8C, 0x88, 0x94, 0x90, 0x9C, 0x98,
0xE3, 0xE7, 0xEB, 0xEF, 0xF3, 0xF7, 0xFB, 0xFF, 0xC2, 0xC6, 0xCA, 0xCE, 0xD2, 0xD6, 0xDA, 0xDE
};

u8 const Adesto_Factory_Programmed[64] = 
{
  // put here AT45DB011D Security Register from byte 64 to 127
};

u8 OneTime_User_Programmable[64];

OneTime_User_Programmable[0] = 0x00; // or whatever you want
OneTime_User_Programmable[1] = 0x00; // or whatever you want
OneTime_User_Programmable[2] = 0x00; // or whatever you want

u8 n, value;

for(n=3;n<64 br="" n="">{
 value = (u8)(Adesto_Factory_Programmed[n] + n);
 OneTime_User_Programmable[n] = GreenDot_Table[value];
}

For more on this whole sad saga, you'll want to read this masterwork by torkelmj, especially if you enjoy reading stuff dripping with spite and vitriol.  Like I do.  Or skip that and just buy a security-coded third party logger from WXForum awesomedude Belfryboy.  Or order up his PCB on oshpark and build your own.  There are options now, and options are good.

Now one might think that everything is just peachy now that DIY loggers are possible once again.  Not me.  I was pretty grumpy that Davis had tried to screw their customers over even if they did release subsequent firmware updates that tried to undo some of the damage.  Once burned, twice shy, so they say.  I started thinking of a Plan B.  I want to build my own console.
Gratuitous Picture of My Dog
My IM-ME thing was a neat hack, but that is about as far as it goes.  IM-ME's aren't actually rolling off the production lines in millions of units per month these days, and their ability to hook up to other little do-dads would not be easily done.  Cheap transceiver boards based off of the popular CC1101 chip that work in the 915 MHz band are hard as hell to come by.  Ask me how I know.  The Seeedstudio RFBee comes tantalizingly close, but others who have tried to go this route run out of RAM and Flash on the limited processor found on this unit before getting too far.

I needed another way and that led me to the Moteino with its awesome little RFM69B transceiver module.  I got it picking up Davis ISS transmissions earlier this year.  Since then I have been coding away in my spare time and have been pulling together the various bits of hardware that I need to get something going.  Something like this.
Beautiful, Isn't It?

Some cool stuff has sprung up from this very breadboard.  My serial connection to the console grew up there on the left side.  The DIY logger design is in the middle.  The stuff on the right half starting from the left is the humidity sensor, the pressure sensor module, the Moteino itself, and finally my USB to Serial Adapter for the project I'm writing about today.

Now I know what you are saying to yourself: "The title of the blog is "Build Your Own Davis Weather Station Console!!!" but that doesn't look like a console at all!  Where is the keyboard?  Where is the display?" Well, I guess I'm not going to get anything past you today.  There isn't a keyboard or a display... unless you want to add one.  The hardware and software here is all open for you to add on what you'd like.  Me, I'm thinking of hooking this bad boy to a serial port on my Beaglebone running Openhab and having all my weather data available from anything in my house that can bring up a web browser.  Very doable, and no display or keyboard necessary for that kind of application.

But why call it a "Console"?  Because I couldn't think of a better name.  And I did indeed try.  It isn't a "Receiver" because it does more than receive given that it has its own sensors hanging off of it.  "Indoor Unit" sounds stupid.  "Thingamahooie" was tempting, I must admit, but didn't quite convey the application well enough.  And hey, the guy that builds the hardware, develops the code, and writes the blog gets to call what he comes up with whatever the hell he wants.  I'm going with "Console", and you'll just have to deal with it.

But I, as I so often do, digress.  Back to business.  On the hardware side of things, here is what I'm working with in more detail:
  • Moteino with an RFM69 module on it.  Pick the 915 MHz RFM69W (standard transmit power) or RFM69HW (high transmit power) at your option.  I have the serial version but it probably would have made sense to pick up the USB variant so I wouldn't need to tie up my USB-Serial converter all the time.  You probably want to add on the male headers option (why even think about it for the one dollar price?) and the flash memory option for four dollars.  I'm not using that yet, but it might come in handy for data logging at some point.  And if you re-dedicate the Moteino to something else later, this memory can be used for wireless remote programming.  Cool.
  • a DHT22 Temperature and Humidity Sensor for less than five bucks.
  • a BMP085 Temperature and Pressure Sensor for less than six bucks.  But it looks like that module is now much cheaper thanks to the compatible BMP180 sensors now out there for under three bucks.  The two are compatible, and I'd probably go with the newer BMP180 myself just because it is smaller.
  • a (yet to arrive and be connected) DS3231 Real Time Clock and Memory Module for less than three bucks (I felt bad for getting something at this price with free shipping so I bought two).  Note that this module also has a 24C32 EEPROM on it that is bound to come in handy.  Don't be dumb and order a DS1307 clock module instead: that one doesn't work well with the 3.3V Moteino and it is horribly inaccurate.  What good is a clock chip that can't keep time?
Arduino is all about little software libraries, and these are the ones I'm working with right now.
  • the LowPowerLab RFM69 library.  Felix has merged the changes I've needed so no need to use my fork of his code any longer.  I don't know if I'll be using this in the long term or come up with my own that is better suited for this application.  Another library that I'm including right now but not really using yet is his SPIFlash library.
  • the JeeLabs DHT22 code.  I have extracted what I need from the JeeLib code and included it into my own library.  JCW's stuff is a leaner and meaner version of some of the other ones floating around on the net.
  • the Adafruit BMP085 library.  I used this as is.  I did look at using the JeeLabs code but it was a little too tied up with JCW's PortsLib for me to bother untangling at the time.
  • the Arduino Serial Command library to parse command input.  This is a really great little library that lets you register callbacks for each separate command and also does command line argument parsing.  I needed to make a few tweaks as noted below, but nothing major.  My changes have been pushed but not merged at this time, so use my fork here.
  • my own DavisRFM69 library, of course.  I've got to be doing something to make all this stuff work together.
Enough chitchat.  Let's cut to the chase, shall we?  Take a look at this thing actually doing something.
Success - I Never Tire of It
This is SandaySoft's Cumulus software talking to my homebrew weather station console.  The outdoor readings are those from my ISS in the front yard.  The indoor readings are from the sensors I have hanging off the Moteino.  On the left you can see the Serial Port Monitor Software recording the RS-232 messages going back and forth.  Cumulus has no idea that it isn't talking to a real Davis console.

Thanks to the Davis Serial Protocol Manual, getting this working wasn't too terribly difficult.   I knew Cumulus used the Davis SDK to talk to the console so the Cumulus author might not be aware of every single command being sent to the station and back.  I also knew that the LOOP command was used to get the data out of the station so I implemented that and some other useful commands on the Moteino.  Then I hooked my real console up and watched some packets go by and found a few more I needed to implement (namely BARDATA, HILOWS, and RXCHECK).  I had a few hiccups of course.  For example, Davis says that a command should always be terminated with '\n' (0x0a) like this...
 except when they don't, like this...
And that is the fun with emulation: you have to emulate not only the functionality, but the bugs as well.  This one caused me to tweak the serial port library to accept multiple command terminators.  Another oddity is that the console itself goes to sleep all the time and is woken by simply sending a lone '\n' that the console is supposed to acknowledge.  The serial library I'm using hadn't actually accounted for that situation.  Now it does.

Want to play along?  Here is what you need to do.   From within your Arduino's libraries directory...
  • Get my version of the SerialCommand library that supports empty commands and a second line terminator.  And of course, you'll want to get a copy of my DavisRFM69 library.  The pared down DHT22 library is tucked away inside the latter.
git clone https://github.com/dekay/Arduino-SerialCommand.git SerialCommand
git clone https://github.com/dekay/DavisRFM69.git
git clone https://github.com/adafruit/Adafruit-BMP085-Library.git Adafruit_BMP085
git clone https://github.com/LowPowerLab/RFM69.git
git clone https://github.com/LowPowerLab/SPIFlash.git
Hardware hookup is like this.

DHT22 Pin 1 -> Moteino 3.3V Out
DHT22 Pin 2 -> Moteino D4 with 10K Pullup resistor to Moteino 3.3V Out
DHT22 Pin 3 -> Not connected
DHT22 Pin 4 -> Moteino GND

BMP085 Module Pin VCC -> Moteino 3.3V Out
BMP085 Module Pin SDA -> Moteino SDA
BMP085 Module Pin SCL -> Moteino SCL
BMP085 Module Pin GND -> Moteino GND
Once you've got all of this hooked up, you'll find a "VP2.ino" sketch in the DavisRFM69 library's Examples directory.  Open that up in the Arduino IDE and select Control-U to compile and download it to the Moteino (be sure to tell the Arduino IDE that you are using an Arduino Uno).  Open up the serial monitor, set the baud rate to 19200 and "Newline" as the line ending.  You should see... absolutely nothing.  Nothing that is, until you type in some (uppercase) commands like "LOOP 1", "TEST", or "STRMON".  The Moteino will respond exactly the same way as a real console would.  Hook it up to Cumulus (with the "Use Data Logger" option unchecked) and revel in a job well done.

I'm off to a decent start here but there is still lots to do.  Here are some things on my list.

  • integrate the real time clock functionality
  • put in the frequencies for consoles in Europe and other regions
  • add some more useful commands like PUTRAIN so you can tell this thing how much rain you've had this year that it can keep a running count of afterward
  • add in some kind of EEPROM functionality so it retains settings like total rainfall between power downs.  Conveniently, there is an EEPROM on my yet to arrive three dollar clock module just begging to be used.
  • make the reception more resilient.  There are some odd things going on here: I get a spurious interrupt on startup (setup issue?) and when I move the board around (need a pulldown on the interrupt line?)  Reception is also poor during the day.  Might be that I just need to adjust the clear channel RSSI value a bit to accommodate more background noise.
  • integrate the real time clock functionality (daily / weekly / monthly / year highs and lows)
  • etc
Anyone that has done much coding on the Arduino gets to know memory limits.  The Moteino has double the RAM and EEPROM than the standard Arduino UNO, so I've got a whopping 2K of RAM and 32K of Flash to play with.  So far I have used roughly half of each, so I've got some headroom.  I haven't done much to try and streamline any of the code besides a few const arrays here and strings stuffed in the F() macro there, so I am sure there is room for improvement if things get really tight.

With this project, I think I've hit the goals I set out to achieve: come up with something functional, inexpensive, and with a low barrier to entry so that other people can dig in.  Have a new feature you'd like to see?  Send me a pull request on Github.  Want to develop a plug in circuit board (kind of like this) with all the dodads attached to lower the barrier to entry even further while making a few bucks on the side?  Be my guest.

I'll wrap this up by saying that the Davis guys make good stuff.  The ISS is very well put together and I love that the thing is solar powered with battery backup.  I've had mine for years and it has been operating trouble free on Battery #1.  The display on the indoor console is nice, big, and easy to read and it also just sips on battery.  I'm guessing that 99.9% of Davis' customers are going to continue buying standard Davis stuff off the shelf despite whatever I come up with here.  Anything I play around with  is ony going to be picked up by that other 0.1% of enthusiasts.  And it is those enthusiasts that are not only going to continue buying Davis products but will recommend them to their friends as long as they are kept happy.  If Davis pulls another stupid Firmware 3.0 thing, they'll do more damage to themselves than I ever did.  I hope they continue documenting their communication protocol and maintaining accessible expansion ports in their current and future products.  That would be a win-win for everybody.

With that in mind, it seems as good a time as any to unveil the new profile picture.
Be Nice


Monday, January 20, 2014

For Our Dogster Friends

Update February 11th: Monster Speedup and Experimental Photobucket Rules

One thing that has become quickly apparent is that HTTrack can be ridiculously slow especially on bigger pages.   I have found a way to speed it up.  Way up. It appears that it throttles the download speed without telling you. There is a spot in the user interface that lets you set the maximum speed, but that does nothing unless you know a trick

Here is a shortcut to getting the speedup. Try putting the following in your Scan Rules
--disable-security-limits
--max-rate 1500000
--assume php=text/html
The first one is the secret to taking the brakes off. The second one is the maximum rate in bits per second. We have a two megabit download rate at our house, so I've set the maximum to three quarters of that, or 1.5 megabits per second. If you set this number too big, it just means it will hog your entire network connection. A rough rule of thumb might be that if you are getting internet through your phone company, try 1500000 as we have. If to the cable company, they are usually faster so try 5000000 (five million).

The "assume" bit is recommended in the HTTrack manual to speed up how it parses the files if you can make certain assumptions, and this assumption seems to work great.  Now it is only taking minutes to download smaller pages. The difference is night and day.  These scan rules are now in the main body of the blog post.

Not in the main body of the blog post is some new filters I have been using that seems to work well for downloading from Photobucket.
+*.photobucket.com/*.jpg
+*.photobucket.com/*.gif
+*.photobucket.com/*.png
-blog.photobucket.com/*
-secure.photobucket.com/*
-support.photobucket.com/*
-tutorials.photobucket.com/*
The lower in the list, the higher the priority, so the order above of these filters is important. The rules start by saying you can only load .jpg, .gif, and .png files from Photobucket. That stops it from downloading all the html pages that then takes you down the rabbit hole and downloading the entire site. After that, it specifically blacklists (with the minus signs in front) the blog, secure, support, and tutorial sections of the site. This is just to be on the safe side.  This has worked well for all the sites we have tried, but keep an eye on it if you decide to try it.  I have not put this in the main body of the blog post but give it a shot.

Finally, another problem we've been having is that some pictures weren't showing up after a successful download. I suspect this is because of occasional heavy traffic clobbering the site. Try the following if you are having trouble:
  • From the same window the Scan Rules are entered in, select the "Flow Control" tab
  • In that tab, enter 5 into the Retries field. 
  • Everything else is as per usual. 
Good luck!

Update February 9th: Disable Photobucket Scan Rules

It sounds like some people are running issues where HTTrack wants to download half of Photobucket.  It looks like you will need to figure out more specific scan rules than the ones I have in there now.  I have removed references to Photobucket from the post.

Update February 8th: Fixes and the Addition of Catster!

I have updated the scan rules to fix a few things and have created a set of scan rules for Catster as well.  The new rules are in the main body of the post.  A shout-out to the Dogster (and Catster!) folks that have helped out so much.
  • The gift archive now works.  A new scan filter "+www.dogster.com/subscr/gift_archive.php?pet_id=590841" (or whatever your pet's number is) does the trick.
  • Thumbnails in photo comments now work.  Dogster, for some bizarre reason, has those thumbnails in a URL that looks like "+files.dogster.com./blahblahblah".  Did you catch the weird period between "com" and "/" in that last sentence?  Neither did I at first.  Tweaking the Scan Rule from "+files.dogster.com/*" to "+files.dogster.com*" did the trick.
  • Added a scan rule"+badge.dogster.com/*" to fix missing badges.
  • Added a few new scan rules that should help images from flickr, photobucket, and one other site I'd never heard of come through: "+*mytextgraphics.com" "+*.photobucket.com/*",  "+farm*.staticflickr.com/*", and "+farm*.static.flickr.com/*".  You can skip these if you don't have images off the main site, but leaving them in doesn't hurt anything.
  • Added a set of scan rules for Catster.  Did you know that the images on Catster are stored on servers named Dogster?  Neither did I at first.  For this reason, the Catster list will have a "+files.dogster.com*" entry in it on purpose.  It will also have a "+files.catster.com*" on it that I doubt is needed, but better safe than sorry.
  • Added a scan rule that pulls in a specific file used to format the display of the photo pages "+www.dogster.com/photos/photo.css", though I didn't really see a difference when it was added.
  • Reordered the list of scan rules to be a little more sane
Attempts by the Dogster folks and myself to display images in the Pup Pals pages have failed miserably.  I suspect that it has something to do with that being a page generated on the fly with PHP and this is a known issue with HTTrack, but the gift history is also PHP and it works.  Weird.

It doesn't hurt to test the archive out by disconnecting your Internet connection, clearing your browsers cache (Google is your friend here), and then bringing up the web page created by HTTrack for your pet to see if anything is unexpectedly missing.  Note that the page might be slow to load this way because it is trying to reach out to some sites like Facebook, Google, and Twitter that I have intentionally not mirrored.  Your browser will wait for a bit, give up, and then load the archive anyway.  These links don't affect the display of your stuff and they'll be there long after Dogster Doomsday.  Once you connect the Internet back up, it will reach these sites normally anyway

If you are an old hand at this archive process, the only substantial changes to this post are in the "Specify The Options" part.  The rest is the same, so you can save yourself some reading.
Finally, some of you might be wondering why this archive takes so damn long sometimes.  I was mystified by this for a while too until I ran a test on a cat that had 36 pages (!) of diary entries.  I checked out the folder where the diaries are stored and there were a ton more entries than that.

WHY?

Well, there might be 36 pages of diary entries, but there are also five or six diary entries per page.  If you click on a "Comments" link, a whole new page opens up for just that diary entry.  Same thing for photos.  So if you have ten pages of diary entries and ten pages of photos, you don't end up downloading twenty pages.  You download twenty pages where entries are grouped together PLUS over a hundred pages dedicated to each individual diary entry and photo.  Yeah, that is a lot, but that is the only way to see the comments.  It also doesn't help that the download speeds from Dogster and Catster seems to be less than speedy in the first place.

"Beepers" on the Dogster forum posted a suggestion on how to speed up an update if you already have an existing download.
  • On the first option page, you can choose "Existing project name".  Click the arrow to choose the one you want to update.
  • On the next page/screen, in the "Action" block, click the arrow again and choose *Update existing download.
  • Then click on "Set options" and the "Scan Rules" tab as usual.  All your rules for the previous save will be there.  Just add the new rules at the bottom and proceed as usual.
Give it a shot!
          

Update February 5th:

The Bad News

Folks on the Dogster forums groups discovered that the archives would not work when they had no Internet connection.  That implied that the download would stop working once Dogster shut down.  I verified that this is a problem and am very sorry to say that all the downloads that everyone has done to this point will be useless once Dogster shuts down. The web browser was doing some stuff behind the scenes that I wasn't aware of.

The Good News

I have a fix that I would like people to try. I have tested this on a couple of pages now with my browser cache cleared and our internet disconnected and it seems to work perfectly. It still has the same limitations as before (can't click on gifts to get details, etc) but all the important stuff will be there. You will still be able to navigate around the page just like you did before.

The Fix

A new set of Scan Rules is now in the main body of the post.  The rest of the steps should be the same.  Try these new scan rules out and post feedback in the comments.  If you are computer savvy, you can double check by clearing your browser cache, disconnecting your Internet connection, and loading the archived page.  If everything looks good, you are ready for Doomsday when Dogster shuts down.

Please test this out and give feedback in the comments.  Once again, I'm sorry for my error. If I knew what I was doing here, it would help!

P.S.

Some people have been worried about seeing stuff like this in their browser address bar.

file:///C:/My%20Web%20Sites/cocopuffondogster/www.dogster.com/dogs/500107.html

The worry is that www.dogster.com is showing up there still.  That isn't actually a problem as long as the start of the address starts with "file:///C:/blahblahblah".  In this case the "www.dogster.com" part in the middle is actually now a directory on your hard drive.  You can use Windows Explorer or whatever to actually navigate to that directory and below.  There is all kinds of neat stuff tucked away in there!

          

Update Jan 25th:

See near the bottom of the page for instructions on how to save your videos!  A couple limitations of this technique are also discussed.  And after all of this, look who was made Dog of The Day for January 25th, 2014!!!


          

One of the saddest times of my life was when our dear dog Abby passed. She was my best friend and a big part of our lives. That was about a year and a half ago now and it still brings a lump to my throat just thinking about that awful time.

In the days that followed, I was struck by the sympathy and compassion expressed by My Lovely Wife’s friends in the Dogster community. Dogster is a website that dog lovers from all over the world congregate to share photos, stories, and advice on their best friends. While I don't doubt that some of these folks might be a few Milk Bones short of a full bag, My Lovely Wife has developed a true friendship with many people on that site, none of whom she has ever met in person. Our dog now gets more Christmas cards than we do.

Unfortunately, word went out a short time ago that Dogster will be closing down. All of the pet pages with the photos, diaries, and associated comments will disappear. People are understandably upset because they don't know how to save all of those memories. Some are printing the pages off, some are resorting to screenshots.  And some are just at a loss.

But I think I know a better way. And I'll show you how to do it yourself. Step by step.  When you are done, you'll have all of your stuff for your dog.  It won't copy forum discussions and stuff like that.  But it will copy all of your diary entries, all of your photos, and all of the comments on that stuff.  And it will store all that away on your local hard drive.

Ready to go?  Good.  Start with...

Read This Whole Thing Through

Like a recipe, I highly recommend reading this through once or twice to get some idea of what you'll be doing.  It does assume you know your way around your computer a little bit, but you don't need to be a guru or anything like that.  If you are comfortable installing a program on your computer and navigating around the various files on your PC, you should be just fine.

Download HTTRack

HTTrack bills itself as a website copier.  Click on this link, download the appropriate file for your PC, and install it on your computer.  There are a bunch of different versions for the various flavors Windows and whether you are running the 32-bit or 64-bit version, so choose wisely.  You'll be out of luck if you are on a Mac (sorry) unless you know how to compile your own programs.  There looks to be versions for various Linux distributions though for the 0.001% of Dogster users on that operating system.

During the installation process, the program will ask you if you want to put an icon on the Desktop and in the Start Menu.  Let it do so.

Start WinHTTrack

Click the icon the installer put on your Desktop.  The program should start up, displaying one of the ugliest user interfaces in existence.  That's OK.  We aren't using it for its looks.  When you get the screen below, hit the "Next >" button to continue.


Start a New Project

We start out by defining a new project as follows:
  • New project name: This is what the program will name the directory where it stores the files you will download.  I'll be using Abby's Dogster page as an example, so I called my project "AbbyOnDogster".  Use whatever name you like here.
  • Project category: Just leave this blank
  • Base path: If you were going to download multiple Dogster pages, then all of them would go into this directory.  You can use the default "C:\My Web Sites" (it works fine) or specify any other folder you like.

Once you're entered that information, hit the "Next >" button to continue.

Specify the Site To Mirror

So far so good, but now it starts getting a little trickier.  Fill out the next window as follows.
  • Action: Stick with the default of "Download web site(s)"
  • Web Addresses: (URL): The best way to fill this in is to navigate to the Dogster page you want to clone, select the address area, copy it (Control-C), and paste it (Control-V).  Or just type it in from scratch.  The window below shows the address for Abby's page.  It will look something like http://www.dogster.com/dogs/590841.  Click that link to see what I mean.
  • The tricky part is coming up.  Do not hit the "Next >" button yet.  Instead, select the "Set options..." button above it.

Specify the Options

(This Section updated February 5th and then on February 8th to fix the Scan Rules as mentioned at the top of this post).

When you hit the "Set options...." button, there will be a bunch of tabs on top.  Click on the tab that says "Scan Rules".  Don't worry about any of the other tabs.  See the area I have circled in blue?  Click your mouse in that area and delete every single character in there.  You want it to be perfectly empty.  Do not hit the "Exclude link(s)..." button.  Do not hit the "Include link(s)..." button.


Now, copy the four thirteen all of the lines of text below and paste it into the now empty area that has the blue circle above.  Cut and paste is your friend here.
-*
+www.dogster.com/dogs/590841
+www.dogster.com/dogs/590841/*
+www.dogster.com/subscr/gift_archive.php?pet_id=590841
+www.dogster.com/static/*
+www.dogster.com/files*
+www.dogster.com/images/*
+badge.dogster.com/*
+files.dogster.com*
+www.dogster.com/photos/photo.css
+*.cdnsters.com/*
+tetra.saymedia.com/*
+*amazonaws.com/*
+use.typekit.com/*
+*mytextgraphics.com*
+farm*.staticflickr.com/*
+farm*.static.flickr.com/*
--disable-security-limits
--max-rate 1500000
--assume php=text/html
Note that there is now a * after "+files.dogster.com".  This is a change from my first and second cracks at this.  Don't miss that one.

What if you have a cat?  Other than having to vacuum hairballs off the carpet on a regular basis, you want to use this set of scan rules instead.  Abby was not a cat, but work with me here.  Again, cut and paste is your friend.
-*
+www.catster.com/cats/590841
+www.catster.com/cats/590841/*
+www.catster.com/subscr/gift_archive.php?pet_id=590841
+www.catster.com/static/*
+www.catster.com/files*
+www.dogster.com/files*
+www.catster.com/images/*
+badge.catster.com/*
+files.catster.com*
+files.dogster.com*
+www.catster.com/photos/photo.css
+*.cdnsters.com/*
+tetra.saymedia.com/*
+*amazonaws.com/*
+use.typekit.com/*
+*mytextgraphics.com*
+farm*.staticflickr.com/*
+farm*.static.flickr.com/*
--disable-security-limits
--max-rate 1500000
--assume php=text/html
As explained up top, there are lines for both "+files.catster.com*" and "+files.dogster.com*" because Catster's images are actually stored on Dogster's servers.

This might look like Greek to you, but every single character here is important.  Make sure that these four thirteen lines get copied EXACTLY into the area circled in blue above.  That first line is indeed a minus sign followed by an asterisk.  All the other lines start with a plus.  All of this is important.

Or almost all of it.  The "+*mytextgraphics.com/*" won't be necessary for most people and could be safely left out.  One of our friends had a link to this website for an animated image of her pets name so this had to be included to get that to show up.  Others had images from Flickr (+farm*.staticflickr.com/*) and Photobucket (+*.photobucket.com/*).  Again, you can leave these out if your page doesn't link to these sites, but leaving them in there doesn't hurt anything either.  You would have to do something similar for any other off-Dogster sites that you link to for special images like that.  Just be sure to start the name with a plus and follow the format I've used.

You should end up with something that looks exactly like this when you are done.  Unfortunately the Scan Rules list is kind of long now so stuff is getting cut off on the bottom of the list.  But you get my drift.

Now if we were to proceed on from this point, it would copy Abby's page on Dogster because her Dogster ID number is 590841.  What you need to do now is replace the number 590841 with your own dog's ID number in all three places: the second, third, and fourth lines of the list.  Your dog's ID number is the one that shows up in the address bar of your web browser whenever you go to your dog's pet page.  It is usually six or seven digits long.  Do not add or subtract asterisks, slashes, or plus signs, or anything else unless you really know what you are doing.  Just change the number.  Get this wrong and you could end up downloading half of YouTube.  Ask me how I know...

Once you have changed 590841 to your own dog's ID number, hit the "OK" button at the bottom of the window and pat yourself on the back.  The hard part is done.

After hitting the OK button in the window above, you'll be back to the window that offered you the "Set Options" button in the first place.  You've already done that, so you can hit the "Next" button to Continue.


Breeze Past Connection Parameters

The next screen you'll see is shown below.  This is just getting fancy now.  You shouldn't have to worry about anything on this window, so hit the "Finish" button at the bottom.  And cross your fingers, because once you hit "Finish" the download will begin.


Watch the Download

Here is where it gets exciting.  The program is now going to go out and fetch your Dogster page, with all the associated diary entries, photos, and comments.  It is going to store them on your hard drive and change all the links around so that everything works on your local computer.  What it won't do is  create a copy of all the forum discussions or download all of your friends pages.  It gets your stuff for your dog.  And that is the most important thing.


How long this part takes depends on how much stuff you put on Dogster in the first place and the speed of your Internet connection.  Our connection isn't that great but there is a fair bit of stuff on Abby's page, and it still took around five fifteen minutes (the bigger set of Scan Rules from my first crack at this causes a fair bit more stuff to be downloaded).  As it runs, you'll hopefully see some familiar text going by.  The window above has some text that says "...bys_adventures", short for "Abby's Adventures".  This kind of thing told me I was on the right track.

Success!

When your pet's page has been copied, you should see a screen like this. Click "Browse Mirrored Website" and hopefully what you'll be looking at is a copy of your pet's Dogster page.

 

The actual page will be found where you specified it in that very first screen.  In this example, it would be in a place called "C:\My Web Sites\AbbyOnDogster".  That file will have a directory in it called "index.html".  Just click on that anytime you like and you should see a replica of your dog's pet page.  For our example, the address bar should look something like this.


See how it says "file:///C:/My Web Sites...".  That means that this file is on my local hard drive.  Contrast this with what it says when I access it over the web:

If everything went well, you'll have a copy of your dog's pet page tucked away nice and safe on your hard drive.  Back it up to a USB stick, move the directory around, or do whatever you like.  Have more than one dog on Dogster?  Just repeat the process again as many times as you like.  The ID number is the key.  To be on the safe side,it might be best to use a different project name each time.

Please note that I've put together this method and this writeup in a hurry.  There might be some cases where the process doesn't work.  If you run into trouble, leave me a message in the comments section and I'll try to help you out.  I'll also try to update this page based on people's feedback to clarify any spots where I haven't been clear.

Now I know that not everybody is going to be computer-savvy enough to do this.  But some will be.  All I ask is that those who can figure it out pay it forward and help out those who can't.  Back up their page, zip up the file, and email it to them.  They'll be incredibly grateful, and you'll feel good for helping them out.

Update - January 25th, 2014

First of all, I've been made aware of a couple limitations of this technique:
  • the link to the "Rosette, Star, and Special Gift History" doesn't work
  • clicking on those items from the main page doesn't give the pop-up message from the sender
  • the link to "See all my pup pals" doesn't work
  • the link to the video page doesn't work and the videos aren't downloaded
Now, I am no web guru but it looks like all these things have something in common: the links are not "static" but are dynamically generated when the link is clicked.  That is something that the web mirroring software used in this technique can't handle.  Other web mirroring software is similarly handicapped.

I was asked in one of the comments about videos and, while I can't get that to work as part of this process, I have found a workaround that will let you save each of your videos to your computer as separate files in case you no longer have the original version.

Basically, we are going to set up a "flash downloader".  This is generally an add-on to your web browser that lets you save the videos that you normally don't have an option to save.  The videos on Dogster are in something called "flash video" format, so you'll need a "player" for that too once the file has been downloaded.  I get into that as well.

Other methods besides the one I have tested below are bound to work and you are free to give this a shot with a different browser or technique, but I probably won't be able to help you much if you veer off this beaten path.  And this method should work for everybody on any operating system.

Want to get your videos back?  Here is how.

Step 1: Download Firefox If You Don't Already Have It

Why Firefox?  Because that is what I use, it worked for me, it is awesome, and it is free.  If you already have Firefox, you are awesome and can skip this step.  If you don't, download it from this link and become awesome.  Go ahead.  I'll wait.

Step 2: Open This Post in Firefox

If you are already using Firefox, skip this step as well.  If not, copy the address link for this post from your current browser to Firefox

Step 3: Get the Download Flash and Video Add-On

Like other browsers, Firefox's functionality can be extended with things called Add-Ons.
Install the Download Flash and Video Add-On from this link and clicking the "Add to Firefox" button that shows up in big letters on something like the snapshot below.

Firefox might give you a big scary warning like the one below.  No worries here.  There will be a number that counts down on the greyed out "Install" button on a window...

After a few seconds, the button will change to say "Install now" and it will no longer be greyed out.  Click on "Install Now".  The add-on will install and Firefox will tell you that it will be enabled when you restart Firefox.  Select "Restart Now".  Firefox should restart and bring you back to this page.

Step 4: Show Add-On Bar

Firefox Add-Ons are shown by default on an Add-On bar that is at the bottom of your browser window.  If you don't see a little box with a down arrow in the bottom right hand corner of your browser, hit the "Control" and "/" keys at the same time to make it appear, like so (I've enlarged it to make it easier to see).

Step 5: Get Your Video

So far, so good.  Next thing you want to do is open up your Dogster page and click on the link to open the page showing all of your videos using the "See my Video Book" link on the left side of your pet page.  What you are going to do is...
  • Start the video
  • Pause the video once it starts playing  (it seems to be important that you do this).
When you do this, you'll see the little icon in the bottom right change color.  Click on the icon and you should see something like the window below.  Ignore the "Flash Files to Download" on top.  Click on the video title under "Videos to Download" and the file should download to your computer!

I expect that the name of the downloaded file will look like some random collection of letters and numbers.  You'll certainly want to rename it to something that is understandable.

Step 6: Play the Video

I won't go into too much detail here.  Some computers might already have the software installed to play these videos.  Try opening / double clicking the file to see if it plays for you.  If it does, you are all set and can proceed with downloading all of your other videos.  If not, you'll want to install a piece of software to do so.  I recommend VideoLAN, as it is free, fantastic, and can play just about any kind of video file you throw at it.  It also runs on just about any computer out there.  Get it from this link.

          

Good luck retrieving the memories of your dog and those of your friends.  Mine are incredibly important to me.

Friday, January 3, 2014

More Than One Way to Skin a Cat

I know what you're thinking: "DeKay hasn't posted in a while.  Maybe he's dead?"  Fear not.  For while the frequency of my posting has dropped, I remain very much alive.  I've just been a little busy at work.
Thankfully the holiday season rolled around and I was able to take a couple weeks off. I took that time to relax and catch up on all the things normal people like to do: enjoy some good food, visit with friends, and hack on microcontrollers with integrated RF transceivers.
This bad boy is the Moteino from LowPowerLab. It is an Arduino clone with a 3.3V Atmega 328p that integrates a HopeRF RFM69 RF module. And let me tell you, this little module rocks. They are small, cheap, and readily available. Felix from LowPowerLab is getting tons of range out of the high power variant of these little guys.  They are easily programmed over a SPI bus so you don't need special hardware like you do with TI's CC1110, and cheap versions operating in the 900 MHz band are easy to get unlike TI's CC1101.  And recently, interoperability with the RFM12B popularized by the JeeNode has been achieved.

Now the RFM12B deserves a bit of discussion.  Like the RFM69 module, it is small, cheap, and easily programmed.  But it sucks in a number of ways.
A small FIFO means that you better have something real time or close to real time listening to the module. Otherwise its buffer overflows and you lose data. Something like an Arduino works well because it isn't doing other stuff like running an operating system. But worse than this is the hardcoded sync bytes. This pretty much limits the RFM12B to talk to other RFM12B's. And that ain't cool.

The RFM69 fixes all these problems and adds some nice features while it is at it:
  • it has a 66 byte FIFO, a fully programmable sync pattern up to eight bytes in length, and a variable length preamble
  • address checking and CRC validation is optional
  • it supports OOK, FSK, or GFSK
  • it can do AES encryption
  • other stuff
In other words, it looks like it can do what other transceiver modules can do and more.

Intrigued, I picked up a Moteino with integrated RFM69W module and a couple standalone modules while I was at it.  Then I got busy (see above).  Then I got some free time (see above).  Then I started hacking (see below).

Devoted readers of this blog will know that I am strangely compulsed to receive wireless transmissions from my Davis weather station using all kinds of strange hardware platforms.  So why stop now?  I've been looking for a platform like this that is available and easy to use so that more people can take a crack with this stuff (it isn't like you can walk into a Toys-R-Us and pick up an IM-ME anymore).  I also don't want to be glued to the Arduino for everything.  While it is great for running as a remote node, I like the idea of something like my BeagleBone Black connected directly to a transceiver module running some home monitoring and control software (HouseMon?) with Internet connectivity.

But, baby steps.  What I decided to start with was my RFM69 equipped Moteino and the LowPowerLab RFM69 Arduino library.  Because once you've got the wireless protocol figured out, hard hard can it be?  Turns out, harder than I thought.  But after slamming my head against a brick wall a time or two, I got this.
Success!
This is the data from my Davis weather station out in my yard and received on my RFM69-equipped Moteino.  The first field is the channel number, the data is the eight data bytes sent by the ISS on every transmission (see here for what those bytes mean), RSSI is the Received Signal Strength Indication, and the CRC is calculated from the first six bytes in the data packet. Every packet is sent with a CRC in the seventh and eigth bytes, so those two values will agree with the calculated CRC if the packet is good.  I've got some problem somewhere where the first packet I receive is bad, as is any packet on Channel 42.  But those are details.  I'm off to a decent start here and I'm sure I can sort those issues out in time.

Want to check it out?  My DavisRFM69 library is on GitHub and the data above was collected using the ISSRx sketch in the Examples folder.  The approach that I've taken is to subclass my driver from the LowPowerLab one, overwriting just a couple of methods that are specific to this application (namely the initialization and the interrupt handler).  That reduces duplication and lets me automatically pick up any improvements Felix incorporates into his main library.  I have made a couple minor tweaks to his main library for some register definitions that I've issued a pull request for.  Hopefully these will be incorporated into his main library soon.  In the meantime, you'll want to use my fork of the library, so everything will compile cleanly.  This and more is all in the README.

This work can be taken a lot further:
  • do a full emulation of the Davis indoor console for use by weather software like WeeWx and Cumulus
  • hang an RFM69 directly off a BeagleBone Black and write a driver to work under node.js
  • why just receive?
Pull requests gladly accepted  :-)

I'm glad to now have this stuff under my belt.  I could have taken the entire break to just sit in front of the TV eating fruitcake.  That would be easy because My Lovely Wife makes a rocking fruitcake.  But what I really find satisfying is that sense of accomplishment when you set a challenge for yourself and make it happen.  This was a good way to kick off 2014, and the year is just getting started.

And don't get me wrong: I had lots of fruitcake.

Saturday, September 14, 2013

Dog Days of Summer No More

Back when my Lovely Wife and I first married, we bought ourselves a nice starter home in the city.  It was great, with friendly neighbors, a good sized yard, and lots of trees.  I love a nice tree.

The weeping birch tree in the front yard was one of the most spectacular in town.  The almost as tall tamarack tree in the front yard was very cool, as it is one of the very few pines that turns orange and loses all of its needles in the fall.  Sadly, both trees succumbed not long after we moved out: the weeping birch from bugs and the tamarack from lightning (!).
My Weeping Birch in Fall with Tamarack at Left
The backyard was nicely treed as well: several big spruce, a small oak, a nice birch, an apple tree, and a lovely Shubert chokecherry.  The latter had been pruned to grow as a tree with several main trunks that arched over your head as you walked under it.  Lovely.

There was also something about the neighborhood that I hadn't counted on: competition.  Many of the houses there had been built thirty years ago, and many of the original owners still lived there.  That meant a lot of these folks were coming up on or already retired and had a lot of time to put into their yards.  The neighbors on our one side were semi-retired and their yard was beautiful.  The neighbors on the other side were even worse: they were both young school teachers and had all summer off to work in their yards, and that they did.  The pressure to have a nice yard in the neighborhood was constant and intense.  I had to kick things up a notch if I were to hold my head up.

So I did.  I bought a really nice cedar gazebo kit and put it up underneath the chokecherry.  It was gorgeous, with its delicate railings and cedar shake roof.  It fit the setting perfectly, and the chokecherry looked like it had grown up around the gazebo.  It was beautiful and it bought me some good cred in the neighborhood but it wasn't enough.
Beautiful, But Not Enough
So I built an enormous pond at the edge of one of the gazebo's edges, and made it as deep as city bylaws would allow.  I stocked it with goldfish and koi, and even built my own floating lights that were powered by the rest of the low voltage lights in the surrounding flower beds.  The fish were really cool, and would swim up to us looking for food whenever we walked by.
Olympic Size
The pond became a veritable black hole of my time and energy.  Things would spin rapidly out of control.  I built a ridiculously long stream from the back edge of the yard that ended with a small waterfall at the pond.  This ridiculously long stream was made even more ridiculous long by meandering it along the way.  A 1200 gallon per hour pump provided water circulation.  The sound of babbling water that crept into our open bedroom window as we went to sleep every night was hypnotically comforting.
This is Only About Half of It
The stream cut the yard in half so that drove me to build a Pete's Pond Page bridge across it.  Its minimalist design was a perfect complement to the rest of the yard.
Heavy Rocks are Heavy
And then at the head end of the stream, I built something made to look a bit like a monstrous beaver dam / waterfall.  This was really big, but it was big for a reason: it hid my single stage Pete's Pond Page biofilter built from a 100 gallon stock watering tank.  Cotton batting served as a filter media and a small bag of barley straw in there prevented algae growth as if by magic.
That Was Easy!
We enjoyed this beautiful yard for several years before moving to another city in search of bigger and better things.  We sold the house to a nurse, and we were hopeful that she would keep the yard up.  What we would learn later was that she actually bought the place for her out-of-control son.  Her son and his friends then turned the basement into a crystal meth lab.  After several years of turning the neighborhood upside down, the bank foreclosed on them.  The chemicals from the drug making process seeped into the walls, and the entire lower half of the house had to be stripped down to the 2x4's and rebuilt before the house could be put back on the market.

Something tells me the drug dealers didn't much worry about cleaning the pond filter on a regular basis.  We moved back into the area a few years later, and I could not bear to drive by the place for many years after, fearing that all the work I had put into the place would have been destroyed by a bunch of stupid dope fiends.  It broke my heart.

Fast forward to present day.  We have been on our acreage outside the city for ten years now.  Much of that time we had The World's Greatest Dog on patrol.  We are just past the one year anniversary of her passing, and our new dog Buddy is now keeping an eye on the place.  Now the thing about Buddy is that he doesn't much like the sun.  Black Siberian Husky crosses would much rather be running across the frozen tundra than soaking up the rays on a hot summer day.

The problem we have now is that we don't have much for mature trees in our yard, especially in the area where my lovely wife spends so much of her time in the summer - the garden.  No big trees, no shade where a dog can hang out to help beat the heat.  There had to be a better way.  There is.  I was taking a stroll through Costco the other day and came upon this bad boy.
Meet the Alexa Playvilion Sandbox
The gears immediately started turning.  I would build Buddy a gazebo to call his own.

Careful inspection of the display unit built up at the store showed an impressive level of quality for the thing given its $100 price tag:
  • It was made of real wood that was already pre-stained
  • There were a ton of screws holding it together.  Screw holes were pre-drilled and counter-sunk
  • There were several nice finishing touches, like small ledges near the bottom and decorative railings at the top
  • The fabric cover was nice and heavy
Construction began the next day.  Normally I don't have too much trouble with the "Put Screw A in Piece 1" type instructions, but I was was thrown off by a couple things.  There were two different types of "Piece 1" pieces for the left and right edges of each side.  And I apparently have just enough dyslexia going on that I kept getting confused as to which piece should be on which side.  Finally, I got the first few sides together and took stock of the situation.

It would not do.  I held a piece of plywood up to the roof line around 11am and saw that the shade cast by the wood fell about six feet back from where the bottom of the gazebo.  The sides were too high to cast enough useful shade over the course of the day.  The happy young children pictured on the box the gazebo came in are doomed to die from skin cancer for this reason.
Sorry Kids, But I Got Bad News For You
Modifications would be in order.  It would be a tall order for someone with the woodworking skills of Homer Simpson, but I'd be damned if I was going to do a half-assed job of this.
His Spice Rack is Also a Classic
It took plenty of cursing and more than a little swearing, but I shortened the supports by around 18".  This would give a gazebo three feet high at the roofline.  The pre-drilling and countersinking of the new screw holes that would attach the bottom boards was the second worst part of it.  The worst part was doing that twelve times for each of the support posts in the kit.  But once that was out of the way and a test fit against the end customer proved successful, things moved along quickly.

I got each of the sides built up and screwed a few of them together.  Everything seemed to be fitting together pretty well.

The six sides are together now.  The picture on the box also had six sides.  I appear to be on the right track.

Here it is with the ledges attached on the bottom pieces and the roof supports up.  They are hard to see, but there are also chunks of wood angled at 120 degrees screwed across each of the corners.  This makes the whole structure surprisingly solid.  Note that I put something on top of the concrete in the driveway to prevent the wood from getting all scratched up.

Like any good engineer would do, we had to do a final test for fit and finish.  The end customer seems to approve.  Note also how my Lovely Wife has to duck down now that I've chopped the roofline down to Buddy size.

And here is the finished product in place.  We put it up on the north side of a small Scots Pine at the corner of our garden to help cast additional shade.  Note the enclosed pool that my Lovely Wife picked up early this summer at 50% off.

So the job is done so I know what you must be asking yourself.  Does he like it?  Well...
Damn Straight, He Likes It
As I tell my friends, I like all the places I live to have a pond and gazebo.  Anything else would be uncivilized.

Monday, July 22, 2013

Gas Problems

I have been having some gas problems lately.

No, not this kind of gas, though that would actually be a lot funnier.  Maybe I need to back up a bit.

I have been playing around with the idea for a new project that would involve doing a little assembly language.  I've done x86 assembly "back in the day" when I contributed some code for Retrocade, an arcade game emulator that was the fastest thing going at the time.
Too Bad It Died
Back then I used NASM, an x86 assembler that is still very popular today.  But I've also toyed around with the idea of getting myself a Raspberry Pi and maybe doing a bit of assembly on it.  NASM is x86 only, so if I wanted to do anything on the Pi, I'd have to learn another assembler while learning another assembly language. That would kind of suck.

This brought me to think about FASM. FASM has a variant called FASMARM that can assemble ARM mnemonics. However, FASMARM doesn't actually run on ARM itself. It can only generate code on a PC that then would have to be transferred to the Pi. That would kind of suck too.

Anybody that does any assembly is aware of the GNU Assembler gas (often written as, well, "as"). gas definitely sucks. It is the only commonly used assembler that uses ATT syntax. In a nutshell, ATT syntax is:
a) backwards
b) full of superfluous % signs
This seminal article on IBM Developerworks provides all the gory details in comparing NASM to gas.  Or forget about beating around the bush and read this quote from the 646 page tome Assembly Language Step By Step:
 The GNU assembler gas uses a peculiar syntax that is utterly unlike that of all the other familiar assemblers used in the x86 world, including NASM. It has a whole set of instruction mnemonics unique to itself. I find them ugly, nonintuitive, and hard to read. This is the ATT syntax, so named because it was created by ATT as a portable assembly notation to make Unix easier to port from one underlying CPU to another. It’s ugly in part because it was designed to be generic, and it can be recast for any reasonable CPU architecture that might appear.
Suffice to say, gas is generally avoided by programmers like Amy Winehouse avoided rehab.
Should Have Gone Into Rehab

However (and there is always a however), the GNU gods saw fit to introduce a compatibility mode to gas that allowed the use of Intel style syntax, the type used by programs such as NASM and FASM. Sounds good. I decided to give it a shot.  I would do so by taking the NASM examples in the Developerworks article and see if I could get them to work in gas using Intel-style syntax.  The binutils documentation led me to believe that this should be a pretty straightforward process.  I was wrong.  There was some headbanging on the way and I nearly gave up at one point, but now I have everything working well.  This is what I learned along the way, presented as gists on gitub that you can dive into and fork as you please.

Here is Listing 1 from the Developerworks article in Intel style syntax and assembled using gas.  This luckily worked right off the bat. The key thing to note in this example is
.intel_syntax noprefix
That is the magic incantation that makes it all happen. This key tidbit from Stackoverflow is what got me going down this road in the first place.  This directive tells gas to use Intel syntax and to not need the % prefix before register names.  Put that in there and besides the directives specific to gas, the code itself looks quite a bit like NASM.

Then I got to Listing 2 and cruised through it to, mostly because it didn't introduce much in the way of new concepts.  This looked like it was going to be easy.
Next up was Listing 3. Listing 3 was a son of a bitch. This code introduced macros and some program strings and constants tucked away in the .data segment (I changed this to the read only data segment - .rodata - because this data was, well, read only).

The problem I had with Listing 3 was that I could not get the strings in .rodata to print to save my life.  Running the code in the debugger shows that I was getting the string lengths read properly.  The problem was getting the address of the strings that were then sent off to the write macro.  My read macro wasn't working right either.  I banged my head against a brick wall for an entire evening before I found this on the net the next morning.  The key text is in the code but I want to put it down again below to drive the point home.
The somewhat mysterious OFFSET FLAT: incantation tells the assembler to figure out the (4-byte) address where the variable x will end up when the program is loaded. Even the assembler does not have all the information to figure this out, since a program may be in several pieces and the assembler does not know where each piece will go. It is up to the loader to figure this out, so in fact all the assembler does with the OFFSET FLAT: reference is to make a note in the object file and it is the loader that will finally fill in the right value in the generated instruction. This is one of the respects in which object code (which ends up in a .o file after assembly) is not pure machine code.
Well, tie me to the side of a pig and roll me in the mud.  This and similar code code
mov ecx, \str
as it was in the Developerworks article would refuse to work, and no amount of BYTE PTR, brackets, dollar signs, or percent signs brought me any joy.  But this code worked just fine
mov ecx, OFFSET FLAT:\str
You'll see that what I actually ended up using here was
lea ecx, \str
just because this version is more clear (I'm trying to load the effective address of that string), I don't need "OFFSET FLAT:", and lea is generally a more powerful instruction for loading an address that I could end up using later.  I would need "OFFSET FLAT:" for a mov, push or any other instruction referencing a memory location in .data or .rodata.  Note that I don't need it to access constants for the string lengths in those .sections.  That is because the assembler can at least figure that much out on its own.  The code below works perfectly.
There are a couple things to note in Listing 3 above.  First, the gas syntax for macros is still used.  Variable substitution is made with backslashed references to the macro parameters.  Just because we are using Intel style syntax doesn't change that part of it.  No problem.  The macro syntax in gas isn't arcane like the ATT syntax itself.

The second thing to notice is how I have coded the string length.  The Developerworks example says I should use something like this
greet_str: .ascii "Hello " 
gstr_end: .set GSTR_SIZE, gstr_end - greet_str 
when in fact I have used this (thanks again, Stackoverflow).
greet_str: .ascii "Hello "
.set GSTR_SIZE, . - greet_str 
Either the Developerworks article was written before the "." was introduced to keep track of the current location or the author wasn't aware of it.  This part of the code becomes cleaner and more NASM like since the extra label "gstr_end" is not required.

On to Listing 4 and we're on a roll.  You'll see the liberal scattering of "OFFSET FLAT:" because of all the non-lea instruction references to the strings in the .data section.  There are a couple things a little different in this one though.  First (shown in the code and mentioned in the Developerworks article), the parameters passed to the linker instruction are changed to link the external C standard library.  Second is the use of "BYTE PTR" where NASM got away with just using "byte".  I tried a search and replace of "BYTE PTR" to "byte" and the program stopped working properly, though gas did not complain.  I could have experimented a bit more with this to see what was going on here but didn't bother.  My initial read of the gas docs suggested to me that "byte" should have done the trick.
Finally, we get to Listing 5.  This was smooth sailing.  Again there is liberal use of "OFFSET FLAT:" to reference memory locations in the .data section.  There is also the introduction of the .rept command to simplify the allocation of ten memory locations, but this translated over straight from the gas code in the Developerworks article without a hitch.
And that about wraps it up. Knowing a few rather arcane tricks makes gas much easier to work with.  gas is also everywhere there is a gcc compiler, so you can use it pretty much anywhere on everything.  All you need to know is a few tricks to make it more usable, and now you know those tricks.  Give it a shot.