Making the Switch from iOS to Andriod

Well, it happened.  I made the switch.

Prior to 2009, I was far from being an Apple fanboy.  Growing up, the first computer I used was a 8088 running DOS 5, subsequent computers I had always ran Windows or Linux and I learned how to program on Microsoft based platforms such as Turbo Pascal, InterDev and Visual Studio.  Although I had to use a Mac running System 7 for a high school job,  Apple products were always a bit foreign and uncomfortable to use when I encountered them.  I’ll admit, this attitude was driven by ignorance in many cases but honestly Apple products just worked a bit differently than my brain wanted them to.  So, my opinion of Apple was largely negative and I didn’t give buying any of their products a second thought.

Then came the iPhone.

Yes, the iPhone was/is a phenomenal product and a true game changer.  Well played, Apple.  After my wife got me an iPhone for my birthday in 2009, my opinion of Apple began to change.  I tore down some of the ‘ignorance walls’ against their products and began to appreciate the design and quality.  I eventually bought a MacBook Air, an iPod Touch, and a few Airport Expresses.  And these were all good products I enjoyed using.

After my wife’s iPhone got stolen I decided to try my hand at a Free iPhone setup to save some money.  After having this setup for about a year, with some tweaks along the way, I had a few holes which kept frustrating me.  The first was the lack of vibrate mode on my iPod Touch.  I know, this seems trivial but it was actually a big deal for me since I would constantly have to manage the volume for meetings and miss notifications when I would forget to turn the volume back up.  When in my car and connected to Bluetooth, if I wasn’t on the Bluethooth input mode on my radio, I would not hear (or feel) any notification, sound, ring, etc.  This was really annoying, to say the least.  When I had a proper iPhone, I would just leave it on vibrate mode and put it in my pocket.  The other hole with my setup was not always being available to receive calls.  Although I had most of my bases covered with wifi at home, work, and in the car, there were still times that I would not receive a call through the Talkatone VOIP app, for various reasons such as my wifi hotspot being out of coverage or the app not running in the background because it had been killed by iOS.  96% of the time, it wasn’t a big deal to miss a call and to just call the person back when I saw the missed call notification.  But, the other 3% of the time it was downright unacceptable.  Once, I basically got stranded with my daughter just because I couldn’t make a simple call to my wife.  Also, it bothered me that my wife would possibly not be able to reach me in an emergency.  Something needed to change.

Then came the Nexus 4.   

A friend of mine told me about the new Nexus 4 that he ended up buying.  He gave me all the specs and explained how it was only $299, without a contract.  Wow, a full blown Andriod phone with all the bells and whistles I was missing on my iPod Touch (like GPS, vibrate, NFC, GSM radio for calls, etc.), a fast processor, 8MP camera and more for only $299 off-contract?  That’s exactly the price I paid for the latest iPod Touch when I bought it.  That got me thinking for sure.  But, the last time I had played around with Andriod on someone else’s phone I was not that impressed.  It seemed subpar and not nearly up to stuff with iOS.  But, my wife had recently gotten a Nexus 7 for Christmas and been telling my how much she liked the newer version of Andriod called “Jelly Bean”.  After I played around with the tablet a bit I agreed with her.  Jelly Bean is quite polished and leaps and bounds better than earlier versions of Andriod.

After considering it for awhile, I decided to make the switch.  The appealing thing for me was that I could have a full-blown phone for reachability, all the other perks of a smartphone, and not be bound by a contract with an expensive data plan .  I decided to purchase a T-Mobile pay-as-you-go SIM card with 1000 minutes for $100 (should last a year with my call usage history) and to use Wifi for all of my data.  Since I still have the FreedomPop in my car, I will have connectivity there.  Also, I still use Google Voice which helps me manage my prepaid minutes because I can just pick up an incoming call on my work phone or wife’s phone when I am able to.

In the end, it came down to cost.  If I were to go out and buy a new unlocked iPhone, it would cost me $649.  I got the Nexus for $300.  That is a significant savings.

Life with the Nexus 4

Cost and biases aside, I have to say that I have really been enjoying my Nexus 4.  The best thing about it for me in the seamless Google experience.  This phone does a great job of bringing all the Google services together in an integrated offering which really shines.  I use so many of Google’s services already and found myself fighting iOS sometimes to be able to do things the Google way.  Things feel easy and intuitive in Andriod Jelly Bean. Also, as for the Nexus 4 hardware itself, I like the form factor, the processor is definitely snappy, the camera is impressive, and it fits nicely in my pocket.

So, I have no ill will towards the iPhone or Apple.  I think the iPhone is still an amazing product.  Such thoughtful design and polish.  I didn’t leave because I wasn’t happy with the hardware or software.  If I could have gotten a new iPhone for much cheaper than $649 I would still be on iOS.  But I must say, I’m becoming more and more fond of my Nexus 4 as I use it more.  I am quickly becoming attached.  Ask me again in a few months and I just might say I prefer it.  I’m not going that far at this point but my opinion is moving in that direction.

Although I don’t have a “free” phone anymore, I do have a full-blown smartphone with call capability and data connectivity for only about $8/month.  Not bad considering my AT&T bill used to be $85 for one phone with basically the same capabilities.

Leave a Reply

Unattended Audio Recordings with a Raspberry Pi

Have you heard about the new Raspberry Pi computer?  From the website: “The Raspberry Pi is a credit-card sized computer that plugs into your TV and a keyboard. It’s a capable little PC which can be used for many of the things that your desktop PC does, like spreadsheets, word-processing and games. It also plays high-definition video.”

Basically, it’s an insanely cheap computer ($35) that you can do all sorts of geeky things with.  Go get one NOW (and then come back).

I waited 3 long months for my Raspberry Pi to arrive after ordering and boy was it was worth the wait!  I love this little thing.  I loaded the recommended image “Wheezy” on it and played around for awhile before quickly moving to Arch Linux ARM which I know well since I’ve had it running on my PogoPlug for almost 2 years.

After tinkering for awhile I kept trying to come up with a project for it.  There are so many cool Raspberry Pi projects out there right now and it’s easy to get excited about tackling one of them.  I’ve had my eye on the High Altitude Ballooning Pi project where you send your Pi to space to take pictures and track its movement with GPS.  Come on now, tell me that isn’t cool.

Anyway, I finally came up with an idea for a project.  I am responsible for recording the Sunday morning lesson in our church Sunday school class.  For the last 3 years the Sunday morning process has been to buy and bring CDRs, queue up the mic and recording system setup in the room, take the CDR home, rip it, encode the WAV to MP3 and upload it so it is available for Podcasting.  Whew, that’s a lot of steps.  Because it’s such a cumbersome process, it’s been hard to keep up with and easy to “forget” from time to time.  Also, it’s hard to train other volunteers on how to do the process end-to-end.  Since we already have a nice high quality microphone and mic amp setup in the room, I thought I would use the Pi to automatically record the audio and make it available as a Podcast.

IMG_0037

So, that’s exactly what I did.  It turned out to be a fun project and has been working for the last couple of weeks.  I found an old USB Belkin Wireless G Adapter (F5D7050/v4000) lying around which I used for wifi connectivity to our church’s wifi hotspot.  Since the Pi doesn’t have mic input capability I bought a $10 USB Sabrent sound card w/ mic input (SBCV/UP-100) from Micro Center which was easy to get up and working with the help of the ALSA guide on the Arch Linux wiki.   I bought an RCA to 3.5mm audio cable to hook up the mic amp output to the 3.5mm mic input on the USB sound card.

I have cron running the following script every Sunday morning at 11:00AM (when class starts) to record for one hour (podcast-record.sh 3600).  Obviously, I took out my credentials and such but this should be enough to give you an idea of what I’m doing.  This script does the following:

  • Record audio from mic using arecord
  • Use sox to convert stereo to mono for smaller file size
  • Use lame to encode to mp3 at 64kbps bit rate
  • Upload to Amazon S3 storage
  • Write a record to my mysql database.  I have a web app running on top of this database that generates the podcast xml feed.
  • Send a ‘success’ email notifying me it all worked
# Unattended podcast record process
# Author: brady@geekytidbits.com
# Notes: Invoke with seconds parameter specifying how many seconds to record, ex. 'podcast-record.sh 10'

FILENAME=$(date +"%Y%m%d_%H%M")

#record with sudo since arecord seems to want root privledges
sudo arecord -f dat -d $1 ./record/${FILENAME}.wav
[ $? -eq 0 ] || exit $?

#change ownership to jdoe
sudo chown jdoe ./record/${FILENAME}.wav
[ $? -eq 0 ] || exit $?

#copy to process directory
cp ./record/${FILENAME}.wav ./process/${FILENAME}.wav

#stereo > mono
sox ./process/${FILENAME}.wav -c 1 ./process/${FILENAME}-mono.wav

#convert to mp3 (64Kpbs) 
lame -b 64 ./process/${FILENAME}-mono.wav ./upload/${FILENAME}.mp3
[ $? -eq 0 ] || exit $?

#remove process files
rm ./record/${FILENAME}.wav
rm ./process/${FILENAME}.wav
rm ./process/${FILENAME}-mono.wav

NOTIFY_EMAIL=MY_EMAIL_ADDRESS@gmail.com
MODTIME=$(stat -c %y ./upload/${FILENAME}.mp3)
DATE=$(date --date="${MODTIME}" +%Y-%m-%d)

#upload to s3
s3cmd put --reduced-redundancy --acl-public ./upload/${FILENAME}.mp3 s3://MY_S3_BUCKET/${FILENAME}.mp3
[ $? -eq 0 ] || continue

#write db record
mysql -h www.mydomain.com -D MY_DB_NAME -u MY_DB_USERNAME -pMY_DB_PASSWORD --execute "INSERT INTO Lesson (Date, Title, Speaker, Audio_File_Name) VALUES ('${DATE}', 'TBD', 'TBD', '${FILENAME}.mp3')"
[ $? -eq 0 ] || continue

mv ./upload/${FILENAME}.mp3 ./complete/

#send email
echo "Done!" | mail -s "Podcast ${DATE} - Upload Complete" $NOTIFY_EMAIL

To install all the dependencies used in this script simply run the following:

pacman -S sudo alsa-utils sox lame s3cmd mysql

There you have it.  That was fun.  Now go get a Pi of your own and do something fun with it.

9 Comments

  1. Thom says:

    Very cool. Thanks Brady. I’m assessing some gift ideas for projects and audio recording is one aspect. I will definitely be lifting a few lines from your code and following the instructions to configure the audio hardware. Cheers,

    Thom

  2. Dave says:

    What size is the file?, what size sd card are you using.?

  3. andrew says:

    Hi, I like this alot, exactly what I’m looking for!
    Any thoughts on slience detect to remove any silence at the start or end of the lesson? Or maybe a manual start/stop button so you can manually start before 11am or manually stop before 12pm?? I guess the later would be alot easier then the prior.
    I understand the concept, but what software/tools do you use to generate the podcast xml feed from the raw MySQL entries?
    Cheers.

    • Brady says:

      Andrew — Have a look at http://digitalcardboard.com/blog/2009/08/25/the-sox-of-silence/ which describes how to use sox to remove silence from an audio file. I did think about implementing this in my setup but just haven’t gone down that road yet. I also did think about a manual stop start but realized that, although possible, there’s a bit of work to get this to work correctly. My thoughts were leaning in the direction of a running a PHP backed website that had a ‘record’ button. The script would check to see if ‘arecord’ was already running to show the user and allow a button press to fire up arecord on demand. My podcast generation happens in an ASP.NET (mono) website hosted on a different server and is very similar to the answer on this SO post: http://stackoverflow.com/questions/2839102/how-to-create-rss-feed-for-a-website .

      • Andrew says:

        Im thinking the need might be high enough for me to make a number of these and sell them cheaply to churches – I’ve come across this problem quiet a bit.
        SOX looks good – exactly what would be required.
        A PHP website is a great idea, even with the ability to change parameters like start and end times, mySQL details and alert email address. Just as an easy way to reprogram the thing over the LAN.\
        Would you be interested in re-writing some of the code for our situation at some stage in the future for an agreed commission price? my email address is: and_rew_story@live.com_.au (without underscores)

  4. Gershwin says:

    This is an interesting solution. We now use a real laptop for this kind of thing, but that also streams the sermon directly (live) to a streaming service using Shoutcast. I guess that it is compressing the audio under fly to MP3. Would such a feature also be possible to build with a Pi?

Leave a Reply

My Free iPhone Setup with FreedomPop

A few months ago, I posted about My Free iPhone Setup which is a combination of software and hardware that allows me to avoid paying monthly fees for an iPhone but instead use my iPod Touch just like an iPhone, including making and receiving calls and texts for free.

Although I have been using this setup for over 6 months and am generally happy with it, a few of the downsides include:

  • Not always being connected
  • Not having maps for navigation while in my car
  • Having to pay for a prepaid cell phone so I can be reachable while in the car

About 6 weeks ago I became aware of a service called FreedomPop which provides 500MB of free 4G WiMax internet each month. The only thing you have to pay for is the device itself. I forked over about $90 (which is interestingly, a refundable “deposit”) for the FreedomPop Photon and started playing with it. I was impressed with this little device right away including the speedy shipping process and nice packaging. It’s really small too!

FreedomPop Photon

Since it offers mobile internet, runs off of Micro-USB power (easy to find car adapter for), and offers 500MB of free data, it became obvious that I could use this device in my car to replace my prepaid cell phone and allow me to have maps for navigation. Also, since it has an embedded battery that can last for up to 6 hours, I could take it out of the car for limited times when needed so I could have internet wherever I am.

The only gotcha I had was the fact that after the device powers off when the battery dies, the only way to power it back on is to physically push the power button. There was no way I was going to remember (or want to remember) to push the power button on this device every morning when I got in my car. I definitely needed it to power on automatically when I started my car.

Fortunately, I found that if I permanently depressed the power button, it would automatically turn on when given power. It would then stay on, even with power button continuously depressed, until the battery died. This means that I could get the Photon to automatically power on when starting my car and then gracefully power off after the car was no longer running and its battery eventually died. So, I found a small pen cap and a rubber band and rigged it up!

Pen Cap Rigging So Power Button Stays Depressed

I’ve had the Photon in my car like this for over a month and it works great. Since I now have Wifi in my car I have access to Google Maps for navigation, can place/receive phone calls with the Talkatone app, and avoid paying for a prepaid cell phone. When I am in the car, my iPod Touch simply connects to the Wifi hotspot offered by the Photon and when I get home the iPod Touch simply jumps onto my house Wifi.

My modified setup now looks like this:

All in all, I am very happy with everything and have saved over $350 in carrier charges ($60 * 6 months)!

4 Comments

  1. Kathy Tankersley says:

    You’re awesome Brady! Thanks!!!!

  2. Travis says:

    How long does the battery last? Is it pretty much a for-the-car thing, or could you potentially carry it in your pocket when you’re out and about? Also, what’s your texting solution?

    • Brady says:

      On a full charge it will last about 6 hours. So yes, you could definitely carry it in your pocket while you are out and about. Since it only last 6 hours it doesn’t make sense to always do this since it would die mid way throughout the day but it is perfect if I am camped out at a location without Wifi. I am using it pretty much as a for-the-car-thing with occasional on-the-go pocket use when needed. For texting, I use the Google Voice app which allows free sending and receiving of texts. The app I use for voice calls, Talkatone, has integrated SMS texting but I prefer the Google Voice app for texting so I use it.

Leave a Reply

Rolling snapshots for EC2

One of the best things about EBS backed EC2 instances is the ability to create snapshots.  To say this makes backing up your instances “easy” would be an understatement.  You can create these snapshots manually through the AWS Management Console and also using the ec2-create-snapshot API command line tool.  The cost of these snapshots is just $0.125 per GB of data stored and incremental snapshots only store changed data so it’s really cheap to make lots of snapshots.

I wanted to be able to setup a daily cron job to automatically snapshot my EC2 instance so I wouldn’t have to worry about backups.  If I ever had a failure or data loss I could simply grab the automatically created snapshot from the previous day and restore it.  Running ec2-create-snapshop with cron is easy enough but there is a 500 count limit to snapshots and I really don’t want to keep hundreds of snapshots.  I really wanted to just keep a handful of ‘latest” snapshots.

Searching around I came cross a script that handled “rolling” snapshots which basically means it will create a snapshot and delete older ones to keep a specified number of latest snapshots.  This is exactly what I wanted.  The script I found was a bit outdated and needed some updates but I was able to make the needed changes fairly easily.  Here’s how you use it:

Usage:

ec2-create-rolling-snapshot [OPTIONS] -d DESCRIPTION VOLUME MAX_SNAPSHOTS

Example (keeps 7 most recent snapshots):

ec2-create-rolling-snapshot -d "Daily backup-vol-fzz00z0z" vol-fzz00z0z 7

This script depends on, and assumes the Amazon EC2 API Tools are installed.  If you are running Amazon Linux these should already be installed.  You’ll need to ensure the following environment variables are set and available at runtime.  If you are running from cron, remember, you’ll need to set these manually in your crontab or a wrapper script as environment variables set in your shell are not available at cron runtime.

  • AWS_ACCESS_KEY
  • AWS_SECRET_KEY
  • EC2_HOME
  • JAVA_HOME

Here’s the script:

#!/bin/sh
 
# Rolling snapshots for ec2 
# Original version: 2010-05-28 by cwilper
# Updated: 2012-10-25 by brady@geekytidbits.com

# (Invoke with -h for more info)
 
showHelp() {
  echo "SYNOPSIS"
  echo "   ec2-create-rolling-snapshot [OPTIONS] -d DESCRIPTION VOLUME MAX_SNAPSHOTS"
  echo "DESCRIPTION"
  echo "   Create a snapshot of a volume and delete the oldest snapshot"
  echo "   in the series, if necessary."
  echo ""
  echo "   The VOLUME parameter is the name of an existing volume."
  echo ""
  echo "   The DESCRIPTION parameter specifies the description to use for the"
  echo "   new snapshot. All snapshots in a series will have the same description."
  echo ""
  echo "   The MAX_SNAPSHOTS parameter specifies the maximum number of snapshots"
  echo "   in the series. If creation of the new snapshot exceeds this threshold,"
  echo "   the oldest snapshot in the series will be deleted."
  echo ""
  echo "OPTIONS"
  echo "   Any option below may be passed a value of '-' to indicate that values"
  echo "   for that option should be read from stdin."
  echo ""
  echo "   -O, --aws-access-key KEY"
  echo "        AWS Access Key ID. Defaults to the value of the AWS_ACCESS_KEY"
  echo "        environment variable (if set)."
  echo ""
  echo "   -W, --aws-secret-key KEY"
  echo "        AWS Secret Access Key. Defaults to the value of the AWS_SECRET_KEY"
  echo "        environment variable (if set)."
  echo ""
  echo "   -T, --security-token TOKEN"
  echo "        AWS delegation token. Defaults to the value of the AWS_DELEGATION_TOKEN"
  echo "        environment variable (if set)."
  echo ""
  echo "   -U, --url URL"
  echo "        Specify URL as the web service URL to use. Defaults to the value of"
  echo "        'https://ec2.amazonaws.com' or to that of the EC2_URL environment"
  echo "        variable (if set). Overrides the default."
  echo ""
  echo "   --region REGION"
  echo "        Specify REGION as the web service region to use."
  echo "        This option will override the URL specified by the "-U URL" option"
  echo "        and EC2_URL environment variable."
  echo ""
  echo "   --dry-run"
  echo "        Don't create or delete any snapshots; just show what would be done."
  echo ""
  echo "   -?, --help"
  echo "        Display this help."
  echo ""
  echo "   --connection-timeout TIMEOUT"
  echo "        Specify a connection timeout TIMEOUT (in seconds)."
  echo ""
  echo "   --request-timeout TIMEOUT"
  echo "        Specify a request timeout TIMEOUT (in seconds)."
  echo ""
}
 
while [ "$1" != "" ]
do
  case $1 in
    -h)
      showHelp;
      exit 0;;
    --help)
      showHelp;
      exit 0;;
    -\?)
      showHelp;
      exit 0;;
    --dry-run)
      dryrun=true;;
    -d)
      description=$2;
      volume=$3
      max_snapshots=$4;
      break;;
    --description)
      description=$2;
      volume=$3
      max_snapshots=$4;
      break;;
    *)
      gen_opts="$gen_opts $1";;
  esac
  shift;
done
 
if [ -z "$description" ]; then
  echo "Required parameter 'DESCRIPTION' missing (-h for usage)"
  exit 1
fi
 
if [ -z "$volume" ]; then
  echo "Required parameter 'VOLUME' missing (-h for usage)"
  exit 1
fi
 
if [ -z "$max_snapshots" ]; then
  echo "Required parameter 'MAX_SNAPSHOTS' missing (-h for usage)"
  exit 1
else
  if [ $max_snapshots -lt 1 ]; then
    exit 1
  fi
fi
 
if [ -z "$EC2_HOME" ]; then
  echo "ERROR: The EC2_HOME environment variable is not defined."
  exit 1
fi

#if [ -z "$AWS_ACCESS_KEY" ]; then
#  echo "ERROR: The AWS_ACCESS_KEY environment variable is not defined."
#  exit 1
#fi

#if [ -z "$AWS_SECRET_KEY" ]; then
#  echo "ERROR: The AWS_SECRET_KEY environment variable is not defined."
#  exit 1
#fi

tempfile=/tmp/ec2-create-rolling-snapshot-$$.tmp
 
ec2cmd="$EC2_HOME/bin/ec2-create-snapshot$gen_opts -d \"$description\" $volume"
snapshot_id=
if [ -z "$dryrun" ]; then
  eval $ec2cmd > $tempfile
  result=`cat $tempfile`
  snapshot_id=`cat $tempfile|grep SNAPSHOT|awk '{print $2}'`
  snapshot_state=`cat $tempfile|grep SNAPSHOT|awk '{print $4}'`
  rm $tempfile
  if [ -z "$snapshot_id" ]; then
    echo "ERROR: Snapshot creation failed"
    echo "$result"
    exit 1
  fi
  if [ "$snapshot_state" != "pending" ] && [ "$snapshot_state" != "completed" ]; then
    echo "ERROR: Snapshot state is not pending or completed"
    echo "$result"
    exit 1
  fi
  echo "Created $snapshot_id from $volume"
else
  echo "Created snap-TBD from $volume (not really; this is a dry run)"
fi
 
ec2cmd="$EC2_HOME/bin/ec2-describe-snapshots$gen_opts"
eval $ec2cmd > $tempfile
if [ $dryrun ]; then
    echo "SNAPSHOT snap-TBD $volume pending 9999-99-99T99:99:99+9999 1 $description" >> $tempfile
fi
result=`cat $tempfile`
series=`cat $tempfile|grep SNAPSHOT|grep "$description"|grep $volume|awk '{print $5,$2}'|sort|awk '{print $2}'`
rm $tempfile
if [ -z "$series" ]; then
  echo "ERROR: Failed to get snapshot series"
  echo "$result"
  exit 1
fi
 
count=`echo "$series"|wc -l|awk '{print $1}'`
echo "Series now contains $count snapshots, max is $max_snapshots"
oldest=`echo "$series"|head -n 1|awk '{print $1}'`
if [ $count -gt $max_snapshots ]; then
  if [ -z "$dryrun" ]; then
    echo "Deleting $oldest"
    ec2cmd="$EC2_HOME/bin/ec2-delete-snapshot$gen_opts $oldest"
    eval $ec2cmd > $tempfile
    result=`cat $tempfile`
    check1=`cat $tempfile|awk '{print $1}'`
    check2=`cat $tempfile|awk '{print $2}'`
    rm $tempfile
    if [ "$check1" != "SNAPSHOT" ] || [ "$check2" != $oldest ]; then
      echo "ERROR: Unexpected output from ec2-delete-snapshot command"
      echo "$result"
      exit 1
    fi
  else
    echo "Deleting $oldest (not really; this is a dry run)"
  fi
fi
 
exit 0

For my own use, I wrapped this script in another script called from cron so I can set the needed environment variables and stop/start MySQL to ensure data integrity at time of snapshotting.

#!/bin/sh

#Stop mysql to ensure snapshot consistancy
echo "Stopping mysql..."
/sbin/service mysqld stop
echo "Creating snapshot..."
export AWS_ACCESS_KEY="AKIDDDDDDDDDDDD"
export AWS_SECRET_KEY="GQYL9/JKCFNbZ/2/DDDDDDDDDDDDDDDD"
export EC2_HOME=/opt/aws/apitools/ec2
export JAVA_HOME=/usr/lib/jvm/jre
~/ec2-create-rolling-snapshot -d "Daily backup-vol-fac00d0d" vol-fac00d0d 7
echo "Waiting for snapshot to finish building..."
sleep 10
echo "Starting mysql..."
/sbin/service mysqld start

I’ve had this setup and running for about 2 months now and it works great. I love a hands-off approach to backing up my server and the piece of mind knowing that I can easily grab and restore a snapshot from last night (or 3 days ago) with a few clicks.

3 Comments

  1. Matt says:

    Nice script. I assume the EC2 command line scripts are included on the Amazon machine imagages. I’m on Ubuntu 12.04, and needed to follow this to get them installed: http://alestic.com/2012/05/aws-command-line-packages. If I get a chance I think I’ll play around with your script above to get one that keeps older snapshots but in lower frequency (e.g. 7 daily snapshot, then older snapshots are 1/week, then older still are 1/ month)

  2. Mike says:

    Great script, works great!

    Was wondering how hard it would be to add an option to copy the snapshot to another region once complete? That would make this awesome script even awesomer!

  3. vandervault says:

    Thanks. Great script, great writup.

Leave a Reply

Nginx with ASP.NET Page Methods

I recently spent some time moving an ASP.NET 4.0 website from Apache (mod_mono) over to Nginx with FastCGI to take advantage of Nginx’s killer performance and low memory footprint.

After following the FastCGI Nginx guide on the mono website I was mostly up and running.  However, my app has some Page Methods I use for AJAX calls via jQuery and they were not working properly.  I was getting 405 or 500 responses from Nginx when jQuery posted to the these methods.  I came across this related article: lighttpd + Mono Asp.Net; albeit related to lighttpd it still deals with FastCGI issues.  It shed light of the fact that mono’s fastcgi-mono-server is expecting parameters to be passed in a certain format from Nginx to properly handle Page Methods.  The default fastcgi config for Nginx does not play nicely with ASP.NET Page Methods.

Then I came across this article: ASP.NET Ajax Error 405 on Nginx, which got me closer to the solution.  Although I thought this would be my fix it didn’t work when I implemented the example config and actually caused a different server error.  After some trial and error I found that the fastcgi_param PATH_TRANSLATED config line should not be included. I’m not sure why including it worked for the author of this article but perhaps he is using a different version of mono.  For the record,  I am using mono 2.10.8 with fastcgi-mono-server4 (i.e. .NET 4.0).  In the end, the following config worked for me:

location ~\.aspx(.*) {
        include fastcgi_params;
        fastcgi_split_path_info ^(.+.aspx)(.*)$;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_param SCRIPT_FILENAME    $document_root$fastcgi_script_name;
        fastcgi_index Default.aspx;
        fastcgi_pass 127.0.0.1:9000;
}

If you are using ASMX Page Methods you simply need to copy this config and change all the aspx references to asmx and it should work.

3 Comments

  1. ClaudioCas says:

    Dear Brady,
    thanks for your post. I’m trying to setup Linux with Mono and Nginx in a Raspberry Pi and this is the solution of all my problem.
    I don’t know why this configuration is not in the official guide on mono site…
    Thanks a lot
    Claudio

  2. martin says:

    since you have used both mod_mono and Nginx. can i ask you, http://www.mono-project.com/ASP.NET#Limitations states that WebParts APIs is not implemented on mod-mono but no mantioning if these limitations are same on Ngix. (i googled for it didn’t find an aswer, thats how i found your site.). i don’t much (jet) about setting up an server, or how they work in general. what im trying to find out is: if a create a compiled controler like object in C++ and host it on a Nginx server using CTpp module and host a .NetMVC4 site on Nginx can i use the C++ object as part of my website?

  3. soner says:

    Thanks a lot man!

    :)

Leave a Reply

AutoMapper With DataTables

When working with data in .NET, I almost always prefer to work with POCO domain objects rather than some type of heavy abstraction layer.  Advantages are many including custom validation, encapsulated business logic and full control and flexibility of what is going on with the entity.  And, using LINQ queries against simple objects is dead easy.

Sometimes I come across an existing system that is using DataTables and DataSets exclusively for handling data and I don’t really have time to convert the data access layer over to using  domain objects.  Fortunately I came across a way to use the nice AutoMapper mapping library to convert a DataTable into a POCO entity very easily.  You just need to:

  • Add a reference to AutoMapper.  You can manually download and reference the .dll or simply install via NuGet by issuing Install-Package AutoMapper from the Package Manager Console (Tools->Library Package Manager) .
  • Create a class that has property names that match the column names in your source DataTable.
    public class Person {
        public string First_Name { get; set; }
        public string Last_Name { get; set; }
        public string User_Name { get; set; }
    }
  • Use the the following syntax to fire up a List<Person>.
    List<Person> people = AutoMapper.Mapper.DynamicMap<IDataReader, List<Person>>(
        sourceDataTable.CreateDataReader());

There you have it.  Now you can go encapsulate some domain specific logic / rules into your domain object (i.e. read-only properties that have domain logic, etc.), run LINQ queries to filter the list, and anything else your heart desires.  It really gives you a lot more power than sticking with a DataTable.

 

2 Comments

  1. Julian says:

    nice, and what about vs. mapping? POKO to IDataReader ?

Leave a Reply

Convert current Epoch time to Local time in SQL Server

Epoch, or Unix time, is a measure of time represented by the number of seconds since  midnight on January 1, 1970 (UTC).  It is used in various places, especially in POSIX systems such as Unix, Linux, BSD, etc.  I recently came across an integration project where I had the need to convert current epoch time to the ‘local’ timezone in SQL Server.  Here is what I ended up with.

DECLARE @epoch int
SET @epoch = 1342189899 --7/13/2012 14:31:39 UTC
PRINT DATEADD(minute, DATEDIFF(minute, getutcdate(), getdate()), DATEADD(s, @epoch, '19700101 00:00:00:000'))
--output: Jul 13 2012  9:31AM

See it in action on SQL Fiddle here: http://sqlfiddle.com/#!3/d41d8/2617/0

Note:  The above approach works well for ‘current’ epoch timestamps  but is flawed if you are trying to convert historical epoch times, because of the daylight savings time (DST) factor.  The UTC offset calculation above (DATEDIFF(minute, getutcdate(), getdate())) is relying upon current system time and is not DST aware.  In my particular case this is not an issue because I have an external system that is writing new records with a ‘current epoch timestamp’ so I convert them to the local timezone soon thereafter and store the result.

Leave a Reply

Efficient Client-side Caching of Dynamic Resource Handlers in ASP.NET

Problem

You have some type of dynamic resource, like an image, you are serving up with an IHttpHandler (.ashx page) in ASP.NET.  You want the client browser to cache the resource locally but also be able to know when it has been updated server-side so it can fetch the latest version.

Options

You could add the following lines to tell the client to cache the resource for only 1 hour and then after that ask the server for the latest version:

Limited Duration Cache

context.Response.Cache.SetCacheability(HttpCacheability.Public);
context.Response.Cache.SetExpires(DateTime.Now.AddHours(1));

But what if you want clients to fetch the latest after a change has been made rather than waiting an hour?  For instance, if you are serving a logo or thumbnail that must be up-to-date on the client.  In my particular case, I have a web application that has different “branding”, including a company logo, depending on the current Url.  When I am setting up a new brand or changing the logo on an existing brand I want clients to know they need to get the latest and greatest right away.  So, another option to accomplish this would be to tell the client to not cache the resource at all, and fetch the latest from the server every time it needs it:

No Cache

context.Response.Cache.SetCacheability(HttpCacheability.NoCache);

But, come on.  That’s not efficient and is just silly.

 A Better Approach – 304 Not Modified

ASP.NET has some good tools for server-side caching (data cache, output cache, static variables, etc.) but when it comes to fine-tune controlling the HTTP  response headers the client uses to determine how to cache a resource client-side, things get confusing fast.

Anyway, a better way to approach this is to respond to the client’s initial request for the resource with the following HTTP headers:

Cache-Control:public, must-revalidate, max-age=0
Last-Modified:Sun, 10 Jun 2012 20:19:21 GMT

This tells the browser to cache the resource but ask the server each time it needs to use it if it has changed. When it asks the server if it has changed, it will take the date specified in the Last-Modified header and send it to the server. Basically, the client will say to the server “Hey, I have a resource you previously gave me, dated “Sun, 10 Jun 2012 20:19:21 GMT”; is there a newer version available? If so send it to me. If not, just tell me there is not a newer version and I will use my locally cached copy. Thanks, have a good day.” If the server knows the resource has changed since the Last-Modified date it will send the newer copy to the client, just as it did initially. If it has not changed, it will respond with a simple “304 Not Modified” HTTP response, and not include the resource data in the response. This 304 response is extremely small in size so it should return very quickly.

This approach has the advantage that it still lets the client leverage its cache for performance gains but leaves the server in control of whether or not the client should show a newer version.

A Working Example

Below is a full-blown ASP.NET working example that demonstrates how to write the correct headers and respond with a 304 Not Modified response. Note: ImageContainer shown below is an arbitrary type containing an image I defined elsewhere in my own solution. In your case, you might be storing your image in a different way but the only thing that matters is being able to determine a timestamp (UpdatedAtUTC property in my case) so you know if the image has changed. If, for example, you are using handling an image stored on the server filesystem you might use something like File.GetLastWriteTime(@”c:\\images\\myimage.png”) to get the timestamp.

public class MyImageResourceHandler : IHttpHandler, IReadOnlySessionState
{
 public void ProcessRequest(HttpContext context)
 {
  //ImageContainer is an arbitrary type that contains image data
  ImageContainer myImage = (ImageContainer)context.Session["myImage"];
  DateTime? ifModifiedSinceTime = GetIfModifiedSinceUTCTime(context);
  DateTime imageLastModifiedTime = myImage.UpdatedAtUTC
  //strip milliseconds before comparison
  imageLastModifiedTime = imageLastModifiedTime.AddMilliseconds(-myImage.UpdatedAtUTC);
  bool clientNeedsLatest = ifModifiedSinceTime == null || (imageLastModifiedTime > ifModifiedSinceTime);

  if (clientNeedsLatest)
  {
    //write latest image to response
    context.Response.BinaryWrite(myImage.ImageBinary);
    context.Response.ContentType = myImage.MimeType;
    context.Response.AddHeader("content-disposition", string.Concat("inline; filename=", myImage.FileNam

    //tell client to cache
    context.Response.Cache.SetCacheability(HttpCacheability.Private);
    context.Response.Cache.SetRevalidation(HttpCacheRevalidation.AllCaches);
    context.Response.Cache.SetLastModified(imageLastModifiedTime);

    //set age/expires so that client doesn't attempt to use cache
    context.Response.Cache.SetMaxAge(new TimeSpan(0, 0, 0)); //max-age=0
    context.Response.Cache.SetExpires(DateTime.Now.ToUniversalTime()); 
  }
  else {
    //tell the client the image has not changed!
     context.Response.ClearContent();
     context.Response.StatusCode = (int)System.Net.HttpStatusCode.NotModified;
     context.Response.SuppressContent = true;
  }
 }

 private DateTime? GetIfModifiedSinceUTCTime(HttpContext context)
 {
  DateTime? ifModifiedSinceTime = null;
  string ifModifiedSinceHeaderText = context.Request.Headers.Get("If-Modified-Since");

  if (!string.IsNullOrEmpty(ifModifiedSinceHeaderText))
  {
    ifModifiedSinceTime = DateTime.Parse(ifModifiedSinceHeaderText);
   //DateTime.Parse will return localized time but we want UTC
   ifModifiedSinceTime = ifModifiedSinceTime .Value.ToUniversalTime();
  }

  return ifModifiedSinceTime;
 }

 public bool IsReusable { get { return false; } }
}

2 Comments

  1. Fernando says:

    Hi there, nice code! Do you know why “If-Modified-Since” is always null in that code in Global.asax?

    Sub Application_PreRequestHandlerExecute(ByVal sender As Object, ByVal e As System.EventArgs)

    If “.png.jpg.ico.css.js”.Contains(Request.CurrentExecutionFilePathExtension.ToLower) AndAlso Request.CurrentExecutionFilePathExtension “” Then

    Dim Filename As String = Server.MapPath(Request.AppRelativeCurrentExecutionFilePath)
    Dim LastModified As Date = New System.IO.FileInfo(Filename).LastWriteTime
    Dim ifModifiedSinceTime As Nullable(Of Date)
    Dim ifModifiedSinceHeaderText As String = Request.Headers.Get(“If-Modified-Since”)

    If Not String.IsNullOrEmpty(ifModifiedSinceHeaderText) Then
    ifModifiedSinceTime = Date.Parse(ifModifiedSinceHeaderText)
    ifModifiedSinceTime = ifModifiedSinceTime.Value.ToUniversalTime
    End If

    If ifModifiedSinceTime Is Nothing OrElse LastModified > ifModifiedSinceTime Then
    ‘tell client to cache
    Context.Response.Cache.SetCacheability(HttpCacheability.Private)
    Context.Response.Cache.SetRevalidation(HttpCacheRevalidation.AllCaches)
    Context.Response.Cache.SetLastModified(LastModified)

    ‘set age/expires so that client doesn’t attempt to use cache
    Context.Response.Cache.SetMaxAge(New TimeSpan(0, 0, 0)) ‘max-age=0
    Context.Response.Cache.SetExpires(DateTime.Now.ToUniversalTime())
    Else
    ‘tell the client the image has not changed!
    Context.Response.ClearContent()
    Context.Response.StatusCode = System.Net.HttpStatusCode.NotModified
    Context.Response.SuppressContent = True
    End If

    End If

    End Sub

Leave a Reply

My Free iPhone Setup

Update: Check out my improved setup: My Free iPhone Setup with FreedomPop.

I have a perennial conflict between wanting geeky toys and saving money. Usually those two things are mutually exclusive but sometimes I am able to have both.

The iPhone is a perfect example of this. I’ve had an iPhone for 2 years and love it. I got it for my birthday so I didn’t have to talk myself into the expensive data plan but after having it for a few weeks I decided the cost was worth it because it is such a useful (and cool) device.

After my wife’s iPhone got stolen and I gave her mine to use I started thinking about alternatives to paying AT&T an arm and a leg for their plan. I had read a few articles over the years in passing about using an iPod Touch as a free iPhone but didn’t ever seriously consider it. However, since I was without a phone and not wanting to renew my contract for 2 years (i.e. spend over $2,000), I decided to give the iPod Touch setup a try.

I did a bunch of reading of blogs about how others accomplished this and decided it was feasible.  Everyone that I found had a slightly different configuration but they all seemed to be happy with the end result.

Overview

The major pieces to my setup are:

The basic idea is that I have a Google Voice number as my main phone number.  When calls come in to that number, I have 4 different physical devices  (“forwarding  phones”) setup to take the call.  Depending upon where I am and whether I have access to WiFi, I will hear the call come in on one of these devices.  When I am at work, my work phone will ring.  When I am at home, my iPod touch will ring, when I am out and about, my wife’s iPhone will ring, and when I am in my car, my prepaid cell phone will ring.  There are very few scenarios with this setup that I am “unreachable”.  Also, outgoing calls can be made from any of these devices.

iPod Touch

This part is easy.  I just went online to Amazon and bought an iPod Touch.  It cost me $200 but I could have easily gotten a used for for much less.  Since the iPod Touch isn’t technically a phone you’ll need to use the speakerphone, ear-buds, or a bluetooth headset to make calls.  I use the earbuds when they are handy and the speakerphone at other times.  It works fine this way.  I have been thinking about getting a bluetooth headset, though.

Google Voice Account

This part is easy too.  I just went and signed up for a free Google Voice account and got a phone number assigned to me.  Then, I simply added my 4 forwarding numbers and configured some options to my liking.  One of the nice options here is the ability to specify Ring Schedules.  Since my wife’s iPhone is one of my forwarding phones and I don’t want her to get unwanted calls (“Honey!  I keep getting calls for you!!”) during business hours when I am at the office, I was easily able to specify that this phone was disabled as a forwarding phone 8-5, M-F.

Talkatone App

Talkatone is an awesome iOS app that leverages Google Voice to give your iPod Touch inbound and outbound calling functionality over WiFi.  You simply download the free app from the App Store and then give it your Google Voice account information when fire it up for the first time.  It runs in the background and has many options so you can fine tune it to your liking.  I was honestly skeptical about this app at first and thought it might not be reliable but I have continually been impressed with it.  Once this app is running and you are connected to WiFi, incoming calls to your Google Voice number will magically ring on the iPod Touch and you will be given the option to accept them.  The app is able to handle intermittent Wifi connectivity so when you go from no Wifi to having Wifi connectivity, it will start listening for calls again.  That feature right there is killer.  It means I can build my list of known Wifi hotspots and the iPod Touch will atomically jump on hotspots and start listening for calls, all without me having to touch it.  Talkatone also supports free texting through Google Voice so add that to it’s awesomeness.

Prepaid Cell Phone (for the car)

This part of my setup is technically not necessary but a very nice to have.  It keeps me reachable and enables me to get a favorable WAF.  Also, it’s important because having a phone in the car can be necessary in an emergency.  I decided to go with T-Mobile’s pay-as-you-go prepaid plan and bought a cheap $10 old-school Nokia phone and loaded it with $50 in credit.  Since it is pay-as-you-go, the cost per minute is high but I don’t intend on using this phone often.  I estimated that I would spend less than $5 a month with this phone.  It is mainly for calls from my wife on the way home (“don’t forget the milk from the store on the way home!”), emergencies, and convenience for those miscellaneous calls that need to be made from the car from time to time.  Once I bought the phone, I just plugged it into the car charger, put it in the arm rest compartment and it stays there.  It’s always hooked up to the charger so it charges when the car is on and since the battery lasts so long, it never dies between the times the car running and charging it.  I simply went into the Google Voice settings and set this phone up as a forwarding phone and it was good to go.

Adding Up the Costs

iPod Touch $200
Nokia cell phone $10
T-Mobile prepaid calling credit for 1 year $50
Total Setup Cost $260
Monthly cost if I still had AT&T service $85
Months to break-even ($260/$85) 3

 

3 month break-even!  Not bad.  After that break-even period, the savings goes straight in my pocket.  Actually, if I had gotten another iPhone on contract (to replace the stolen one) I would have ended up spending $200 on it anyway so I could argue my marginal out of pocket cost for the “free” setup is much less (~ $60).

My Experience

I’ve had this setup for a couple of months now and can honestly say it is working well.  I am reachable, able to make calls almost everywhere I am, and am saving bookoos of $.  I basically get an iPhone setup for practically free.

Yes, it is not “as good” as a full blown iPhone setup, however and for the following reasons:

  • Performance.  The iPod Touch is not as snappy as the latest iPhone because it doesn’t have as beefy of a processor.  So, the iPod Touch can be sluggish at times but I find it to be acceptable and not too bad.
  • No Vibrate.  Ok, some people might not care about this but I was disappointed to find that the iPod Touch does not have a vibrate option.  I used to turn my iPhone ringer off and just rely of the vibrate feature to know when calls were coming in because the phone was always in my pocket.  I liked not having to worry about silencing the phone during meetings and such.  If, however, you jailbreak your iPod Touch you can install programs like SBProfiles to help.  All in all, this is not a deal breaker but just a downside.
  • Not Always Connected.  With my 4 forwarding phones I am pretty darn reachable.  But, since the iPod Touch only has Wifi there are some times when I am out and about and don’t have Wifi.  If I go to get my haircut and the place doesn’t have a hotspot, I don’t have phone or data service.  I could get my prepaid phone from the car and take it in but it would be nice to pull of mobile Safari and surf around while waiting to get my ears lowered.  If I am on the road and there is a huge traffic jam, I can’t pull up my phone to see the traffic info or find an alternate route.  I could call the wife for help, though, with the prepaid phone.   Also, I could pull into a McDonald’s parking lot to jump on Wifi to check traffic. Yes, this sucks that I’m not “always connected” but honestly, it doesn’t suck as bad as I thought it would.  I don’t find myself saying “Arrrgggh!! I don’t have Wifi here!”.  Every once in awhile it is a bit painful not being connected but it’s not that bad.  I try to look at the bright side and realize it’s good to be disconnected sometimes.  And, there are so many places with Wifi so chances are I can find a hotspot somewhere near by.

That’s it.  I’m loving my Free iPhone setup and the $ I am saving!

5 Comments

  1. aaron says:

    very cool! i think the lack of maps and navigation on-the-go is what might have me crawling back to service. still a cool idea. did you hear that apple might be offering their own service for iphones?

    http://finance.yahoo.com/news/apple-announces-plans-offer-wireless-122000980.html

  2. Love this site. When do you find time for to write all of this?

    Odell

  3. Brent says:

    I still think I want to do this. My contract is up this month with Sprint.

  4. Jeremt says:

    love google voice and talkatone…
    i have a similar setup. however, i go with a mobile hotspot, jailbreak iPhone for use of SMS GV Extention, and even on places with 3G networks it does the job. Got an iPod 4th gen and basically can use as a phone, but the only down part
    of the iPod is it doesnt have an earpiece so, without earplugs needss to be on speaker phone.

Leave a Reply

Surveillance Camera

Last year my house was burglarized.  I was away from home when it happened and got an iPhone Prowl notification from my AlarmServer app running on my Pogoplug immediately after my alarm went off.  I called my next door neighbor, who was home at the time, and they went over right away but the thieves were already gone.  Based on alarm logs I know they were only in one part of the house and were in and out in less than 60 seconds.  Anyway, it was pretty cool getting the notification on my phone and being able to call my next door neighbor so soon after they broke in my door.  I have to assume my response time was faster than a paid monitoring service would be.

Although my alarm performed as it should have I realized that I needed video surveillance so I could identify the thieves to the police next time.  I decided to use a TRENDnet wireless IP netcam, my always on Linux server , and the open source program called Motion to accomplish said surveillance of my driveway and increase the security of my home.

Camera

The TRENDnet ProView Wireless Internet Surveillance Camera TV-IP501W (left) is a nice looking camera and mounted discreetly on the overhang of my roof.  It was easy to setup as all I needed to do was run a power cable to it.  It uses 802.11g so I didn’t need to worry about a separate ethernet cable.  It has a built in web server which is pretty easy to use.

Motion

Although the TRENDnet camera has the ability to upload images to an FTP server, it doesn’t detect motion and drops many, too many,  images when configured to do so.  I decided to set up Motion, “a software motion detector”, which intelligently detects motion and only saves images that show a material change in the picture.  The Motion website has a page with details on interfacing with TRENDnet camera which was helpful in getting everything configured: http://www.lavrsen.dk/foswiki/bin/view/Motion/TrendNet.  I installed Motion on my Pogoplug server running Arch Linux by simply running pacman -S motion. The two important config settings in /etc/motion/motion.conf were:

netcam_url http://192.168.1.7/VIDEO.CGI #192.168.1.7 is the IP address of the camera
threshold 3000 #trial and error determined this to be best when camera is at 640x480 resolution

Archiving

Motion does a great job of dropping only images that display motion but I wanted to have a bash script run nightly that would zip up all the images for the day and also create a time-lapse video so I could quickly see the motion for an entire day.  Also, I wanted it to only keep 10 days of history.  Here’s what I came up with:

#!/bin/bash

# Format: YEAR MONTH DAY
DATE=$(date +%Y%m%d)
PATH="/srv/surveillance/driveway"
TARGET_FILE="$PATH/$DATE"
cd $PATH

#create timelapse video
/bin/rm $TARGET_FILE.mp4
x=1; for i in $(/bin/ls *jpg); do counter=$(printf %03d $x); /bin/ln -s "$i" img"$count$
/usr/bin/ffmpeg -r 5 -i img%03d.jpg -pix_fmt yuv420p $TARGET_FILE.mp4
/bin/rm img*.jpg

#archive images into zip
zip -m $TARGET_FILE.zip *.jpg -x \*.zip

#remove image archives and videos that are at least 10 days old
find /srv/surveillance/driveway/* -mtime +10 -exec /bin/rm {} \;

iPhone Integration

In another project last year, I wrote several components to monitor my alarm system.  One of the components was something I called AlarmWeb which is basically a HTML5 application that displays nicely on the iPhone (and Andriod for that matter).  After setting up the surveillance camera I decided it would be nice to integrate it with this interface.  I simply included <a href=”http://192.168.1.7/VIDEO.CGI”><img src=”http://192.168.1.7/IMAGE.JPG”/></a> on the page so that an image still will show (IMAGE.JPG) and then clicking the image will take me to the live video feed (VIDEO.CGI) directly from the camera.


Samples

Single image

Time-lapse video

Leave a Reply