Tuesday, June 19, 2012

Importance of testing

I just finished the excellent Coursera/Berkeley Software as a Service (SaaS) class yesterday. There were two major themes in the class: engaging customers in creating user stories to make sure that you build the right thing, and use BDD and TDD that relies on continuous testing. In particular I enjoyed learning how to use Cucumber that has two parts: an english (or other natural language) style DSL that customers and developers can work on together to create user stories with expectations of what the system should do and an underlying set of steps that map the customer facing DSL descriptions to real testing code.

Cucumber is very popular in the Rails world but I had never tried it before. I now like using Cucumber and as I get more used to it for Ruby and also for Rails I look forward to integrating it into my development process for other programming languages.

A few nights ago I spent some time with the lein-cucumber project for Clojure. It is still a little rough around the edges compared to the Ruby version but seems to be very workable.

This morning over coffee I have been looking at the Java version Cucumber-JVM that lein-cucumber is based on. I use IntelliJ and there is a Cucumber plugin. I couldn't get my tests to run inside IntelliJ this morning but the IntelliJ editor understands the syntax of features files, does syntax color highlighting etc. Looks promising for the future. For now I think that I will just add the Maven Cucumber-JVM dependency and run the tests from the command line. For Ruby and Rails development running RSpec and Cucumber tests inside the IDE is very convenient so I look forward to figuring out how to get in-IDE tests running in IntelliJ.

Monday, June 18, 2012

More on PaaS: new dotCloud pricing and services

I have had a dotCloud account for a while, but so far only to experiment with. dotCloud has announced new pricing models that look very developer friendly with free lower performance sandbox support and "live" support for production. They may have hit the sweet spot for supporting free development while nudging developers to not deploy small demo apps using their free sandbox model. I would love to see Heroku's and dotCloud's customer stats on free versus paid hosted web apps.

The basic idea is that as you add services you may by the amount of memory that those services use. I like their handling of horizontal and vertical scaling. I have not tried it yet, but from the documentation it looks like when you horizontally scale a persistent service like PostgreSQL, MongoDB, MySQL, etc. they automatically set up master/slave, replica sets, etc. as appropriate. You can also vertically scale any service by adding memory.

Another interesting PaaS that I have experimented with is Cloud Foundry that is, like Redhat's OpenShift, an open source stack that you can also install on your company's private servers.

Managing servers is a form of technical debt. Using PaaS costs a lot more for raw resources but "set and forget" web apps deployed to PaaS can end up being a lot cheaper to deploy over long term.

Sunday, June 17, 2012

A better tool for private social and working networks? Using Open Source Apache Wave (used to be Google Wave)

OK, maybe now we should just call is Wave.

As I talked about a few years ago in my blog, I really enjoyed using Wave for interacting with family and friends, and also experimenting writing Wave Robots hosted on AppEngine. Those good times have ended :-(

Fortunately, the Apache Wave incubator project provides the code and directions for running Wave on your own servers.

If you are lucky, installing Wave is as simple as:

git clone git://git.apache.org/wave.git wave
cd wave
ant compile-gwt dist-server
ant -f server-config.xml
./run-server.sh
The default data store is the file system but you can use MongoDB instead. After you have started the server, you can use two different web browsers (e.g., Firefox and Chrome) to create two different accounts using the Register a new account link on the right side of the welcome page. After logging in under two accounts with two web browsers then create a new note under one of your Wave accounts and invite the other user account to that note. As you type on one account you see the characters echoed on the other.

Wave has a beautiful UI, cleaner than the original Google Wave. A beautiful example of a GWT web application. The code base is moderately large with about 1800 Java source files and 450 Java test files. There are 26 GWT modules. So, lots of code, but it is well organized and written, and so not too scary: certainly possible to make custom modifications for different business uses.

Apache Wave in its current form seems stable (but I like to back everything up before doing a git pull, and I don't git pull updates very often) and with a MongoDB back end, Wave looks like it would be a nice tool for private workgroups, families, groups of friends, etc.

Anyone considering an open source private Facebook-like system might want to take a long look at Apache Wave. While I wish the Diaspora project (and other open source Facebook alternatives) well, I can't help but think that Apache Wave is a better place for people to combine their energies so we all have a good tool to set up private social and working networks.

Redhat OpenShift is another interesting PaaS

My friend Alex Ott left a comment on my blog yesterday asking me if I had looked at OpenShift.

I had created an account a while ago but never did anything with it. This morning I briefly tried getting Clojure (not yet supported) running but promptly gave up and switched to Ruby, which along with Java, Python, Node.js, and Perl is supported. It only seemed fair to test Redhat's PaaS with a supported stack.

By default Ruby 1.8.7 is supported with a lot of gems pre-installed. I preferred to use RVM and Ruby 1.9.3 so I:

  • I created a new empty "Do-it-yourself" Cartridge using the web console. I was prompted to add a public key, etc. Easy setup.
  • A new empty cartridge runs a trivial Ruby web app.
  • I followed Mark's instructions to install RVM and Ruby 1.9.3.
  • After that, a git commit and a git push redeploys your app.
  • You should take a careful look at .openshift/action_hooks/* that are places where you can customize builds and deployments.
Using Mark's instructions, I temporarily modified .openshift/action_hooks/pre_build to look like this:
cd $OPENSHIFT_DATA_DIR
curl -L https://raw.github.com/xiy/rvm-openshift/master/binscripts/install-rvm-openshift.sh | bash -s
After that, I re-edited it to look like:
#cd $OPENSHIFT_DATA_DIR
#curl -L https://raw.github.com/xiy/rvm-openshift/master/binscripts/install-rvm-openshift.sh | bash -s

source $OPENSHIFT_DATA_DIR/.rvm/scripts/rvm
rvm use 1.9.3-p125@global

gem install sinatra rails mongo mongo_ext
After installing some gems I commented out everything in this file for future reference. I then edited .openshift/action_hooks/start to look like this:
source $OPENSHIFT_DATA_DIR/.rvm/scripts/rvm
rvm use 1.9.3-p125@global

gem list
ruby --version
nohup $OPENSHIFT_REPO_DIR/diy/testrubyserver.rb $OPENSHIFT_INTERNAL_IP $OPENSHIFT_REPO_DIR/diy > $OPENSHIFT_LOG_DIR/server.log 2>&1 &
The last line was originally in the generated file. Once this is all working, you can replace the placeholder mini-app with a Sinatra or Rails app.

There have been no announced costs for using OpenShift. For now it is in a free beta period. It is reasonable for Redhat to want a lot of test users so they can measure what it costs for them to provide the various services and then announce costs.

Saturday, June 16, 2012

Deciding between two premium hosting options for a new side project

Over the course of a year I spend about 2/3 of my working time consulting for customers and the other 1/3 doing my own projects. I view doing personal projects as a form of continuing education. My last major personal project was my natural language processing (NLP) web service KBSportal.com and at the same time I spent a day writing SleepyBird.us for my wife. The project before those projects was an experiment in a live note taking and scheduling system for consultants ConsultingWith.me which I still use but I have turned off the ability for people to create accounts. I never implemented the sister project ConsultingWith.us. Six years ago my side project was CookingSpace.com which I do still use. For me side projects are a great way to try out new ideas and it is well worth it to me to pass on less interesting consulting gigs to free up enough time for my own stuff.

I am in general interested in the combination of knowledge management and NLP and my new side project combines these interests with a tool that I want for myself: a system to register web resources, notes, and tasks and getting periodic summaries. One of the things I want to experiment with is three front ends (web app, mobile web app, and email) accepting input and displaying one a day and once a week status summaries to help put my time in context - to stimulate reflection on what I have been doing. The other thing that I want to experiment with is a web worker and backend worker architecture.

In the past, I combined web front end code and a background work thread into a single JVM process: basically I wrote Java web apps that had a specific worker servlet whose init method started a work thread. That always worked really well but for my new project I want to have a better architectural fit with platform as a service (PaaS).

Which brings me to the main topic of this article: I view both Google's AppEngine and Heroku to be Premium Services. Premium because no way you get a fraction of the raw resources as using a physical server from a provier like hetzner.de or even from AWS. What you get is the tradeoff on saving admin resources and easier scaling.

AppEngine got a lot of bad press for their price increases but the result is a service that they are unlikely to stop providing. One of the implementation options I am considering is a front end web worker written in Python and a backend worker written in Clojure. For this experiment I doubt that I will have to scale past that, but at least the path to scaling out would be straightforward. The other implementation option I am thinking about is using Heroku with a front end rails app and a background worker using either Ruby or Clojure.

Monday, June 04, 2012

I am a living advertisement for JetBrains products :-)

Full disclosure: for years JetBrains comped me with free products, but for a few years now I have been buying my own licenses.

I have been working since 5am this morning on three customer projects in three languages:

  • Python - using PyCharm
  • Ruby - using RubyMine
  • Java - using IntelliJ
Switching languages and supporting platforms and frameworks sometimes causes a little cognitive dissonance, taking a few minutes to get my head into a different toolset.

With the same (almost) user interface, similar code completion, etc., switching between three languages and projects today was easy with no stress.

I will also thank other companies that make it easier for me to do my work:

  • Amazon AWS - Amazon is my favorite technology company. AWS has changed my working life. Since I live in a remote mountain area, I also appreciate getting stuff from Amazon. The Kindle platform with synched reading accross all of my devices is also great.
  • Apple - my MacBook Air and iPad are great devices that I don't even have to think about while I am working and playing (and the iTV is pretty cool also)
  • Heroku - I am a big fan of PaaS and Heroku is my favorite
  • Google - for GMail, Docs and Calendar
  • github - really makes remote customer interaction simple and productive
  • Evernote - my own personal memory box
  • Dropbox
  • Netflix - covers most of my non-sports and non-reading recreation
I work and live in a digital world and these great companies deserve my business for making my life simpler and more fun.