Thursday, April 29, 2010

Sweet: Ubuntu 10.4 released

I have been running 10.4 beta for a while so I am just doing an update. Before the update I made a nice discovery: I had problems getting the Realtek wireless working in my Toshiba Satellite U505 but this morning I noticed that Realtek had new Linux drivers available that work fine for me. Best to either buy slightly older hardware for using Linux, or to develop patience.

I only have two servers (usually I work with customers' servers on their projects) and I am trying to decide if I need to update those. I host my own web apps on Google AppEngine and Heroku so my servers are largely unused.

Wednesday, April 28, 2010

I wish the Play framework was available when I did server side Java

Before I started using Ruby about four years ago, almost all of my web development was either J2EE or plain old JSP apps (with some fun experiments with Lisp, Scheme, Prolog, etc.)

Even though Rails is my "go to" platform for efficiently writing web apps, I like to keep up with Scala + Lift, Java options on AppEngine, Clojure + Compojure, Seaside, etc. I just revisited the Play framework and I must admit some real regret that something like Play was not available when I did a lot of server side Java work.

Play borrows a lot from opinionated frameworks like Rails and a quick study is enough to be productive largely because of the similarities to Rails in the ease of setting up routes, database access, nice MVC separation, etc. Probably the neatest feature of Play is that Java source code is compiled on the fly using the Eclipse Java compiler so if you edit a Java controller of model file and refresh a web page you instantly see the effects of the code change. There is a Scala sub-project of Play also.

Tuesday, April 20, 2010

My Objectify-Appengine setup

I wrote last week on using Objectify-Appengine to decrease the Google AppEngine request loading time. I did not mention my development setup so I'll do that now, hopefully saving you some time:

Start by building a JAR file from the latest source:
svn checkout http://objectify-appengine.googlecode.com/svn/trunk/ objectify-appengine-read-only
cd objectify-appengine-read-only
ant
You should now have a JAR file in target/objectify-508.jar (build number may be different for you). I use IntelliJ and after setting up a fresh AppEngine project, I copy in the generated Objectify JAR file and delete some of the generated JAR files so I have the following JAR files in web/WEB-INF/lib:
geronimo-jpa_3.0_spec-1.1.1.jar
appengine-api-labs-1.3.2.jar
objectify-508.jar
This is all you need. You can then follow the examples in the Objectify Wiki. You should re-think your data modeling, away from a relational database mindset. The AppEngine data store may seem limiting from a programming perspective but its scalability and performance advantages make the learning curve very worthwhile.

Wednesday, April 14, 2010

A great way to get around the long loading request times on AppEngine

I wrote yesterday about my experimental project my-foc.us that is currently a prototype app running on Heroku. In parallel, I wanted to experiment with doing an AppEngine deploy but I got rather frustrated this morning by the both long and unpredictable cold loading request times. Even with some clever new work to manage the load time of JRuby apps, there is still a lot of work to be done to get the cold request load times under a reasonable limit (perhaps a three or four seconds?).

I think that Python AppEngine apps have the fastest cold request load times but I am not Python developer (very rusty!). I tried a Java app and the load times were very much better that JRuby, but then I ran across a great project that reduces the load time for Java apps by not using JDO and instead using the light weight objectify-appengine project. I am not sure what is the best way to accurately measure cold loading request times except for letting an app get swapped out (wait a few minutes with no requests) and manually time a browser refresh. Anyway, it looks like my loading request times using objectify-appengine are now almost always about a second with the worse time that that I measured being just over two seconds. The URL for my AppEngine experiment is my-focus.appspot.com.

The obvious problem with this "solution" is that it takes me about twice as long to develop a JSP based web app than using Rails or Sinatra. Still, if you want to get good performance with and reduce costs on AppEngine, then I feel that you need to conform to the requirements of the platform. Perhaps occasionally giving up the use of a favorite programming language and web framework is well worth the effort. (BTW, the loading request time is not much of a problem with Scala either but it is with Clojure - not quite as much overhead for initialization as JRuby but cold loading request times are about 10 seconds for me.)

Tutorial on using Map Reduce to analyze log and web site click data

This is a useful tutorial that walks you through the steps of analyzing log and user interaction data using Hive and Elastic MapReduce on AWS. If you need to set up a data warehousing system, this should get you started.

Tuesday, April 13, 2010

my new project: http://my-foc.us

We all have things that we are proud of professionally. For me this is largely being proud of process and some achievements. My process is one of focussing down on a (very) few things and getting them done efficiently and then spending time "playing" with new technologies and working on new ideas. These are two very different kinds of work and I approach them differently.

I have read a lot of material on "getting stuff done" and I have used other people's productivity software. For to-do lists I like the simplicity of just using GMail's "Tasks" features. All other projects that I have tried have many features that I don't need or care about and miss some things that I need. Since I focus on just one or two things per (work) day, I want a system that helps me to manage just a few things. This will definitely be a "scratch my own itch" style project.

It will take at least a month until I get a public beta ready, but I do have a placeholder web app set up: my-foc.us.

Sunday, April 11, 2010

Very useful book: "REWORK"

Jason Fried's and David Heinemeier Hansson's new book "REWORK" is great - a must read for web developers and entrepreneurs.
Their basic message is to stay focused on what you really need to get done and eliminate as many distractions as possible. As a result of reading this book I have changed a few non-optimal habits:
  • I made a big push to finish my current book so that I could concentrate on my current development project.
  • For my current project of version two of my cookingspce.com web portal, I reverted to Ruby 1.8.7 to avoid a few small ongoing Ruby 1.9.1 problems, updated to a newer version of Rails to get some bug fixes, using Heroku and not deploying it myself, removing almost as many old features as the number of new features I am adding, adding mobile phone support for easy recipe access and a shopping list manager, and more "AI" in presenting data and auto-organizing shopping lists (scratching my own itch: working on the features that I want myself)
  • Turning down more consulting offers that do not closely match my areas of specialization
  • Spending less time learning new technologies (trying to cut back from about 10 hours a week to about 5 hours a week - time enough to stay on top of new stuff, but more time for meeting short term development and writing goals)
  • Deployments: as much as possible I am going to stick with Heroku and AppEngine for my own business projects, even though almost all of my customers have me deploy their stuff to Amazon. If customers want to pay me for custom deployments I enjoy that, but for my own business projects I don't want to spend the time.
  • Turning off my cellphone and email client while I am writing and working. I need to get up and walk around every 30 or 40 minutes to avoid sitting for too long. I use this frequent 5 minutes of walking outside time to turn on my Android phone, check for voice mails and email, and respond immediately if I have to. It takes me about 5 minutes to get in the flow when working, so this gives me many (roughly) 30 minute highly productive sprints each day.
Thanks to Jason and David for writing such a useful book!

Friday, April 09, 2010

Mobile device web apps

I used the advice on this Rails Cast to start converting one of my Rails apps to nicely support mobile devices. Working on my MacBook, I use the iPhone Simulator (since I don't own an iPhone) and my Droid phone to test on a real device.

So far most of the work has been deciding what I don't want to show when web requests come from a small screen device and tweaking the mobile device CSS file.

Thursday, April 08, 2010

My Clojure, MongoDB, Ring, and Compojure development setup

I wrote a few days ago about my Clojure and Compojure setup that automatically reloads modified files while I am developing. I have added support for accessing MongoDB using the congomongo library. My new project.clj file:
(defproject kbsportal "0.1.0"
:description "test using Compojure for KBSportal.com"
:dependencies
[[compojure "0.4.0-SNAPSHOT"]
[ring/ring-devel "0.2.0-RC2"]
[ring/ring-httpcore-adapter "0.2.0-RC2"]
[ring/ring-jetty-adapter "0.2.0-RC2"]
[ring/ring-servlet "0.2.0-RC2"]
[org.clojars.liebke/congomongo "1.0.0"]]
:main kbsportal)
My Compojure setup file kbsportal.clj:
(ns kbsportal
(:use compojure.core
ring.adapter.jetty)
(:use ring.middleware.reload)
(:use ring.middleware.stacktrace)
(:use somnium.congomongo)
(:use mongo)
(:use nlp))

;; set up test use of congomongo + MongoDB:
(mongo!
:db "notes", :host "127.0.0.1") ; notes contain title, content, and words

(defroutes test-routes
(GET "/" []
(str "<h1>Hello World" (foo) "</h1>"
(get-all-notes)))
(ANY "*" []
{:status 404, :body "<h1>Page not found</h1>"}))

(def app
(-> (var test-routes)
(wrap-reload '(kbsportal))
(wrap-reload '(nlp))
(wrap-reload '(mongo))
(wrap-stacktrace)))

(defn dev []
(run-jetty #'app {:port 8080}))
The example file nlp.clj has not changed since my previous blog, and here is the source file mongo.clj for reading all notes from an existing MongoDB data store:
(ns mongo
(:use somnium.congomongo))

;; expect notes to contain :title and :content
(defn get-all-notes []
(let [notes (fetch :notes :only [:title :content])]
(apply
str
(for [note notes]
(str "<b>" (:title note) "</b> " (:content note) "<br/>")))))
Please refer to my previous blog for directions for running this.

Sunday, April 04, 2010

My Clojure, Ring, Compojure development setup

I had to search several sources (like this) to get this setup, so I thought it would be useful to summarize my setup here. Unlike most developers, I use five different editors (emacs+slime+swank, IntelliJ IDE, GEdit, e, and Textmate) depending on which computer and OS I am on. I wanted to be able to leave a Compojure based web app running in development mode and have source files refresh regardless of which editor I am using. In the following example, I have two Clojure source files: one with Ring and Compojure code, and one a utility file, containing for this example an extremely useful function foo. Start by creating an empty Leiningen project:
lein new KBSportal
cd KBSportal
Then edit project.clj to look like:
(defproject kbsportal "0.1.0"
:description "test using Compojure for KBSportal.com"
:dependencies
[[compojure "0.4.0-SNAPSHOT"]
[ring/ring-devel "0.2.0-RC2"]
[ring/ring-httpcore-adapter "0.2.0-RC2"]
[ring/ring-jetty-adapter "0.2.0-RC2"]
[ring/ring-servlet "0.2.0-RC2"]]
:main kbsportal)
In the empty src directory create the file kbsportal.clj:
(ns kbsportal
(:use compojure.core
ring.adapter.jetty)
(:use ring.middleware.reload)
(:use ring.middleware.stacktrace)
(:use nlp)) ; nlp.clj is my other source file

(defroutes test-routes
(GET "/" []
(str "<h1>Testing foo: " (foo) "</h1>"))
(ANY "*" []
{:status 404, :body "<h1>Page not found error</h1>"}))

(def app
(-> (var test-routes)
(wrap-reload '(kbsportal))
(wrap-reload '(nlp))
(wrap-stacktrace)))

(defn dev []
(run-jetty #'app {:port 8080}))
and the file nlp.clj:
(ns nlp)

(defn foo [] " - This is from foo.") ; a very useful function!
Notice how I use the Ring wrap-reload function to force checking the date stamp on the two source files and reload them if they are changed. You would not want to check for automatic reloads in a production environment. This is a skeleton project, but in a real project you could add wrap-reload calls for new source files that you add to your project's src directory. To run in development mode, you can type:
lein repl
to start a Clojure repla and then inside the repla load the main source file and run the (dev) function:
(use 'kbsportal)
(dev)
Hit the URL http://localhost:8080 to load the test web page, edit either source file, and reload the web page to see the effects of your edits. This setup provides a quick and effective interactive development environment. If you just use emacs, then for a better alternative look for one of the excellent emacs+slime+swank Compojure-specific tutorials on the web.