Mark Watson's blog

Citrusleaf: an interesting (non open source) NoSQL data store

2012-01-21T13:27:00.000-07:00

I have been using Citrusleaf for a customer (SiteScout) task. Interesting technology. Maybe because I am excessively frugal, but I almost always favor open source tools (Ruby, Clojure, Java, PostgreSQL, MongoDB, Emacs, Rails, GWT, etc., etc. that I base my businesses on). That said, I also rely on paid for software and services (IntelliJ, Rubymine, Heroku, AWS services, etc.) and it looks like Citrusleaf is a worthy tool because of its speed and scalability (which it gets from Paxos, using lots of memory, efficient multicast when possible for communication between nodes in a cluster, etc.)

Yes, the DynamoDB managed data service is a very big deal

2012-01-18T22:12:00.000-07:00

Just announced today: DynamoDB solves several problems for developers:

No administration except for creating database tables (including some decisions like using simple lookup keys or keys with range indices and whether reads should be consistent or not)
Fast and predictable performance at any scale (but see comment below on the requirement for provisioning)
Fault tolerance
Efficient atomic counters

The probable hassle for developers that I see is in knowing how to provision tables for reasonable numbers of allowed reads and writes per second. When you create tables one option is to get warning emails when you hit 80% of provisioning capacity; I interpret this to mean that you really had better not go over the capacity that you have provisioned. Amazon needs to know how much capacity you need in order to allocate enough computing nodes for your tables. The capacity that you pay for can be raised and lowered to avoid getting runtime exceptions when you go over your provisioned number of reads and/or write per second.

The lastest AWS Java SDK handles DynamoDB. For Ruby, the latest aws-sdk (gem install aws-sdk) supports DynamoDB. I signed up for DynamoDB, looked at the Java example, and wrote a little bit of working Ruby code using documentation - I had to slightly change the example code to get it to work for me:

require 'aws-sdk'

dynamo_db = AWS::DynamoDB.new(
    :access_key_id => ENV['AMAZON_ACCESS_KEY_ID'],
    :secret_access_key => ENV['AMAZON_SECRET_ACCESS_KEY'])
table = dynamo_db.tables.create('my-table', 10, 5)

begin
  sleep 3
  puts "Waiting on status change #{table.status}"
end while table.status == :creating

# add an item
item = table.items.create('id' => '12345', 'foo' => 'bar')

# add attributes to an item
item.attributes.add 'category' => %w(demo), 'tags' => %w(sample item)
p item

# update an item with mixed add, delete, update
item.attributes.update do |u|
  u.add 'colors' => %w(red)
  u.set 'category' => 'demo-category'
  u.delete 'foo'
end
p item.attributes.to_h

# delete attributes
item.attributes.delete 'colors', 'category'

# get attributes
p item.attributes.to_h

# delete an item and all of its attributes
item.delete

I used the AWS web console to then delete the test table to avoid charges.

DynamoDB is a big deal because while it is easy enough to horizontally scale out web applications and back end business applications, it is a real pain to scale out data storage for session handling and application data. Except for paying for the service, Amazon is trying to remove these hassles for developers.

I think that in addition to deployments to EC2s, DynamoDB will be a very big deal for Heroku users because it gives them another data store option in addition to Heroku's excellent managed PostgreSQL service, MongoHQ, Cloudant, and other 3rd party data service providers.

Web 3.0 and the Semantic Web, a slight return

2012-01-11T12:14:00.000-07:00

After talking with a friend and a friend of his about the Semantic Web and healthcare yesterday, I re-watched a great video on Web 3.0 by Kate Ray that I bookmarked and blogged about a couple of years ago. I like this video because it frames the problems that the Semantic Web is trying to solve. My last published book (for APress) had Web 3.0 in the title, a term that did not really catch on :-)

At least a little bit of my enthusiasm for Semantic Web technologies has diminished over the last ten years because of problems that I have had on customer projects trying to collect linked data from disparite sources and merge it into something useful. There are (apparently) no silver bullets and any data collection and exploitation activities involve a lot of difficult work.

I would not be surprised if this problem of merging different data sources is not solved by using Ontologies and webs of linked data sites, but rather, by vendors curating data in narrow domains and selling interfaces to this curated data.

In a world of too much information the activity of curation can have a very high value and this value and the market price for these services will determine the amount of resources invested in combinations of automated and manual curation of information.

sleepybird.us site is online

2012-01-10T16:34:00.002-07:00

Yesterday I wrote about two web portals I have been working on in Clojure. One of them is online: our stock photo web site.This is a simple web app written with Clojure and Noir. I use the excellent stripe.com system for accepting orders for JPEGs (and soon hi-def video clips). In my tests it seems easy enough to buy JPEG files: you just check the ones you want, go to the purchase page, and in a few seconds you are downloading a ZIP file with the JPEGs you purchased. A simple little web app but I think that my wife and I will have fun with it: we are avid photographers.

My two new projects: both web portals written in Clojure

2012-01-09T17:08:00.000-07:00

I have three web portal projects that I have wanted to develop for quite some time. I am close to releasing two of them (a text analytics web service and a stock photos and video clip store. My wife and I are avid photographers and we have been wanting to travel more and do more photography; I started putting together the photo site yesterday morning and hope to have it fully on line in the next day of two - simple to implement. The text analytics web service will be publicly available within a month or so (right now, just the demo page is active - I short circuited the new account login for now).My third project is a web portal for a single consultant to manage multiple customers. Last year I prototyped this for my own use using Java + GWT + AppEngine and then ported it off of AppEngine, using MongoDB for the data store. I have had such a fun and productive time using Clojure and Noir for my two recent projects that I am considering porting this third project to Clojure. I might leave it as-is except that I have already done most of the design for a more complex web app to manage multiple consultants with multiple customers. I know that the development would go much faster using Lisp.

Using Emacs and org-mode in OS X

2012-01-02T13:25:00.002-07:00

I recently ran across David O'Toole's org-mode tutorial and I have been experimenting with using org-mode with Emacs instead of the little utility web app I wrote for my own use a few years ago. I decided that I like org-mode better after learning the basic commands even though I can no longer access my to-do lists from ay web browser. Org-mode is useful for more than simply managing to-do lists and tasks but that is what I am mostly using it for.

To make org-mode always easily available I added this to my ~/.profile:

alias orgmode='echo -e "\033]0;org-mode\007";Emacs -nw ~/Documents/org-mode/.'

This will open org-mode in a the current term window tab and change the tab title to "org-mode." I keep all of my org-files in /Users/markw/Documents/org-mode/ so change that bit of bash script to reflect where you want to keep your org data files. You might also want to substitute emacs for Emacs -nw which is what I use for command line Emacs because I prefer the latest version 2.4.x of Emacs.

(re) learning Clojure

2011-12-05T12:27:00.001-07:00

I have been using Clojure for about 25% of my consulting work in the last 2 years, read two books on Clojure, and I had some Clojure examples in a book I wrote last year.

That said, I don't really feel "expert" at the language the way I do with Java, Ruby, and Common Lisp.

I am trying to fill in some gaps by carefully reading through one of my customer's Clojure code, and all Clojure libraries that I use like Noir, Compojure, etc. I am trying to pick up more idioms. I enjoy it when I see a new trick in someone else's code and going back to my code to improve it.

Using the New York Times Semantic Web APIs

2011-12-04T09:29:00.001-07:00

I am working on a side project of my own in Clojure using the AllegroGraph 4 and Stardog RDF repositories (thanks to Franz and to Clark & Parsia for licenses to use their products!) and my own NLP code. I am using the excellent NYT data access APIs to get research/test data.

I am going to show you some simple examples in Ruby for accessing the NYT Semantic Web APIs that are free to use up to 5000 API calls a day.

I also use other NYT APIs. Each API has an access key that you need to sign up for. I set my access keys as environment variables that I access in my code; for example in Ruby:

# New York Times API Keys:
NYT_SEMANTIC_WEB = ENV['NYT_SEMANTIC_WEB']
NYT_SEARCH = ENV['NYT_SEARCH']
NYT_NEWSWIRE = ENV['NYT_NEWSWIRE']
NYT_PEOPLE = ENV['NYT_PEOPLE']
NYT_TAGS = ENV['NYT_TAGS']

In the following code snippets, I am only using the Semantic Web APIs. I want to first search for available concept types and concept names, based on keyword search:

require 'simple_http'
require 'json'

def semantic_concept_search query
  uri = "http://api.nytimes.com/svc/semantic/v2/" +
        "concept/search.json?" +
        "query=#{CGI.escape(query)}&api-key=" +
        NYT_SEMANTIC_WEB
  JSON.parse(SimpleHttp.get(uri))
end

def  pp_semantic_concept_search query
  json = semantic_concept_search(query)
  puts "Results:\n"
  json["results"].each do |result|
    puts "\n\tconcept_name:\t#{result['concept_name']}"
    puts "\tconcept_type:\t#{result['concept_type']}"
    puts "\tconcept_uri:\t#{result['concept_uri']}" if result['concept_uri']
  end
end

pp_semantic_concept_search("Obama")

The second method "pretty prints" the JSON data that I am interested in. Some of the sample output looks like:

concept_name: Obama, Barack
concept_type: nytd_per
concept_uri: http://data.nytimes.com/47452218948077706853

concept_name: Obama, Malia
concept_type: nytd_per

concept_name: Obama, Michelle
concept_type: nytd_per
concept_uri: http://data.nytimes.com/N13941567618952269073

Once I have a concept type and concept name I can then look up articles:

def lookup_concept_data concept_type, concept_name
  uri = "http://api.nytimes.com/svc/semantic/v2/" +
        "concept/name/#{concept_type}/" +
        "#{CGI.escape(concept_name)}.json?&" +
        "fields=all&api-key=" + NYT_SEMANTIC_WEB
  JSON.parse(SimpleHttp.get(uri))
end
 
def pp_lookup_concept_data concept_type, concept_name
  puts "** type: #{concept_type} name: #{concept_name}"
  json = lookup_concept_data(concept_type, concept_name)
  puts "Results:\n"
  json["results"].each do |result|    puts "\n\tLinks:"
    result["links"].each do |link|
      puts "\t\trelation: #{link['relation']}"
      puts "\t\tlink: #{link['link']}"
      puts "\t\tlink_type: #{link['link_type']}"
    end
    result["article_list"]["results"].each do |article|
      puts "\tTitle: #{article['title']}"
      puts "\tDate: #{article['date']}"
      puts "\tBody: #{article['body']}\n\n"
    end
  end
end

pp_lookup_concept_data('nytd_per', 'Obama, Barack')

Some sample output looks like:

Links:
 relation: sameAs
 link: http://rdf.freebase.com/ns/en.barack_obama
 link_type: freebase_uri
 relation: sameAs
 link: http://dbpedia.org/resource/Barack_Obama
 link_type: dbpedia_uri
 relation: sameAs
 link: http://en.wikipedia.org/wiki/Barack_Obama
 link_type: wikipedia_uri

Title: U.S. Urges Egypt To Let Civilians Govern Quickly
Date: 20111126
Body: WASHINGTON -- Ever since tens of thousands of protesters converged on Tahrir Square in Cairo for the first Day of Revolution exactly 10 months ago, the Obama administration has struggled to strike the right balance between democracy and stability. In the early morning hours on Friday, President Obama came out on the side of the Arab street, issuing

Title: EDITORIAL; The Solyndra Mess
Date: 20111125
Body: The Republicans on the House Energy and Commerce Committee appear to have hit the pause button on their investigation into the failure of Solyndra, a solar panel maker that entered bankruptcy proceedings in September, defaulting on a $528 million federal loan. What have we learned? Nobody comes out of this looking good. Not the Obama

Great to see useful linked data/Semantic Web data sources being made available! Hopefully these little code snippets will save you some time in getting started using the NYT APIs.

Closer to the metal: Clojure, Noir, and plain old Javascript

2011-11-26T16:14:00.001-07:00

I am wrapping up a long term engagement over the next five to six weeks that uses Java EE 6 on the backend, and SmartGWT (like GWT, but with very nice commercially supported components) clients. As I have time, I am starting up some new work that uses Clojure and Noir, and it is like a breath of fresh air:

I keep a repl open on the lein project and also separately run the web app so any file changes (including the Javascript in the project) are immediately reflected in the app. Such a nice development environment that I don't even think about it while I am working, and maybe that is the point!

As I have mentioned in previous blog posts, I really like the Clojure Noir web framework that builds on several other excellent projects. Developing in Noir is a lot like using the Ruby Sinatra framework: handles routes, template support options, but it is largely roll your own environment.

Ruby Sinatra web apps with background work threads

2011-11-21T21:37:00.001-07:00

In Java-land, I have often used the pattern of writing a servlet with an init() method that starts up one or more background work threads. Then while my web application is handling HTTP requests the background threads can be doing work like fetching RSS feeds for display in the web app, perform periodic maintenance like flushing old data from a database, etc. This is a simple pattern that is robust and easy to implement with a few extra lines of Java code and an extra servlet definition in a web.xml file.

In Ruby-land this pattern is even simpler to implement:

require 'rubygems'
require 'sinatra'

$sum = 0

Thread.new do # trivial example work thread
  while true do
     sleep 0.12
     $sum += 1
  end
end

get '/' do
  "Testing background work thread: sum is #{$sum}"
end

While the main thread is waiting for HTTP requests the background thread can do any other work. This works fine with Ruby 1.8.7 or any 1.9.*, but I would run this in JRuby for a long-running production app since JRuby uses the Java Thread class.

Using the Stardog RDF datastore from JRuby

2011-11-21T14:14:00.001-07:00

I was playing with the latest Stardog release during lunch - the way to quickly get going with the included Java examples is to create a project (I use IntelliJ, but use your favorite Java IDE) and include all JAR files in lib/ (included all nested directories) and the source under examples/src.

I took the first Java example class ConnectionAPIExample and converted the RDF loading and query part to JRuby (strange formatting to get it to fit the page width):

require 'java'
Dir.glob("lib/**.jar").each do |fname|
  require fname
end

com.clarkparsia.stardog.security.SecurityUtil.
      setupSingletonSecurityManager()
com.clarkparsia.stardog.StardogDBMS.get().
      createMemory("test")

CONN = com.clarkparsia.stardog.api.
        ConnectionConfiguration.to("test").connect()
CONN.begin()
CONN.add().io().format(org.openrdf.rio.RDFFormat::N3).
  stream(java.io.FileInputStream.new(
            "examples/data/sp2b_10k.n3"))

QUERY = CONN.query("select * where {?s ?p ?o}")
QUERY.limit(10)
RESULTS = QUERY.executeSelect()

while RESULTS.hasNext() do
  result = RESULTS.next()
  result.getBindingNames().toArray().each do |obj|
    puts "#{obj}: #{result.getBinding(obj).getValue().stringValue()}"
  end
  puts
end

This is mostly just a straight conversion from Java to Ruby. The first few lines enumerate all JAR files and require them. The last part, of interpreting the results, took a few minutes to figure out. I used IntelliJ to explore the result values of class MapBindingSet, looking at available methods to call to get the binding names of the variables in my SPARQL query and the values (as strings) for these three variables for each returned result.

Output will look like:

s: http://localhost/vocabulary/bench/Journal
p: http://www.w3.org/2000/01/rdf-schema#subClassOf
o: http://xmlns.com/foaf/0.1/Document

s: http://localhost/vocabulary/bench/Proceedings
p: http://www.w3.org/2000/01/rdf-schema#subClassOf
o: http://xmlns.com/foaf/0.1/Document
...

If you want to run this bit of code, put it in a file test.rb in the top level Stardog distribution directroy and just run

jruby test.rb

I wanted to be able to use Stardog from both JRuby and Clojure. My lunch time hacking today is just a first step.

Experimenting with Google Cloud SQL

2011-11-15T19:05:00.001-07:00

I received a beta invite today and had some time to read the documentation and start experimenting with it tonight.

First, the best thing about Google Cloud SQL: when you create an instance you can specify more than one AppEngine application instances that can use it. This should give developers a lot of flexibility for coordinating multiple deployed applications that are in an application family. I think that this is a big deal!

Another interesting thing is that you are allowed some access to the database from outside the AppEngine infrastructure. You are limited to 5 external queries per second but that does offer some coordination with other applications hosted on other platforms or host providers.

Their cloud SQL service is free during beta. It will be interesting to see what the cost will be for different SQL instance types.

It was very simple getting the example Java app built and deployed. I created a separate SQL instance (these are separate from other deployed AppEngine application instances), made a new IntelliJ AppEngine project, pasted in the example code, and it all worked.

Perception of quality is often influenced by price. Since developers now have to pay more for using AppEngine, I find myself looking more at AppEngine as a premium service, which it is. Despite my dislike for MySQL (I use PostgreSQL when given a choice), Google's hosted and managed MySQL cloud data service looks good and provides developers with more options. Their SQL service is synchronously replicated between data centers automatically for you.

It has been a few years now since I had to either set up a physical server or a leased raw server for any deployments. I like that! Thank you Platform as a Service (PaaS) providers like Heroku (built on AWS) and AppEngine - they are the future. I still do a lot of work on "plain AWS" but that is still much more agile than provisioning my own servers.

The quality of new programming languages is apparent by looking at projects using the language

2011-11-12T08:57:00.001-07:00

The community growing around the Clojure language is great. While the Clojure platform is still evolving (quickly!) browsing through available libraries, frameworks, and complete projects is amazing.

My "latest" favorite Clojure project is Noir that simply provides a composable mechanism for building web applications (using defpartial). I get to use Noir on two customer web app projects (and some work with HBase + Clojure) over the next month or two, and I am looking forward to that. The simpler of the two web apps is an admin console exposing some APIs on a private LAN and the Try Clojure web app is a great starting point, as well as an example of a nicely laid out Noir application.

Since Clojure is such a concise language I find it easy to read through, understand, evaluate, and use projects. Since I am still learning Clojure (I have just used Clojure for about 6 months of paid work over the last couple of years) the time spent reading a lot of available code to find useful stuff is very well spent because reading good code with an open repl is a great way to learn new idioms.

Writing a simple SQL data source for the free LGPL version of SmartGWT

2011-11-07T15:14:00.000-07:00

While travelling back from a vacation I cleaned up some old experimental code for writing a fairly generic SmartGWT data source with the required server side support code. The commercial versions of SmartGWT have support for connecting client side grid and other components to server side databases. For the free version of SmartGWT you have to roll your own and in this post I'll show you a simple way to do this that should get you started. Copy the sample web app that is included in the free LGPL version of SmartGWT and make the modifications listed below.

I also set up a Github project that contains everything ready to run in IntelliJ.

The goal is to support defining client side grids connected to a database using a simple SQL statement to fetch the required data using a custom class SqlDS. I had to strangely format the following code snippets to get them to fit the content width for my blog:

    ListGrid listGrid = new ListGrid();
    listGrid.setDataSource(
      new SqlDS(
         "select title, content, uri from news where " +
         "content like '%Congress%'"));
    listGrid.setAutoFetchData(true);

The following datasource looks for the column names (i.e., "title", "content", and "uri") in the SQL query and creates fields in the constructed SqlDS instance with those column names. I also assume that there is a servlet defined to process the HTTP GET fetch at the bottom of the constructor:

package com.markwatson.client;

import com.smartgwt.client.data.DataSource;
import com.smartgwt.client.data.DataSourceField;
import com.smartgwt.client.types.DSDataFormat;
import com.smartgwt.client.types.FieldType;

import java.util.Arrays;
import java.util.List;

public class SqlDS extends DataSource {
  public SqlDS(String sql) {
    setID(id);
    setDataFormat(DSDataFormat.JSON);

    List<String> tokens =
        Arrays.asList(sql.toLowerCase()
           .replaceAll(",", " ").split(" "));
    int index1 = tokens.indexOf("select");
    int index2 = tokens.indexOf("from");
    for (int i=index1+1; i<index2; i++) {
      if (tokens.get(i).length() > 0) {
         addField(new DataSourceField(tokens.get(i),
               FieldType.TEXT, tokens.get(i)));
      }
    }
    // should do a better job at UUENCODEing SQL:
    setDataURL("/news?query="+ sql.replaceAll(" ","20%"));
  }
}

The only thing left to do is write a servlet that processes web wervice requests like /news?query=... and returns JSON data with fields from the SQL query for each returned row for display in the list grid:

package com.markwatson.server;

import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;

public class DbRestServlet extends HttpServlet {
    @Override
    public void doGet(HttpServletRequest req,
         HttpServletResponse resp) throws IOException {
      PrintWriter out = resp.getWriter();
      try {
          // remove "query="
          String sql = req.getQueryString().substring(6); 
          int index = sql.indexOf("&");
          sql = sql.substring(0, index);
          out.println(
             DbUtils.doQuery(sql.replaceAll("20%", " ")));
      } catch (Exception ex) {
        ex.printStackTrace(System.err);
        out.println("[]");
      }
    }
}

The utility class DbUtils returns JSON data which is what the client side SqlDS DataSource class expects from the server:

package com.markwatson.server;

import org.codehaus.jackson.map.ObjectMapper;

import java.io.StringWriter;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class DbUtils {
  static String dbURL;
  static Connection dbCon;

  static {
    try {
      Class.forName("org.postgresql.Driver");
      // Define the data source for the driver
      dbURL = "jdbc:postgresql://localhost/test_database";
      dbCon = DriverManager.getConnection(
                     dbURL, "postgres", "password");
    } catch (Exception e) {
      e.printStackTrace();
    }
  }

  public static String doQuery(String sql)
                               throws Exception {
    ObjectMapper mapper =
       new ObjectMapper(); // should cache and reuse this!
    List<Map<String, String>> ret =
        new ArrayList<Map<String, String>>();
    Statement statement = dbCon.createStatement();
    ResultSet rs = statement.executeQuery(
                            sql.replaceAll("20%", " "));
    java.sql.ResultSetMetaData meta = rs.getMetaData();
    int size = meta.getColumnCount();
    while (rs.next()) {
      Map<String, String> row =
         new HashMap<String, String>();
      for (int i = 1; i <= size; i++) {
        String column = meta.getColumnName(i);
        Object obj = rs.getObject(i);
        row.put(column, "" + obj);
      }
      ret.add(row);
    }
    StringWriter sw = new StringWriter();
    mapper.writeValue(sw, ret);
    return sw.toString();
  }
}

I had to add three JAR files to the SmartGWT sample project:

jackson-core-lgpl-1.8.1.jar
jackson-mapper-lgpl-1.8.1.jar
postgresql-9.0-801.jdbc4.jar

SmartGWT's DataSource abstraction is a real improvement over how I connect to databases in GWT apps where I tend to write a lot of small RPC services to fetch and save data as required. My simple DataSource subclass SqlDS does not support writing data back to the database from the client; it can either be extended or you can use a RPC service call to save edited data.

Annoyed by anti-MongoDB post on HN

2011-11-06T10:51:00.000-07:00

I am not going to link to this article - no point in giving it more attention. The anonymous post claimed data loss and basic disaster using MongoDB. I call bullshit on this anonymous rant. Why was it posted anonymously?

I am sitting in an airport waiting to fly home right now: just finished extending a Java+MongoDB+GWT app and I am starting to do more work on a project using Clojure+Noir+MongoDB.

I do have a short checklist for using MongoDB:

For each write operation I decide if I can use the default write and forget option or slightly slow down the write operation by checking CommandResult cr = db.getLastError(); - every write operation can be fine tuned based on the cost of losing data. I usually give up a little performance for data robustness unless data can be lost with minimal business cost.
I usually use the journalling option.
Use replica pairs or a slave.
I favor using MongoDB for rapid prototyping and research.
I use the right tool for each job. PostgreSQL, various RDF data stores, and sometimes Neo4J are also favorite data store tools.

Notes on converting an GWT + AppEngine web app using Objectify to a plain GWT + MongoDB web app

2011-11-04T10:16:00.000-07:00

There has been a lot of noise in blog-space criticizing Google for the re-pricing of AppEngine services. I don't really agree with a lot of the complaints because it seems fair for Google to charge enough to make AppEngine a long term viable business.

That said, I have never done any customer work targeting the AppEngine platform because no one has requested it. (Although I have enthusiastically used AppEngine for some of my own projects and I have written several AppEngine and Wave specific articles.) I still host KnowledgeBooks.com on AppEngine.

I wrote a GWT + AppEngine app for my own use about a year ago, and since I always have at least one EC2 instance running for my own experiments and development work I decided to move my app. It turns out that converting my app is fairly easy using these steps:

Copy my IntelliJ project, renaming it and removing AppEngine facets and libraires.
Add the MongoDB Java required JARs
I had all of my Objectify datastore operations in a single utility class on the server side - I converted this to use MongoDB

Sure, a complex application would take a while, but my app only has 6 model classes (all POJOs) so the whole process took less than 90 minutes.

Recent evaluations of web frameworks while on vacation

2011-11-04T08:54:00.000-07:00

My wife Carol and I have been visiting family in Rhode Island this week and since our grandkids are in school on weekdays, I have had a lot of time to spend writing the fourth edition of my Java AI book and also catching up on reevaluating web frameworks.

Although my main skill sets are in data/text mining, general artificial intelligence work and Java server side development, I do find myself spending a lot of time also writing web applications. In the last few years, I have done a lot of work with Rails (and some Sinatra), GWT, and most recently with SmartGWT because one of my customers really liked SmartGWT's widgets. (Note: if you are in the San Jose area and want to work on a SmartGWT project with me, please email me!)

For my own use, because I have strong Java and Ruby skills, the combination of Rails, GWT, and SmartGWT works very well for me when I need to write a web app.

That said, I have spent time this week playing with Google's Closure Javascript tools and less time with ClojureScript that uses Google's Closure. Frankly, both Closure and ClojureScript look fantastic, but I have a personal bias against making Javascript development a career and although ClojureScript works around this issue by compiling a nice subset of Clojure to Javascript I am concerned that the market for developing with ClojureScript is probably small. If you do want to write Lisp code on the server and client side definitely spend a few evenings playing with ClojureScript because it may be a good fit for you. I have also recently had a good experience with Clojure and the Noir web framework.

Anyone know any SmartGWT and Java developers looking for a job?

2011-11-01T12:09:00.000-07:00

A call out for some help: one of my favorite customers is looking for a SmartGWT and Java developer in San Jose area - anyone know anyone good + available?

Common Lisp example code for my Semantic Web book is now LGPL licensed

2011-10-23T08:35:00.000-07:00

A few days ago I re-released the Java, JRuby, Clojure, and Scala example code for my JVM languages edition of my Semantic Web book under the LGPL.

I just did the same thing today for the Common Lisp edition of this book:

Github repository

Changed license from AGPLv3 to LGPLv3 for example code in my book "Practical Semantic Web and Linked Data Applications, Java, Scala, Clojure, and JRuby Edition"

2011-10-20T20:16:00.000-07:00

Here is the github repository for the source code and all required libraries.

My open content web page" where you can download a free PDF for my book or follow the link to Lulu to buy a print version.

Enjoy!

Semantic Web, Web 3.0, and composable systems

2011-10-20T12:23:00.000-07:00

I really enjoyed Steve Yegge's long post last week about the shortcomings of Google's architecture. Google provides great services that I use every day but building systems as Amazon does of composable web services (AWS) to build more complex products and services seems like a better approach.

I have been experimenting with Semantic Web (SW) technologies since reading Tim Berners-Lee, James Hendler, and Ora Lassila's 2001 Scientific American article. I have not often had customer interest in using Semantic Web technologies and I think that I am starting to understand why people miss the value-add:

Just as AWS provides composable web services SW helps information providers to provide structured and semantically meaningful data to customers and users who decide what information to fetch, as they need it. These consumers of SW data sources must have a much higher skill set to build automated systems compared to a user of the web who manually navigates around the web to find information that they need.

So I think that the issue becomes how can to make it relatively easy for system designers and software engineers to fetch and consume information. The easy answer is to point them to a good book on SPARQL and RDF data sources. A better answer is probably to provide examples using common programming languages, the "best" libraires for making SPARQL queries, and small sample applications tailored to the types of data that an information provider provides and what type of inferencing makes sense to discover implicit data that is not explicitly in the provider's data store.

I would describe the SW as building and using composable data sources that are defined in terms of ontology's that make it possible to merge data from different sources and to discover implicit data through inference/reasoning.

A letter to my friends and family: the death of American democracy: not dying, but already dead

2011-10-19T19:39:00.002-07:00

Hello family and friends,

Democracy in our country is dead, but you would not know it from reading the highly censored corporate-owned and controlled "news"/propaganda media.

If you look to foreign news or youtube or the general Internet or talk to friends in foreign countries that have a free press, you will understand that is is not rank and file cops, but their supervisors commiting what I think can only be called illegal brutality against the "occupy" movement.

The high-ranking police do this because they are ordered by their puppet-masters to do so. There is a huge disparity between what the general public wants and what the corporate lackeys in Congress and the corporate lackey Obama (following in the ubber corporate lackey W.Bush's footsteps) do. As Warren Buffet said in a recent interview, the USA is now a plutocracy, and that is a shame. Good writeup on a writer's arrest: I enjoy Naomi Wolf's work - a writer with reasonable views.

Our founding fathers warned us about banks and control of currency, control by the rich, etc. taking control of our country, but the right-wing extremists have removed civics from school curriculums, used total control of the media to scare people into giving up their rights as American citizens, and acted against the common good.

One Republican agenda is to make it difficult for the poor, the elderly without transportation, and students to register to vote, and their anti American "values" disgust me. Anyone who has anything to do with keeping American citizens from voting by making registration more difficult is an asshole. These are people who don't understand what our country is supposed to represent, or don't care.

One last comment: the un-educated are most easily swayed by right-wing propaganda, which is why I believe that our educational system has been deconstructed by those on the far right politically.

It is time for people who have self-identified as republicans and conservatives to speak up against the un-American right wingers who corrupt our political system. Barry Goldwater had great conservative ideals which I largely agree with, but I bet he is rolling in his grave in disgust by the current bunch of "conservatives."

Appreciating Steve Jobs and the people taking part in "Occupy Wall Street"

2011-10-05T19:31:00.000-07:00

First: my condolences to Steve Job's family and friends. He was an awesome guy who lived on his own terms and made the world a better place by doing things that he loved and was proud of.

I would also like to give a shout out of appreciation to the broad spectrum of Americans who are taking part in "Occupy Wall Street." They are facing state sponsored brutality: the elite class doesn't like the legal protests so they put pressure on the government and government influences police to do things that in their hearts they know are not right. I have been reading a lot of strong criticism of the police for their brutality in New York City against mostly peaceful American citizens exercising their first amendment rights - I personally try to not blame the police because I think it is more accurate to blame the people who control them. There are shocking videos on youtube of police brutality against US citizens in New York City during these protests and I thought about putting some links here, but these videos are violent and may be upsetting to many people.

The erosion of rights for American citizens is shocking even though I have been watching the same thing happening already in England so I have been expecting a similar collapse of basic American rights and values in favor of the powerful.

Experimenting with Clojure 1.3 and Noir 1.2

2011-10-02T17:34:00.000-07:00

Noir is a Clojure "mini framework" that is built on top of Compojure. Chris Granger released a new version today that is updated for Clojure 1.3. After working mostly in Clojure last year but using Clojure not very much this year (lots of work for a Java shop) I decided to check out both Clojure 1.3 and Noir 1.2 this afternoon - and I liked what I saw.

The Noir example application uses a recent version of clj-stacktrace and stack traces are much better: Noir prints a well formatted stack trace on any generated web page if an error occurs. This stack trace is very good, filtering out information that you really don't want to see, identifying where the error occurred, and with usually a useful error message.

This eliminates the only major complaint I have ever had with Clojure. Very cool!

The Noir web site had a link to an article written by Ignacio Thayer on running a Clojure Noir MongoDB app on Heroku, using a free MongoDB account. Worked great. I made a trivial change to src/noir_mongo_heroku/views/welcome.clj to also work with a local MongoDB service:

 (let [mongo-url (get (System/getenv) "MONGOHQ_URL")]
   (if mongo-url
     (let [config    (split-mongo-url mongo-url)]
       (mongo! :db (:db config)
               :host (:host config)
               :port (Integer. (:port config)))
       (authenticate (:user config) (:pass config)))
     (mongo! :db "db"))

Noir roughly supplies the same general level of functionality as Sinatra. Noir's development environment, like Compojure that it is layered on, supports live code reloading so if you are used to an interactive dev style like that of Rails and Sinatra, and if you like Lisp (:-) then give it a try.

I am an old Lisp hacker, starting with Lisp on a Dec 10, and getting my first Lisp Machine in 1982. I think that Lisp (Common Lisp, Scheme, Clojure, etc.) mostly appeals to those of us who would (mostly) like to build up our own infrastructure. I say this even though some Lisps have huge libraries: Franz Lisp, Racket, Clojure (both good native libraries and stuff inherited from Java-land), etc. are certainly "batteries included" languages/platforms, but I still characterize Lisp'ers as build it ourselves types.

JPA 2 is the only part of Java EE 6 that I like a lot - how it compares to ActiveRecord

2011-10-02T13:18:00.000-07:00

First, in Ruby-land: I am a huge fan of both Datamapper and ActiveRecord. Here I am only going to talk about ActiveRecord because it is freshest for me because I have been reading through a few Rails specific books that Obie Fernandez's publisher Addison-Wesley sent me review copies of earlier this year: these books use ActiveRecord 3.*. Recently I created two small throw-away learning apps using Rails 3.1 to kick the tires on new features and I used ActiveRecord for each.

One of my customers is a Java EE 6 shop (although we now do use SmartGWT for web apps) and I have been using JPA 2 (Hibernate provider) a lot. In Java-land, I can't imagine using anything else to access relational databases unless you want to use the Hibernate APIs directly, and I would not be inclined to walk that path.

I used to approach object modeling and design differently in Ruby: I would usually start with a relational database and use ActiveRecord's (fairly) automatically generated wrapping APIs. I now think that this is a generally "less good" approach (unless you are using a large legacy database) and now I start with generating models and migrations for the first cut of my object models and then use new migrations to manage changes to my models' schemas. Perhaps a small difference, but I am happier thinking about Ruby model classes than database schema.

So, my development approach is now very similar to using Ruby+ActiveRecord and Java+JPA 2.

My belief is that it is much faster to do object modeling and data persistence, including time for changing object models and associated business logic in Ruby and ActiveRecord. Not even the most die-hard Java developers should seriously argue with this. Given more expensive development in Java + JPA 2, I think that it is worthwhile listing some things I like less about ActiveRecord:

It is more difficult to read code and understand the models because access methods are not defined in the code. For example, if a class User has a many to many relationship with the class Project, evaluating something like (User.first.public_methods - Object.methods).sort will indeed show that ActiveRecord provides a method projects that returns an array off associated projects. However, if you don't know the conventions that ActiveRecord uses for inherited class methods like has_and_belongs_to_many, has_many, belongs_to, etc. then reading other people's code may be confusing. I find that navigating large numbers of JPA annotated model classes in a good IDE like IntelliJ is quick and makes understanding a very large codebase manageable.
A quick 100 hour reading of the Hibernate Reference Manual shows more capability and options than I am aware of in ActiveRecord.
For large "enterprise" applications where increased cost of development does not matter as much, the availability of off shelf distributed caching options and other highly scalable infrastructure components favors, in my opinion, the Java platform.

That said, valid counter arguments are:

The learning curve for ActiveRecord is relatively small so after writing a few complete web applications a developer will know most of what they need and well written Rails apps should be easy to read and understand. I also find that reading though large Rails applications using RubyMine provides almost as good of a developer experience as working with large Java projects with IntelliJ.
Most Java developers will never need to use most of what JPA 2 and Hibernate offer.
Worrying about very large scale optimization is almost always a premature optimization. I am reading a great book "The Lean Startup" that makes this point very well: avoid making very long term plans, instead favoring short iterations of planning, measuring, then pivot or keep the current plan.

One thing that I believe that both JPA 2 and ActiveRecord do very well is managing transitive relationships. Both can be set to perform cascading deletes, etc. I think that JPA 2 and Hibernate offer a finer control for maintaining associations between objects but for my work I have found ActiveRecord to always be adequate.

I enjoy using both ActiveRecord and JPA 2 enough that I don't really care which platform my customers prefer, and I almost never try to talk my customers into switching away from their platform of choice.

Finally saw movie "Crazy Heart" - thinking that an AI could write country western songs

2011-09-29T19:23:00.000-07:00

Inspired by James Meehan’s Tale-Spin program and thesis, I have to say: writing an AI program to write country western music seems possible. Prof$t!

For work I have been using GWT/SmartGWT. For fun: Seaside and Pharo

2011-09-25T11:01:00.000-07:00

I have been enjoying working on two customer web apps written in SmartGWT. SmartGWT is built on Google's GWT with the addition of Isomorphic's smart client library that implements very nice data grids and other UI components and also has good support for wiring rich clients to data sources. Still, there is a lot of ceremony involved in GWT and SmartGWT development so I would recommend these technologies for large projects. For me this ceremony and large learning curve is well worth it because I like coding both rich client and server side components in Java in one development environment (IntelliJ).

For side projects that require a web UI I like using both Play! and Rails (and a lot of Rails development work in the last 3 or 4 years).

Just recently as another side learning project I have been revisiting the Seaside continuation based web framework for Smalltalk. This week I bought the PDF version of "Dynamic Web Development with Seaside" and when I get bits of free time I have really been enjoying reading through the book with a Pharo Smalltalk Seaside image open next to the book to work along and experiment as I read. You can read through this book online also if you want to check out Seaside.

Seaside provides a very agile development environment, both editing code in a Smalltalk browser and hooks in web apps themselves.

Google+ Developer APIs

2011-09-15T13:22:00.000-07:00

I received an email from Google today about the release of APIs to access public data. I looked at the Ruby example (a Sinatra App) and the Java JSP example during lunch. Looks like a good kick-start for using public Google+ data in our own web apps. There are also examples for other languages.If I have time this weekend I would like to try deploying an app of my own.

Getting set up to work on the 4th edition of my Java Artificial Intelligence book

2011-08-28T11:26:00.001-07:00

For the 3rd edition, I used Eclipse for both development of the Java examples and to prepare the Latex manuscript (using the Eclipse Latex plugin TeXlipse).

I really prefer using IntelliJ for Java development and TeXShop (Mac OS X only) for editing Latex files so I just converted my writing setup.

I have a fairly good idea of what new topics I want to cover but I am still deciding what material from the 3rd edition I want to remove.

Since I released the 3rd edition over three years ago, I have averaged about 300 downloads of the free PDF version a day with a few sales of the print edition each month. I like making a free version available for people to read and generating traffic for my web site where I advertise my consulting services is great. Unfortunately, in the last couple of years when I see my book in search results it is very often on someone else's web site which violates the conditions of the Creative Commons non-commercial use license for the PDF version of my book.

My Latex setup supports output of PDF, HTML, and HTML with automatically generated embedded Google ads. I might change my mind, but I think that the free version of the 4th edition will be HTML pages on my web site, perhaps with a few Google ads, but probably not. I'll continue to sell print copies of my book on Lulu and also offer a PDF version for a few dollars.

Changing the way we use the Internet

2011-08-28T10:06:00.001-07:00

Unless searching for online docs, looking up error codes and error messages, etc., I do relatively little web search and browsing anymore - compared to even a year ago. I usually rely on good links from Twitter and Google+ to find things worth reading, keep up with new tech, and sometimes even read the news.

In the 1980s, I was a "find useful stuff at public FTP sites" resource at SAIC. I spent time maintaining lists of useful FTP sites and what they contained so I could help people quickly find stuff. Gopher was a step up. Good search engines were a huge improvement for finding stuff on the web.

Now I find myself mostly depending on what interesting people recommend. Even though I am a techie and don't represent a typical Web user, I still think that the trend of using social media to find interesting (and even useful!) material is widespread. It will be interesting to see how the major web companies like Google, Amazon, Microsoft, Yahoo, etc. perform financially in the future because it seems very difficult to predict new disruptive technologies that will capture peoples attention and interest.

What I have been working on lately

2011-08-16T17:18:00.000-07:00

It has been a while since I blogged. Carol and I are leaving soon on a driving trip with our grandkids, daughter and son in law - it was requested that we both leave our laptops at home, a request that we are planning to honor. So, since I will be without a computer for about 10 days, here is a quick catch-up on what I have been doing:

I have been fairly busy lately working for two customers. At a friends company we are writing rich client web applications using SmartGWT and Java EE 6 on the backend. I have also been working for Compass Labs on building a large graph database and also doing some data mining of Freebase data. Google bought Freebase last year and is working on new features and APIs. I am in a private beta for their new Freebase APIs - good stuff!

Second edition of "Semantic Web for the Working Ontologist"

2011-08-02T21:18:00.001-07:00

First thanks to Morgan Kaufman Publishers for sending me a copy of the second edition. Dean Allenmang and Jim Hendler did a great job of updating the examples and fixing a few small glitches from the first edition.

I have been extremely busy with work so I have only been able to spend about 90 minutes so-far with the second edition but I hope to give it a careful reading when I am on vacation in a few weeks.

This book is an excellent guide for anyone who wants to invest a fair amount of time to learn how to write semantic web enabled applications: a very comprehensive book.

I recently bought a eBook copy of Bob DuCharme's short book Learning SPARQL
Querying and Updating with SPARQL 1.1 that I would also like to recommend as a fairly easy introduction to using SPARQL repositories and generally using SPARQL in applications.

If two good new semantic web books were not enough, I have also had fun recently experimenting with Clark & Parsia's new Stardog RDF datastore that offers fast data loading, fast SPARQL queries, OWL 2 support, and built in search.

Working on a new GWT application for a personal project

2011-07-09T10:05:00.005-07:00

I have been using SmartGWT a lot for a customer's project so I have been generally digging into both the Google Web Toolkit (GWT) and the SmartGWT system that builds on top of GWT.

My personal project is a reimplementation of my Rails application for ConsultingWith.me (very old placeholder page) in GWT. I plan on writing a few long blog articles about GWT and/or SmartGWT in the near future but for now I have learned a few things that are worth sharing:

Plan ahead on designing data models that essentially live on the client side, that support caching on the client side, and efficiently support asynchronous data fetches from the server. The view pages (written in Java, compiled to compact and efficient Javascript) must always make asynchronous calls to the local (in the browser) data model because it is unknown whether the local model has the data or must itself fetch asynchronously from the server.
Stop thinking about session data in the same way that you do in an old style web application: any class variables in your application are always available in the compiled client side code. The only reason to store session data on the server is security: remember that in principle someone could hack the Javascript code in their browser so for some operations the client needs to pass a security token (defined during login/authentication) to the server and the server must verify access privileges for an operation.
Developing in your IDE's debugger is a good idea. I use IntelliJ and there are separate panels for server side log views, warnings generated by the Javascript code on the client, and the debugger itself. Very useful!
Once you have designed your (mostly) client side data model and determined its interface to your server side datastore (which is the AppEngine datastore in my current project) try to get as much of the server side support written and debugged as you can before seriously starting to write the client side UI. The main reason for this is that if you are only editing client side Java code (that gets compiled to Javascript) you can edit, save changes, refresh your browser and immediately see the results. Whenever you need to modify server side code you must restart the application which really slows down development.

Writing a rich web application in GWT certainly takes a different mindset than simpler AJAX applications but it is worth the effort in some cases. For my current personal project, the original Rails implementation had great functionality but was not fluid enough to use continuously throughout my work day to track time and write work notes. Because this is for my own use and there is no deadline for finishing it is worth my time to get it just right, working on it an evening or two a week.

At the beginning of this year I was very interested in SproutCore for the same general developer use cases as GWT. SproutCore uses the same sort of asynchronous event handling style as GWT with a big difference that you write the client in Javascript instead of Java. This year my largest customer evaluated both SproutCore and GWT (and derivatives like SmartGWT and GWT-Ext). Since they are a Java shop they decided to use SmartGWT which gave me an incentive to dive more into GWT related technologies. As with most of you, my technical interests are sometimes (but not always!) driven by what people pay me to work on.

Apache Google Wave in a Box project is starting to look good

2011-07-05T07:52:00.002-07:00

It has been a long time since I tried to run the wave protocol stuff. I just grabbed the latest source and followed these directions on my MacBook. I was quickly up and running with no problems.

The web UI is a lot simpler than Wave but looks similar. Wave in a Box is looking good as a development platform - something to customize for your organization.

I used two browsers, Chrome and Safari to create two test accounts, and as I expected, the real time messaging, etc. worked fine.

Google+ seems to be very well done

2011-06-29T20:36:00.000-07:00

Thanks to Marc Chung for the Google+ invite. The web UI is very slick and I know I am going to have fun with g+. Anyone on Google+, let me know if you want to be in my Artificial Intelligence, Clojure, and/or Ruby groups.

I am using SmartGWT on two projects

2011-06-27T08:27:00.001-07:00

My currently largest customer uses the commercial version of Isomorphic's Smart GWT and I have spent a few evenings working on one of my own projects (that I will probably open source when it is done) that uses the free LGPL version. 7/9/2011 edit: I ended up re-writing this in straight-up GWT. I should also mention that I have signed a consulting agreement with Isomorphic (svn commit rights :-) but I have not had time in my schedule to do any work for them yet.

SmartGWT uses Google's Java to Javascript compiler but instead of using the standard GWT UI components it uses Isomorphic's SmartClient Javascript library (suitably wrapped for extending GWT). The commercial version's sweet spot is reasonably easy integration with server side data sources like relational databases, JSON web services, etc. The free LGPL version provides an example of a client side data source that you can hook up with custom code to web services that you write yourself. For my for-fun side project my back end server processes requests and returns JSON data: fairly easy but not as nice as using Isomorphic's proprietary data sources.

The developer experience is similar to using Google's GWT. You program the UI in Java and it gets compiled to Javascript. If you are only modifying client sode code, then run in the debug/developer mode and your changes to Java UI code get recompiled and loaded into your web browser when you do a refresh.

Prelude to learning Clojure and Scala: learn some Haskell

2011-06-19T11:52:00.000-07:00

I worked through part of the "Real World Haskell" book a few years ago, but settled on mostly using Clojure as a functional language, with some Scala also. I bought "Learn You a Haskell for Great Good!" this week and I have been enjoying the gentle approach to learning the language. Miran Lipovača did a good job writing this book. ("Real World Haskell" is also excellent.)

One thing that occurred to me is that since Clojure and Scala borrow so many good ideas from Haskell that learning some Haskell before diving into either Clojure or Scala might be a good idea.

Largest public SPARQL endpoint: sparql.sindice.com

2011-06-15T13:26:00.000-07:00

The Sindice project has transitioned from a university and consortium project of DERI to a commercial company. Check out the SPARQL endpoint web form - very impressive. During lunch I tried using the Sindice Java client library and it was easy to use but does not for some reason support direct SPARQL queries.

Programmer study time

2011-06-12T13:04:00.003-07:00

I love both of my jobs (programming and writing) as long as I don't overdo it and take a lot of time off for other activities like hiking, kayaking, playing musical instruments, and cooking.

I have another "down time" activity that is both fun and relaxing for me: studying things that help me with my jobs. For example, when I first adopted Ruby as my primary scripting language and also started developing using Rails, I spent a lot of time reading through the C implementation of Ruby, the Ruby libraries, and the Rails source code. I find this kind of study relaxing because there are no deliverables and things learned studying the implementation of tools I use really pays off in increased productivity and learning new programming idioms and techniques. I used to base a lot of my work on the Tomcat server and ten years ago I made a real effort to understand its implementation. When I was very young I worked as a systems programmer and kept source listings of interesting parts of the operating system at home.

My "reading" activities today included two hours looking through some of the code in the EJB container sub-package of the Glassfish web application server. One of my customers runs much of their business on Glassfish so I am motivated to understand the platform. I learn things reading through Glassfish code that I never would reading books on EJB 3.1.

Time is our most precious possession and certainly it is not to be wasted. That said, spending at least several hours a month carefully studying the code in a few of the open source software tools I use is time well spent.

Polyglot programmers: setting up multi-language access to data

2011-05-04T07:50:00.001-07:00

For those of us who tend to use several programming languages doing some up front work to make sure that we have access to all required data stores in the languages we use then makes it possible to really pick the best language for each task. I have a side project that I have been coding on for over ten years, with lots of code in Common Lisp, Scheme, Java and Ruby. If I can ever get a long enough break from my consulting business I have a few good ideas how to monetize some of this work. I was working on this project during a recent cross-country flight and I started adding a bit of new functionality in Ruby, switched to Common Lisp, and the implementation was a bit easier. I needed access to a PostgreSQL database (using PostgreSQL's built in text indexing and search) I use and once I had Internet access after the flight (for some quick reference), I worked out the few lines of code to interface with my annotated news data collection. Here is a small code snippet in case you need to do the same thing (using the lower level pg library instead of clsql because I only need to hit PostgreSQL):

(ql:quickload "pg")

(pg:with-pg-connection
  (conn "kbsportal_development" "postgres"
         :host "127.0.0.1" :password "password")
  (postgresql::pgresult-tuples
    (pg:pg-exec conn "select * from news")))

;; search support:
(pg:with-pg-connection
  (conn "kbsportal_development" "postgres"
        :host "127.0.0.1" :password "password")
  (postgresql::pgresult-tuples
    (pg:pg-exec conn "select * from news where to_tsvector(content) @@ to_tsquery('obama | congress')")))

I added this snippet to a directory full of snippets for common tasks for each language that I use.

Text search in SimpleDB: a Ruby example

2011-04-30T11:24:00.007-07:00

You might want to use SimpleDB for storage and to support text indexing and search if you did not want to manually run and administer Solr yourself. Here is a little snippet that shows how to store searchable documents in SimpleDB:

require 'rubygems'
require 'aws_sdb'

SERVICE = AwsSdb::Service.new

# assuming that this domain is already created
DOMAIN = "some_test_domain_7854854"

class Document

  def initialize name, text
    words = (name + ' ' + text).downcase.split.uniq
    attributes = {:words => words, :text => text}
    SERVICE.put_attributes(DOMAIN, name, attributes)
  end
  
  def Document.search query
    # The last inject takes the intersection and
    # insures that all search terms are present:
    keys = query.downcase.split.collect {|x|
      SERVICE.query(DOMAIN,
                    "['words' starts-with '#{x}']")[0]
    }.inject {|x, y| x & y }
    keys.collect {|key|
                  SERVICE.get_attributes(DOMAIN, key)}
  end

end

Document.new('title1',
             'The bird flew to the lake for water')
Document.new('title2',
             'The dog chased the cat')

p Document.search 'flew lake'

The formatting of this code snippet is odd because I was trying to get short lines to fit the page width. This code snippet is not terribly efficient but since the first 25 Amazon SimpleDB Machine Hours consumed per month are free for your Amazon AWS account using this code example in your applications can end up being almost free (there are small data storage and bandwidth charges) and you get the advantage of no administration hassles. The output for the above code snippet is:

[{"text"=>["The bird flew to the lake for some water"],
  "words"=>["bird", "flew", "for", "lake", "the",
            "title1", "to", "water"]}]

There are two improvements that you can implement: remove noise/stop words from the words attribute and make the code multithreaded to execute the individual SimpleDB queries in parallel when possible to do so. I was trying to make this example code snippet concise. For simple and/or moderately used applications these improvements aren't necessary.

If you run this example remotely from your laptop, notice that remote SimpleDB access is a little slow. When run on a small EC2 instance, it takes about 0.05 seconds to add a "document" to SimpleDB and about 0.1 seconds to search using two search terms.

And the best JVM replacement language for Java is: Java?

2011-04-21T08:31:00.000-07:00

Although I use Ruby (mostly Rails) and Common Lisp on many customer projects, I am heavily invested in the Java platform and I don't see that changing in the next ten years or so.

Java is more than a little heavy on ceremony however, and I would like a really agile language for the JVM. I have used Clojure a lot in the last year for work on one customer's project but at least for now the lack of concise and useful runtime error backtraces kills some of the joy of using Clojure. Really nice language and community however, and I expect in a few years Clojure may be my primary JVM language. I love coding in Ruby and the JRuby developers do a great job moving the sub-platform forward. However, except for large Rails applications, I don't see myself writing very large applications in Ruby: for me Ruby is a scripting language for getting stuff done quickly and easily. I do like Scala but the learning curve is steep and that means that it is difficult to find pre-trained highly skill Scala developers.

Java has the sweet spot of lots of great tools and a rock solid infrastructure. So, how to make Java more agile? I do a few things that help: I use public attributes so I don't bother with getters/setters anymore unless I am using a framework that needs them for introspection. I very much like JPA, but I am growing less fond of the rest of the Java EE 6 stack - really a lot of layers between designing and writing code and runtime; too much abstraction for my tastes. The Play! Framework is great, in general, and I am using it on my personal project and I look forward to seeing how Play! develops as an agile platform over the next few years.

Some new Platform as a Service providers: cloudfoundry.com and dotcloud.com

2011-04-19T12:51:00.001-07:00

I am on vacation so I have not had much chance to try the beta invites I just received for cloudfoundry.com and dotcloud.com but both look promising as works in progress.

For now, Cloud Foundry is set up for Ruby Rack applications (like Rails and Sinatra) and Java Spring apps. They currently support MongoDB, MySQL and Redis. They will release the core software if you want to run a cloud on your own servers.

Dotcloud supports a wide range of platforms and data stores. Their roadmap shows what is available right now and what is planned.

Both beta programs are free for now. It will be interesting to see what the costs are.

(Roughly) comparing Play! version 1.2 with Rails

2011-04-13T17:41:00.005-07:00

Both the Play! and Rails frameworks implement MVC and have very agile development environments. Play!, being written in Java (but also supporting Scala development) accomplishes this agility by using the Eclipse incremental Java compiler so if you edit any Java code or HTML template files (with embedded Java/Groovy expressions) you immediately see the results after refreshing your web browser.

While Play! is not nearly as complete of a stack as Rails, it does include modules for

MongoDB
AppEngine
Objectify
GWT
Search
PDF generation of any view
Scala use
CoffeeScript
OpenAuth working with Google, Yahoo, Twitter, etc.
Simple CRUD scaffolding
Facebook Connect and Graph API
Lucene search of JPA models
etc.

I have several years of Rails experience and I am using Java EE 6 for a customer project. With this background, I put Play! in the sweet spot between Java EE 6 and Rails: easy to learn if you know Java and supports agile development. My favorite part of Java EE 6 is JPA, which Play! supports.

I have played with Play! off and on for over a year, but just for a few hours at a time, and never any serious projects. (This is largely because most people who hire me usually want me to do Lisp or Ruby development.) I have more or less decided to use Play! for one of my own projects because I already have so much reusable Java code I have written for it and I like the interactive Play! development process. My wife and I are just starting a vacation, and after finding myself in a quiet place with time on my hands (we catch an early flight tomorrow morning and are staying near the airport) I just reimplemented a bit of data modeling code that I wrote in Clojure, Ruby, and Common Lisp last weekend, this afternoon in Java + JPA + Play! I really have been struggling with the decision of which language and framework to use, so I am in experimentation mode! BTW, I am using PostgreSQL with its native indexing and search functionality and I find that JPA and Java object models mix fairly well by mostly using JPA with some native quieres like this contrived example using PostgreSQL's indexing and search functionality:

List results = News.em().createNativeQuery("select * from news where to_tsvector(content) @@ to_tsquery('japan | nuclear')", News.class).getResultList();

that maps results back to my Java POJOs.

Amazon Cloud Player: make sure you take advantage of their introductory offer

2011-04-03T11:48:00.000-07:00

I just purchased an MP3 album "Johnny Winter And / Live" for $5 and got a $20 one year upgrade of 20 GB of cloud storage - a sweet deal, but considering that you always get 5 GB free this may not be much of an added value. Amazon has a nifty uploader application that looked at my iTunes MP3s and playlists and is cloning that on Amazon Cloud Player automatically. My entire iTunes library will only take up a few GBs after it is automatically uploaded. Sometimes Amazon kills Apple's iTunes store on price: I was about to buy a few tracks on iTunes last year and then realized I could buy the entire album as MP3 on Amazon for for not much more.

Amazon seems to be investing in introductory offers like the upgrade for Cloud Player and the first time AWS developer's package (basically free to develop and deploy for one year). Certainly expensive for them to provide as free services but Amazon is playing the long game. My ordered list of the most impressive technology companies:

Amazon
Google
Apple
Netflix

Notice that Microsoft is not on my list :-)

The Cloud, The Cloud

2011-03-22T19:52:00.000-07:00

Redit was down for over 5 hours last week because of problems with EBS volumes on AWS. Netflix was down a few hours today: another AWS user but I don't know yet what the difficulties are. So Amazon has problems and the web is full of people complaining about occasional problems with Google's AppEngine cloud hosting service.

No one likes to not support users 24x7 and the users don't like interrupted service, but I think that these occasional outages are just growing pains as we move towards a new way to deploy applications that costs less money, requires fewer staff resources, and likely is more energy efficient.

I think that I have only had one customer in 3 years who did not at least partially deploy on Amazon's AWS. This is the future and we need to learn how to work around problems and take advantage of resource savings when we can.

Node.js

2011-03-20T10:05:00.000-07:00

I read this morning about a new beta book from O'Reilly Up and Running With Node. Before skimming through the beta book, I took a half hour to review the standard Node.js Manual & Documentation. The beta book is very nice (recommended!) and I especially enjoyed the discusion about using the Node REPL. I was lucky enough to have received a Heroku Node beta invitation last year but so far all I have done with Node is to build new versions every few months and play with the examples in the documentation. I think that this is worth the time, even though I have a very full consulting schedule, because Javascript is probably going to become popular for server side development as it already is for browser development. This will happen because of the efficiency of V8, the ability to share code between server and client side, and the inherent scalability of event, rather than multiple thread systems.

David Rumelhart passed away. RIP to a good guy.

2011-03-16T17:10:00.001-07:00

Before David won a MacArthur Grant he was a professor at UCSD and co-wrote a few great books on artificial neural networks that really helped me a lot. My company also hired him as a consultant and he gave me advice when I implemented the 12 currently popular neural network learning and recall algorithms in the 1980s for our ANSim software product. It was a great experience sitting in his living room talking about NNs.

He was a nice guy and I am sure that he helped many people with his work.

MongoDB 1.8 released - and there is joy throughout the land

2011-03-16T16:28:00.000-07:00

I have been running 1.7.5 on my laptop and just upgraded to 1.8 stable. While the normal way to run MongoDB (at least in my work) is to use read-only slaves for analytics, etc., I am still glad to see the single server robustness changes, including optional journaling. I also noticed that in the admin shell, showing databases provides database size estimates.

Another useful change is replica set authentication using identical key files that are placed on each server. You then let one server know about the others (as before). You can read about other improvements here.

Nourish and manage your career, not your job

2011-03-13T09:41:00.002-07:00

I have been working close to full time since the beginning of this year for two customers. This is unusual for me since I have usually capped my work week at 32 hours maximum over the last 25 years. I have been enjoying the work and extra earnings but I believe that working too many hours carries a real cost and some risks:

It is far more important to manage our own careers than any particular job. You don't own your job but you do own your career. Just as you maintain your home and your car, careers require fairly much constant maintenance, including:

Life long learning of new technical skills.
Developing skills that enable people you deal with to also be successful: strive for win-win outcomes.
Networking that supports finding work, getting second opinions on important decisions, and leads on new interesting and useful technologies.
Time for self analysis: what has worked for you in your career (and life!), possible improvements, and understanding situations and attitudes to avoid in the future.

Individual jobs, including consulting gigs, are just tools for improving a career. While we owe employers our best effort, honesty, and transparency (e.g., share bad news earlier rather than later), we also owe it to ourselves to nourish and care for our own careers.

Don't stray too far from well supported language, tool and platform combinations

2011-02-20T13:29:00.002-07:00

I have been doing a lot of customer work lately using Java EE 6 and Glassfish. Fairly nice development environment even if Java EE 6 is heavy on ceremony (but better than J2EE).

Just for fun today I took small play apps I had written in Rails and Clojure + Compojure and shoe-horned them to run on Glassfish. Interesting exercise, but unless there is an overwhelming need to create a custom deployment setup, it is so much better to go with the flow and stick with well crafted and mature setups like

Java and Java EE 6
Clojure + Compojure running with embedded Jetty behind nginx for serving static assets
Rails app hosted on Heroku
Web apps written in either Java or Python hosted on AppEngin using the supported frameworks
etc.

I must admit that I enjoy hacking not only code but also deployment schemes - enjoy it too much sometimes. Sometimes it is worthwhile, most often not.

Curated data

2011-02-07T09:15:00.000-07:00

It is difficult to predict what data will have long term value so it is often safest to archive everything. With data storage costs approaching zero I think that we can expect high value data to last forever, baring a nuclear war or the crash of society.

Curated data has a higher value than saving "everything." I think that the search engine Blekko is interesting and useful because of what it does not have: human powered curation yields fewer results but very little SPAM. The Guardian's curated structured data stores have much higher value than the original raw data (from government sources, etc.). I can imagine The Guardian curated data becoming a permanent part of our history as for example are ancient stone tablets we see in museums.

I have long planned on providing curated news and technology data that has semantic markup either on my ancient knowledgebooks.com domain or a new placeholder kbsportal.com but I seldom have free time slots because of my consulting business. Hint: I would like having a few partners who are into statistical natural language processing and general data geeks to help me with this. I don't know if it would end up being a viable business or just a public service portal.

Big Data

2011-02-06T15:47:00.003-07:00

For the last few decades, it seems like I work on a project every few years that stretches my expectations of how much data can be effectively processed (starting in the 1980s: processing world-wide seismic to detect underground nuclear explosions, all credit card phone calls to detect theft, etc.)

I was in meetings three days last week with a new customer and as we talked about their current needs I made mental notes of what information they are probably not capturing that they should because it is likely to be valuable in the future in ways that are difficult to predict.

To generalize a bit, every customer interaction with a company's web sites should be captured (including navigation trails through web sites to model what people are personally interested in), every interaction with support staff, every purchase and return, etc.

Amazon has set a high standard in user modeling with Amazon suggests for products that you might want to buy. Collecting data on your customers should not make them feel creepy about interacting with your company but rather should make them feel important and well-serviced.

I am a voracious reader both for fun and to continue a life long education. I have found myself shifting some of my reading attention from computer science to statistics, math, and general business intelligence. (That said, I still read several computer science books per month).

I was disappointed that I could attend the Strata 2011 data conference but I did enjoy watching the keynote speech videos: fairly good material and worth some viewing time.

Two good books on AppEngine development

2011-01-31T16:20:00.002-07:00

The publisher PACKT sent me a review copy of Google App Engine Java and GWT Application Development by my friend (via email correspondence) Daniel Guermeur and co-author Amy Unruh (thanks!) and I bought Code in the Cloud Programming Google AppEngine by Mark C. Chu-Carroll.

Both are very good books and complement each other.

Mark's book gives an interesting insite into AppEngine from someone who works at Google. He covers both Python and Java development. I relied on the Python sections of the book when I wrote a Python based AppEngine application last December. I am not much of a Python programmer but his book got me going quickly and I had few problems. He uses Google's Django support for AppEngine for the Python examples and Google Widget Toolkit (GWT) for the Java examples.

Daniel's and Amy's book is a hands-on guide to using the Eclipse IDE and GWT to develop AppEngine applications in Java. They use JDO for the book examples which is probably best for the general reader but my own preference is the lighter weight objectify-appengine datastore wrapper. The example application is neat: Connectr gathers social media and provides some aggregation services. I like that the application is interesting and contains useful code. I would particularly recommend this book if you want to use Eclipse and GWT to build AppEngine applications and you want everything you need in one tutorial and reference book.

Social networking: why fewer connections may be better

2011-01-21T08:52:00.001-07:00

I have a very public web presence from this blog and my web site. I enjoy sharing information and communicating with people via email and occasionally (with a heads-up email first) talking on the telephone. I also spend an hour a week giving free advice to students on their projects, employment hints, and to a more limited degree give feedback on technical ideas. I can enjoy doing this because email is asynchronous: I can handle these interactions when they don't interfere with my work or research.

In the past I have accepted connections on LinkedIn and Facebook from people who I don't know, just t be friendly. However, there is a cost to this.

LinkedIn frequently sends out email statuses of what colleagues (current and present) are doing. I like this for people I know well either personally or through years of email interactions. However, status updates from people I am not closely associated with take time even to ignore.

The situation is worse on Facebook. I used to always accept friend connections from anyone who looked like they were a computer scientist from their public profile (in addition to people I know).

I am a native English speaker with reading knowledge of French. One of the great things about the web is communicating with people from around the world with similar interests. Due to my limited natural language skills however, this has to be in English. Sometimes half of the updates on my Facebook are in languages I can't read.

I am leaving all of my LinkedIn connections as-is but in the last few weeks I have been removing friend connections on Facebook except for close friends, family, and people I have worked with in the past. Checking my Facebook page a few times a week is much more enjoyable.

For other communication, email is best.

Java EE 6 is actually pretty good

2011-01-09T09:33:00.004-07:00

I do most of my development in very agile programming environments like Ruby and Rails (with Datamapper or ActiveRecord), Clojure with MongoDB, etc. I like languages that have an interactive repl.

Recently, I have taken on for a new customer helping (a life-long friend's company) do some conversion to Java EE6 and I must say that Java EE 6 is very well done. I wrote a J2EE book many years ago and used to be into Java server side development but drifted to other platforms partly because that is what customers hired me to work on and partly because of my own technical interests. In the last 5 years I have probably spent about 30% of my time working with Java (customers want Lisp and Ruby development).

Java EE 6 is so much better than J2EE. Writing POJOs (with EJB annotations), unit testing them as simple POJOs, and then integration testing them in an EJB container makes for a reasonable programming environment. I still think that you get much more bang for your programming buck with Ruby and Rails but there are applications that fit very well into the Java EE 6 infrastructure.

Recommended: Niall Ferguson's "The Ascent of Money"

2011-01-02T19:59:00.000-07:00

I just finished watching tonight the DVD of the PBS series but the book covers the same material. Harvard professor/historian Nial Ferguson puts down and in its place the (in my opinion also) misguided view that governments can, long term, spend their ways out of problems.

The PBS series is especially fun to watch (in addition to being very educational) because as Ferguson traces "bubbles" throughout history, there is local video that helped me picture what happened in ancient, medieval, and modern times.

Worth watching! (There are many excerpts on youtube if you don't have time to watch the 4 hour series.)

I rented the DVD from Netflix, and their summary is good:

British historian and author Niall Ferguson explains how big money works today as well as the causes of and solutions to economic catastrophes in this extended version The Ascent of Money documentary. Through interviews with top experts, such as former Federal Reserve Chairman Paul Volcker and American currency speculator George Soros, the intricate world of finance, including global commerce, banking and lending, is examined thoroughly.

One bit of advice is that there is no firm economic security in anything but income. For individuals this the ability to earn money and for governments the ability to raise sufficient taxes to pay for most expenditures. The PBS documentary makes this point very well.

Happy New Year

2011-01-01T11:04:00.002-07:00

I start each day by enumerating the good things in my life (computer scientist-speak for "counting my blessings") followed by meditation and relaxation techniques. A nice way to start each day.

I would like to do the same today, the first day of 2011 (note that 2011 is the sum of consecutive primes: 157+163+167+173+179+181+191+193+197+199+211).

I am grateful for my family and friends, living in one of the most beautiful places in the world (Sedona Arizona), having interesting work with great customers, resources to self-fund my own research as a computer scientist, and time to enjoy my hobbies (cooking, hiking and reading).

Happy New Year!

Nice: Neo4j version 1.2 final released

2010-12-31T08:57:00.000-07:00

Neo4j is a high performance graph database that I usually use with the JRuby neo4j gem and sometimes with Clojure (documentation here).

Neo4j is open source (AGPL v3) and is alternatively available for a reasonable fee with a commercial license (where you don't need to AGPL your project). I took advantage of the free offer to get your first Commercial Basic Server license, even though I am likely to open source my project anyway.

Good SimpleDB performance tips

2010-12-29T11:57:00.000-07:00

I don't usually write blogs that just reference other people's material, but this three part article by Siddharth Anand (first installment, then follow links to other articles) is really worth reading if you use SimpleDB.

BTW, while the local SimpleDB simulator simpledb-dev works OK, for development I usually just access SimpleDB remotely from my laptop. One word of warning: while properly constructed SimpleDB queries can run very quickly from EC2 hosted applications, remote access tends to be 5 to 10 times slower.

WebServius wrapper for web services and data services

2010-12-29T10:44:00.002-07:00

Webservius is a wrapper that provides for your web service APIs and data sources. Webservius handles billing, tools for your customers to use to monitor their use of your services, logging and use statistics, etc. They charge you 10% of what is billed through their service.

When I have a chance I will try the free version of this service (limited number of API calls per day and access to end users must be free) and write up the experience.

I noticed WebServius on Amazon's blog but Webservius is an independent company that uses the Amazon AWS platform.

Control of news media == ability to set public opinion

2010-12-27T11:43:00.003-07:00

In the late 1800s Western Union was able to alter the results of a presidential election (reference: The Master Switch: The Rise and Fall of Information Empires - recommended!)

A similar situation exists today in the USA: all major news sources provide a very slanted pro-corporate agenda that results in a large percentage of the population simply not understanding things like the benefits of a social safety net (a trade of tax money to keep society civil and safer), how we incarcerate a much larger percentage of our population than other countries, that Wikileaks has released a very small percentage of the diplomatic cables and those cables that have been released have been redacted to minimize the chance of putting people in danger, that we spend as much on our military as the rest of the world combined, how military spending enriches only a very small number of powerful people, the level of corruption in Congress (both democrats and republicans), etc.

It is sad that better sources of news come from the Comedy Central channel (for example, The Daily Show), Al Jazeera, foreign services like German Der Spiegel, small companies like Salon.com, etc.

The USA is in rapid decline. Part of the decline is due to failing infrastructure, large scale corruption, massive military costs, and also a much less informed population. All of these means to decline are fueled by a tightly controlled dishonest news media.

Christmas came early: my Google TV arrived at 7pm tonight

2010-12-22T21:32:00.002-07:00

Setting it up took 45 minutes because it immediately downloaded a new version of the OS when I set up the wireless Internet connection. I use Directv and it synced up with my DVR and Sony TV with no problems. The keyboard is nice, with a trackpad and mouse button top right corner. It is much nicer using the keyboard rather than the remote for Directv guide and DVR control.

I have experimented with writing Android cellphone apps with the SDK and now I want to try some HTML 5 apps for both the Android cellphones and Google TV. Fun! I had a consulting job two years ago writing some Java blu-ray example apps and the development environment for that was, frankly, painful. Both Android and Google TV seem much more developer-friendly.

You personalize Google TV by logging into your Google/GMail/Apps account. I had to enter my account information multiple times for Youtube.com and Picasa photo albums. There are some wrinklles to iron out but the platform has a lot of promise.

We watched part of a Netflix movie on demand and it looked good. I have a Hulu Plus account but when I went to Hulu I got a pop up dialog warning me that Hulu is building out for Plus access on Google TV. I am looking forward to that because Hulu Plus is a pretty good service. Along with all of the Directv and Netflix material, it seems like there is an infinite amount of good stuff to watch.

A few improvements I look forward to:

When looking at my Picasa photo albums it seemed like the Google TV did not cache-ahead very well. I would like a picture to stay on screen until the next one is ready.
When watching one of my wife's documentaries for a charity she helps, Youtube playback was pretty good because we uploaded a fairly lowres video to Youtube and it played back smoothly. I then tried one of my videos on Youtube that was fairly hires and there were too many pauses for buffering. Also, if I paused the video it did not seem to buffer ahead like on my laptop.

I am very excited with the Google TV platform and I thank Google for giving me a free Logitech unit. I am not always good at predicting which nacent technologies succeed in the marketplace but to me, at least, the Android platform and the Google TV built on very similar technology show a lot of promise.

Excellent product: RubyMine 3.0

2010-12-22T14:18:00.003-07:00

I bought RubyMine when it was first released and recently paid for upgrading to version 3.0. I switch between using TextMate (or GEdit on Linux) and RubyMine for both Ruby and for Rails development but since getting version 3.0 I spend almost all of my time using RubyMine.

Rails 3 support is very good and working with RVM is a nice new feature. I never used to use the Ruby debugger (at all!) but I have used it twice briefly in version 3.0. At least for Rails 3 development I still separately run both rails server and rails console outside of the IDE - a matter of personal preference.

I was using PyCharm (also by JetBrains) a lot for 2 weeks and the autocompletion hints and instant syntax warnings really helped me because my Python and Django skills are light-weight. I don't much need autocompletion hints and instant syntax warnings for Ruby and Rails development but it is unobtrusive and often useful.

The HTML, JavaScript, Erb, and Haml support seems better also. One feature that I don't use in daily development much but that I really like is View->Show Model Dependence Diagram that gives a great overview of your database model classes, their attributes, and their associations. Great for documentation and to serve as a reminder if I have not worked on a project for a while and need to get back into it. Quick navigation support to jump to definitions, etc. also saves me time.

EDIT: Since I wrote this blog I have started using the Model Dependence Diagram fairly often. It would be great to also have an option to show a UML class diagram.

My only real disappointment with version 3.0 is that it does not implement the tear-off editing tabs (i.e., tear-off to create a new and separate window) that the latest version of IntelliJ supports.

That said, using RubyMine is a comfortable experience and the IDE features stay out of my way until I need them.

Getting the Dojo 1.5 rich text editor dijit.Editor working nicely with Rails 3

2010-12-22T09:54:00.001-07:00

I could not find much on the web and it took me a little time to get everything working (especially saving edited content) so I'll share some code, hopefully to save you some time. In apps/views/layouts/application.html.erb I added some boilerplate (with non-editor stuff not shown) to be included with each rendered page:

<head>
  <link rel="stylesheet" type="text/css"
        href="http://ajax.googleapis.com/ajax/libs/dojo/1.5/dijit/themes/tundra/tundra.css">
  <link rel="stylesheet" type="text/css"
    href= "http://ajax.googleapis.com/ajax/libs/dojo/1.5/dojox/editor/plugins/resources/css/Save.css">
  <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/dojo/1.5/dojo/dojo.xd.js"
          djConfig="parseOnLoad: true">
  </script>
  <script type="text/javascript">
      dojo.require("dijit.layout.BorderContainer");
      dojo.require("dijit.layout.ContentPane");
      dojo.require("dijit.Editor");
      dojo.require("dojox.editor.plugins.Save");
  </script>
</head>

Then in my view I create an inline rich editor using:

<div dojoType="dijit.Editor" id="editor1"
     extraPlugins="[{name: 'save', url: 'task'}]">
  <%= raw(@task_current.content) %>
</div>

It is very important to use the raw method because Rails 3 automatically makes safe HTML encoded strings for any result returned by <%= ... %>. Also notice the url value of 'task': this sends the content data to the index method of the task_controller. This method checks to see if an incoming request is an XHR POST and if so gets the raw content:

if request.xhr?
   if session[:task_id]
     task = Task.find(session[:task_id])
     task.content=request.raw_post
     task.save!
   end
end

For brevity I did not show any error handling.

Enjoy!

Reading "The Rails 3 Way"

2010-12-21T13:48:00.001-07:00

Obie had his publisher send me a review copy of his book Rails 3 Way that was just printed. It is a very good reference for Rails 3, and it really is a reference book meant to be accessed for specific problems. That said, I am reading it straight through because although I have a lot of Rails development experience, I would like to understand Rails at a deeper level.

While Rails itself is "opinionated" Obie's book is even more so: he bases his business on Rails and the book reflects the way things are done in his company. The book uses Haml and RSpec exclusively. I still mostly use Test::Unit and Erb, and the book will probably not change that (but it might!).

I found Paolo Perrotta's "Metaprogramming Ruby" to be a great resource for getting more into the low level details of Ruby and I expect that "The Rails 3 Way" will serve the same purpose for Rails.

Using cloud services when services like Delicio.us and Wave get cancelled

2010-12-17T16:20:00.000-07:00

You can still access my public delicio.us bookmarks if you want them. I will miss the service, as I miss Google Wave.

I tried several cloud based to-do and getting things done style web sites but ended up writing my own my-foc.us web app but left the door open to other people by releasing the app as open source. I used ran the online kbsdocs.com word processor for a few years but made sure to provide data export options.

People and companies who provide free services have no real obligation to continue the services forever but they have a responsibility to give users a good exit strategy. I exported my delicio.us bookmarks and I suggest you do the same.

Suggestions for Python SDK and AppEngine

2010-12-08T10:30:00.001-07:00

Partly because I don't often code in Python, I have been feeling some pain using the Python AppEngine SDK. I thought that I would pass on a few things that have made the process easier for me:

Use an IDE. I have been using PyCharm but other people have mentioned liking Eclipse with the Python and AppEngine plugins. Live hints when I get a local variable name wrong, suggestions to automatically import one of my own modules, and autocompletion have really helped. If I were more familiar with Python and the Python AppEngine SDK then this would not be as beneficial.
Unit tests: I have been using gaeunit.py to support unit testing and that has been helpful. I get all of the model code and controller helper functions working and tested before doing the UI
I find that the default Python SDK template engine (from Django) to be adequate and to be fairly easy to use.
I have found Mark C. Chu-Carroll's book Code in the Cloud, Programming Google AppEngine to be a useful tutorial.

Platforms and Infrastructure as service

2010-12-08T08:36:00.001-07:00

Good news for the people at Heroku (Salesforce just paid 200+ million for the company). This is definitely a kick up for platforms as a service. I have long considered Heroku to be the best platform as a service offering because it is so developer friendly - very different from AppEngine which I would label as "scalability friendly."

I had to recently make a decision between developing for Heroku or AppEngine for my new business ideas (consultingwith.me and consultingwith.us). Heroku and Ruby on Rails, along with all of the useful plugins and auxiliary services offered by Heroku, make the most agile web application development and deployment story right now.

In contrast, developing for AppEngine is a pain in many ways. That said, I think that the new AppEngine SDK and services are a solid improvement and since I hope for more than small scale success the relatively inexpensive hosting costs and automatic scalability of AppEngine won me over (at least for these projects).

Infrastructure as a service is also becoming more accepted. My largest consulting customer is considering hosted infrastructure options for very large high performance datastores. As easy as it is to set up a cluster using MongoDB, HBase, or BigCouch, etc. there are so often good business and technical reasons to pay specialists companies to do it for you.

AppEngine SDK 1.4 release is likely a game changer

2010-12-02T16:19:00.002-07:00

I use AppEngine to host my own projects but not for my customers (everyone who has hired me for a few years has wanted Amazon AWS deployments - no exceptions).

I think that the story for using AppEngine is definitely better with the 1.4 release. A charge of $9/month for keeping 3 compute instances always spun up for an application seems like a modest cost to workaround slow loading request times. (I have written twice about this: 1 and 2.)

Ten minute CPU limits for background processes is a nice increase over the old 30 second limits.

I have recently been using the Python AppEngine SDK for my latest side project (something I need for my own use and I am also planning on offering it to other consultants for a small fee).

Platform as a service: I have had the opportunity to help many individuals and companies in 14 years of running my own consulting business and I feel like I have a fairly clear understanding of the opportunity costs of manually managing servers. Small teams especially get hit hard when too much available engineering time is spent on deployments and administration of servers. I expect managed deployment platforms like AppEngine and Heroku will be a strong growth market. Managing servers and deployments may be fun but often does not make business sense.

Python is not such a bad language

2010-11-30T10:20:00.001-07:00

I have used Python off and on for about 10 years and have never particularly liked it. This is funny since Ruby is my favorite programming language and Ruby and Python are very similar both in features and in the types of applications that they are best used for. For me the largest shortcoming of Python is the lack of blocks.

That said, I have two small side projects (consultingwith.me and consultingwith.us) that I wanted to host on AppEngine. I have a fair amount of experience with the Java AppEngine SDK but my gut instinct was to go with the Python SDK using the default webapp library. After spending about 5 hours writing code, I find that Python is fairly comfortable. 7/9/2011 edit: I enjoyed getting a basic system working with Python (a great learning experience) but ended up writing a feature complete version in Rails and using that for several months. In the last week or so I have re-implemented it in Java + GWT for the Appengine.

Must-have tool for understanding your web site and blog: Google Analytics

2010-11-28T10:42:00.003-07:00

I use Google Analytics as a feedback mechanism for what people actually read on my blog and to know which pages on my web site people read (which I equate to which pages people find most useful). As an example, 50% of visitors go directly to my Open Content web page so this gives me a strong incentive to write more free web books and post the PDFs.

As a consultant I interact with many developers and companies and it always surprises me when people don't bother to measure performance, in this case understanding what content people find interesting and/or useful.

Speaking of measuring performance:

Another must have tool if you do deployments: get a free 1-server account at cloudkick.com. (Disclosure: if you end up eventually signing up for a non-free multiple server account then I get a small perk.) If one of your web applications starts to slow down it can save a lot of time being able to tell at a glance what the I/O, CPU, network, etc. activity has been in the last few days.

Wonderful book: "Land of Lisp" - Conrad Barski is a great author and communicator

2010-11-23T16:30:00.003-07:00

I have been enjoying Conrad Barski's web based tutorials for years and I recently received his new book Land of Lisp: Learn to Program in Lisp, One Game at a Time! in the mail.

I have been writing Lisp code since the late 1970s and in ancient times I wrote two Lisp books for Springer Verlag. To be honest, I just bought the book for enjoyment but I find myself getting a new perspective and learning more about Common Lisp.

Recommended!

Distributed NoSQL datastores: Cassandra and Cloudant's BigCouch

2010-11-21T13:53:00.005-07:00

In my work for customers in recent years almost everything that I do uses PostgreSQL (sometimes PostGIS) and/or MongoDB. (I write a lot about the Semantic Web but so far no one has paid me to work on a project with an RDF data store like Sesame or AllegroGraph.)

While I think PostgreSQL and MongoDB are great, their replication stories have not been great. MongoDB's master/slave and replica pairs work OK, and replica sets (MongoDB 1.6 and above) look to be a big improvement (it only takes a few minutes to try the MongoDB 1.6.x replica set tutorial example; follow the instructions.) I have not tried replica sets yet in a production environment but I am looking forward to it! I find MongoDB to be extremely developer friendly with convenient client libraires in Clojure and Ruby (I don't like dealing with JSON data and hashes in Java).

PostgreSQL 9 replication is easier to set up and administer than Slony but I have not had to use it in production. The replication supports master/slave hot stand-by but it is not a distributed data store with no master process.

Cassandra was designed to be distributed with no specific master server. I am almost done reading "Cassandra, The Definitive Guide." I have been enjoying experimenting with Cassandra a lot recently on both my laptop and transient EC2 instances. One negative about Cassandra for me is that I find the Java and Clojure client libraries to be inconvenient to use compared with their MongoDB counterparts. The Ruby client library is very developer friendly!

CouchDB was the first NoSQL datastore that I used (except for RDF data stores) but I have never found it to be as convenient for my work as MongoDB. That may change thanks to Cloudant's BigCouch open-source version of CouchDB that has built in clustering capability. It is extremely easy to set up a test system following the directions on BigCouch github. On both my MacBook and also using EC2s, it only took about 15 minutes to set up a cluster. If I had to set up a fault tolerant distributed data store on small (or even micro) EC2 instances cluster, BigCouch would be a strong candidate because of the relatively low RSIZE memory footprint compared to Cassandra (for empty systems, about 15MB for CouchDB and 150MB for Cassandra - but expect these memory requirements to increase rapidly with large data stores). BTW, check out the people working at Cloudant: interesting that so many physicists work there. Cloudant bases their CouchDB hosting business on BigCouch.

Good to be a programmer. Or: custom solutions are sometimes better

2010-11-21T11:04:00.002-07:00

I signed up for a free Evernote account a year ago last April but never really used it. After reading an article in the New York Times this morning about sharing data across computers and handheld devices I decided to install the Evernote app on my Android cellphone. I also installed the web clipper Evernote Chrome browser extension. After setting up notebooks for major interests (writing tools, writing ideas, different technology areas), I spent some time using Evernote (OS X app, browser interface, and Droid app). Really nice.

I thought of using Evernote for my to-do tasks but my own custom web app my-foc.us works better for me. This reminded me how great it is to be a programmer and make things just the way you want. Now, I must admit that my-foc.us is very simple (it has just the functionality that I want and nothing else). This project is open source and took me perhaps 6 hours to write and deploy on Heroku. It is really a great feeling to make something just for yourself (although other people are welcome to use it).

Some background for another "just for me" web app:

About four years ago I almost died very quickly from two large pulmonary embolisms. Fortunately I am almost fully recovered (e.g., I can do 5 or 6 hour hikes, but not the 8 or 9 hour hikes I used to do) but I need to monitor my intake of vitamin K because it interferes with the small doses of blood thinner I take daily. Now, vitamin K is absolutely vital to have in your diet (one example: keeps calcium deposits out of arteries and heart valves and promotes bone growth and general health). So, four years ago I created a simple web app cookingspace.com that contains many of the recipes that I use and some convenience foods that I like. What is different about cookingspace.com is that it uses the USDA National Nutrient Database for Standard Reference, Release 20 to show the nutrients in food I eat. At first, I referred to the nutrients per serving for individual recipes and for complete meal plans but after a while, I trained myself to remember the approximate amounts of vitamin K and about 20 other nutrients in food. You can see displays for random recipes by clicking this link a few times. This web app turned out to be a large project, taking me about 40 hours to write because the recommended USDA database had a lot of errors so I ended up using their much larger "research" database and then trimming out things I didn't need. Anyway, for me this was time well spent.

Anyway, it is great to be a programmer :-)

Is it better to spend time learning new programming languages or study languages and tools you already use?

2010-11-20T08:40:00.003-07:00

I have been thinking about a discusion on Hacker News yesterday about which new programming language people want (or need) to learn. While I definitely enjoy learning new programming languages by writing small applications, I think that I personally get a better productivity boost by reviewing languages and tools I already use.

In the last year I have mostly used Ruby and Clojure for my work. In the last few months I have read two books on Clojure and one on Ruby. Sort of: the more you know, the more you can learn and more deeply understand something.

Recently I reviewed the commands and read another tutorial for the screen utility that I have been using almost everyday for three years. Well worth the time.

I do a lot of work on remote servers and emacs is not always installed so I have also used vi for years. This morning, I saw a reference to learning vim by using vimtutor and spend 20 minutes working through the complete tutor program (learn by doing: you edit the tutorial as part of the tutorial). Time very well spent.

New AppEngine 1.4.0 features a game changer for JVM languages

2010-11-19T10:03:00.002-07:00

As I wrote last April you can use Objectify in Java apps to reduce loading request times (avoiding JDO overhead) but running JRuby and Clojure applications on AppEngine has been hindered by still long loading times. The new feature forthcoming to AppEngine (SDK available, but server side support is not in place yet) of allowing paid for apps to keep three compute instances always running will open up the AppEngine platform for other languages and frameworks. On the development side I have had good JRuby and Sinatra experiences with the AppEngine SDK but have never wanted to deploy anything other than experiments - that will change now.

Publishing to the Kindle, Android, iPhone, and iPad

2010-11-18T16:10:00.002-07:00

I just bought a Kindle and I like it more than I thought that I would. The screen is easy to read and it is much lighter than an iPad.

I have a plan to add support for Kindle, Android, iPhone, and iPad file formats in my writing/publishing pipeline. Currently, my "development system" for writing is Latex with some custom Ruby scripts and a Makefile for each writing project. From Latex source files, I can currently generate:

Lulu print books
PDFs for laptop viewing
HTML pages with automatically inserted Google Adsense ads
HTML pages

I have been generating fairly good revenue from publishing my Open Content books and selling print copies (some generous people also pay for the PDF instead of using the free downloads). Because I am earning more money from my Open Content writing, I plan on devoting much more time to writing next year. I plan on always making the large format PDFs free for download, and selling print books and versions for hand-held devices.

Benchmarks: memory use can be as important as runtime for some applications

2010-11-18T06:03:00.001-07:00

I often look to the benchmark game results. While I am pleased that Clojure is now included, I find the memory use (at least for these benchmark programs) to be disturbing (e.g., Clojure vs. Ruby 1.9 and Clojure vs. Java). When deploying to small servers, VPSs, small EC2 instances, etc. memory use can be critical.

Clozure Common Lisp 1.6 has been released

2010-11-13T16:19:00.001-07:00

Although I have used SBCL for more consulting work than Clozure CL, I have started using Clozure for more of my own projects. One thing I prefer is that standalone applications built with Clozure are about 10 megabytes smaller than apps built with SBCL. Release notes

Thanks to Tom Munnecke for portrait photographs

2010-11-13T15:59:00.003-07:00

My friend Tom Munnecke recently took some casual portrait pictures of me, and I am now using one as my Twitter picture. Sometime I will also replace some of the pictures on main web site.

Contact your Congressional Representatives and Senators and ask them to have their families publicly go through the TSA grope experience

2010-11-12T16:53:00.003-07:00

Use this link to contact them.

Also, how about President Obama letting his wife and daughters get groped by TSA? Our Senators and Representatives let their wives and kids get groped?

Hate to break it to you, but the government officials who we elect and who get paid off by corporate lobbyists don't live in the same world we do.

People get the government that they deserve. Are you going to do anything about this? An email is good, but also consider taking the extra time to insist on talking to your elected officials.

BTW, I don't think that TSA has every caught a terrorist, or done any good at all. (Not like the FBI and other intelligence services who do worthwhile work.)

I am softening my position on Oracle's stewardship of Java

2010-11-12T08:25:00.003-07:00

I have to admit that I got a little carried away with my criticism, fueled largely by my agreeing in substance with the position of the Apache Foundation. However, Apple's announcement and IBM's earlier announcements about working with Oracle and OpenJDK have caused me to soften my position.

To be fair to Oracle, my complaints about lack of tolerance for Apache Harmony and the FSF Gnu Java implementation apply equally to the now defunct Sun Microsystems.

The GPL license for OpenJDK is not all that comforting given the patents wrapped up in Java implementations: only passing the Java Compatibility Kit tests gets a patent waiver. Still, for Java developers, I think that all is right in the world, on a practical level.

My nephew died this morning: rest in peace Anthony

2010-11-10T12:33:00.002-07:00

My nephew Anthony was hit by a car last night and died early this morning.
The following picture shows Anthony, my sister in law Anita, and me in front of my house in Sedona. Anthony was about 17 years old in this picture.

Here is another picture of Anthony and my Dad when they were visiting Sedona. Anthony was 20 years old in this picture.

Anthony loved talking about politics, his family, and music.

Question: what would be the legal ramifications of forking Java, but not calling it Java?

2010-11-09T18:06:00.000-07:00

I understand that Oracle owns the trademark and there are patent issues. Still, an alternative pure open source platform named wombat or whatever might end up being necessary.

I understand that Oracle would like to monetize Java but there is that "killing the goose that laid the golden egg" metaphor becoming real life :-)

Ideas?

Downloading all of your data from Facebook

2010-11-08T10:29:00.000-07:00

Nice: I just tried this: go to your Facebook Account menu on the upper right corner of the home web page and select Account Settings. Near the bottom of the page, click the learn more link for Download Your Information. Follow the instructions and you will get a ZIP file with your home page, friends list, all messages, wall, and photographs.

Free PDF for the Common Lisp edition of my "Practical Semantic Web and Linked Data Applications" book

2010-11-05T14:40:00.004-07:00

I have been working on this book on and off for two years and finally finished it recently while on a long vacation. You can get a free PDF on my Open Content web page or if you want to encourage me to write more material on niche topics, you can buy a print copy.

Although this book covers a small market, I believe that the combination of Common Lisp and the AllegroGraph RDF data store is a great combination for developing knowledge intensive software.

Wow: I think QuickLisp will change my Common Lisp development setup

2010-11-01T16:08:00.001-07:00

I have been experimenting with the beta for QuickLisp.org and it looks very good. I never disliked the ASDF package management system, but QuickLisp looks to be easier to use and with a few hundred common Common Lisp already in the repository, getting most dependencies installed is quick and easy. Good job Zach!

For many years I have been using a brute force approach to package management: in every project I work on, I have a utils directory where I un-tar the source code to all dependencies, and their dependencies, etc. I then locally

(push "utils/PACKAGE_DIR/" asdf:*central-registry*)

for all dependencies. This has always worked really well for me because I can ZIP up a project directory, rsync it to a server, and I am good to go with all dependencies. However, it is a pain to keep multiple copies of libraries in multiple projects. For reasons I don't even remember anymore, I never liked to use a global ASDF cache.

It will be a long weekend day project, but if I don't run into any serious problems with QuickLisp, I am going to clean up all of my Common Lisp projects to not use local copies of dependencies. I'll wait until the Common Lisp edition of my Semantic Web book is done, to avoid delaying that effort.

Convergence on HTML5 for user facing software development

2010-10-29T12:55:00.000-07:00

News this morning on ZDNet about Microsoft moving away from Silverlight towards using HTML5 is more good news. I expect (hope!) to see more iPhone and Android development to be based on HTML5 rather than native apps.

As we see improvements in device independence (i.e., getting to read email, tweet, watch video, video teleconference, etc. on our smart phones, netbooks, laptops, Google TV, Wii, XBox, etc.) I hope to see real improvements in development platforms for universal applications.

Java support on the Mac

2010-10-26T03:06:00.000-07:00

Long term, this is a big issue for me: I spend a lot of my time developing Java, Clojure, and Scala code using IntelliJ so I need support for desktop Java apps like IntelliJ. For now, my customer (CompassLabs) is a Clojure shop (at least the work I do) and Emacs+swank+Clojure is a sweet combination, so Clojure development will not really be impacted because this does not require Swing desktop app support. But I am pretty much addicted to IntelliJ for Java and Scala coding.

I would be interested in an official statement from JetBrains how they will handle this issue, long term.

An easy solution is to just use my i5 Windows+Ubuntu laptop in the future.

review of new Hulu Plus service

2010-10-25T13:36:00.001-07:00

I got a beta preview invite today, and so far it looks really good: HD, no opening commercials (at least for what I chose to watch during my lunch break), and additional material. Hulu Plus is $10/month so I may not use it permanently but I will give it a try for several months. Just my opinion, but Hulu Plus at a reduced rate of $5 or $6/month would make it more compelling since for $15/month I get 2 Netflix blu-ray movie checkouts (at the same time) and their very good streaming service.

My wife and I are also Directv customers and although we really like the service, it is expensive. Directv must be worried about competition from direct Internet viewing options because they called up several nights ago and offered a small reduction in my monthly rate and a free 6 month promotion for all their channels (except for pay per view). Directv is a good user experience.

If you are like me, you use movies and a few TV shows for something to do while either bored or too tired at the end of the day to do much else. Unless viewing is a social/group activity with side conversation, I feel a little guilty sitting around enjoying media that other people created when I could be getting exercise, working on my current book project, or making some walking around money by consulting.

A little off topic, but I re-watched late last night two of the GoogleTV presentation videos from the last Google I/O and I am excited about more seamless integration of all media that I have purchased (or is free with advertising) and all devices that I use (MacBook Pro, Droid, HD TV with blu-ray and Wii). BTW, I did a small Blu-ray Java app for a customer two years ago that was really just a demo: it put a user's Twitter stream in the upper right corner while playing from a blu-ray disk.

* edit: apparently there are no commercials: I finished watching a recent episode of the office and the screen went black for a second in places where there were commercial breaks.

Future society: an optimum strategy for flourishing

2010-10-23T10:05:00.001-07:00

I just watched this interview with Tim Wu. He nails it re: the tendency of information industries to move from open to closed systems. I just pre-ordered his new book: The Master Switch: The Rise and Fall of Information Empires There is little doubt in my mind how our society is going to evolve in an era of consolidated corporate power and ubiquitous information systems.

Although I don't subscribe to the idea that history repeats itself, I do believe that history does inform us about human nature and how powerful people fight to consolidate power and influence. This tendency is firmly stapled into our DNA.

There will almost certainly be strife between what used to be the middle class and financial and political elites. I read yesterday that one of the international rating agencies predicted a loss of "social cohesion" in the USA. Right now, there are large strikes in France over raising the retirement age from 60 to 62. It is interesting that here in the USA, the waning middle class seems silent in comparison. In my country the consolidation of power by control of news by just a few corporations has been brilliant in execution, but not in the public good: we see people swayed by corporate controlled news to back political agendas that are very much against their own interests.

I have always had a strong interest in strategy games like Go (I wrote the first commercially available Go playing program) and Chess (I wrote the simple Chess program Apple shipped with very early Apple IIs). The point is: when playing strategy games, you play the current board position. Now, life is not a perfect knowledge game like Chess and Go but as individuals it still makes sense to "play the position" with what knowledge and resources we have.

What knowledge do we have? I would argue very little if you rely on USA filtered and biased news sources. You can improve your knowledge a little by reading multiple international news sources. I also rely on what people I trust write in their blogs and books.

What resources do we have? As individuals our resources are specialized job skills, education, family and friends, real property (hopefully not debt encumbered), owning your own business, cash assets, and social and professional networking. Negative resources are things like excessive personal debt.

In playing Chess and Go, it is almost always useful to understand what moves your opponent wants you to make, and avoid these moves! In our lives, I believe that it is useful to try understanding what the financial and political elites want the masses of non-elites to do. In recent history, using advertising and media control, a vision of extravagant lifestyles has been pushed at people and this has been carefully combined with control of cheap borrowing to induce people into taking on excessive private debt: optimum elite strategy for increasing control, wealth, and influence. The obvious strategy for non-elites is to avoid debt, avoid getting sucked into a materialistic lifestyle, and to invest personal resources in acquiring specialized job skills and making education a life-long process.

Social network based authentication done right

2010-10-21T20:41:00.001-07:00

Although there are valid privacy concerns using social networks like Facebook, and to a lessor degree Twitter (because almost all tweets are intended to be public), for most of us the value proposition of shared user identity between web sites provides advantages of consistent login/authentication without multiple accounts and also enabling web sites to potentially show you more things that are interesting based on your online behavior.

I have been an occasional Hunch.com user since they went beta. Today I was looking at their login/authentication scheme that uses either Twitter or Facebook authentication. I tried using both Twitter and Facebook for authentication and liked that Hunch recognized that my previous Hunch account, my Facebook account, and my Twitter account belonged to the same person and immediately offered to merge the accounts.

Giving a site like Hunch the ability to access some Twitter and Facebook data on users opens up even more opportunities for using machine learning to further personalize the user experience for Hunch.com.

Most of my work in the last year has been using machine learning to process some form of user data (although I also get a lot of plain Java/Ruby/Lisp development work) and understanding the behind the scenes infrastructure and techniques as well as user experience benefits has softened my previous stance against corporations collecting too much information. A few days ago when my wife and I were in Hong Kong, we were talking about the ubiquitous advertising (the night time skyline is a beautiful canvas for advertisers) and customized advertisement displays in the movie Minority Report. Custom ads that are interesting and useful are a good thing. Just be sure to understand what information you share and how it is used.

For more of a learning experience I have started doing some up-front research for adding Facebook and Twitter authentication to my cookingspace.com web site (something I wrote for my own use, but it has users). I would like to have my own "full stack" environment for collecting user preferences and using machine learning for recommendations. For customers, I tend to touch only parts of their systems so work on cookingspace.com is motivated by my desire to understand more of the entire process rather than build a popular web portal (although that would be nice also!)

Nice: Clojure results now in Computer Language Benchmarks Game

2010-10-21T19:03:00.000-07:00

Examples:

Clojure vs. Ruby 1.9: median results 8 times faster

Clojure vs. Java 6 server: median results 4 times slower

Clojure vs. Python 3: median results 10 times faster

My travel journal notes for my Siberia, Japan, and China trip

2010-10-21T00:49:00.005-07:00

Here are my rough notes that I was emailing to my family and friends. As I edit them I will post a few of my best pictures here on my public Picasa web alblum - just look for recent photo albums with "2010" in the title. I did not make notes for the first week as we were going north west through the Aleutian Islands and into the Bering Sea.

Fun excursion: 1.25 hour drive Petropavlovsk to Indigenous village

We went way off the beaten track today, but had a lot of fun. Except for a lot of driving on very bad roads, most of the day was spent inside an Indian style lodge similar to what a large family would live in during the long winter. Entrance was a long very low tunnel. The lodge had a small hole in the ceiling directly above the fire pit.

Three women sang and danced inside the lodge, one spoke a little English and told stories and legends, etc. They also cooked us a meal that was pretty good.

I have some fantastic video of the singing and dancing (similar to Southwest Indians, but more complex, musically). I'll post a few of the video clips when I have a chance. Carol took video of me dancing, but that will be censored.

We are now steaming towards Japan

We have today and tomorrow at sea, then the next day we land in the port of Sendai. Countryside and temples, here we come!

As per the email I sent last night, Carol and I really enjoyed our "Indigenous people's village tour" yesterday. The roads were so very bad that they had to transfer us from a robust tour bus to an extremely robust mini-monster truck (huge wheels, 4-wheel drive) for the last few miles and even then getting to our destination was dicey. Carol just showed me some of the videos she took; using both of our videos, we should be able to put together a good presentation. We just used our digital cameras, but the sound and video quality is good. Carol just showed me a great video sequence from the mini-monster truck going down a very bad road with a large field of yellow wild flowers on our right. Right now she is editing a photo from our ship showing snow covered mountains/volcanoes surrounding the harbor we were in yesterday. Stay tuned for production-ready product :-)

Our tour guide yesterday was a little apologetic for the condition of their city, roads, etc. Definitely, not a lot of wealth to be seen (she said the ultra-rich people lived in guarded enclaves well outside of the city). Still, they have lots of natural resources and I think enough jobs - their life style may compare OK to non-rich people living in the USA in a few years.

We saw an amazing magic show last night, "Moscow Magic." The couple had been married for 30 years and their act together was very smooth and well executed. They have a 25 year old daughter who is a magician with the Moscow Circus.

Carol and I are hanging out in the library this morning at a stone table that is covered with semi-precious stones: reminds me of the inlaid semi-precious stones on the sides of the Taj Mahal. Carol is editing trip pictures and video on her laptop and I am getting ready to work on my current book project. I have only been spending about 90 minutes a day writing, but have been, I think, very productive. I'll say it again: if I were wealthy I would live and work on cruise ships about half the time.

bird rescue (?)

We are passing the far northern Japanese islands this morning, arriving in Sendai early tomorrow morning. It is getting warmer: we are now about the same latitude as Portland Oregon, so we have come far south compared to being in the Bering Sea. I was out on deck very early this morning at the very top of the ship in a short sleeve shirt, so it really is warming up (probably 40 degrees at 6am).

Most days we go to both a history lecture on the areas we are visiting (lecturer is awesome) and also listen to the naturalist on board. Yesterday the naturalist was telling us that large birds land on the ship and can not take off so it is a kindness to pick them up and toss them over the railing. Carol and I went outside on the Promenade deck last night at about 9pm and what do we see but a large bird (the wings looked like a frigate bird but the color was different) trying to take off. Now, these birds have these tiny stubby little legs and he was not getting any lift-off speed. I picked him up (heavy bird, lots of fat on his chest) and carried him to the railing but I could not bring myself to toss him overboard into the darkness. You guessed it: I handed him to Carol and she did the bird tossing. It was so dark that we have no idea if we saved him or tossed him to a watery death. I hope the former option. In any case, it was a thrill to hold such a large bird; as some of you know, I am more than a little fond of birds, and miss my ornery little parrot. We saw four more birds of the same species later in the evening and they too looked really scared to be stuck on deck. I picked up one of them, but Carol and I decided to not toss him, so I set him down near three other birds that were nearby. Early this morning, two of them were still on deck, hiding behind some lounge chairs. edit:It turns out these were water birds and I would have done them all a favor by spending 5 minutes tossing all of them back into the ocean. (This is what the naturalist told me a few days later when I talked with him at lunch time.)

We had a formal dinner last night and the ship's chief medical officer sat with us - interesting to hear about his life. edit: He joined us on two shore tours so we ended up getting to know him fairly well.

I am very excited about landing in Japan tomorrow. We have reservations on four awesome sounding shore excursions (one for each port in Japan we will be visiting). Carol has been taking many good pictures, and eventually we will organize an online photo and video album (our target date for this is February 2011 :-)

Sendai: first port in Japan

Yesterday was very special. I have always wanted to visit Japan and the tour yesterday was the perfect start.

We first went to a tourist area that is one of the most popular destinations for Japanese: a few hundred beautiful stony little islands with a few pine trees on each. Our view point was a three hundred year old tea house on a bluff.

Nearby, there was an ancient forest with a long formal entry, and the main gate framing the same islands. On one side were about a dozen meditation caves and on another an early Shogun's residence. All beautiful and interesting.

Our last stop was a Shinto Shrine on a small mountain, also looking out over the sea in the distance. Our guide was wonderful, with her spiel, giving me more information when I asked questions, and teaching us all the proper way to pray at Shinto Shrines (yes, I will probably convert :-)

A bit of a disappointment today: we were going to go on a self guided tour through caves and catacombs where Buddhist monks have over the centuries made carvings and pictures. Each person was to be given a candle, and they would send us in by ourselves. Anyway, the cruise tour lady on the ship warned people during a talk of claustrophobia problems, and so many people bailed on the tour that they cancelled it. I said how much I liked the your yesterday, so they are sending us on something similar today, but it will be in Yokohama (we should dock in about 1 hour).

Yokohama: 2nd port in Japan

Yokohama is very different than Sendai; Yokohama is a huge port, but well laid out and very clean. Even the water in the bay is clean. So far, all of Japan has been spotless and tidy.

We took a tour today to the ancient city of Kamakura, made capital of Japan by the Shogun in 1192. Our first stop was at a giant bronze Buddha: taller than you would believe. I went inside it also - sort of interesting.

We also spent time at a Shinto Shrine and walked through town. I bought two t-shirts, some Japanese cookies our guide recommended, and a hand painted bamboo bookmark. Carol and I also walked down several small alleys/walkways in the residential areas: odd because in the middle of the city, it was absolutely quiet.

After the tour, Carol and I had a late lunch on the ship, then spent the afternoon and early evening in a park near the ship that was a popular hangout spot for locals. We leave at 10:30pm for the port of Shimizu.

After dinner, the same music group that was performing in the morning on the pier was on board entertaining. They had a calligraphy artist doing "real time" huge drawings, interpreting the music. Then, with a translator, the calligraphy artist explained the nuances of what she was doing.

Shimizu: port 3 in Japan

We started the day going to one of the Japanese national treasures + places: the Nihondaira Park and Kunozan Toshogu Shrine. The shrine was in a very remote spot: had to take a 2000 foot cable car to get there. It was built for the Shogun Tokugawa Ieyasu who was an early and especially powerful leader. I can't really describe the area and our pictures don't do it justice. There were beautiful buildings, very old cedar groves, the monument where his body has lain for 800 years, and hawks flying overhead. The breeze made the trees seem like a live animal around the shrine. Sitting there quietly has been the absolutely best time so far on this trip. The park also had an excellent museum of ancient Samarai armor and weapons.

After some more sight seeing, we had a fancy traditional Japanese lunch in a nice hotel - really strange stuff, but I ate everything. Watson traditions to uphold, and all of that.

In the afternoon, we saw something fairly much awesome: the small museum that has most of the existing work of the wood block artist Utagawa Hiroshige. He worked around 1830 to 1840 so this is not really old like most of what we have seen. You have seen his famous "Ocean Wave" wood block print, and the collection of his entire work was amazing.

It is worth noting that on all tours we have taken in Japan, the places we visited have had almost all Japanese tourists and a relatively small number of foreign tourists. The Japanese are very big on visiting their own places of interest. Easy to understand because of the beauty and general historic interest.

Kobe + Kyoto: 4th port in Japan

Carol and I are sad to be leaving Japan in an hour. We did a 10 hour tour today, leaving our ship in the port of Kobe and driving to the ancient capital city of Kyoto. I would like to come back to Japan and rent an apartment for a few months sometime when I am not consulting and just writing. Wonderful place, full of polite and very happy and contented looking people.

The first stop in Kyoto was the part time residence of the Shogun Yoshimitsu that was built in 1397. It was very dangerous being Shogun: enemies were often trying to assassinate the current Shogun. Yoshimitsu's residence had squeaky "nightingale" floors that creaked as we walked on them. A little noisy, but no sneaking up on anyone without a lot of effort. There were a series of rooms that were used for audiences with the Shogun, and the security setup was intense looking.

Calvin: our guide today was an old woman who seemed very knowledgeable about Ninjas. She said that they would hire themselves out to the highest bidder, either to protect the Shogun or assassinate him. She said that they were like the American CIA (not entirely sure what she meant by that). In Yoshimitsu's residence, he paid local Ninjas to masquerade as gardeners to spy on people approaching the residence and generally collect intelligence.

We then went to the Golden Pavilion, a Shinto park + shrine with lots of water, reflections, etc. Lunch was a buffet, and yes, I did eat some Kobe beef. Kyoto was an interesting place, and we spent the afternoon going to several more historic sites.

Two day tour to Beijing

Just a short note because I am tired. Carol and I just got back from a 35 hour trip to Beijing. We left the ship first thing yesterday morning for a 2 1/2 hour drive to Beijing. We had lunch and then headed for the Great Wall. One note: the weather was beautiful and sunny: blue skies except for a few scattered clouds. The Great Wall was most interesting because it gives some indication of the power of the Chinese emperors. We got extremely tired climbing the stairs on one segment of the Great Wall and you will see why when I send out some pictures.

After the Great Wall we moved on to the Ming Tombs in the hills outside of Beijing. This was an interesting lesson in how Feng Shui is done on a very large scale. We then went at dusk on what is called the 'sacred walk' where dead emperors were carried in a procession. Interesting because of large animal sculptures along the way.

We then had a painfully long drive through Beijing traffic to what is reputed to be the best duck restaurant in Beijing. I have no real knowledge if this claim is true but there were lots of pictures of famous people and politicians eating there. The food was tasty.

On the way to our hotel, the InterContinental, we passed the lit up Olympic sites for the swimming structure and the "Bird's Nest" - looked really cool from the bus, but when Carol and I walked back there at about 11pm, the lights were out. It was great taking a long nighttime walk in Beijing anyway.

Our hotel room was unbelievable, probably the nicest room I have ever had. Unfortunately, we had to meet for breakfast at 6:30am (full on Chinese food buffet with all kinds of strange stuff to eat) and get an early start to Tian'anmen Square: odd but interesting experience - people were in a 3 or 4 hour line to see Mao's tomb. We walked around and gawked. Don't underestimate the patriotic fervor in China: we saw it everywhere. Our guide talked about every person pulling together to help the economy. Single minded and to the point. Very scary, us being economic competitors. Also, I didn't mention before how beautifully splendid Beijing itself is. I know that there are poor areas in China, but Beijing was awesome and people just had a happy look about themselves living or working there.

We then went to the Forbidden City. I can not describe how huge this is because we covered such a tiny part of it. The open squares are immense and the imperial buildings, well, are big and imperial. Our guide told me that if a baby was born there and spent each night in a different room, then he would be 30 years old by the time that he slept in all major rooms. Our guide took us to the "Emperor's lovenest" (as he called it) area. The area where he lived with a dozen rooms nearby for his favorite concubines was small compared to the rest of the Forbidden City.

After lunch we went to the Temple of Heaven which was nice after the Forbidden City because the temple area only covered (about) 25 acres and did not numb the mind trying to contemplate the place. Everyone on the tour was happy enough to start the 2 1/2 hour drive back to the ship, after the second long day in a row.

We now have two days at sea before spending two days in Shanghai - looking forward to some down time and working on my current book project.

"Steaming" towards Shanghai

You might remember my previous email about "bird tossing." Sort of a followup: I was laying by the pool this morning doing some writing and I felt a little thump on my stomach. I looked down and there was a tiny canary-type bird that had landed on me. We looked at each other and it jumped down on the lounge chair right beside me. After a pause of a few seconds, it ran up my right pant leg, took a sharp right at my laptop, ran across my stomach, and flew away. Yesterday a similar bird flew right up beside Carol and I with a large moth in its mouth and last night we watched two of them bathing in a pool of fresh water (from the decks being washed down). Cheerful little birds.

Judy Garland's daughter Lorna Luft was the performer for last night. I saw her early yesterday morning while walking on deck and we talked briefly, but I didn't know then who she was. Her show last night was good and we just spent an hour listening to her answering people's questions in the lounge. Frank Sinatra was her godfather and a major influence in her life, and her best friend is Barry Manilow - she had some good stories.

We head up the Yangtze River at 1:30am tomorrow morning and from what I hear the captain will earn his salary. (Actually, he is charming, and has already earned his salary.) We fly home in 6 days and I am looking forward to getting back to real life.

Yahoo! Shanghai!

Patti: Happy Birthday!! I hope you had a great time in Hawaii.

We arrived in Shanghai early yesterday morning after about 3 hours of river navigation. We took the "Zhujiajiao Water Town" excursion that lasted all day yesterday. The water town was like a little Venice: lots of small bridges and very small sanpan boats to get around. Our ride on a little sanpan only lasted about 10 minutes but it was fun. Lots of narrow pedestrian only streets and we both bought a lot of inexpensive stuff in small shops. Later we had a mediocre lunch and visited a silk factory. I have some interesting video clips I took of the process of getting the silk threads off of the cocoons. Carol bought a beautiful green silk jacket; she is thrilled with it.

Last night we had dinner outside on the tail end of the ship and enjoyed the spectacular Shanghai skyline for several hours. Really nice. We had rack of lamb and a couple of chefs were cooking it up fresh nearby so it did not get cold coming all the up from the kitchen area. Since I didn't eat much lunch I also had a second entree of fresh mussels and two nice appetizer plates. As expected, food on the ship is fantastic, especially lots of high quality shasimi and sushi every lunch.

After the super-tidy futuristic feel of Beijing, I feel that we are now seeing more of the 'real China', except that Shanghai historically has such a strong western influence, and you see that in the 100 year old western style buildings. This morning we went into the city on our own and walked through the areas that we only drove through yesterday: the Bundt financial districts, lots, lots, lots of shopping, etc. I had a small cold yesterday, feeling better today, but I wanted to take it easy so after a while I gave Carol all of my Yuan money and I caught a bus back to the area where the ship is.

We now have two sea days starting tomorrow and then a long day in Hong Kong (two tours, totaling 10.5 hours, then we get up early the next morning and fly home). I am really looking forward to getting home - our normal life is pretty sweet, and this trip seems somewhat like a fantasy.

Hong Kong, and a Typhoon

Today has been a long but fun day. We woke up and looked out the window as we were entering Hong Kong bay. Beautiful city.

We took an 8 hour 'best of Hong Kong' tour and saw just about all the famous places. We started in the Bird Market where there were a zillion birds for sale and generally a lot of Chinese bird lovers hanging around. We then went through a food market that was super interesting, if a bit bloody, gory, etc. We then took a funicular to Victoria Peak and ate in a very good restaurant; Carol made sure that I got the seat at our large table right up next to a floor to ceiling window, looking down over Hong Kong - felt like I was dining on the edge of a cliff. We then went to the other side of the the Island to Aberdeen and took a sampan ride during which I shot a lot of video. We then went to Stanley Market, and then back to the ship.

We had a hurried but tasty dinner and then headed out for a 2+ hour nighttime harbor cruise that was a lot of fun also.

We just put our large bags outside our stateroom for the porters, and we are now going to get some sleep since we get up early to fly home tomorrow.

re: the Typhoon: the ship's captain has said that the cruise might have to be diverted, changing the itinerary. Our new ship-board friends who are on the full 69 day cruise (we just did the first 25 day segment) did not seem to mind a possible change of ports visited. Hopefully we will have no weather problems flying out tomorrow morning.

I am back home after a 4 week vacation: blog comments are now enabled

2010-10-20T23:46:00.000-07:00

I did not want to deal with blog comment SPAM while travelling so I had temporarily turned off comments. I will post my travel log when I get a chance.

Big productivity gain: not having an Internet connection in the middle of the Pacific Ocean

2010-09-26T16:05:00.002-07:00

Carol and I are on a long cruise, and because of the high cost of Internet connectivity, I am only getting on the web for about 5 minutes every other day.

I have been spending about 2 hours each day working on the Lisp edition of my Semantic Web book, and I must say that my productivity seems a lot better when I am not distracted with an Internet connection.

So far, we have been very good about not over eating on this trip - enjoying the food but eating small portions. Except for some complementary Champaign the first night we have avoided alcohol, making it easier to not over-eat!

We will be onboard for 25 days so we don't feel pressured to engage in all activities that might be fun. So far, we have been enjoying a series of onboard lectures, the movie theater, and lots of walking on deck.

I am going to be on travel for 4 weeks: temporarily turning off blog comments

2010-09-21T08:17:00.001-07:00

Carol and I are leaving on a long trip. Unfortunately, I get SPAM comments on my blog which are easy enough to remove, but I will be off of the Internet for long periods of time. I'll turn comments back on when I get home.

I have my laptop setup to work on the Common Lisp edition of my Semantic Web book so that will probably be available in final form in about 6 weeks.

Rich client web apps: playing with SproutCore, jQuery, and HTML5

2010-09-15T09:51:00.001-07:00

In the last 14 years I have worked on two very different types of tasks: AI and textmining, and on (mostly server side) web applications. Putting aside the AI stuff (not the topic for today), I know that I need to make a transition to developing rich clients applications. This is not such an easy transition for me because I feel much more comfortable with server side development using Java, Ruby on Rails, Sinatra, Merb, etc. On the client side, I just use simple Javascript for AJAX support, HTML and CSS.

As background learning activities I have been working through Bear Bibeault's and Yehuda Katz's jQuery in Action and Mark Pilgrim's HTML5 books. Good learning material.

When I read that Yehuda Katz is leaving Engine Yard to work on the SproutCore framework I took another good look at SproutCore last night, worked through parts of the tutorial with Ruby + Sinatra, and Clojure + Compojure server backends. I find Javascript development to be awkward, but OK. I need to spend some time getting setup using IntelliJ on both jQuery and SproutCore learning projects. If anyone has any development environment suggestions, I am listening.

MongoDB "good enough practices"

2010-09-14T07:18:00.003-07:00

I have been using MongoDB for about a year for customer jobs and my own work and I have a few practices that are worth sharing:

I use two levels of backup and vary the details according to how important or replaceable the data is: I like to perform rolling backups to S3 periodically. This is easy enough to do using cron, putting something like this in crontab:

5 16 * * 2 (cd /mnt/temp; rm -f -r *.dump*; /usr/local/mongodb/bin/mongodump -o myproject_tuesday.dump > /mnt/temp/mongodump.log; /usr/bin/zip -9 -r myproject_tuesday.dump.zip myproject_tuesday.dump > /mnt/temp/zip.log; /usr/bin/s3cmd put myproject_tuesday.dump.zip s3://mymongodbbackups)

The other level of backup is to always run at least one master and one read-only slave. By design, the preferred method for robustness is replicating mongod processes on multiple physical services. Choose master/slave or replica set installations, but don't run just a single mongod.

I often need to do a lot of read operations for analytics or simply serving up processed data. Always read from a read-only slave unless the small consistency hit (it takes a very short amount of time to replicate master writes to slaves) is not tolerable for your application. For applications that need to read and write, just either keep two connections open or use a MongoDB ORM like Mongoid that supports multiple read and write mongods.

Another thing I try to do is to place applications that need to perform high volume reads on the same server that runs a MongoDB slave; this eliminates network bandwidth issues for high volume "mostly read" applications.