Sunday, December 30, 2007

2008 predictions

Technology

  • The use of Java will continue to migrate to the edges of application space: use on cellphones and small devices and large scale server side system. Except for developer tools like Eclipse, NetBeans, and IntelliJ Java on the desktop will grow slowly (with a few exceptions like LimeWire)
  • The trend of using scripting languages like Ruby and Python for small and medium size systems will continue.
  • Most innovative software development will continue to be done as web based applications with browser and small device clients.
  • New web based applications will continue the upwards trend to outsourcing infrastructure to Amazon (EC2, S3, SimpleDB), Google (GData, applications), etc.
  • Pressure to reduce IT costs will drive most organizations even more towards open source.
  • There will be more progress in implementing the Semantic Web as more organizations adopt open standards for sharing structured data and start to see the value of opening up their own data to outside users.
  • The shift towards hi-tech development in Asia rather than in the USA and Europe will continue.
  • Low energy networks and computing devices will become more prevalent and important in the face of large scale energy shortages (long term trend).
  • Security will continue to be a problem as trojans/viruses enable organized crime to effect large scale theft.
  • Social network web applications will continue to evolve into open development platforms.
  • Microsoft will continue to have problems with Vista, Windows 2000 and XP use will continue strongly, and the company will have increasing difficulties reconciling the push for open standards with their "lock in" business models.
  • A "real AI" will not be developed in 2008 :-)

Social and Economic

  • Random web surfing will decrease as savvy users learn to more quickly find relevant information linked from RSS feeds that they trust. I believe that more people will realize how much time they spend, with small value to them, reading general sites like Slashdot, Digg, and Reddit. People will focus more on what helps them in their work and specific interests.
  • In the USA and other developed countries, the middle class will adjust to greatly shrinking disposable income by looking more towards inexpensive and lower energy use activities like social interactions with family and friends, exercise, reading, and watching TV (yuck!) and movies.
  • The growing middle classes in developing countries will spend their increasing disposable income on higher protein diets, consumer goods, cars, and travel.

Political

  • Developed nations will continue the current trend of moving towards corporate controlled governments - this will continue to be made more possible by massive consolidation of news media by just a few very large corporations that can then strongly affect democratic elections and thus control elected officials.
  • In general, people in most countries will be more concerned with safety and their economies than with civil rights and freedom.
  • Long term trend: the USA and the UK will become less important as world powers as economic problems undercut their military ability. Russia will continue the trend of decreasing civil rights and freedom, but the population at large is unlikely to care in the face of improved economy and standard of living. Long term, China will have severe structural problems due to pollution; they will have some problems with energy and other resources, but their current plan of long term energy treaties will give them an advantage over other industrialized nations that need to import huge amounts of energy.
  • There is a long term trend towards meritocracies strongly favoring people with high value skills and/or large amounts of capital.

Friday, December 28, 2007

Very cool: backing up Google Office documents and outsourcing infrastructure

If you use Google Office Applications, check out gdatacopier. One of my only complaints about using Google Documents has been having to manually export and backup my online documents - now, problem solved.

Although there is some sensitive information that I would not store in Google Documents, for most information having it online and available is a deal maker for me - I also like being able to share specific documents for group writing.

There is a question whether storage and software services should be "outsourced". For most of my business and research interests, I would answer that question with a "yes". For me the issue is in reducing labor costs. By using leased managed servers, Amazon EC2/SimpleDB/S3, Google Data, DabbleDB, etc. does precisely this: it allows systems to be run and maintained much more inexpensively by outsourcing infrastructure.

One firm requirement for outsourcing infrastructure is a plan for recovering from the loss of an outsourced service. Such a plan must include periodic data backups under your own control as well as a strategy for duplicating services - not always easy to do.

Friday, December 14, 2007

Consulting and working at home: when does the work day end?

Slashdot has a discussion this morning on this topic, and it is very relevant to my lifestyle. Most of my customers, since they use me in a telecommuting mode, also use knowledge workers in Vietnam, India, Russia, South America, and Eastern Europe. Because of the timezone differences it can be very productive to do some work right before I go to bed and if, for example, I wake up very early in the morning.

I balance this "always on" mode by taking frequent long breaks during the day for hiking with friends, seeing movies with my wife, playing with my pet Meyers Parrot, turning off my computer and reading, cooking, working on CookingSpace.com, etc.

Besides taking these frequent long breaks, I have three tricks that I use to feel good about my "always on" mode:
  • Feel good about providing service: always equate the loss of personal time with feeling good about helping someone
  • Bill for all time spent, even small increments: if I am interrupted by the need to handle something ASAP then I also bill for time spent getting back into my previous task
  • Limit total work time: unless there is a drop-dead emergency, I limit the number of hours total that I work each day and week. I just started doing this in the last year, and it makes consulting even more fun
It all really comes down to having a good sense for balancing one's own personality and needs with working lifestyles. Some of my friends simply prefer working hard in an office and then having their evenings and weekends free.

Saturday, December 08, 2007

Hiding source code: so bad!

For my own research and development I have almost totally committed to Ruby (yes, I decided to ignore that little "poor run time performance" problem). For many years, one of the attractions of C++, Java (with byte code obfustication), compiled Common Lisp, etc. was that intellectual property could be protected.

If you write and distribute code in a scripting language like Ruby, clearly everyone gets the source code.

I am over this desire to hide source code. First, I think that in most cases (for me) using open source licenses makes the best business sense. Second, in cases where proprietary software does make sense, it seems to me that customers who buy products get shortchanged if they do not also receive source code for what they pay for.

I am updating CookingSpace.com to Rails 2.0.x

I am home from vacation (just drove in from San Diego, arriving in Sedona today - we have snow :-). After doing some quick tasks for my customers I turned straight away to updating my local development version of CookingSpace.com to Rails 2.0.x

I had to do a few simple things. First, I had to edit environment.rb, adding a definition for config.action_controller.session. Actually, Rails will warn you about this and tell you exactly how to do this. I also had to run two commands because Rails has been made more modular:
script/plugin install auto_complete
script/plugin install acts_as_list
I have been making my web apps (Rails, Java, and Common Lisp) more "RESTfull" in the last few years and the Rails 2.0.x REST focus was enough for me to decide to update sooner rather than later.

Thursday, December 06, 2007

Back online after my vacation

Carol and I, after a year of health problems, celebrated our return to good health with a 15 day cruise from San Diego to Hawaii and back (pictures).

Vacations are a funny thing for me: usually I love my work, research, and studies but after a vacation it always takes me a week or two to enjoy normal life as usual. Oh well. I enjoy cruising a lot: great relaxation and I think less expensive than other forms of vacations where you constantly rack up separate transportation, lodging, and food costs. Some tricks for maximizing the enjoyment of a cruise: occasionally skip meals to avoid always feeling full, hang out at many different places aboard a ship to both experience different environments and meet more people, and when ashore look for opportunities to use public transportation to great sites to visit and just use packaged tours as needed. We had a great time and look forward to visits in Sedona from some of our new friends who we met on the cruise.

Friday, November 16, 2007

Google's view of UI development: do it in Java

I have enjoyed building a few AJAX prototypes using GWT and now I am just now starting to look at Google's Android SDK - also built on Java technology.

I have done a couple of consulting jobs using the Java ME platform, and I would guess that eventually I will do something (professionally) with Android - what is not to like about an open cellphone platform built on Linux and programmable with Java :-)

Friday, November 02, 2007

Quality Open Source: Squeak 3.9 final and new Seaside 2.8

I don't use Smalltalk professionally but I wanted to try the new Seaside 2.8 release. I did a fresh Squeak 3.9 final installation on my MacBook running Leopard. Installing Seaside was as simple as starting the SqueakMap package loader, doing an upgrade to all installed packages, then do a Seaside install answering yes to all optional package offerings. Everything 'just works' and has a very polished feel to it - important when I am giving up an evening or two to play with something very cool.

One very important aspect of this "feeling of quality" is an active community that maintains a central package web site - good examples are SqueakMap (Squeak), Gems (Ruby), and CPAN (yuck, Perl :-) Having a central package repository is more than just a time saver: time spent dealing with too many installation and configuration options disrupts what I would call problem solving mode or thinking. I used to take great pleasure in staying on top of most J2EE technologies, but I got over that :-)

The way that Seaside gets around stateless HTTP issues is very cool but I suspect that the future of rich client applications may be something like Adobe AIR. For my work, I am happy enough for now just using Rails and AJAX. I have also experimented a lot with Dojo and Lisp on the back end and a little with OpenLazlo. The question to ask is how much "rich client" support do you need for a web portal - applications like GMail show how much can be done with Javascript/HTML/CSS.

Software development: smaller is better

Small is better at all levels of software development: Small projects have lower failure rates than larger projects. Smaller classes are easier to understand and use. Small functions and methods are easier to write, understand, test, and reuse.

I think that the first point is the most interesting. It is difficult to get requirements right and reducing the size of projects by breaking them up reduces risks involved with poor understanding of requirements. To use web applications as an example, by implementing a minimal system that still provides useful functionality, we learn early on what are the most important requirements. We also keep team size small - more efficiency and less risk. We also get something useful to users developed faster.

I think that ego is the negative force that fights against the goodness of small and focused projects. At the customer level, they often want all of their ideas implemented sooner rather than later instead of setting priorities. At the developer level, they may want to push technology choice over satisfying users' needs in a timely and inexpensive way.

While frameworks certainly save development time they also add their own kind of complexity in the form of steep learning curves, selection, and maintenance to upgrade to newer versions.

In many cases I think that a better way to simplify systems is to extend programming metaphors to support application development a higher level. Two very good examples of this are Google's GFS distributed file system and their map reduce libraries. These tools change they way developers think because they do not need to worry so much about scaling their applications. Another good example is Ruby's ActiveRecord: it changes the way I think about persistence and Rails migrations are now my favorite "modeling language" for specifying class definitions even though I am really defining database schema. I argue that the combined notation of using Rails migrations and adding has_many, one_of, etc. calls in ActiveRecord classes captures the useful information in UML class diagrams in a very terse, easy to write, easy to understand, and easy to maintain executable notation. (BTW, the only type of UML diagram that I still use frequently is sequence diagrams.)

Wednesday, October 31, 2007

OS X Leopard

I have 4 computers (1 laptop and 3 desktops) packed away - I now do everything on one fully loaded MacBook: for medical reasons(*), I either need to have my legs elevated or be walking - sitting at a desk for more than a few minutes is not a good idea. Fortunately, I can still hike for up to 3 or 4 hours at a time :-) Anyway, having everything in one laptop works best for me.

Anyway, since the work and research part of my life is bound up with one nicely configured laptop, I was hesitant to change anything, but after burning 4 backup DVD-Rs and cloning my laptop's disk drive to an external disk, I did a fresh install last night and restored my user files from the cloned disk.

It took a few hours to get all the tools working that I use (and I am probably still not there yet), but the time was well spent. I think the Time Machine idea is awesome - you really need to try it to appreciate it. I plan on alternating between two external disks, keeping one at a friends house for off site backup.

I am still deciding if the Finder changes are all that great. The icons don't seem as distinctive as the old ones. I do like the document automatic preview options.

The Java JVM support (version 5) performs well, and the NetBeans IDE (used only for Ruby and Ruby Rails development) seems more responsive. Ruby is installed as a shared library framework so if you should run many simultaneous Ruby processes, some real memory would be saved (like Apple does for multiple JVMs running). I was going to do my own Ruby install and decided against it.

Leopard seems very much faster than Tiger but that may be partially because I wiped the disk and did a clean install (the ultimate disk defragmentaton :-)


---

(*) A bit of advice: last year I was working really long hours sitting at my desk without getting up often enough - that lead to deep vein thrombosis and 2 pulmonary embolisms. Set a timer to remind yourself to get up every 30 minutes or so!

Thursday, October 25, 2007

ZFS: disuptive technology?

Many years ago I thought that CORBA was a disruptive technology that would change the world for distributed systems. I sort-of guessed wrong - in retrospect I explain my poor prediction on the relatively high cost of early CORBA software. CORBA became 'free' too late.

I also think that ZFS (a new "file system" - yet so much more than a file system) also could be a disruptive technology that changes the world for storing and retrieving information, supporting: multiple levels of data protection, a transactional model for committing changes, on the fly expandability, etc. As open source software, wide and free availability is not a problem with ZFS, or at least should not be when ZFS is available for OpenBSD, FreeBSD, Linux, OS X, etc.

That said, I think that ZFS is more of a platform than a 'file system'. That is, I think that developers with specific data storage and retrieval problems (e.g., large scale graphs, databases, permanent record archival, etc.) will look to ZFS as a possible bottom data layer instead of file systems on the low end or relational databases on the high end.

As ZFS storage pools grow to span multiple (file) servers, we can start thinking in terms of using distributed data queries and accumulating results (a little like using map reduce operations on GFS). ZFS seems like such a wide open platform that I expect that we will all be surprised at future novel applications layered on top of ZFS.

Tuesday, October 23, 2007

NetBeans 6.0 beta 2 is released

I have been using the Ruby and Ruby on Rails specific NetBeans 6.0 beta 1 for about 2 hours a day for Rails development. Beta 2 (just released today) has good improvements for code completion, popup documentation, and general RHTML editing. Good stuff!

I have blogged about this before: I am tired of using so many different programming languages (Common Lisp, Java, Ruby, Python, C++, Prolog, etc.) on customer projects. I am starting up a new (non-consulting) business so I am reducing the amount of consulting work I will accept. I would like to just do Ruby and Ruby on Rails development - turning down other work unless it is very interesting.

Algorithm development and solving people's problems - that is the fun stuff, not keeping current with a half dozen programming languages and dozens of frameworks.

Friday, October 19, 2007

Ruby debuggers

Although I do not actively use Smalltalk, I check the Smalltalk blog feeds several times a week because Smalltalkers are often into interesting things. James has been talking about Smalltalk debuggers a lot recently.

While the state of Ruby debuggers is no where near those found in Squeak or VisualWorks Smalltalks, debugging controllers in Ruby on Rails web applications is fairly slick: in development mode code and RHTML templates are dynamically reloaded (CSS style sheets are not, but that is OK). The NetBeans Ruby environment with the "fast debugger" installed is pretty good. That said, I feel that Ruby has some big advantages over Smalltalk for some types of applications, so it is really a personal decision based on what development and deployment requirements are.

Using a debugger to step through how Rails controllers are called is also pretty interesting, if not educational.

Netbeans IDE 6.0 Beta 1: (J)Ruby and Rails support is looking good; mixed experience with IntelliJ 7.0 + Ruby plugin

I did have an installation problem: I have every possible version of Java installed on my MacBook so I had to manually edit
/Applications/NetBeans/NetBeans 6.0 Beta 1.app/Contents/MacOS/netbeans
to explicitly set jdkhome to my 1.5.0 directory. Hopefully most people will not have that problem.

I found it most convenient to have local JRuby and Ruby installations in my home directory and use those so the NetBeans GEM tools had write access.

I still find using TextMate to be more convenient for development, but code completion and browsing features may win me over in the long run.

I have a 30 day IntelliJ 7 trial and installed the first Ruby plugin for version 7. I ran into errors running the generate script and could not create a Rails project from an existing Rails application directory. I would bet that these problems will be fixed quickly, so I will give version 7 another try before my trial license expires. (I am trying to decide whether to upgrade to version 7 and Ruby support will affect my decision).

Tuesday, October 16, 2007

"100% Rails all the way, baby!"

A few years ago my wife and I each spent about a week working on a healthy recipes web portal. I have been promising myself that some day I would set aside a large block of time and make a "social network" style cooking web portal, and recently decided to just do it. I am finishing off my existing large consulting tasks, and for about 6 months plan on spending most of my time developing CookingSpace. Currently I just have a Rails place-holder at CookingSpace.us and since Carol and I are planning a lot of travel in the next few months, I will probably not have a full prototype in place until early next year.

I had been planning on using Java, JSPs, struts 2, and Hibernate Search - but decided to go with Ruby on Rails. I made this decision partly because I have been doing a lot more Ruby work than Java work in the last 18 months, and partly because I have learned some new Rails architecture tricks, and wanted to use them.

In any case, Carol and I want to create a healthy cooking web portal that will make a real positive difference in people's health and in their enjoyment of cooking and eating. I believe that the concept of seeing nutritional ingredients for recipes coupled with the ability to change ingredients on the fly and see changes in fats, sodium, vitamins, etc. will help people who either have dietary problems or simply want to eat healthier. If we end up making some money on CookingSpace that would be nice also :-)

Friday, October 12, 2007

It is important to check web sites and web applications for usability by readers with disabilities

I have blogged about this before but it is an important issue: some web sites use some technologies that might not work well with screen readers for people with impaired vision or are blind. I just ran through my own web sites to make sure that they are usable with the text-only lynx web browser and they worked OK - if anyone needing assistive technologies has problems with my sites then please let me know.

Wow: what a difference caching makes!

I just spent a day adding MemCached to a customer's web application. Incredible what a difference it makes! The great thing about MemCached is that many client libraries are available and caching works with handling user sessions, web page fragments, some web service calls, etc. Depending a lot on the application, MemCached can take a load off of database servers, back end web services, etc. I was also pleasantly surprised by how simple it was to implement (after carefully analyzing the web application itself - gaining an understanding of what should be cached is the tricky part). Good stuff!

I don't think that I will ever deploy a web application again without analyzing it for MemCached use.

Tuesday, October 09, 2007

Joost: Internet TV done the right way

I have blogged about Joost a few times before, but the system now works even better than before: better looking video, very few pauses in video stream, and more interesting material.

One thing about Joost: the commercials are short and unobtrusive. I actually wish that they would include more commercials to ensure that as a company they stay in business. Good stuff!

I think that ABC also does Internet video very well (on my MacBook, they is a one time Java plugin installation). However, I like the more "Indy" style material on Joost. I also prefer Joost's technical approach of using a "rich client": not everything has to happen in a web browser. That said, I would hope that all major networks follow ABC's lead.

Tuesday, October 02, 2007

More fun with JRuby: using the PowerLoom reasoning system

I just wrote a complete example, including all required source code and a Makefile on my AI blog.

JRuby continues to please me: fantastic to be able to switch from C-Ruby to JRuby when I need to use large existing Java libraries.

Good PDF book: "Ferret" by David Balmain

Slashdot had a discussion yesterday on indexing and searching documents - a subject of particular interest to me. After reading the comments, I revisited the indexing and search tools that I have used over the years: Ferret (a Lucene clone) is my favorite library for several reasons: it uses the Lucene API (which I have used for years), it is very fast, and coding in Ruby is faster for me than Java (Lucene) or Common Lisp (Montezuma). I bought Dave's book on Ferret yesterday, and it is a good reference with lots of good examples.

I have a "semi alive" open source project (KBSPortal) written in Java, uses Lucene and my own clustering and analysis libraries. I have been mulling over switching to Ruby and Ruby on Rails because it would be easier developing the web interface, I like to code in Ruby more than Java, and there are some very nice text analysis Ruby Gems that I could use in place of some of my own Java analysis code (in the spirit of building on other people's libraries, when possible, to take advantage of shared work). I get consulting work setting up custom document management systems and I would like to have a complete stack that could be set up and customized in less than a day.

Sunday, September 30, 2007

Plain text and other simple document types

I limit work to about half time so it is especially important for me to make my work processes as efficient as possible (I also spend about 10 hours a week researching and playing with new technology, but that is generally an unpaid for pleasure, driven by my own interests, not my customers').

Almost everything I deal with is plain text: Ruby, Lisp, Java code, detailed work log notes, design notes, etc. For information that I need to record just for myself, I get no benefit from styled text formatting. For other people I work with or want to communicate with, Latex does the job of generating great looking documents.

The Pier content management and Wiki system looks very interesting to me because it supports Latex output. That said, I am finding Google Docs to be a good alternative to Latex for very small documents. Anything to save time. Pier is open source and deployment using Squeak looks simple. (I have written here before about easy deployment of Squeak web applications.)

Wednesday, September 26, 2007

I am loving Amazon's new MP3 musc store

I am finding very low prices on some music when buying entire albums. A few caveats:
  • For Mac users: buy music using Safari and not Firefox or Camino
  • Downloading is a one shot deal: Amazon encourages you to immediately make an archival copy of purchased music
The album down loader is nicely integrated with iTunes - I hope that Apple does not try to break this convenient integration.

In addition to saving money, this is a huge time saver for me: when I buy music from the iTunes store I always make an archival copy by burning an audio CD, then using Tunes to rip to MP3s, then add the MP3s to my permanent backups - this was a time consuming process.

Tuesday, September 25, 2007

"Why would you use Ruby when you could use Smalltalk???"

Patrick Logan poses the reasonable question "Why would you use Ruby when you could use Smalltalk???" Looking at things from a development rather than a deployment perspective, I would have a difficult time arguing with Patrick. Squeak provides a good open source platform, many libraries and third party projects, easy headless deployment of Seaside web applications, etc. Commercial Cincom VisualWorks adds the ability to create small executables, great web services libraries, good support for commercial licensing, etc.

Ruby on the other hand also has a lot going for it: many instant install gem libraries, easy installation of Ruby itself, Rails is a nice web app framework, and the best of all: Ruby is already installed on most servers (and is easy to install otherwise) and deployment of Ruby applications and utility scripts is simple. Also JRuby is a very interesting technology for those of us who need to work in "Java land".

One other big win, Ruby-wise, for me is that (like Python) is such a readable and concise language: meaningful algorithms/programs are often just one or two pages of code - great for digesting other peoples code or when you need to look at old work.

Monday, September 24, 2007

Iranian President Mahmoud Ahmadinejad is not crazy, but his speech irritated me today

The problem that I have with Iran is not that I view them as a threat to my country (USA) or the Middle East: it is that Iran is the second most despised government in the world (behind #1 Israel, and slightly ahead of #3 North Korea in just about every international opinion poll).

I would expect a more conciliatory approach from a leader of a widely despised government (internal human rights, suppression of free speech, an economy that is so bad that their educated population leaves in great numbers, etc.) I don't think that any rational person really views Iran as a real security threat (although Iran is a convenient political issue in my country), but I have real problems with their internal (to Iran) behavior. Re: nuclear power technology in Iran: as long as they continue to be a member in the IAEA and allow inspections, that is good enough for me, but I understand the arguments of other people who don't feel comfortable with this. Unfortunately, incompetent foreign policy by the Bush administration has certainly given Iran more influence in their part of the world, but that is something that we now have to learn to live with - no taking back mistaken actions and policies, and "doubling down" on the bad bet of attacking Iraq, by attacking Iran, would be a horrible mistake (a long shot neoconservative wager that is a true nightmare scenario for the whole world).

I would like to make a more general point:

The world has become a much safer place over the last several decades, based on fewer civilian deaths and a decreasing general level of violence in the world. Also, compared to the constant threat of nuclear annihilation during the cold war, things are definitely looking good right now: something we should be grateful for.

One reason the world has been progressing into a safer place is globalization and interlocking economies. If it were up to me, looking out for the welfare on my own country (USA) but also caring about all people on this planet, I would have more trade, talk, student exchange programs, tourism, etc. with all countries in the world (OK, I don't really don't want to go on a vacation to North Korea, but you get the general idea). The better the flow of ideas, dialog, tourism, and education abroad programs, the better off the world is, and for that reason I would like to commend Columbia University for inviting Ahmadinejad to speak and answer questions today. Ahmadinejad did not make a good impression on me, but I was still glad to hear him. You can bet money that some politicians will try to make political mileage out of this, but they will be acting out of self interest, not for the general good.

Sunday, September 23, 2007

Interesting article on graph/lattice theory leads me to a good looking library

After reading Mark Chu-Carroll's extremely interesting article on using lattices for representing information, I started a hunt for good graph representation and analysis libraries. I looked for Lisp, Ruby, and Java libraries, and found a great looking library written in Ruby (gratr). One caveat: I spent 30 minutes enjoying reading through the library source code, but otherwise I have just experimented with the tests and examples. Thanks to Shawn Patrick Garbett, Luke Kanies and Horst Duchene.

I often find interesting/useful software this way: I get excited reading a good technical paper or blog, and then go and look for relevant software tools.

Saturday, September 22, 2007

Glassfish v2: Update Center and port 4848 Web Admin are cool, but...

Although I have been paid to work on both the Enhydra Enterprise Java application server and on the JBoss based Jaffa framework, I must admit that Tomcat has always been my favorite platform: lean, and add just what I need.

As a result of my 'build up just what I need' preferences, I just got around to experimenting with Glassfish. The Update Center is great, and as more instant install frameworks and applications become available, this will be a time saver. The web admin tool is refined - no complaints there.

I do have one complaint about the 200 MB resident memory footprint. A lot of what I do involves deploying to low cost servers, often semi-managed VPS systems. Smaller memory use is cheaper, but for most large server deployments, an extra 100 MB makes no difference.

I thought that it was very cool that one of the available instant install components is JRuby with the most excellent Goldspike. I have written before in this blog about the ease of running Rails web apps with Goldspike and JBoss. With Glassfish and the Update Center, it is even easier. Cool stuff.

Programming languages: advantages of both specialization and being a generalist

In the 1970s, I very much enjoyed working through Jean Sammet's classic book "PROGRAMMING LANGUAGES: History and Fundamentals". For one thing, it was my first real exposure to Lisp, and it was fun learning many new languages at once. At one point I had over a dozen programming languages on my resume (due to needing to use some one-off languages for military hardware, and several different assembler languages).

Generalization is good because the more experience with different languages, libraries, frameworks, and development styles that you have, then the easier it is to choose a good technology to solve new problems. I would argue that broad experience is at least a little better than narrow but deep experience.

Unfortunately there is another side to this issue: whenever I see really great design and code, it almost always seems to be written by someone who deeply specializes in one, or perhaps two, programming languages.

In the last few years, due to customer requirements, I have had to work in Java, Common Lisp, Ruby, and Python (and a little work with Prolog, and even less fortunately in C++). I am now in my mid-50s, and semi-retired (I try to limit myself to working no more than 25 hours per week). I would like to specialize in just one or two languages and not have the overhead of staying current with a half dozen programming languages.

The language that I enjoy the most is Ruby, and with the maturation of JRuby, it is great to be able to use the same language for scripting and general software development (using Matz's C based Ruby) and when I need to use existing Java libraries and frameworks (JRuby). While I really enjoy Lisp and Smalltalk (never professionally), my hope is that as JRuby continues to get faster and integration with the Java platform improves that the combination of (C or J)Ruby and Java will cover everything that I need for my work. BTW, the comment about future work on a "compiler that allows compiling a given Ruby class 1:1 to a Java class, producing a class you can construct, with specific signatures you can call from Java code" posted by Charles Nutter a few days ago to my JRuby blog article was very good news indeed.

Friday, September 21, 2007

Great combination: nginx, Mongrel, and Rails for secure HTTPS

I had to set up a customer's Rails application to run using SSL+HTTPS this morning. Based on a few positive web blog articles I decided to try Igor Sysoev's nginx web server. If you first build and install OpenSSL and the Perl regular expression library, then build nginx with
--with-http_ssl_module
--with-openssl=/OPENSSL_SOURCE_DIR
--with-pcre=/PCRE_SOURCE_DIR
you should be all set to use HTTPS. Clustering mongrel is also simple; I used this nginx.conf file from Brainspl.at as an example and I was set up and running very quickly. Good stuff!

Thursday, September 20, 2007

JRuby and Jython: a one way street?

This is just personal experience: JRuby and Jython are great when you want to use existing Java libraries and utilities from inside Ruby and Python programs. (Warning: Jython does not implement all of Python: many of my Python programs that use list comprehensions, generators, etc. simply do not work yet.)

However, I am skeptical of the utility of doing the reverse: using modules of Ruby or Python code inside large Java applications. I would like to hear about examples of this that don't just use Ruby or Python as embedded scripting languages.

BTW, since I usually prefer using the native C versions of Ruby and Python, I set up my bash environment to create aliases for running the JRuby and Jython command line tools:
## for Jython and JRuby without messing up PATH and native Ruby and Python:     

alias jython=/Users/mark/bin/jython2.2/jython
alias jruby=/Users/mark/bin/jruby-1.0/bin/jruby
alias jirb=/Users/mark/bin/jruby-1.0/bin/jirb
alias jgem=/Users/mark/bin/jruby-1.0/bin/gem
export JRUBY_HOME=/Users/mark/bin/jruby-1.0
I also edit my jirb and JRuby gem shell scripts to explicitly set the path to jruby. Now I can easily run the JRuby and Jython for just those occasions when I want to use Java class libraries.

Tuesday, September 18, 2007

1. Shameful behavior: ruining our grandchildren's lives because of our generation's greed 2. a story about my Grandmother

This is shameful behavior: the Federal Reserve (privately owned, not part of the US government) dropping interest rates to bail out stock market investors who have made bad investments, Bush's running up the deficit to line the pockets of his cronies, and now the US Congress talking about bailing out stupid homeowners who have acquired bad mortgages and spend stupidly beyond their means. Selfish, stupid behavior that will ruin the lives of so many in future generations.

Past generations were made of better stuff. My Dad sent me an email today with a story that his Mother told him about the death of her father because he was trying to save the financial future of his family:
My Mother told me this story when I was a small child. This was very personal for her, and I remember the emotion in her voice as she related it.

My Mother was born in 1886 on a farm in Illinois. Several photographs have survived of this time. She, with her parents and two smaller brothers, lived in a large, attractive frame house. Photos show a horse-drawn plow working a corn field. I remind you that they would not have had automobiles, telephones, radio, or electricity and central heating in the home. A trip into town meant a ride in the one horse carriage.

During the growing season the work day for a farmer and (older) sons began at daybreak and ended at sunset. The wife and daughters canned food for the long winter months, sewed clothing, and prepared meals for the men. When winter storms came, families could be isolated for extended periods.

My story begins when my Mother was thirteen or fourteen years old (I do not remember the exact date.) It was a very harsh winter, with bitterly cold storms following in a seemingly endless procession. The roads were blocked with huge snow drifts. Without central heat, the family huddled near the cherry red kitchen stove, stoked with coal. Hot water bottles warmed the feet.

Isolated from the rest of the world, each day all anxiously hoped for an end of the storm. Grandfather would put on his warmest clothing and venture out to tend the livestock. It was essential that the cattle and horses be fed and protected from freezing. When Grandfather succumbed to the flu, the family faced a crisis. With a high temperature he became too weak to go out and the family could only pray that the storm end. It did not end, and faced with financial ruin with the loss of the livestock, Grandfather knew that he must go out.

My mother and her smaller brothers waited apprehensively while Grandmother helped her husband up and to dress as warmly as possible. He went out into the howling blizzard-- to return triumphantly. The livestock were saved!

Their relief was shattered when that night his flu turned into pneumonia and Grandfather died. With no one to tend them, the livestock also perished. It was then necessary, come spring, for Grandmother to sell the farm and to move into town.

My maternal grandmother married another farmer and they both lived long lives. My grandmother was in her 90’s when she died.
My father's mother ended up getting a Masters Degree (rare for a woman in the early 1900s), married a minister, lead a financially poor, but happy life, and raised two sons who ended up teaching at Harvard and Berkeley.

Yes, previous generations were made of "sterner stuff" than weak willed people who care more for owning a "McMansion" and a new car every year than leaving a decent world for their descendants.

The New York Times 'gets it'

Oh, the pleasures in life: The New York Times now allows free access to almost all of their material. I used to spend lots of time reading the NYT online, and now that they have re-opened up their material (opinions, etc.) they are back on my very short Bookmarks menu (I use del.icio.us to manage most bookmarks: here are my personal del.icio.us bookmarks).

The NYT has clearly made the decision to go for more readers, and hope that advertising revenue makes up for subscription fees. In a small way, I do the same thing: after writing 14 published books, I put almost all of my writing effort into material that I simply give away for free - not totally altruistic: I earn my living as a consultant and having more people visit my site probably helps my bottom line about as much as book royalties.

While I understand that people working for companies like SAP or Microsoft might have a different viewpoint than mine because of where they see their personal revenue stream coming from, I believe that business and commerce works better when we all build on each other's open source projects, Creative Commons material, etc.

Chris Petrilli's second brain

I always enjoy Petrilli's blog entries (both technical and political), and his discussion today on augmenting our brain's memory capacity with indexed and cross referenced programming manuals, PDF books, etc. seems right on.

I have been more frequently buying PDF books: not as nice to read, but in the future it is great to have indexed and online. Because I mostly use a MacBook with open connections to my customers' servers for most of my work, my MacBook is the home for my "second brain". I have experimented with doing my own indexing and document clustering (naive O(N^2), hierarchical, and k-means), using Spotlight, and using Google Desktop for the Mac. I have settled on using Spotlight, but using the text, HTML, and PDF's in my "second brain" for my own research programming for IR, etc.

Monday, September 17, 2007

Classic "slip of the tongue"

My wife and I were just watching CNN and the host was apparently interviewing a security expert who was giving what I thought was an amazingly one-sided take on the private military company Blackwater's recent bad publicity in Iraq. After sounding like an advertisement for how great Blackwater is she had a "slip of the tongue": when asked what some of the services Blackwater provides in Iraq she said "we protect ...". Notice the use of the word "we". It looks to me like CNN may have been interviewing a Blackwater employee without divulging that bit of information to viewers.

Monday, September 10, 2007

IBM's support for OpenOffice.org is important

Microsoft Office is one of the principle reasons why so many businesses feel like they are locked into using Microsoft Windows. While I support the rights of Microsoft as a business to try to lock people, companies, and governments into using their proprietary data formats, and thus their software, it is also the rights of people, companies, and governments to fight back for their own best interests.

Proprietary data formats are a losing proposition for users. The lack of assistive technologies for people with disabilities has been a sticking point for wider adoption of OpenOffice.org - IBM's donation of the time of 35 engineers to work on assistive technologies is a large contribution to both open source and the infrastructure that so many of us build our businesses on.

Wednesday, August 29, 2007

Good book: "Programming Collective Intelligence"

This book is a great introduction to the techniques that I use almost daily in my own personal research and work for customers, and I can recommend it without reservation. The choice of Python for the examples is not optimum for me, but OK, especially because the techniques in the book for machine learning, categorization, clustering, filtering, optimization, support vector machines, etc. are mostly short and can be used as is or converted to whatever programming language that you need to use. The data used to present the book material is mostly from collaborative web sites. The book relies heavily on existing Python libraries and I like this approach since it mirrors rational software development practice: build custom code on top of existing libraries and software tools. Good book!

Saturday, August 25, 2007

Gambit-C Scheme 4.0 final released today

Gambit-C Scheme 4.0 is now available with pre-built binaries (or build from source code). Mac OS X and Windows versions are available with the OS X compiler generating fat universal, PowerPC, or Intel code.

When I use Scheme, I use MzScheme/DrScheme slightly more often than Gambit-C Scheme because of the existence of more 3rd party libraries, but Gambit-C Scheme compiles faster applications and the Snowfort library site now has many canned libraries for Scheme development. Good stuff!

Friday, August 10, 2007

Erlang 'mindshare'

I bought Joe Armstrong's new Erlang book as a beta PDF early this year and have been enjoying the material (the book is now in print). Erlang definitely has a lot of hacker mindshare but I have been unable to convince my customers to use it (so far). This may be a generalization, but those of us who love to program in Prolog are very likely to also enjoy working with Erlang. Erlang is certainly a great tool but I think it is unlikely to be very popular for two reasons: it does not provide instant gratification like Ruby on Rails and there is no large company promoting it (e.g., like Sun, IBM, etc. promote Java). That said, Erlang has a great open source community behind it and learning Erlang is very worthwhile if you occasionally need scalable applications. A comparison with Java is interesting: Java (especially with the new concurrency support in JDK 6) scales well on single servers with large numbers of cores while Erlang probably has the advantage when scaling to multiple servers.

Wednesday, August 01, 2007

Franz Allegro Common Lisp 8.1 has been released

I just received the 8.1 update from Franz yesterday afternoon, and there are several nice changes/updates, including:
  • New versions of read-line that do much less consing
  • Improved jLinker Java bridge
  • Improvements to Common Graphics (I don't have a license for this so I could not try it)
  • AllegroGraph Free Lisp Edition: limited to 50 million RDF triples - I had asked for this and I think that it is a good idea to let people experiment for free with small Semantic Web applications. The full 64 bit version of AllegroGraph is designed for very large stores of relatively static data (almost everything in an AllegroGraph triple store is indexed by default for fast lookups, so inserts and modifications are more expensive computationally).

Sunday, July 29, 2007

"Getting Stuff Done" GMail FireFox plugin

The GTDInbox FireFox plugin is very cool!

I use GMail, Google Calendar, and Google Documents in my work flow, but one thing that I was missing was a good way to schedule fine grain work tasks as active actions, as something to look at in the future, completed actions, etc. Basically, a light weight Getting Things Done(tm) system.

I have just started to use GTDInbox to set up work tasks and track them. Tracked tasks show up as short text only GMails sent to yourself and as you would expect you can search on them, list in order, etc. I like GTDInbox because it basically stays out of my way until I need it, and if I have GMail open I can review tasks quickly. I like organizational tools that don't themselves take up a lot of my time.

This is a good writeup on using Emacs to organize work flow - good advice and information but I now only use Emacs for a few ours a day (just Common Lisp development, with some Ruby). In the late 1980s and 1990s I "lived" in Emacs all day long. Now, there are other specialized tools like IntelliJ, TextMate, and TexShop that work better for for me.

Wednesday, July 25, 2007

Adoption of GPL v3: looking good

I was glad to see that the SugarCRM is adopting the GPL v3. Probable non-adoption of GPL v3 by the Linux kernel is an unfortunate side effect of the large number of contributors to the kernel. However, since so many of the software projects that make Linux useful will adopt GPL v3, I believe that GPL v3 will eventually be the most widely adopted OS license. Apache, MIT, and BSD licenses also have their place, but I (personally) believe that the future of software is a commons, where everyone (but proprietary software vendors) wins big by building on a GPLed infrastructure.

Friday, July 13, 2007

The shoe-maker's family has no shoes?

One trouble with being a consultant is getting time and energy for your own projects.

My wife and I spent a busy week or so a couple of years ago building a healthy cooking web portal. Since then, with just a few changes, the site has been running on auto-pilot. I wrote the original implementation in Java+JSPs, and now my wife and I are talking about investing more time into what is for us a fun hobby (we are both extremely good cooks).

I need to decide how to proceed: I have worked through several Seaside tutorials in the last few years (good stuff), and I have done several Ruby on Rails web applications for consulting customers in the last 18 months, so I had seriously considered moving off of the Java platform. In the end, assuming that I have time for a version 2 of our recipes web portal, I will probably stick with Java. I would like to upgrade to using Struts 2 and Hibernate with Lucene search. I have, just for fun, been improving my Javascript skills, so version 2 will be AJAX enabled.

Anyway, Carol and I have been talking enough about the recipes web portal that I think that we will start work on it again. We only have a few hundred registered users, but we get a fair number of visitors who do not create a free login, and an average of about 5 page views per site visit, so people hang around. I originally used the USDA nutrition database to estimate nutrition data for recipes - I took that off because the data seemed inaccurate (USDA often sites multiple studies that vary a lot!) However, enough people have asked for nutritional information that I will probably add it back in with a warning to accept nutrient data per recipe as estimated information. One thing that I implemented in the original site that people seldom use is a free custom database of "food ingredients on hand" - with a nice option to only show search results for recipes that you have the ingredients for. A good lesson here: sometimes users will ignore features that developers think are great! I might remove that entire module of code and web UI stuff - if users don't use something, toss it out.

Thursday, July 12, 2007

Squeak Smalltalk 'goodness'

I sometimes tell my tech-friends that if I did not have to earn a living, then I might spend all my time in Squeak. I would use Smalltalk more, but requests for consulting work always come for Java, Lisp, and Ruby. Torsten talks about some great new UI tools that will be donated back to the open source Squeak community. Squeak is a very fine platform, and projects like Seaside and Open Croquet will not be the last high visibility projects we see built on Squeak.

Saturday, July 07, 2007

REST rules

I am a software architect who also writes a lot of code. I very much like using different programming languages for different system modules based on language features and available libraries. In other words, I have moved past trying to write everything in C++ or Java.

For my current project, I need to do a lot of development in both Ruby (small utilities, database, text utilities, Rails) and back end Common Lisp AI processes. After decades of building custom socket interfaces, CORBA, SOAP, JMS, etc. the simplicity of REST is just right for most of what I need to do that consists of stateless service requests. REST services can return plain text, JSON, XML, etc., depending on what makes sense for any given service.

Implementing Ruby REST clients is simple (what isn't in Ruby :-), and writing generic REST services in Lisp is simple enough using any of the common Lisp web servers (I use Portable AllegroServe, but any that you like will do).

Fiat currencies like the US dollar

I agree with this article almost 100%

I say I agree "almost 100%" because when a crash of the fiat dollar does occur, I don't believe that it will be as bad as this article suggests: too much is riding on making any crash "gentle" and drawn out. If the dollar lost 60% of its value in one year, that would be a catastrophe, but if the dollar loses 75% of its value slowly and steadily over 10 to 15 years, then I think that society will slowly adapt: lots of cutting back on expenses, sharing homes, eating a less expensive vegetarian diet, and generally cutting living expenses as much as possible. People are resilient and adaptable, and for things that really matter, most people will still lead a good life.

I believe that this article is correct: our government and big business are doing everything that they can to postpone the fiat dollar collapse (invaded Iraq and currently threatening Iran, Venezuela, and even Russia with our threat of a missile shield around their borders). My personal opinion is that our elite class should let the crash start to happen in a slow, steady, controlled way -- and perhaps that is what is exactly happening right now with diminishing purchasing power of the dollar for food, housing, education and transportation (thankfully offset somewhat by cheaper imported goods).

Saturday, June 30, 2007

Education is a life long path

For both personal growth (and fun!) and to stay current as a consultant I find myself spending about 10 to 12 hours a week in general learning activities. I addition to good books, ACM Portal papers, etc., one of the most enjoyable sources of new and/or interesting ideas are the Google Video Lectures. If you have not already done so, install the Google video player; higher resolution viewing than in-browser viewing that makes it easier to read view-graphs and live demos. On OS X, the video player stores downloads in ~/Movies where I keep a text file of notes on what I found most useful/interesting in the videos, and approximate time locations.

Friday, June 29, 2007

I upgraded my open source projects to use version 3 of the GPL and LGPL today

I agree with the new provisions, so why not! I believe that more companies and organizations should adopt wider use of the GPL (or Apache, BSD, etc.) to lower cost and improve quality. I argue that a small part of IT infrastructure can be proprietary to protect intellectual property while most is layered on top of (and supports with contributions) open source infrastructure software.

Thursday, June 28, 2007

I revisted the scene of my 2/2/2006 accident today

One of my friends (who is a psychologist) suggested this morning going back to the hiking location of my 2/2/2006 accident (broken bone in shoulder, torn rotators cuff, shoulder dislocation). I had some initial hesitation about going back, but I had a great time today: we climbed the mountains separating Boynton and Fay canyons. I had a difficult time getting up the mountain but the experience was very good. This picture (taken today) is near the area where I fell: above_Fay_Canyon_looking_towards_Bear_Mountain In addition to the views, we enjoyed visitng two Indian ruin sites today. I have many pictures of the Sedona area on my Flickr photo site.

Saturday, June 23, 2007

My best Ruby coding hint

One programming trick that I use to reference Ruby source code for the standard libraries and locally installed gems is to set up TextMate projects (Eclipse Ruby projects also work well) for:
  • /usr/local/lib/ruby/1.8 - location of Ruby source code for standard libraries on my system
  • /usr/local/lib/ruby/gems/1.8/gems - location of local gem installs (often contain examples/tests and documentation files) on my system
When you are using a standard library or a gem it is great to have the source right in front of you for reference.

Friday, June 22, 2007

"Everything is Miscellaneous" - book review

I just finished David Weinberger's great book Everything Is Miscellaneous: The Power of the New Digital Disorder during a picnic with my wife today. For me, the main "take away" ideas from the book were:
  • In a digital world, the same information can be in multiple places, and organized the way users need or want it.
  • Rigid tree-structured indexing allows parent/child associations but fails to allow general associations between objects. Example: multiple tags allow users to identify information the way they want to access it.
  • When applying tags indicating that an object belongs to a group, it is good to allow a numeric value. Example: Mary is a: manager (0.75), technical guru (0.9), good in meetings (0.25)
  • Technologies like Wikipedia will continue to make information and knowledge a commodity without the limits placed on paper documents like the Encyclopaedia Britannica. There are no limits to the number and size of articles in Wikipedia. Wikipedia supports rich links that can identify relationships between articles. As articles in Wikipedia mature and stabalize they transition from being information to representing knowledge.
  • The web is messy: anyone can write and link to anything. RDF and the Semantic Web can provide some degree of order while still allowing messy bottom up development while providing opportunities to share schemas and ontologies.
A fun book to read and a good source of ideas.

BTW, here is a picture near where we had our picnic today (Red Rock Crossing in Sedona Arizona):DSC00007

Why GPL version 3 is important

Just my personal take: Compatibility with Apache 2 license is huge. I would guess that most developers of software under GPL and Apache licenses would like to enable people and organizations to build new open source software with both GPL and Apache licensed components.

Another reason why GPL v3 is important: there are too many open source licenses right now. If GPL v3 is successful, then I hope that an even larger percentage of open source projects use the GPL. While some corporations do everything that they can to lock in business by spreading FUD about open source and using non-standard file formats, it is in the general public interest to have a strong open source infrastructure and standard document formats. Standards in open source licenses are also important which is why I would encourage standardization on GPL (and LGPL), Apache, and BSD - whatever developers feel is right for their projects and their personal philosophy on open source. I believe that there is also no problem mixing BSD and GPL modules in the same system as long as individual licenses are included with any distribution.

While my projects are currently LGPL I prefer the GPL. LGPL has one huge advantage: LGPL licensed code can always be used under the GPL license. I LGPL my projects to let more people and organizations use my work but I am considering changing to GPL v3 in the future.

Thursday, June 21, 2007

Hibernate Search: good integration of Lucene and Hibernate

In the last 2 years, when not too busy consulting, I have been working on a knowledge management system KBSportal. While I have prototyped some ideas in Ruby and Common Lisp, my final target implementation language has always been Java (trying to make something that will be very widely used).

In the Java version, I implemented a threaded asynchronous system for maintaining both a relational database and manage Lucene indices and search. Last night I spent some time reading the documentation and playing with the Hibernate Search example programs. Hibernate Search supports both an asynchronous update mode and a simpler synchronous mode where objects are created in a relational database and immediately indexed for search. The search API returns either object IDs and search scores or simply returns Java objects matching a search query. The important thing to me is that by using Hibernate Search I can remove a lot of my own code making my system easier to understand and modify, and take advantage of future improvements made by the Hibernate Search developers. Writing my own version was fun and educational, but I like the Hibernate Search implementation more than my approach that was custom fitted to my application. Great stuff!

For Ruby developers: check out acts_as_ferret that provides the same kind of integration between ferret (Ruby/C port of Lucene) and ActiveRecord: also good stuff!

Wednesday, June 20, 2007

Cool: Ruby Google GData client library

This library is still being developed (right now just calendar and blogger support is in place). If you need a complete client API implementation right now, use the Java libraries: the examples are very complete making it easy to get started. I am going to wait for the rest of the Ruby client library to be released before I get serious with GData applications because find Ruby to be a more convenient "hacking" language to get (mostly small) projects done quickly.

Fantasy that may likely come true: Prometeus (experience is the new reality)

This video, at least on a technology level, syncs up very well with my own expectations: 5 minute Video on YouTube - well worth watching if you have not already seen it

Artificial Intelligence (AI) information assistant avatars, compelling Virtual Reality (VR) with haptics (*), and shared information will almost certainly be in our future (assuming that Bush or someone else does not start using nuclear weapons and destroy civilization).

I think that many of you would agree on this view of our future technology and how it will affect life and work. What is less clear to me is what our society will look like; I think that assuming our civilization lasts, we have 2 probable outcomes:
  • Novelist William Gibson's view of the future: governments are sidelined, corporations form the structure of economic and work life, "dog eat dog" meritocracy at all levels of society - this is the outcome I expect
  • A system like we have now, but where "competitive" governments compete for skilled workers and citizens by competing on low tax rates, public safety, fair and balanced arbitration between public and corporate interests - probably not going to happen, but this would be the best outcome
(*) haptics: force feedback on human/computer interfaces, sense of taste, smell, etc. I implemented a force feedback steering wheel for a VR racing simulator about 12 years ago: if you would drive the vehicle off of the racing track, you would feel the off road vibration differently than the on road steering wheel vibration; hit something and feel a realistic effect in the steering wheel. Coupled with good sound experiences with spatial effects (head related transfer function) this kind of thing is very compelling, or "suspension of disbelief" as it is known in the VR business.

Sunday, June 17, 2007

Using Lucene with JRuby

I use the Ruby Ferret indexing and search library a lot. Ferret is a port (some Ruby, mostly C) of Lucene. I have recently been getting into using JRuby. A few days ago, I discovered that it was reasonable easy to run a simple Rails web application using the Java application server JBoss using JRuby (this took me an hour - next time will be easy). Today, I spent a short while getting Lucene and JRuby working together:
require "java"
require "lib/lucene-core-2.1.0.jar"

class Lucene
@index_path = nil
def initialize(an_index_path = "data/")
@index_path = an_index_path
end
def add_documents id_text_pair_array # e.g., [[1,"test1"],[2,'test2']]
index_available = org.apache.lucene.index.IndexReader.index_exists(@index_path)
index_writer = org.apache.lucene.index.IndexWriter.new(
@index_path,
org.apache.lucene.analysis.standard.StandardAnalyzer.new,
!index_available)
id_text_pair_array.each {|id_text_pair|
term_to_delete = org.apache.lucene.index.Term.new("id", id_text_pair[0].to_s) # if it exists
a_document = org.apache.lucene.document.Document.new
a_document.add(org.apache.lucene.document.Field.new('text', id_text_pair[1],
org.apache.lucene.document.Field::Store::YES,
org.apache.lucene.document.Field::Index::TOKENIZED))
a_document.add(org.apache.lucene.document.Field.new('id', id_text_pair[0].to_s,
org.apache.lucene.document.Field::Store::YES,
org.apache.lucene.document.Field::Index::TOKENIZED))
index_writer.updateDocument(term_to_delete, a_document) # delete any old docs with same id
}
index_writer.close
end
def search(query)
parse_query = org.apache.lucene.queryParser.QueryParser.new(
'text',
org.apache.lucene.analysis.standard.StandardAnalyzer.new)
query = parse_query.parse(query)
engine = org.apache.lucene.search.IndexSearcher.new(@index_path)
hits = engine.search(query).iterator
results = []
while (hits.hasNext && hit = hits.next)
id = hit.getDocument.getField("id").stringValue.to_i
text = hit.getDocument.getField("text").stringValue
results << [hit.getScore, id, text]
end
engine.close
results
end
def delete_documents id_array # e.g., [1,5,88]
index_available = org.apache.lucene.index.IndexReader.index_exists(@index_path)
index_writer = org.apache.lucene.index.IndexWriter.new(
@index_path,
org.apache.lucene.analysis.standard.StandardAnalyzer.new,
!index_available)
id_array.each {|id|
index_writer.deleteDocuments(org.apache.lucene.index.Term.new("id", id.to_s))
}
index_writer.close
end
end
This code assumes that the Java Lucence JAR file lucene-core-2.1.0.jar is in the subdirectory lib. A short test program is:
require "lucene"
require 'pp'

ls = Lucene.new
ls.add_documents([[1,"test one two"],[2,'testing 1 2 3'], [3,'this is a longer test string']])
ls.delete_documents([1]) # optional: test document delete from index
pp ls.search("test")
I had some hesitations about JRuby: I was concerned that using JRuby would lack the light weight feel of hacking in native Ruby. No worries though: JRuby is easy and quick to work with.

Tuesday, June 12, 2007

GMail PowerPoint file viewing

If you use GMail, try searching your email for "pps" - the extension for PowerPoint slideshow attachment files - then select "View as Slideshow". Very cool. Now if Google adds a service like Gliffy online diagram editor to the Google Office suite, then they will have a real office replacement: web applications that are less capable that desktop applications like OpenOffice.org but are a big win when it comes to group access, search, etc. I have started writing papers less than 3 or 4 pages, notes on how to configure servers for Tomcat, Seaside, etc., ideas for technical and creative writing, etc. on Google documents. For me, the one last 'must have' feature for Google documents is a technical diagram drawing tool with the ability to embed figures in Google word processing documents.

Monday, June 11, 2007

Apple Safari for Windows

I am surprised: Safari for Windows :-)

I just installed the new Safari beta 3 on my MacBook, but did not grab the Windows version. I see the big win distributing a Windows version of iTunes, but I don't see the win for tossing another web browser into the mix available for Windows unless Apple is planning on making Safari into a platform with "proprietary" extensions. I quoted "proprietary" because the core of Safari is open source.

Saturday, June 09, 2007

JRuby 1.0 released

Great news! Although I also use Common Lisp and Java a lot, Ruby is a favorite programming language because code is so concise and easy to read. If I never had to worry about runtime performance, I would use Ruby even more. I suspect that JRuby will evolve into a reasonable fast language even if the JVM is not the best runtime platform for a dynamic language like Ruby.

Monday, June 04, 2007

Why the complaints over purchaser's name embedded in Apple's non-DRMed music?

I do not understand the complaints! If I buy non-DRMed music that I am free to personally use the rest of my life on any device that I own, why should I care if my name is watermarked in the music? This music is not supposed to be shared with other people anyway - fair enough.

Apple did do something recently that I did not like: they made it more difficult to generate MP3 files. I use 2 cheap generic MP3 players, and not iPods so I do need MP3 files generated from what I purchase. I have seen some workarounds listed on the web, but it is an inconvenience. I suspect that there is a strong connection between the watermarking and the difficulty in converting to MP3s that would obviously lose the watermarking. It seems to me that Apple should have made the new version of iTunes make it very easy to generate MP3s, but have the conversion process copy the watermark - I would not care about that. Looks like Apple does not want to make using generic MP3 palyers with iTunes an easy process - I understand this but don't like it.

Sunday, June 03, 2007

Emacs 22.1 is released - first major update in 5 years

Check it out!

For Mac OS X users, I built the new version using:

./configure --enable-carbon-app=/Users/mark/bin --without-x

to also build an Aqua application in my personal bin directory in addition to a command line version in /usr/local/bin/emacs

Again, for Mac users: I like to put this in my .emacs file:

(global-set-key "\C-xt" 'tool-bar-mode)

So I can toggle the graphic toolbar icons on and off when running the Aqua application.

The URL for this blog has changed

Please use http://markwatson.com/blog/ now to access this blog. The old URL does a redirect. Please also use the new RSS feed: http://markwatson.com/blog/atom.xml

Saturday, June 02, 2007

Is the Common Lisp 'loop' macro evil?

I must admit to being "old school" when it comes to Lisp coding. I like s-expressions. The loop macro introduces a non s-expression syntax to the language that I find a little uncomfortable. I spent 20 minutes this morning looking for interesting loop example uses and generally playing with them. Definitely some cool stuff, but I think that I will continue to pass on using the loop macro, except when it is already in other people's code - no point in rewriting stuff that works.

Different programming languages have different coding styles that seem to work well and are language specific. As an example, yesterday I wrote some Ruby code with two nested collection collect blocks. I had been coding non-stop in Common Lisp for weeks, and before I checked my new Ruby code into svn, I took another look at it: looked strange at first even though it was a natural Ruby coding style. I looked at it again this morning, after my "thinking in Lisp" mode was temporarily suspended, and the nested collect blocks looked just fine. Amazing how choice of programming language affects the way we think to solve problems.

Thursday, May 31, 2007

Google Gears: a sea change for web applications?

I thought that Adobe Apollo might have hit a sweet spot for allowing developers to create hybrid desk top and web applications. I think that Google may have hit a sweeter spot with Gears. Gears installs as a FireFox add-on and uses the SQLite database to store data locally on your file system. Each site that you visit that is 'Gears enabled' causes a pop up permissions dialog to appear - I recommend being careful of which web sites you allow to use the Gears add-on. I have reviewed the developers' documentation and it looks straight forward to set up and a Javascript SQL database API makes it easy to use SQL in your Javascript.

Saturday, May 19, 2007

Why the ODF is better than Microsoft's document formats

It takes a few lines of Ruby code to process OpenOffice.org document files:
require 'rubygems'
require 'rexml/document'
require 'rexml/streamlistener'
require 'zip/zipfilesystem' # install with gem
include REXML

class OOXmlHandler
include StreamListener
attr_reader :plain_text
def initialize; @plain_text = ""; @last_tag_name =""; end
def tag_start name, attrs; @last_tag_name = name; end
def text s
@plain_text << s << "\n" if @last_tag_name.index('text')
end
end

class ReadOpenOffice
attr_reader :text
def initialize file_path
Zip::ZipFile.open(file_path) { |zipFile|
xml_handler = OOXmlHandler.new
Document.parse_stream((zipFile.read('content.xml')), xml_handler)
@text = xml_handler.plain_text
}
end
end

puts ReadOpenOffice.new('KBrecipes.odt').text
I have spent too much of my time over the last 10 years dealing "programatically" with Microsoft document formats. I am tired of wasting my time when open document formats are so much easier and less expensive to use.

Monday, May 14, 2007

Data representations: the more the better

A friend of mine, also a long time Lisp hacker, stated the opinion about 15 years ago that object oriented might be the "last great programming paradigm". A good thought, but object representation is just one way to think about and manipulate data. I admit that my favorite style of programming is with an object oriented language with transparent object relational mapping (e.g., Java with Prevayler, Ruby with ActiveRecord, or Lisp with something like AllegroCache).

The ability to write programs (preferably quickly :-) allows us to experiment with data representations. The point I am making here is that different software tools can not only help solve different problems with different levels of effort, but the tools that we choose inform us and change the way we think.

Sometimes a relational data model just works best, both for efficient access (not letting an object relational mapping system build an entire object in memory when you are only interested in a few columns in a table) or for thinking about and browsing data.

When there are too many attributes characterizing data that you need to explore or use, then faceted browsing helps a lot: pick the most important attribute, then the second, etc., eliminating large parts of a data space.

For some problems graphs are the best data representation and languages like Lisp and Prolog that allow list data structures to be cut up and put back together are most effective.

Wednesday, May 09, 2007

Use of arrays considered harmful?

Pardon me for playing with the title of Edsger Dijkstra's famous paper, but as a Java developer I am starting to believe that arrays should be seldom used, given Java generics and the revised collection classes. I started thinking about this while reading someone else's code and noticing some artificial looking conversions between arrays and collections. Then for fun, I did some measurement and noticed over 15% of processing time was spent in toArray() methods.

The big cost in IT is usually in development and maintenance, not in deployment costs. Most projects are small, not Amazon size deployments so I believe that unless you are sure that you are building a very large scale system, it is best to optimize development to reduce costs for small and medium size systems. Smaller code size means smaller costs. Java programs should be as short and concise as possible and generics and use of collections helps a lot. Sometimes when I read the source for Java programs it looks like the authors were paid by line of code :-)

Saturday, May 05, 2007

Interesting technology: AllegroGraph

I am using Franz's AllegroGraph for two proof of concept projects for a customer: one using the Java APIs (free version) and one using the Lisp version that is unlimited in the size of stored data. RDF storage and querying is not easy technology to use (at least for me) but looks very promising.

The thing that I find interesting about using AllegroGraph is that you are dealing with disk-based persistent data, but not dealing with objects - not dealing with object relational mapping, etc. Instead, you work with graph data structures that are stored on disk, with parts cached in memory. Interesting stuff.

Still, dealing with RDF is not optimal, compared to dealing with graphs in memory. As an example: I used to work a lot with Rete networks using Lisp (hacking Charles Forgy's Lisp code) and dealing with graph data structures built up with Lisp lists, cons, etc. is just easier to do. In memory graphs, semantic networks, etc. are just easier for me to wrap my thoughts around. However, approaches like AllegroGraph have the advantage of scalability.

Wednesday, May 02, 2007

My article "A Java Developer's Guide to Ruby" was just published

Check it out on DevX.

In this article I write about why a Java developer might want to also use Ruby and I cover a few cool Ruby features.

Saturday, April 21, 2007

Deploying a Squeak + Seaside web application to a Linux server in 3 easy steps

I am sure that most developers who have tried DabbleDB have experimented with the web framework Seaside that DabbleDB is built with. I have worked through Seaside tutorials several times in the last couple of years and last month I wanted to deploy a tiny Seaside based experiment to one of my leased Linux servers. It took a short while to get it deployed because I didn't get it right the first time, so here are the simple steps that I eventually used:
  • Get your Seaside application running interactively in Squeak; make sure the WAKom service is running on a desired port number with:
    WAKom startOn: 9090
    and save image and quit Squeak. Do not stop WAKom before saving your image and quitting Squeak.
  • ZIP up your image and source changes:
    zip -9 -r seaside_running_image.zip Squeak3.9-final-7067.*
    copy to your Linux server, and unzip in your Linux Squeak directory.
  • Run Squeak in headless mode:
    nohup squeak -headless Squeak3.9-final-7067.image &
OK, maybe that was more than 3 steps. If you should ever need to do a scalable deployment Ramon Leon's directions look good but I have not tried them.

How much does web framework choice really matter?

Based on experience with consulting jobs developing web applications using several Java frameworks, Ruby on Rails, and Portable AllegroServe with WebActions (open source Common Lisp frameworks), I believe that choice of framework is less important than:
  • Programming language: choose a language that both fits the application domain and has good library support for your application
  • Data modeling: while I believe in interactive bottom up development, spending time up front getting object models 'right' makes development easier
  • Object persistence: there are lots of good choices (prevalence, object relational mapping for relational databases, distributed memory only, etc.) but choose a scheme that makes sense both for development and deployment
I think that these 3 issues are all more important than choosing a web UI framework.

I have been investing a fair amount of time learning Erlang this year and the ErlyWeb framework (that uses the high performance Yaws Erlang web server) looks very good for both interactive development and distributed deployment. For web applications that map well to Erlang, ErlyWeb allows Erlang to be the development language of choice, but again the important choice is programming language selection rather than web framework.

Tuesday, April 17, 2007

The Semantic Web, Parrots, and AI

Two different subjects today: I just added a blog entry on the semantic web on my AI blog and our pet parrot. One (possible) route to understanding how to do AI is to appreciate problem solving abilities in the natural world. Our young Meyers parrot is a good problem solver but it takes him a while. Earlier this morning, I was reading in bed and had fetched our parrot so he could run around like crazy on and under our bedspread - good for burning off energy. Our parrot wanted to get at some of my stuff on my night stand, but his way was blocked, except for a space between two water bottles which, try as he might he could not squeeze through and he could not move the water bottles. He spent about 2 minutes walking back and forth thinking about the sad situation he was confronted with when he suddenly lowered one wing, raised the other, moving his shoulders close together and then simply walked right through the "water bottle gap" :-)

Our small parrot must have some abstract world model of objects and his own body. Why and how he thought of raising one shoulder while lowering the other to compress the width of his shoulders is a mystery to me, but I believe that this was possibly an example of abstract thinking.

Monday, April 16, 2007

Landscape page layout orientation should be the new standard

Am I the only one tired of viewing vertical "portrait" formatted PDF files on a computer screen? Since it is fairly trivial to support generating either horizontal or vertical page orientation, there seems little excuse to not at least offer readers the choice. If you want to print a PDF, landscape is also OK.

I have purchased three "books on PDF" in the last few months: not a bad deal since this costs publishers a lot less than producing physical books, so they tend to be about half price. One of these purchased PDFs has a landscape (horizontal) layout: what an improvement for on-screen reading! I wish that all publishers would take a clue and that people who produce papers as PDFs would also provide landscape options. BTW, I still have a few vertical (portrait) PDFs on my web site, but I am working on re-publishing them in landscape mode: I have switched to a Latex+some custom code writing system that produces HTML, and either format for PDF, but it is taking a while to converting some of my older projects to Latex.

Saturday, April 14, 2007

Can't believe I missed this Unix utility: screen

I have been using Unix systems for over 20 years: I can't believe I missed the Unix screen utility for using multiple screens in a shell window. (I was browsing the web late last night on my new N800 web pad, and saw a post on Digg about screen - I tried it first thing this morning when I turned on my laptop.)

Here is a short introduction to its use
but all you really need to know to get started is to run the program, use control-a c to create new views, control-a a to send a control-a to a running program (useful for emacs :-) and use control-a n and control-a p to cycle through "next" and "previous screens.

Screen comes pre-installed on OS X and is an easy install on Linux if your distro does not already include it. If you love the command line, screen is definitely worth a few minutes of experimentation to see how you like it.

Friday, April 13, 2007

Review of Nokia N800 'web pad'

A N800 is a very small device that uses WiFi to access the web, including good support for Google GMail and instant messaging, and has a built in video camera.

Although it is a very nice device, I have two issues with it:
  • I thought that it came with video conferencing software (from what I just read, I need to wait a few months for additional software to be available).
  • The screen is very small - the resolution is a tasty 800x480, but the physical size is small. I find myself holding it about 14 inches from my eyes - probably too close to avoid eyestrain with long uninterrupted use.
I bought it for portability for reading both online technical material and fun stuff like Slashdot and reddit - without having to hold a laptop. After using it for about 1 hour I became very used to it. Also, did I mention that it runs Linux :-)

Monday, April 09, 2007

Google Data APIs

I have been experimenting with online data applications (for example, my DabbleDB FactBook RSS feed and experiments with Metaweb's freebase.com system).

I am adding Google's Data APIs to this cornucopia of structured data stores. If you are a Java developer, Google's GData APIs download is a good quick start.

GData and DabbleDB are great for organizing your own data online and then using that structured data in your own applications. Freebase.com offers a wealth of "other people's data", as does DBpedia: Wikipedia information culled as RDF data.

Tuesday, April 03, 2007

Interesting way to build rich client web applications: mjt

As I wrote a few weeks ago, someone working at Danny Hillis's start up Metaweb recently gave me an alpha invite to freebase so I have been experimenting with both their web services APIs and web application as I get some free time. Very cool stuff.

The freebase developers have released the mjt Javascript template library under a BSD-like license. This library enables you to write rich client applications based on web services that return JSON data. It takes more than a few moments to get used to the idea of client side templates but if you are used to server side tools like Java Server Pages (JSP), etc., the ideas are similar. I have done nothing with mjt (so far) except for reading the documentation and playing with the demos, but my first impression is that mjt would be very useful for developing web applications using back end services that are implemented in several languages/platforms that accept REST requests and return JSON.

I have been getting very comfortable with Javascript in the last year. I wrote an AJAX enabled web application for my customer yesterday and today using Common Lisp on the back end with client side implemented as Javascript, CSS, and HTML - a nice combination of REST web service calls to process AJAX calls and Lisp code to generate HTML as needed. If I was already fully up to speed on mjt, a reasonable alternative would have been to limit the back end Lisp code to just providing the REST web services and serving up static content. Anyway, mjt is well worth looking into.

Saturday, March 31, 2007

Less is more: advantages of compact programming languages

Compact languages and libraries have the advantage that one person can understand most of an implementation. Two of the most elegant programming languages, Squeak Smalltalk and Ruby, have sufficiently compact implementations to understand, given some effort. And this effort is worthwhile - a lesson I learned in the 1970s when I worked as a systems programmer: I kept listings of key parts of the PrimeOS (Multics based) operating system and for the Lisp and FORTRAN implementations handy at home and in my office. Time spent reading through the code paid huge dividends whenever I had to do any systems level programming. Reading the code helps create a mental map of whatever software that you are using.

The implementation of Ruby and the standard Ruby libraries is compact, basically divided into the C implementation of Ruby itself and the Ruby code for the standard libraries. If you are going to be working much with Ruby then consider creating at least a project (using Eclipse, IntelliJ, TextMate, etc.) with the library source code. Good for library reference, looking up APIs, and for just reading code.

I would make the same suggestion to developers who use Squeak Smalltalk (or other implementations): you have the source code to the standard libraries: read it!

I also enjoy and am very productive using the Java and Common Lisp frameworks, but these languages with their standard libraries are so huge that I would personally never attempt to dive in and understand their implementations. Not having a mental road map of the implementations of Common Lisp and Java reduces the effectiveness of these platforms.

Saturday, March 24, 2007

GWT and Seaside: steps in the right direction for increasing developer productivity

Although most of my work is in AI development and general web services, I greatly enjoy writing user interfaces - this interest started in 1982 when SAIC bought me a Xerox 1108 Lisp Machine - the windowing system and general development tools were so good that they make most recent software development environments look weak.

Google Web Toolkit (GWT) and Squeak Smalltalk based Seaside are pleasant exceptions. Just like using Rails, GWT and Seaside provide that pleasant "everything just works" experience and great interactive development environments.

Last night I used GWT and IntelliJ to prototype some web UI ideas. If you have not tried working with the GWT Development Shell, give it a try: changes to client side resources (style sheets and HTML) and to Java client side source code (automatically compiled to Javascript during development) are immediately reflected in the test web app. I don't have any current tasks dealing with Java web applications right now, but the next time I do, GWT looks like a very good framework choice.

Seaside also supports the same fluid interactive development style: changes made in a Squeak Smalltalk source browser are immediately reflected in running web application. Seaside uses continuations to provide a more linear programming model. Blocks of code are used instead of URLs; inside a code block, you can call another code block - after a "return", you continue where you left off in the first code block. There is a performance penalty for using continuations, but it looks straight forward to run multiple Squeak processes behind Apache so it look like Seaside scales the same way that Rails scales. DabbleDB is written using Seaside - a great proof of concept for scaling.

Wednesday, March 21, 2007

Balancing the use of Open Source (especially GPL) and proprietary software

The GPLv3 is likely to have stronger requirements for sharing back code used in web applications. While I personally think that this is fair, it will be "interesting" for companies using GPLed components in proprietary web applications.

As a consultant, it is often frustrating when customers do not want to simply use a license like the GPL because they worry about protecting their intellectual property. I believe that in almost all cases, any proprietary code in a system should be associated with custom data handling. I am not a lawyer, but I believe that the GPL (even v3) allows GPL systems to share data with proprietary systems via a relational database (or some other type of persistent storage).

Why bother dealing with the GPL, assuming that you don't buy into the social and philosophical ideas of the FSF.org? Because it will end up saving you money! For example, using a GPLed content management system and donating improvements back to the project will save you money and effort every time there is a new release that already has your improvements incorporated.

What about maintaining a competitive advantage over competitors? Any advantage will probably be in both how well you serve your customers and in your proprietary data and data processing software.

Monday, March 19, 2007

Apollo alpha released to developers: offline enabled web applications

I have spent a lot of time in the last year studying and using AJAX technologies: Dojo (when I want a rich set of components), Prototype (when I want simple, lean AJAX support for forms), OpenLaszlo, some time spent with GWT, less time spent with Flex 2. For web applications that never have to be used "off line", these technologies meet my needs.

The Apollo developer pre-release looks compelling for applications that need to continue running when the user does not have an Internet connection. That said, I think that Adobe will get real competition from the open source OpenLaszlo system. Reason? Because OpenLaszlo is open source, great funding, and lots of Fortune 100 customers using it. I would like to see Adobe compete by releasing the Flex 2 SDK as open source (it is now free as in beer).

Friday, March 16, 2007

Factor

I just read about Factor on Dave Roberts' Finding Lisp blog. While I don't expect to do anything serious with Factor, I must say: it is very well done - at least the Mac OS X version that I tried. Add creative to well done. Read what Dave has to say about Factor (he also has good things to say about Erlang, for which I have been investing some learning time in the last month) and install Factor to see an interesting light weight environment. Factor reminds me slightly of using Oberon (a long time ago): a different way to work.

metaweb.com and freebase.com

I am always on the lookout for freely available sources of data in useful formats. Metaweb was founded by Danny Hillis and their first public system is at www.freebase.com.

"Freebase is a vast, free, open online database of structured knowledge" - from their web site.

One interesting thing, besides the interesting technology for storing and querying structured data where both the user can define her own categories and use system wide categories, is that the content that hosted is freely licensed under Creative Commons, GNU documentation license, or in the public domain.

You need to request an invitation, and then the documentation provides information on accessing Freebase. I experimented during lunch time with their Python client APIs - cool stuff.

Thursday, March 15, 2007

Trip down memory lane: I installed Minix 3 tonight

I was an early Linux user (downloaded Slackware over a slow modem in early 1994) but I installed Minix before that - probably in 1992 or 1993.

While I was reading tonight, I installed Minix 3 with Parallels on my MacBook. The system worked OK, but ran really slowly (over 5 seconds to start Emacs). I have seen a few blogs lately on Minix 3 and I decided to install it, and it was fun to kick the tires for 20 minutes. I like the idea of Minix (an operating system simple enough for a hobbyist to understand) but the slow performance convinced me to delete the Parallels disk image for it. I don't know why it ran so slowly since Windows 2000 and Ubuntu Linux run very well under Parallels.

Wednesday, March 14, 2007

I have released some NLP (natural language processing) tools with a LGPL license

Here is the download link. These tools come in a few 'flavors': Java, Ruby, C++, and C#. I expect to add two larger NLP projects in the next month.

BTW, I consider the LGPL to be "business friendly". You are allowed to mix my LGPL software with your own commercial products without open sourcing your products. You may also mix my LGPL software with open source with projects with Apache, BSD, MIT, or Mozilla style licenses. If you have any questions, ask me. If the LGPL license prevents you from using this material, please let me know about it.

Integrating PHP and Ruby with Java server side deployments

I dislike re-writing code that works well without a very good reason. Sometimes there is a temptation to rewrite code in order to move to a new language and platform. The Quercus PHP run time WAR file from Caucho looks like a good way to mix and match existing server side Java and PHP projects. Several years ago, a customer asked me to loosely integrate (just share a common login) SugarCRM and the JSP+POJO+Prevayler based web app that I had written for them to manage their research papers and devliverables. Not difficult to do, but my integration solution was a hack. Compiling PHP to byte code for the Java platform and easier (non-web service) calls between PHP and Java is an obvious win for developers. When JRuby's performance gets better (and it will :-) I look forward to the same type of integration with both Ruby and Rails.