I cover a wide range of topics including machine learning, linked data, network programming techniques for IoT, and some ideas for knowledge management using cloud data.
Friday, November 13, 2015
Thursday, October 01, 2015
I will continue using this blog for personal posts, programming languages (mostly Haskell, Java, Clojure, and Ruby) and general discusions on technology.
Monday, September 28, 2015
As the universe unfolds, I have been spending most of my time recently working with machine learning and for the forseeable future that will not change. Lets face it, many of the really great ML libraries are written in Java and Python.
I still love development with Clojure and Ruby, and I am still on my long term quest to become passably proficient with Haskell. That said, it is crazy to not simply use languages that have the best library and framework support for any task.
Sunday, August 02, 2015
I worked (mostly) at SAIC in the 1980s and 1990s. In the groups I worked in we developed large software systems (sometimes with hardware components) for customers. Software reuse was a dream back then that was largely unfulfilled. Our procedure for reuse was mostly cut and paste from old projects, with some effort to write reusable libraries. There was also a movement to use commercial off the shelf (COTS) software.
It occurred to me recently that we are now much closer to the dream of widespread software reuse. What has changed is a healthy open source (and libre) software ecosystem of trusted and vetted libraries, frameworks, and complete applications. I tend to trust software from FSF and the Apache Foundation, for example. Organizations and individuals are motivated to release software for a variety of reasons: for help in development and bug detection, for good publicity and self promotion, and sometimes for ego. All good reasons!
My process when starting a new project is to first identify existing open source software that I can build on. My choice of programming language is often dictated by the language used in the open source software projects that would be most beneficial to my project. It is a thrill to build a new project using mostly existing software. This greatly reduces the cost of projects and I think also greatly increases developer satisfaction. Who wants to spend 6 months writing a project mostly from scratch when it can be done much more quickly building on other people's work.
For me the magic that makes this all happen is public repositories like github and bitbucket. The cost of evaluating an open source project for reuse can be very low: a git clone, build, run the tests, look at the tests, and read through the documentation and source code.
So I believe that we have made incredible progress in software reuse in the last 30 years.
Wednesday, July 29, 2015
Installing Windows 10 on my 5 month old HP Stream 11 was easy.I have no comments on that process.
Visually two things stand out: windows are all white except for a very thin aqua blue margin and my slow laptop seems to run the UI faster. I don't know how much of the speed bump is making the code more efficient and how much is doing away with some animation effects.
The desktop now seems like a mixture between Windows 7 and 8.1. The start menu is back and the bottom icon navigation bar is always visible along the bottom of the screen unless you put an application in full screen mode. Clicking the windows start menu in the lower left corner of the screen brings up a combo: classic looking menu on the left and the metro large icon interface like Windows 8.1 on the right side of the popup - but this popup only covers about 30% of the screen. It all seems a bit odd to me but I really like it - after using it for ten minutes (it took a little while to adjust to the new interface). I think the new interface in general is very well done. I haven't spent enough time with the new Edge web browser to have a firm opinion yet, but it seems functional and the reading view does a good job of reformatting web pages without advertisements for comfortable reading.
Windows 10 Cortana, similar to Google Now and Apple Siri, is always available just to the right of the Windows start button. Just like the search utility on Windows 8.1 this is the way to quickly find stuff in the system. Need to change the PATH, then just type 'environment variables' and instantly the environment edit utility is shown. I think this actually works a little better than Spotlight on OS X and quite a bit better than hot key search on Ubuntu. I tried using Cortana for searching for things in my community. It did OK, but will hopefully get better as it gets access to more of my life context.
I will spend time locking down some of the privacy settings. I alreaded deleted the Skype application because it leaks a little too much personal information for my taste. Cortana is configuraable for setting which types of information are collected. It pays to take the time checking possible privacy settings for products and services from Microsoft, Google, Facebook, Apple, etc. I tend to keep my Linux laptops more locked down, privacy wise, than my Windows and Apple laptops. I try to strike a balance between having some privacy and also enjoy available products and services.
Saturday, July 25, 2015
Even though I mostly use Java, Ruby, and Haskell, I have also been getting my head back into using Scala in my spare time. I took Martin Odersky's Functional Programming Principles in Scala class three years ago, and although I really enjoyed the class (he is a great lecturer!), I didn't much care for the tooling for Scala at that time. I ended up mostly using Clojure (with a little Haskell) for my day to day most used programming language.
This morning I was looking for an interesting template project using Scala for the backend and Scala.js for client code and ran across the impressive Widok project which supports a reactive web framework for the JVM and Scala.js. I have been experimenting this morning with the Widok skeleton client server starting project which provides niceties like automatic asset (SCSS to CSS, etc.) handling, auto re-compilation, etc. I don't have a need for using Widok right now but it is on my personal "watch list." Widok is really nicely done and the client server skeleton project is easy and fun to hack on.
Still, after three years I still find the tooling for Scala to be a bit heavy. I have a much faster laptop on order and that will help reduce the 5 or 10 second delay in processing changed assets and code for my work and generally experimenting with Scala, playing with Widok, etc..
Tooling is where Clojure and Clojurescript development really shine in my opinion. My Clojure setup allows me to edit server side code, Clojurescript client code, change hiccup template code, etc. and basically immediately see the results of code changes. I like that my hiccup templates are also Clojure code. It is fun to work on projects where everything is simply Clojure. Nice. I also like that I am not using any heavy weight frameworks, just a few libraires. This contrasts with richer development kits like Rails and Meteor.js which are fantastic when your application fits their application models. I like just using libraries and keeping the technology stack simple unless there is a particularly good fit for using something more complex.
Monday, July 20, 2015
sudo sh ~/Downloads/crouton -t cl-extra
After installing Ubuntu I had about 10.5 GB of free local storage (remember, Chrome OS is also installed). After installing the gnu command line development tools, Ruby 2.1, Java 8, lein for Clojure development and several of my Ruby and Clojure projects I still have 7.5 GB of storage. I still want to install Haskell so I will have less space to work with.
When developing web apps, you can test using the Chrome web browser since crouton runs Ubuntu right along side Chrome OS. For example, I was running a Sinatra app using port 4567 and could just hit the URL http://localhost:4567 on the Chrome browser in Chrome OS. Easy. I thought that I might have to start a ssh tunnell beyween the two Linux environments, but that was not required.
Opening several terminal windows using ctrl-alt t and then starting a shell in each with:
Saturday, July 04, 2015
My curiousity about Chromebooks has persisted. I am going to start teaching free classes at my local library in about two months on Internet secutity and privacy and I used this as an excuse to buy a Tosiba Chromebook 2. I had already bought earlier this year a little HP Stream 11 Windows 8.1 laptop using the same excuse :-)
I will start out this "review" with a list of the good and not so good things about the Chromebook 2 (all just my personal opinions). The good:
- The number 1 reason: security. Nothing (except for a few SSH keys) is stored locally and I think that if a Chrombook gets compromised it is automatically reset with a factory image.
- The screen is fairly high resolution 1080p and looks really nice. Some of the user comments on Amazon complained about reflections in the screen, and this would be a problem if it was used outdoors.
- 4 GB of main memory which helps the performance. Most Chromebooks seem to only have 2GB of memory.
- The keyboard and trackpad work well - no complaints.
- Contrary to some reviews, the battery has been lasting about 8 hours - not bad at all.
- This laptop is inexpensive, given the nice display.
The not so good:
- The laptop case is plastic. It looks OK, but I expect it to get dinged up easily. I also bought an iPearl hard shell cover that should help prevent some case damage.
- Only 16 GB of storage. How much would it have added to manufacturing costs to make this 32 GB? I did try using a 32 GB SD Card for a while which worked fine except for the card sticking slightly out of the case. I think that I will use the external memory card only for movies while flying or other cases where I temporarily need a lot of local space.
- Privacy: I find myself using Google, Dropbox, and Microsoft web services a lot on the Chromebook. There are issues using these services that I have blogged about before: FSF and practical Internet privacy and security.
Using the Toshiba Chromebook 2:
Writing: As you might expect, I find this laptop great for web browsing and watching Netflix. It is also pretty good for writing. I use leanpub for writing. My markdown manuscript files are stored in a book specific Dropbox folder, and I use a web interface to generate preview PDF, Mobi, and ePub files (which get put back into Dropbox). I use the Chrome StackEdit app to edit the markdown files. This is a simple and effective workflow that lets me concentrate on writing.
I am very happy that I bought the Chromebook - no buyer's remorse :-)
July 5, 2015 (next day) edits):
For web based Haskell development, I left fpcomplete.com off of my list - an excellent web based IDE and Haskell learning center.
Also, I spent some time this morning working on a Clojure project using Emacs + cider instead of IntelliJ. To be honest, since I usually use IntelliJ, I forgot how great cider has become. Beautiful integration with lein, nice auto-complete, etc.
Tuesday, May 19, 2015
I recently signed up for the pro version of nitrous.io since they are phasing out the old version that I used for free. So far I am very pleased. Now you get isolated containers to work in with root access. I use the $15/month version that gives you 1G ram and 20G of disk space for a total of two containers. Since I only use nitrous.io for development, I found it easier just use one container (using the entire 1G ram) and cloning the git projects that I am working on.
Some of the things that I particularly like are:
- The editor built into the web based IDE handles just about all programming languages and file types.
- There is a separate app than can do two things: sync files between your laptop and your container and also forward ports so you can test run apps in the container and use your local web browser. If you don't want to forward ports, you can use their preview option that opens a new browser tab and lets you test in your browser without any port forwarding.
- The battery on my laptops last longer if I do builds and test runs in the container. No (or less) fan noise :-)
- It is easy to spin up a new container for experiments and just delete it.
- When I am travelling, I don't have to load up a particular laptop with all of my projects - everything is in my cloud container so all I really need is a web browser.
- Plays nicely with Heroku. I haven't tried it, but nitrous.io is also set up to play nicely with Google Cloud Services and Microsoft Azure.
- With a little setup, you can SSH into your container from your phone, tablet, and laptops.
- The nitrous.io web IDE reminds me of the Cider web IDE that I used internally at Google.
- In general I think I save a fair amount of time having a container always available with my setup.
- I tend to use my Mac and Windows 8.1 laptops more often that my Linux laptop, and even though I have SSH, git, IntelliJ, etc. set up on all of my laptops, always having a Linux development environment no matter what device I am on is a more consistent development experience.
Tuesday, May 12, 2015
I am pleased that Heroku has introduced a new low volume pricing tier. I never felt very comfortable freeloading on their free tier and free tier web apps also timed out leading to longer loading request times. Now for $7/month per app Heroku supports a "hobbyist" mode for lower traffic sites that never get timed out or swapped out. So far I have redeployed three of my low traffic sites to Heroku's new plan, moving them from a dedicated server. They reduced their free tier hosting of a site to only being active just 18 hours a day - in other words: great for testing deployments but not good for hosting sites for free 24/7. I think this is a good move on their part although I did hear some complaints on Hacker News about this. Under this new pricing tier my three low volume sites cost about $21/month to host. Under the old paid plan the cost would have been about $105/month.
BTW, I would like to thank everyone who took the survey (or emailed suggestions directly to me) about topics for my new book project Power Java. When this book is released, hopefully by August 2015, I will then continue work on and finish what is the same book, but using Clojure for the example programs (Power Clojure). Thanks! I appreciated the input.
Tuesday, April 14, 2015
In addition to using several programming languages, I also like to experiment with different web infrastructures.
A few weeks ago I switched from using gmail as my primary email service to using fastmail.com. I still use gmail as a backup email and for my Google identity but I decided that I liked Fastmail a bit better and the yearly cost is not much.
The other change I have made is switching my www.markwatson.com web site from a Ruby + Sinatra web app on my own server to a PHP app running on Google's AppEngine. My absolutely favorite feature of AppEngine is the rolling system logs that can be checked easily from the AppEngine console. When I worked at Google in 2013, I loved the internal development environment (Borg, online system logs, the Cider IDE, and much more). Using AppEngine is, in a small way, reminiscent of Google's internal environment - at least enough so to make me nostalgic :-)
As much as I like devops and in general configuring and running Linux servers (often VPSs on Azure, Digital Ocean, and AWS), I sometimes feel a little guilty spending time on operations: perhaps my time could be better spent. My preferred PaaS providers are AppEngine and Heroku, with Cloud Foundary technology services (like IBM's Bluemix) also fairly nice. One thing that has always bothered me with PaaS however are "free" usage tiers. For one thing I like being a paying customer with SLAs, support, etc. Also, free tiers have to affect to some degree pricing for paid users.
Wednesday, April 08, 2015
Well, actually, I had just one Clojure project until today. I refer to my project as KB2 (KnowledgeBooks.com 2) and it is basically a kitchen sink for everything that I thought that I wanted in a personal (and perhaps small group) research and content management system:
- A personal version of Evernote: allows me to collect eBooks, web pages snippets and notes in a personal repository that is searchable. I use a Firefox add-on I wrote to capture multiple selections on web pages and send them to the web app.
- Uses NLP to identify entities in eBooks, web pages and notes and add an information icon that provides DBPedia (WikiPedia) information on the fly.
- Uses the Bing search API to find information on what my NLP analysis code considers if the main topic of eBooks, web pages and notes.
I enjoy meditation (also practice Yoga since about 1975 and Qigong for about two years) and after my early morning mediation this morning I had one of those ah-ha moments:
Life in Sedona Arizona
My wife Carol and I have been enjoying the early spring time in the mountains of Central Arizona. My friend Bill Bohan (one of the authors of the book Great Sedona Hikes) took this picture of me while we were hiking on Bear Mountain last week:
I have also been enjoying gardening. Several friends and I volunteer to keep a 1 mile historic irrigation ditch functioning to provide water to a historic farm and Crescent Moon Red Rock Crossing Park. The following three pictures (the first two taken by Don Fyffe) show me and some friends unloading some wood chips someone gave us for the farm. The third picture shows me holding up a bok choy I grew with the community garden in the background:
My new book project: "Power Java"
I was 'scientific' in my approach to choosing the material for this book: I had 11 topics that I wanted write about and I used a Google survey (it is located here) to get feedback from people who follow me on social media. Both the survey results and some great suggestions emailed to me by Alex Ott really helped me narrow down the topics for the book. Thanks to everyone who helped! (You can still add to the survey if you want.)
It was a difficult decision choosing Java for this book project. Most of my development in the last year has been in Clojure and Haskell (with a little Ruby and Java) but I decided that the book would have a wider audience written in Java.
I might provide an appendix and additional sample code showing the use of some of the examples with Clojure and JRuby wrappers but it is so easy for Clojure and JRuby developers to reuse Java code that I am not sure if it is worthwhile adding this material to the book. (Feedback on this will be appreciated, BTW.)
Monday, March 02, 2015
It is, IMHO, a very good thing that net neutrality now has the force of law behind it. The Internet is the most important artifact for sharing information ever and is worth protecting as a neutral platform. Even though there is a lot of value in walled gardens created by Google, Facebook, and Apple I still hope to see many more systems developed that support individual control of our own data and better support for privacy.
I think that Ward Cunningham's new Federated Wiki (on github: github.com/fedwiki) is a very interesting idea for combining local storage with federated sharing of content. Something to keep our eyes on!
The first version of my product will be for individual use (simple install on a personal server, or run locally on a laptop) but I am interested in evolving it into a federated system for use by small teams. A federated system is also useful for the single user use case: for example, sync content from a laptop to a server.
Friday, January 16, 2015
I have good intentions of starting to give talks at my small town's local library on safe internet browsing, privacy issues, and any topics that I get requests for. I bought a $200 HP Stream 11" laptop to get up to speed on Windows 8.1 since I only use OS X and Linux.
I am pleasantly surprised at how much I like this laptop! The display is pretty good, the keyboard is also OK, and the no-fan design is nice. It works well with a second 1080p external monitor also. I ordered the laptop directly from Microsoft: their "signature editions" come with no crap-ware; just Windows 8.1 and nothing else.
- Only 32 GB of solid state disk space. After installing Ruby, Java 8, IntelliJ, git command line tools, Pharo Smalltalk, Chrome, and cloning several git projects I have about 11 GB free.
- Slow CPU. For using Pharo Smalltalk this is no problem. It is a small problem for Clojure development. I use IntelliJ with the Clojure plugin for editing and have a lein repl running in a command window. Not too slow, but not great either.
I bought this laptop just to experiment/play with Windows 8.1 but I think that it will become my travelling laptop. It is very light and small to carry around. The build quality so far seems very good for a $200 laptop.
Saturday, January 03, 2015
I would like to wish everyone a Happy New Year!
I am not going to make new year technology predictions, but I do have a few comments on what technology I am using and what I expect to be using this year:
First, I am surprised at how much Microsoft technology I am using. This started last spring when for political reasons I wanted to stop using Dropbox (I did not like their appointment of Condoleezza Rice to their board of directors). At the time, I could not find a good Dropbox replacement but after getting a Microsoft BizSpark business development grant last fall, and very much liking the Azure services, I recently decided to take another look at Office 365. It turns out that for the same $100/year I paid Dropbox for a terabyte of storage, with Microsoft I now get for the same cost one terabyte of cloud storage per family member and everyone in my household also gets Office applications and use of the online Office tools (that run really well in Chrome with Chrome apps). The versions of Microsoft Word, OneNote, and OneDrive for my iPad and my Samsung Note 4 Android "phone" are also excellent. I bet that I end up being a long term customer of Azure and Office 365.
My BizSpark sponsored (with a generous free Azure usage tier for up to 3 years) business plan involves writing a web service that I won't go into right now, but I have already started replacing the Haskell + Ruby prototypes with Clojure code.
The other big change in my life, technology wise, is my Samsung Note 4 Android "phone." We have been listening to the hype surrounding a mobile device lifestyle for years, but for me having a very capable computing device with a very high resolution screen always with me has been a major improvement because I can always write, communicate, do photography, and in a pinch open a SSH window and write a little code or do some admin on a server. It is liberating to not always need to have a laptop with me.
Thursday, December 18, 2014
I want to wish everyone a happy holiday season!
Carol and I have been traveling a lot this year and we are staying home this Christmas, hanging out with our friends in Sedona Arizona instead of traveling to see family in California and Rhode Island.
I was going to buy a new laptop for my work and writing (my MacBook Air is 3 years old) but in spite of my laptop being dinged up it is otherwise in good shape. I decided to do something radical with it yesterday:
When I bought my laptop 3 years ago I initialized it with a Time Machine backup from my older MacBook Pro. Needless to say, there was a lot of cruft on my system. I was also running a developer's preview of Yosemite (with all available updates). Yesterday I reformatted the disk drive and did a fresh install of OS X Yosemite - without restoring anything from Time Machine.
I did manually (command line) restore several directories from the latest folder in the Time Machine backups to save the time of not doing fresh git clones of all of my projects, work and writing. I also decided to just install a minimal set of writing tools that I currently use (mostly I now use leanpub.com so all I need for writing is a plain text editor). For programming languages I only installed Haskell, Java 8, node.js, and Ruby.
Restoring OS X systems with Time Machine works very well, but it was time for a fresh start. Everything seems to run much faster and my SSD drive is 75% empty.
I am "sort of" retired now. I try to limit myself to about 10 hours a week of consulting and about 20 hours a week working on a commercial software product. Currently I am helping one customer integrate IBM Watson into their product and I am doing some development work for another customer (online sports game: Ember.js front end with a Haskell back end). Fun stuff!
With two friends I volunteer maintaining a historic farm owned by the US Forest Service. Part of this is keeping land clear of weeds and planting and harvesting food, and another part is keeping a 1 mile irrigation ditch (supplies water for the farm and a local park) clear. Tim took these two pictures of Don and I working on the irrigation ditch last week:
Saturday, December 06, 2014
Carol and I just got back from a cruise from San Diego to Hawaii to Ensenada Mexico and back to San Diego. We have done this cruise before but several people in our extended family wanted to go and we enjoyed the family time. The best part of this particular cruise is that you get about 10 "sea days" which I enjoy.
I took this picture from the Promenade Deck where I spent most of my time reading (and I hate to admit, hacking some Haskell NLP code):
Carol and I don't gamble but she received a birthday gift from the cruise line of some free slot machine time, as seen in this picture:
We had a lot of fun on shore also. Brother Ron, sister Anita, Carol, and I went snorkeling in Maui on a reef 2/3 mile offshore where we saw many green sea turtles. Here is a picture of me that my brother Ron took and a picture of a sea turtle that I took with Ron's camera:
I have a new phone, a Samsung Galaxy Note 4, that was great for the trip. It takes 4K ultra high definition video with video stabilization and great pictures. I put together an 11 minute video for my family that is really pretty good. The advantage of using a phone is that you always have it with you. The huge size of the Note 4 also made is great for reading Kindle books and listening to music and audio books. Now that I have the Note 4 I don't use my iPad mini very often. Except for 3 or 4 one hour Haskell hacking sprints, I never had to use my laptop; everything (email, web, reading, etc.) was done with my phone.
Friday, October 17, 2014
I originally wrote cookingspace.com in Ruby and Rails about 8 years ago, and a few years ago I rewrote it from scratch in Clojure.
This week I made some major improvements. First, I cleaned up some technical debt by rewriting cookingspace.com as a plain Compojure and Hiccup app, removing all reliance on the deprecated Noir library. I also did some major code cleanup.
I also rewrote the code for calculating and displaying the nutritional information for the recipes. The nutritional information used on this site is derived from the USDA Nutrition Database. Nutrition information is shown for each displayed recipe. This includes total percentage of minimum daily requirement and for each nutrient the recipe ingredients supplying most of the nutrient (ingredients providing less than 1% of contribution to daily requirement are not shown). The cookingspace.com system tracks 42 nutrients including most vitamins and minerals important for good health.
I originally wrote cookingspace.com to help me track the amount of vitamin K in my diet. I now have the relative vitamin K levels in common food memorized and it is easy to eat (approximately) the same amount or vitamin K each day. Mission accomplished! I now use cookingspace.com to get a better understanding of which recipe ingredients contribute good and "bad" nutrients.
Sunday, October 12, 2014
For a personal project I want to make a web app with a "rich client" interface. I had originally planned to write this app in Haskell with the Yesod web framework. However, as much as I like Haskell, I do still have occasional time wasting problems with cabal, Yesod, and sometimes with non-pure Haskell code. My gut feeling is that I will get things done faster if I use Clojure.
In the past I have experimented using Clojure and Ember.js but until today I have not spent much time with ClojureScript and Om (I have written web apps using ClojureScript, so the learning curve is trying to use Om).
Getting started with Om is straight forward. I used the chestnut lein plugin to create a new Clojure + ClojureScript + Om project. The chestnut plugin is very nice - it set up a reasonable development environment without having to go through a learning curve. After experimenting with the generated project, I then starting substituting in code from David Nolen's Om tutorial into the skeleton project that the chestnut lein plugin created.
I am not much of a user interface expert, and I am not sure how much time I will devote to learning Om. If I earned my living doing web apps, this learning curve would be very worthwhile. I usually use simple tools to make web apps like: Ruby + Sinatra, sometimes Rails, often Clojure + composure + hiccup.
Saturday, October 11, 2014
If you sign up for a (free for 30 days) account on IBM BlueMix, it is simple to use a pre-canned IBM Watson instance that contains medical information and travel information. Code samples are provided for Java, node.js, and Ruby. I wanted to use Ruby so I used these setup instructions.
IBM BlueMix uses the Cloud Foundry PaaS tools. If you have any experience using Cloud Foundry then setting up a (free for 30 days) BlueMix account, and deploying one of the sample web applications that uses the pre-canned IBM Watson medical and travel instance can be done in about 20 or 30 minutes. This is a worthwhile exercise because once you deploy your sample web app you can experiment with IBM Watson's ability to parse natural language questions and return relevant data. Very nice stuff!
In order to build a custom application using IBM Watson you need to supply training documents and training questions. I am helping a customer do this right now. Currently you need a partnership arrangement with IBM to train your own IBM Watson instance but I believe that the ability for anyone to do this via BlueMix will be available in the near future.