I love both of my jobs (programming and writing) as long as I don't overdo it and take a lot of time off for other activities like hiking, kayaking, playing musical instruments, and cooking.
I have another "down time" activity that is both fun and relaxing for me: studying things that help me with my jobs. For example, when I first adopted Ruby as my primary scripting language and also started developing using Rails, I spent a lot of time reading through the C implementation of Ruby, the Ruby libraries, and the Rails source code. I find this kind of study relaxing because there are no deliverables and things learned studying the implementation of tools I use really pays off in increased productivity and learning new programming idioms and techniques. I used to base a lot of my work on the Tomcat server and ten years ago I made a real effort to understand its implementation. When I was very young I worked as a systems programmer and kept source listings of interesting parts of the operating system at home.
My "reading" activities today included two hours looking through some of the code in the EJB container sub-package of the Glassfish web application server. One of my customers runs much of their business on Glassfish so I am motivated to understand the platform. I learn things reading through Glassfish code that I never would reading books on EJB 3.1.
Time is our most precious possession and certainly it is not to be wasted. That said, spending at least several hours a month carefully studying the code in a few of the open source software tools I use is time well spent.