Tuesday, August 30, 2005

MySQL optimizations; free Opera browser license

OK, I know that there is a large overhead for doing single table inserts, even maintaining a pool of connections. This little fact was very obvious this morning: I had a one-shot little program that generates data for about 5 million rows in a database table. The calculations themselves take a while (about 5 minutes on my PC), ignoring database access. Last night, right before I went to bed, I added a few lines of ad-hoc JDBC calls to write out my 5 million rows after everything was calculated in memory. I woke up this morning and it was still running.

So much for being a little sloppy, even for a one-shot (throw it away after it runs) program. I re-wrote it batching up just 2000 inserts at a time (insert into... values (..), (..), (..)...) and the database access code ran in just 20 minutes. After the data was added, it took another two minutes to index the table (obviously you want to do this *after* adding the data). I made a note to myself to review my production code - I think that I almost always batch up inserts when appropriate, but I want to make doubly sure!

re: free Opera browser license: as seen on Slashdot: a short term free license giveaway. I grabbed a free license so I have one more popular browser to test my web applications against.

No comments: