Planet Perrygeo

• ... find a Starbucks...
• ... we're releasing an API ...
• ... friends list ...

These events allow us the opportunity to engage in conversations and dialogs with various technologists because we want to gain feedback about the needs of open source developers and users. The objective of course is to channel this information back to development so we can reflect this in future products and business decisions in order to best support our customers.

i) Case:

i) If <point binary representation> immediately contains a <wkbpoint binary>, then <point binary representation> is the well-known binary representation for an ST_Point value that is produced by <wkbpoint binary>.

ii) Otherwise, <point binary representation> produces an empty set of type ST_Point

q) Case:

i) If <linestring binary representation> immediately contains <num>, then <linestring binary representation> is the well-known binary representation for an ST_LineString value. Let APA be an ST_Point ARRAY value with cardinality of <num> that contains the ST_Point values specified by the immediately contained <wkbpoint binary>s. <linestring binary representation> produces an ST_LineString value as the result of the value expression: NEW ST_LineString(APA).

ii) Otherwise, <linestring binary representation> produces an empty set of type ST_LineString.

Sidenote: As a result of WKB not having SRID support, it's not possible to round-trip a geometry through WKB without losing the SRID value. Try this standard SQL and see what happens:

SELECT ST_SRID( ST_GeomFromWKB( ST_AsBinary( ST_GeomFromText('POINT(0 0)', 4326 ) ) ) )

Then try the bastardized PostGIS EWKB format instead:

SELECT ST_SRID( ST_GeomFromEWKB( ST_AsEWKB( ST_GeomFromText( 'POINT(0 0)', 4326 ) ) ) )

(ED: Enabling comments might help.. sorry about that, should work now. TM)

I talk a fair bit with various folks about book ideas they or I have. I'm curious on your thoughts for potential new books that might be popular - what's the next big "geo" book that you think you, your colleagues, the world at large needs?

Where do these books leave off? Or what new waters need to be charted?

Hope to see you at the MapServer introduction workshop at FOSS4G 2010 in Barcelona this September.

As usual, it's a team teaching approach for beginners, delivered by Jeff McKenna, Perry Nacionales and myself.

This workshop always sells out and people beg to join in or stand at the back - so sign up soon or you might miss out.

See you there!

No, not lazy as in REST … Lazy as in “Lazy evaluation“:

In computer programming, lazy evaluation is the technique of delaying a computation until the result is required.

Take an example raster processing workflow to go from a bunch of tiled, latlong, GeoTiff digital elevation models to a single shaded relief GeoTiff in projected space:

1. Merge the tiles together
2. Reproject the merged DEM (using bilinear or cubic interpolation)
3. Generate the hillshade from the merged DEM

Simple enough to do with GDAL tools on the command line. Here’s the typical, process-as-you-go implementation:

1. gdal_merge.py -of GTiff -o srtm_merged.tif srtm_12_*.tif
2. gdalwarp -t_srs epsg:3310 -r bilinear -of GTiff srtm_merged.tif srtm_merged_3310.tif
3. gdaldem hillshade srtm_merged_3310.tif srtm_merged_3310_shade.tif -of GTiff

Alternately, we can simulate lazy evaluation by using GDAL Virtual Rasters (VRT) to perform the intermediate steps, only outputting the GeoTiff as the final step.

1. gdalbuildvrt srtm_merged.vrt srtm_12_0*.tif
2. gdalwarp -t_srs epsg:3310 -r bilinear -of VRT srtm_merged.vrt srtm_merged_3310.vrt
3. gdaldem hillshade srtm_merged_3310.vrt srtm_merged_3310_shade2.tif -of GTiff

So what’s the advantage to doing it the VRT way? They both produce exactly the same output raster. Lets compare:

The Lazy VRT method delays all the computationally-intensive processing until it is actually required. The intermediate files, instead of containing the raw raster output of the actual computation, are XML files which contain the instructions to get the desired output. This allows GDAL to do all the processing in one step (the final step #3). The total processing time is not significantly different between the two methods but in terms of the productivity of the GIS analyst, the VRT method is superior. Imagine working with datasets 1000x this size with many more steps - having to type the command, wait 2 hours, type the next, etc. would be a waste of human resources versus assembling the instructions into vrts then hitting the final processing step when you leave the office for a long weekend.

Additionaly, the VRT method produces only small intermediate xml files instead of having a potentially huge data management nightmare of shuffling around GB (or TB) of intermediate outputs! Plus those xml files serve as an excellent piece of metadata which describe the exact processing steps which you can refer to later or adapt to different datasets.

So next time you have a multi-step raster workflow, use the GDAL VRTs to your full advantage - you’ll save yourself time and disk space by being lazy.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

A few people heard me on This Week in Startups (starting at 15:45) asking Jason if we should take money from the first VC who fell into our laps, or spend time doing the Sand Hill Road rounds, meeting more VCs, and doing a road show for the other firms that might be interested in investing.

Jason (and his guest James Segil) both agree that we should take more time picking the right partner. We’re going to be in bed with these guys for years, they say, and we have to approach this like picking a spouse.

Anyway, people emailed me in shock and surprise that we would even consider VC, considering the things I’ve written.

Almost ten years ago, I wrote about two kinds of companies, the Ben and Jerry’s type of company, which grows carefully and organically, and the Amazon type, which uses huge amounts of investment capital to get big quickly.

At the time, I had no doubt that I wanted Fog Creek to be a Ben and Jerry’s type of company, and that model has served us well. By staying profitable and growing carefully, we’ve managed to survive two big downturns and we’ve grown into a stable, 34-person company that’s a great place to work and is likely to remain stable, and a great place to work, for a long time.

StackOverflow, though, is a bit of a different story.

There are a few indicators for the type of company that I believe can benefit from, and should take, VC.

1. There’s a land grab going on. The business is in a new field with no competition, but the field has proven itself, and is obviously going to get very crowded very soon, so the faster you can grab territory, the better.
2. There is a provable concept that’s repeatable. I always point to the example of the Starbucks IPO, which was brilliant because it was so simple. Every new Starbucks store that opened in Seattle became profitable in a matter of months. They tried a couple of stores in Chicago and Washington just to make sure it wasn’t a Seattle thing, and those worked even better. Thus, the formula of opening as many stores as possible was as close to a sure-thing as possible, so raising money was a no-brainer.
3. The business itself could benefit from the publicity of getting an investment from someone who is thought of as being a savvy investor.
4. The investor will add substantial value to the business in advice, connections, and introductions.
5. The business can potentially have a big exit or become a large, publically traded company.
6. The founders are not in it for their own personal aggrandizement and are happy to give up some control to make the business more successful.

There are counter-indicators, of course: signs that you shouldn’t consider VC. Here are just a few off the top of my head:

1. If the founders are risk-averse and are willing to trade a much smaller payout for lower risk.
2. If the founders are technical without substantial business experience and wish to maintain absolute control forever.
3. If the investor is mostly “dumb money,” i.e., someone who doesn’t know about the field. The proverbial dentist, who is happy to give you a half million bucks, but doesn’t know the first thing about CPMs and CPCs and CTOs, so you might as well not bother.
4. If you’re going into an established field with a lot of competition, there’s no benefit to speed; you’re better off slowly building a niche business and growing from there, quietly taking one customer at a time away from the competitors.
5. If the product is immature and unproven, in which case, expensive marketing efforts will be wasted proving to the world how bad your product is.
6. If the founders don’t have enough of the right kinds of industry connections, or the idea is not compelling enough, so that raising VC would take months or years
7. If there is any other way to raise the kind of money you need, for example, by selling actual products to customers.

When I put all these things together the conclusion is that StackOverflow is one of those rare companies for which VC can really work. Jeff and I started out with a goal for StackOverflow of changing the way programmers and system administrators get answers to their questions on the Internet, which was deeply broken. In 18 months we’ve accomplish that: we’ve got 6 million unique visitors every month. Now we’re biting off the bigger goal of changing the way everyone gets answers to their questions on the Internet, and that’s something we can’t do alone.

So, off I go, on a StackOverflow road show. I’ll be in Silicon Valley Feb 24-Mar 3; drop me a line if you want to get together.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

I just spent some time figuring out how to set up denyhosts on Snow Leopard. I’ve used denyhosts before, but never felt like I had things set up properly for Mac OS. Now I think I have it figure out, so here it is. This is for 10.6, your mileage may vary on earlier versions.

I had three goals – get denyhosts working, get it to start automatically at boot time, deal with rotating the logs (a general problem I’m happy to have figured out).

Installation

Easiest first – installing denyhosts. Note that you need to be root to do this. Pretty much just follow the directions. These are the three main settings to worry about.

SECURE_LOG = /private/var/log/secure.log
LOCK_FILE = /var/run/denyhosts.pid
DAEMON_LOG = /var/log/denyhosts

Log rotation

Mac OS 10.6 uses newsyslog for some logs (I’m not sure why, but apache logs don’t seem to be dealt with by newsyslog). To add your own to the mix, just put a file into /etc/newsyslog.d following the format for newsyslog.conf(5). I called mine local.conf

# logfilename          [owner:group]            mode count size when  flags [/pid_file] [sig_num]
/var/log/denyhosts                              640   5     *    $D0     J
#

The trouble is, this rotated the log just fine, but then denyhosts stopped logging because newsyslog essentially pulls the rug out from under denyhosts by moving the file. The fix for this leads me to the start at boot time section, because the same tweaking dealt with both.

One design difference between newsyslog and logrotate is the way they deal with notifying processes that logs have been rotated. Logrotate uses prerotate and postrotate scripts, which would be ideal for denyhosts. The way you start and stop it is with

daemon-control start
 
daemon-control stop

daemon-control

stop actually sends a

SIGTERM

to the denyhosts process, but that won’t do any good in the newsyslog config file since once stopped, you need a command line to start it.

Start at boot time

# logfilename          [owner:group]            mode count size when  flags [/pid_file] [sig_num]
/var/log/denyhosts                              640   5     *    $D0     J  /var/run/denyhosts.pid 15
#

In the early days of a technology startup, you tend to have a lot of software developers, and you feel like you could never have enough. If you hire sales and marketing staff too early, they don’t really get much traction, and you may start to think that sales and marketing are a waste of time. This led me, in the early years, to believe that a healthy software company should have a lot of real software developers and maybe no sales and marketing.

At one point I entertained the quixotic and, retrospectively, stupid idea of requiring every employee at Fog Creek to be a programmer... even the receptionist would have to have done some BASIC programming in high school to qualify. In the US Marines everyone, even the cooks, is a rifleman. Of course that’s because the cooks are in friggin Afghanistan getting shot at so they better be riflemen, whereas our receptionist almost never has to drop into the source code and bang out a class. Almost never.

Over time, though, as your product gets better and better, the more sales and marketing people you hire, the more you sell.

That’s because programmers multiply salespeople, and vice-versa.

I’m a nerd, so I’ll be real math-y about this. Define the quality of your code on a scale from 0 to 1.

0 means your product solves absolutely no problems for anybody so nobody in their right mind would ever buy it. Microsoft Bob.

1 means that every single person on Earth, if they bought your software, would be net happier, even after paying your fee.

Your software starts at 0 and slowly climbs up the hill.

If everybody in the world knew about your software and was encouraged to evaluate it, the number that would buy it would be (Earth population) x Quality.

Sales and marketing functions exist to encourage earthlings to find out about your software and evaluate it. These functions will have no effect on sales if your quality is extremely low. But as the quality gets higher, the value of sales and marketing goes up, commensurately. Double the quality, and the same sales effort yields double the revenue.

The population of the planet is so large, and the effect of sales and marketing so hard to scale, that by the time your product is really great, the optimal ratio might be very heavily tilted in favor of sales and marketing. Large software companies might have 5 or 10 or 20 people in the sales organization for every developer.

This explains, among other things, why US software companies can’t expect to get sustainable advantage by offshoring software development to cheaper countries. If a developer in Russia, India, or China costs 50% as much as a developer in Seattle, San Francisco, or Boston, but software development is only 10% of your costs, you can only get a 5% advantage from offshoring development. The offshoring that does happen is strongly biased to custom software development which, by design, can only solve one person’s problem, so more developers than marketers are needed.

It is not the case (as commonly believed by nerds) that marketing is a substitute for code quality. The best marketing in the world cannot force people to pay for a useless product.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

KML is linked. It is self-descriptive, and can rely entirely on following of links to obtain more information, whether that is styles or additional data.

However, the most common way of packaging KML is as KMZ — which is sort of like packaging an HTML page inside a zip file with all of its component parts. When this is done, web-based tools — like the Javascript support in browsers — lose all access to the data other than through a server side proxy (and even that isn’t a trivial thing to achieve). Styling information and related parts are not stored as separate resources on the web. The information available in the KML has suddenly become just another application-specific format.

If this were uncommon, it wouldn’t be such a shame; it’s certainly possible to distribute data like this for use cases where it is necessary, including offline use and other use cases. However, this is not a limited situation — in fact, more than 80% of KML made available on the web tends to be primarily available as KMZ. This packaging of KML leaves much to be desired, and limits the use of such data in web-based tools.

The web already has ways to compress data — gzip-based compression is common on many web servers (a tradeoff of CPU time for bandwidth), and works fine in all KML clients I’m aware of (including Google Earth and Google Maps). This lets your data exist on the web of resources and documents, rather than in a zipped up bundle.

My interest in this matter should be obvious: I work with mapping on the web. Ideally, I work with tools that don’t require server-side code — every piece of server side code you have to build is another heavy requirements placed on the users of any software. Browsers, as a common platform across which developers can code, are a worthwhile target, and trapping your data in KMZ hides it from browsers.

Free your KML! Publish on the Web! Don’t use KMZ!

One of the most difficult thigns to do in time of disaster is to quickly organize, marshal, and present resources. This applies across all aspects of disaster response — whether it be managing and distributing food, organizing volunteers, or setting up technical resources to assist with the relief effort.

The last is the field I obviously have the most experience/ability to help with, especially with regard to mapping. In past situations, I have put some of my map expertise to work in helping to create a resource for the disaster; the last significant case for me was in 2007, when I managed a ton of imagery made available as part of the efforts with regard to the San Diego wildfires. (That map is still available, though it’s a bit worse for the wear at this point.)

When the Haiti Crisis happened, I let it slide; I figured that someone else would step up to manage the data this time. After a while, though, I saw an increased number of imagery sources, and little coherent organization of the resources by a single party — one of the key things that made the 2007 fires map successful. As a result, and combined with some data that was being more narrowly published, I decided to set up a map. The first day I did any significant work on this was over the weekend of the 15th.

At first, the map wasn’t particularly great; it was primarily just a tool to view a bunch of satellite data that was being made available. This was primarily just a quality control check for users of OSM who needed access to the data to complete the map of Haiti. Over time, more data became available — and more importantly, the OpenStreetMap map data became a primary map for the area and rescue efforts. Suddenly, the Haiti Crisis Map — then just the “UAV map” — was being used more and more.

As more and more data became available, the old map, using a simple OpenLayers layer switcher, became unwieldy; never a user-friendly layout to begin with, adding 20 layers to an OpenLayers map with an unplanned mix of base and overlay layers leaves much to be desired.

By Wednesday, it was clear that the hodge-podge of available disk space attached to the hosting machine wasn’t going to cut it; though we started with just over 4TB available spread over 3 different drives, managing the data was becoming unwieldy at the same rate as the UI. Thankfully, by Wednesday the 20th, John Graham was able to get access to another Sun X4500 and set it up, giving us a clean 16TB drive to put new and old imagery on. (About 6 hours later, the NFS machine to which all of the current data was stored began to fail, most likely due to heavier than normal load on the machine; I spent most of that day moving data off the old drive and onto the new.)

In addition to the data migration, at this time, Aaron Racicot was able to step up and offer his help in building a GeoExt based UI for the map. His efforts turned my hack into a reasonable UI for browsing the map, and it is really only because of that that I was able to keep going.

Over the weekend, at CrisisCamp, I was able to add additional features to support Ushahidi; the code was moved into Github, haitibrowser. In the middle of this week, the code was integrated into APAN, the All Partners Access Network, to support the efforts of SOUTHCOM in maintaining a high quality Central Operating Picture of events in the area.

Over the past two weeks, data has continued to pour in, in the hundreds of gigabytes a day. This is in part thanks to the wonderful availability of imagery thanks to the generosity of the commercial providers, in addition to the data made available by organizations like NOAA, companies like Google, and more. The extremely high quality imagery produced by RIT/ImageCat/WorldBank, for example, is an example of what is possible with the hard work of people with great hardware and a great team.

Using my knowledge — gleaned from my efforts in the earlier days of OpenAerialMap — I have been able to process this data and make it available as tiles and WMS to all consumers, primarily targeted towards OpenStreetMap editors. Over two dozene layers are available via what is now called the Haiti Crisis Map, each one adding a different viewpoint of data. In addition, the map contains links to other files like KML collections from Ushahidi and Sahana, and as recently as yesterday, gained the ability to create your own layers, which you can access in the map and provide as a link to someone else, as well as export as KML.

As part of the process of making the site more readily available, it is now available from haiticrisismap.org.

The most difficult part of this is attempting to manage the large sources of data. Thankfully, the resources that I have available have allowed me to be a bit lax in my conservation of disk space, CPU time, etc. Many thanks to CalIT, SDSU/SDSC, and Telascience for organizing these resources. In addition, a lot of the ‘hard work’ in the UI has been done by Aaron Racicot of Z-Pulley. I’ve done a lot of minor work, but the major UI layout and work has been done by him.

Thankfully, I’ve had the support of a lot of good people in this effort, and a lot of good tools to use. Using GDAL + OSSIM in the background for image processing, MapServer + TileCache for mosaicing and serving, OpenLayers + GeoExt for a UI, and OSM for a base map data layer have all made this effort possible.

The haiticrisismap will continue to see improvements. It shows a lot about what a dedicated small group of people can do with an investment when properly motivated; I can honestly say that because of the resources made available through these efforts, we have saved lives. Whether it is through maps produced through OSM being loaded onto Volunteer GPS systems, or the use of the data to determine an accurate location in a map by Ushahidi volunteers, this tool has been an effective aid to the relief effort in Haiti, and will continue to do so as much as is possible in the coming days and weeks.

My sister got her kids a little puppy, and they’ve been trying to train it. To live with a dog in the house, you need to teach it not to jump on people, not to poop in the house, to sit on command, and to never, ever, ever chew on the iPad. Never. Good girl.

With dogs the main trick to training is that feedback has to be immediate. If you come home to discover that, hours before, the dog tipped over the garbage can in the kitchen, it’s too late for training. You can yell at her but she just won’t get what you’re going on about. Dogs are just not that smart.

For programmers, getting better at what you do requires quick feedback, positive and negative, on what you’ve just done. The faster you get the feedback, the faster you’ll learn. With long-cycle shrinkwrap software, it can take a year or more to hear feedback from customers.

That’s one of the reasons we have testers. A great tester gives programmers immediate feedback on what they did right and what they did wrong. Believe it or not, one of the most valuable features of a tester is providing positive reinforcement. There is no better way to improve a programmer’s morale, happiness, and subjective sense of well-being than a La Marzocco Linea espresso machine to have dedicated testers who get frequent releases from the developers, try them out, and give negative and positive feedback. Otherwise it’s depressing to be a programmer. Here I am, typing away, writing all this awesome code, and nobody cares. Boo hoo.

Who should be a tester? That’s tricky! Software testing is one of those careers that isn’t that well known, so a lot of people who would be great at testing and would probably enjoy it a lot never consider applying for jobs as testers.

Signs of a good tester:

• Scientific
• Loves a good puzzle, even the kind that takes days to solve
• Likes to think about things methodically
• Generally likes working with software and computers

You don’t have to be a programmer to be a tester. A lot of companies want testers to be programmers who write automated test suites. It seems more efficient that way. This reflects a misunderstanding of what testers are supposed to do, which is evaluate new code, find the good things, find the bad things, and give positive and negative reinforcement to the developers. Sure, automated test suites are a time saver, but testing software covers so much more than that. If you put too much emphasis on those scripts, you won’t notice misaligned text, hostile user interfaces, bad color choices, and inconsistency. Worse, you’ll have a culture of testers frantically working to get their own code working, which crowds out what you need them to do: evaluate someone else’s code.

A particularly terrible idea is to offer testing jobs to the programmers who apply for jobs at your company and aren’t good enough to be programmers. Testers don’t have to be programmers, but if you spend long enough acting like a tester is just an incompetent programmer, eventually you’re building a team of incompetent programmers, not a team of competent testers. Since testing can be taught on the job, but general intelligence can’t, you really need very smart people as testers, even if they don’t have relevant experience. Many of the best testers I’ve worked with didn’t even realize they wanted to be testers until someone offered them the job.

If you:

• Love software and computers
• Want to work on a software team, and
• Don’t particularly like programming

you should consider being a tester. (We’re hiring! What a coincidence!)

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

Steve Krug has written a follow up to his usability classic Don’t Make Me Think. The sequel, Rocket Surgery Made Easy, is a terrific, short, concise, fun guide to running simple “hallway” usability tests to improve the usability of your software and websites. Highly recommended.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

“As companies expand, the people within them start to specialize. At such a point, some managers will conclude that they have a ‘keep everyone on the same page’ problem. But often what they actually have is a ‘stop people from meddling when there are already enough smart people working on something’ problem.”

From my latest Inc. column: A Little Less Conversation

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

Microsoft Careers: “If you’re looking for a new role where you’ll focus on one of the biggest issues that is top of mind for KT and Steve B in ‘Compete’, build a complete left to right understanding of the subsidiary, have a large amount of executive exposure, build and manage the activities of a v-team of 13 district Linux& Open Office Compete Leads, and develop a broad set of marketing skills and report to a management team committed to development and recognized for high WHI this is the position for you!”

This is ironic, to use the Alanis Morissette meaning of the word [NSFW video].

The whole reason Microsoft even needs a v-team of 13, um, “V DASHES” to compete against Open Office is that they’ve become so insular that their job postings are full of incomprehensible jargon and acronyms which nobody outside the company can understand. With 93,000 employees, nobody ever talks to anyone outside the company, so it's no surprise they've become a bizarre borg of "KT", "Steve B", "v-team", "high WHI," CSI, GM, BG, BMO (bowel movements?) and whatnot.

When I worked at Microsoft almost two decades ago we made fun of IBM for having a different word for everything. Everybody said, "Hard Drive," IBM said "Fixed Disk." Everybody said, "PC," IBM said "Workstation." IBM must have had whole departments of people just to FACT CHECK the pages in their manuals which said, "This page intentionally left blank."

Now when you talk to anyone who has been at Microsoft for more than a week you can’t understand a word they’re saying. Which is OK, you can never understand geeks. But at Microsoft you can’t even understand the marketing people, and, what’s worse, they don’t seem to know that they’re speaking in their own special language, understood only to them.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

Been a while.  But I am back.  Doing alot of Python, Javascript, Geo-Hacking and plan to dump my brain about it all soon.  Stay tuned

I had previously bitched and moaned about the licensing restrictions on the TrainingPeaks WKO+ software. Truth be told, the reason I was so put off by their crappy licensing scheme was that my cycling training relied so heavily on their software. It was not perfect but it was the best tool available. I’ve since discovered Golden Cheetah which is a viable open-source alternative but it still lags behind WKO+ in many critical features.

Now, fresh in time for the 2010 training season, Peaksware has released a new version 3.0 of WKO+ which, amongst many UI and functionality improvements, has made considerable progress on the licensing front.

We know, our licensing has been a challenge to deal with for our customers in the past, but we’ve always tried to be as helpful as possible getting you back up and running after a hard drive crash or new computer. To remedy this, we’re pleased to announce an all new flexible licensing system. First, with every purchase we now allow you to install WKO+ 3.0 on up to two computers; second, we’ve built an online activation/deactivation system so you are free to move your active licenses from machine to machine. Are you leaving on a 2 week trip? Just de-activate your home computer, activate your laptop, and you’re on your way. When you get home, de-actiavate your laptop, re-activate your desktop and you’re all set.

It ain’t open source (there is still a place in this world for proprietary software if they can push the boundaries and innovate) but the sensitivity to the licensing issue just may have restored my faith in their company.

Database client drivers intended for the same database can do drastically different things. By Python standards, the Postgres driver situation is completely schizo. There are a lot of them available - there are five dedicated Postgres drivers listed on the wiki, as opposed to just one for MySQL. People might choose different drivers for licensing reasons, for religious reasons, randomly (because they never did any analysis like I am about to do), or for completely inscrutable reasons because they are just plain out of their minds. You really would not believe how much blood I have seen spilled over Postgres client drivers.

Fix incorrect logic for GiST index page splits, when the split depends on a non-first column of the index (Paul Ramsey)

Is your desktop backed up?

Did you backup that server?

Are your backups on a different machine?

Do you have offsite backups?

All good questions, all best practices.

But let’s stop talking about “backups.” Doing a backup is too low a bar. Any experienced system administrator will tell you that they have a great backup plan, the trouble comes when you have to restore.

And that’s when you discover that:

• The backed-up files were encrypted with a cryptographically-secure key, the only copy of which was on the machine that was lost
• The server had enormous amounts of configuration information stored in the IIS metabase which wasn’t backed up
• The backup files were being copied to a FAT partition and were silently being truncated to 2GB
• Your backups were on an LTO drive which was lost with the data center, and you can’t get another LTO drive for three days
• And a million other things that can go wrong even when you “have” “backups.”

The minimum bar for a reliable service is not that you have done a backup, but that you have done a restore. If you’re running a web service, you need to be able to show me that you can build a reasonably recent copy of the entire site, in a reasonable amount of time, on a new server or servers without ever accessing anything that was in the original data center. The bar is that you’ve done a restore.

Let’s stop asking people if they’re doing backups, and start asking if they’re doing restores.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

The higher someone’s Stack Overflow reputation, the more likely they are to have submitted a CV to Stack Overflow Careers:

This is not entirely surprising, of course: the more time someone has invested in Stack Overflow, the more likely they are to (a) know about Stack Overflow Careers, (b) be willing to invest $29, after all the hours they’ve already sunk, and (c) have the confidence that their CV is going to impress the kind of employers that are using the site.

Still, the participation rate in Stack Overflow Careers is pretty impressive, and it somewhat confirms the claim we’re making to employers, which is that when you search for CVs on Stack Overflow, you are looking at some pretty gosh darn good programmers.

While I’m rattling on about statistics, here’s a little bit of data about Stack Overflow traffic itself that you may not have seen.

We use Quantcast to measure our traffic. Currently, they’re showing us as the 740th ranked site in the world (of all sites), with 6 million monthly unique visitors, 1.9 million from the US. And the growth is pretty steady, except for a couple of weeks at the end there which reflect the holiday season:

Comparing our traffic to our big competitor is difficult because they don’t use Quantcast, so we have to rely on Alexa, which has a reputation for particularly terrible data, but here’s what that looks like:

Are there any sites out there for programmers with more traffic than Stack Overflow? I haven’t found any, using the available data... even msdn.microsoft.com has less, according to Quantcast, but I find that hard to believe.

In either case, having decided that Stack Overflow was the biggest programming site in the world, I thought, “hey, it should be easier for us to get ads.” I asked our ad guy, Alex “DailyWTF” Papadimoulis, if Microsoft had bought any ads. They’re about to launch Visual Studio 2010, which is probably going to have the biggest marketing campaign (in dollars) in the history of developer tools, and you’d think they’d want to spend something at the biggest programming site in the world. Here’s what he wrote back:

“Microsoft is doing huge spends, but they’re going through McCann for the VS2010 launch (IIRC). Agencies really don’t like us. Now if we go to video units with fly-over… oh they’ll start loving us!”

What he’s referring to is the fact that we don’t accept any kind of animated ads on Stack Overflow, because, well, they’re evil, so we lose a lot of revenue from advertising agencies who are looking for the most aggressive possible ways to get in people’s faces. Whatever. Don’t care. We hate animated ads and I’m pretty sure our users do, too.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

This is a truly spectacular piece of work. We have often been asked by clients to buffer and merge point datasets with several million points. We attempted this using ArcWhatever (could barely open the points, let along buffer them) and FME, which ran for a week and then gave an out of memory error. So, I do the whole configure, make, make install thing, 4 times, for postgres, goes, proj4 and postgis. After a lot of swearing and running ldconfig a few million times I eventually get postgis to accept that geos really is installed -- MySQL might have more limited spatial functionality, but it sure is a lot easier to build from source. Anyway, I digress. I run a few random queries using the excellent generate series capability in postgres, and manage to create, buffer and merge 100,000 points in a few seconds. Finally, I try this on a real world dataset, namely all of the postal addresses in Wales, 1.4 million or so. With a 200m buffer, this ran on a reasonably pokey 64-bit linux box in 19 minutes. Truly astonishing. Well done. Much as I love MySQL, this was a bit of St. Paul on the road to Damascus moment.

“Like most entrepreneurs, Ryan and I are still learning about how to manage people and teams. And we’re both used to hiring very smart and dedicated people who will get things done to a high standard if you give them some general direction and set them free. But on this trip, we started to notice that this style of hands-off management, which works so well with our own staffs, just wasn’t working when we had outside vendors involved.”

From my December column in Inc.: “When and How to Micromanage”

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

For as long as I’ve been in the industry, which is, I think, about 74 years now, the problem I’ve had with hiring programmers was not interviewing them or deciding if they’re smart—it’s been finding them in the first place.

What I’ve dreamed about is a programmer search engine.

The ideal programmer search engine would only include programmers who are actually looking for jobs. If you’ve ever emailed someone based on a resume you found through a traditional search engine, you’ve probably discovered that they’re not actually on the market.

It would only include people willing to work in your neck of the woods.

It would show you CVs right away, and, ideally, it would show you something about their programming skills besides the usual resume blahblah.

Well, OK, that day is here, and I’m like a kid in a candy store. Nom nom. Announcing the other half of careers.stackoverflow.com: the employer’s side!

Right now, there are about 928 candidates on there. That’s a start. What’s more interesting is whether there’s a candidate who meets your needs.

Let’s say you’re searching for a full time Java programmer within 40 miles of Palo Alto. Right now there are 11 candidates listed. All but one are active on StackOverflow... one even has reputation over 4000 points.

Want a bit more choice? Check the box that indicates that you’re willing to relocate. Now there are 80 matches, all of whom have the legal right to work in the states. Candidates have a lot of flexibility indicating where they’re willing to work. Even if you need a Ruby on Rails programmer in Oklahoma City, as long as you’re willing to pay for relocation, you’ve got 7 choices. You’ve got 14 choices in London (with the legal right to work.) If you think that a Python programmer could learn Ruby, you’ve got 51 choices. There are plenty of choices whether you’re hiring in Tel Aviv, Sydney, Silicon Valley, or New York. There are four programmers in Copenhagen right now. No relocation required. All of them highly qualified, actually; any one of them would qualify to interview at Fog Creek.

Stack Overflow Careers is something of a chicken-and-egg business. We have to get a big audience of programmers and a big audience of employers all at the same time, and then it’s like a junior high school dance, with the boys on one side of the gym and the girls on the other side, and for a while you just sit there holding your breath to see if anyone will dance. We invited a few hundred employers as beta testers... these were the companies that have been listing jobs on StackOverflow over the last six months, and so far, they’ve found a few dozen candidates that they liked. Once it gets to that point, we’re out of the loop, so we don’t really know how many people are actually finding jobs, but please email me your success stories and failure stories so we can keep working to make it better.

In the meantime, Jeff and the StackOverflow crew have done something brilliant: they’ve made it possible to do searches and see how many candidates match even before you have to pay. So if you want to try it out but are afraid that there aren’t students looking for OCaml internships in Houston, you can try it, and find that there is, indeed, one. So, try it out right now. There’s no obligation, and we’re happy to give you your money back if you don’t think you got good value.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

In an address aimed at rank-and-file Americans, Mr. Obama did not call for sacrifices.

You get what you pay for.

Note: If you post any specific 9.4 Beta information (such as quoting forums posts on the Beta forums), expect ESRI to personally contact you. They appear to be monitoring this blog post. You’ve been warned.

Do you like your job?

Do you enjoy the people you work with?

Would you want to have lunch with them? Every day? Alex Papadimoulis thinks that Fog

Tyler Griffin Hicks-Wright Creek’s free lunches are “cultish,” but everyone at Fog Creek loves them. Maybe it’s the mandatory brain implant we install in each new worker, but I like to think that we just enjoy eating together because we genuinely like each other and like spending time together. If you can’t imagine eating lunch every day with your coworkers, I hate to break it to you: you might not like them. Is it OK to spend most of your waking hours with people you don’t like?

Do you actually enjoy doing your job? If you wake up an hour early in the morning, do you think, “Yay! I can go in early and get another hour of work in!” Or does that sound ridiculous to you?

Are you learning? When was the last time you had to learn a new skill? Is this year kind of like last year, or are you doing something new, stretching yourself, challenging yourself to be better?

At one of the recent DevDays events, I asked the audience (almost 100% programmers) how many of them were incredibly satisfied with their job, found it fulfilling, and were treated well by their employers. Only about 25% of the hands went up. I asked how many people either hated their job and couldn’t wait to find something better, or were actually actively on the job market. Again, about 25%. The rest were somewhere in the middle: maybe they can tolerate their job, but they’re keeping an eye open for something better.

Who is this DevDays audience? They’re the elite of the elite of the best programmers out there. They’re the people who participate in Stack Overflow, the people who read, the people who are constantly trying to learn more about programming and software development. More than half of them paid their own money to attend a one day conference. They’re the most desirable software developers on the planet. And 75% of them are not delighted with their job.

That’s unacceptable. I’ve been saying for ten years that the top developers have a choice of where to work, and the top employers need to work harder to attract them, because the top developers get ten times as much work done as the average developers.

And yet, I still keep meeting ridiculously productive developers working in shitholes.

We’re going to fix this, right now. Thus, Stack Overflow Careers.

We’re going to completely turn the job market upside down, for the best software developers and the best companies.

This is a talent market. Developers are not even remotely interchangeable. Therefore, recruiting should work like Hollywood, not like union hiring halls of the last century.

In a union hiring hall, downtrodden workers line up like cogs, hoping to make it to the front of the line in time to get a few bucks for dinner.

In Hollywood, studios who need talent browse through portfolios, find two or three possible candidates, and make them great offers. And then they all try to outdo each other providing plush work environments and great benefits.

Here’s how Stack Overflow Careers will work. Instead of job seekers browsing through job listings, the employers will browse through the CVs of experienced developers.

Instead of deciding you hate your job and going out to find a better one, you’ll just keep your CV on file at Stack Overflow and you’ll get contacted by employers.

Instead of submitting a resume, you’ll fill out a CV, which links back to your Stack Overflow account, so that you can demonstrate your reputation in the community and show us all how smart you really are. To a hiring manager, the fact that you took the time to help a fellow programmer with a detailed answer in some obscure corner of programming knowledge, and demonstrated mastery, is a lot more relevant than the Latin Club you joined in school.

Employers can see how good you are at communicating, how well you explain things, how well you understand the tools that you’re using, and generally, if you’re a great developer or not. And they can see your peer reputation, so all that hard work you’ve been putting into helping people on Stack Overflow can karmically come back and help you upgrade your job to the latest, state-of-the-art, great place to work.

Stack Overflow has grown incredibly fast. After a year in business, it gets over a million page views most weekdays and currently stands as the 817th largest site on the Internet, according to Quantcast. It reaches 5.2 million people a month. But Stack Overflow Careers doesn’t have to be massive. It’s not for the 5.2 million people who visit Stack Overflow; it’s for the top 25,000 developers who participate actively. It’s not for every employer; it’s for the few that treat developers well and offer a place to work that’s genuinely fulfilling.

Read the FAQ, then go file your CV now, and upgrade your career.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

My new Inc. column is up. “For a guy who wrote a book on how to hire great programmers, it’s mortifying how incompetent I’ve been at enlarging the sales team, which, right now, consists of one terrific account executive and a dog. (I’m just kidding. There’s no dog.)”

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

What is your company about?

Recently I got inspired by Kathy Sierra, whose blog Creating Passionate Users and Head First series of books revolutionized developer education. She kept saying the same thing again and again: help your users be awesome.

Kathy taught me that if you can’t explain your mission in the form, “We help $TYPE_OF_PERSON be awesome at $THING,” you are not going to have passionate users. What’s your tagline? Can you fit it into that template?

It took us nine years, but we finally worked out what Fog Creek Software is all about, which I’ll tell you in a moment, but first, some backstory.

In the early days, we were all about making a great place to be a software developer in New York City.

Yep, that was all there was to it. Almost every software job in the city was terrible. You had a choice of which kind of terrible. Want to wear a suit and work long hours under crummy conditions? Take a job at a bank. Want to report to a manic-depressive creative who demands that you stretch HTML in ways that would have you put to death, in certain countries? Take a job at a media company. Want to work 24/7 in a basement with water pipes dripping on your head and get paid in worthless stock options? Take your pick of the revenue-free dotcom startups.

Why New York, then? There are lots of great product companies where software developers are treated very well in Redmond, Washington. But I was sick of trying to live in lesser cities. Sure, the Seattle area is beautiful, and green, and clean, and possesses great coffee, and I understand that there are even a couple of grocery stores open late now. But I’m staying in New York, because it’s the greatest city in the world.

I gave up the search, and decided to start a company with my buddy Michael Pryor. Making a nice place to work was our primary objective. We had private offices, flew first class, worked 40 hour weeks, and bought people lunch, Aeron chairs, and top of the line computers. We shared our ingenious formula with the world:

The tagline was “building the company where the best software developers want to work.” It was, to say the least, awkward. It didn’t make for a good elevator pitch. It didn’t really have the right format. “Abercrombie and Fitch: building the apparel store where the hottest teenagers will want to work.” Who cares? Not the hot teenagers, I’ll tell you that.

Anyway we accomplished that goal. Cross it off the list. What’s next? We needed a new mission statement.

And it has to be something of the form, “We help $TYPE_OF_PERSON be awesome at $THING.”

Bells went off. Everything we’ve done successfully has one thing in common: It’s all about helping software developers be awesome at making software.

That includes Joel on Software, Stack Overflow, all the books I’ve been writing, the conferences like DevDays and Business of Software, the Jobs Board and Stack Overflow Careers.

It includes our flagship product, FogBugz, which is all about giving developers tools that gently guide them from good to great. It’s the software implementation of the philosophy I’ve been writing about for a decade, lacking only one thing: the feature to replace exceptions with return values, while adding Hungarian prefixes to all variable names. THAT IS A JOKE, PEEPLE. Put DOWN the bazooka.

Helping you make more awesome software is why I write endlessly about what we’re doing at Fog Creek, despite the fact that people accuse me of shilling. I’m not writing to promote our products. You don’t have to buy our products to get the benefit of reading about my experience designing them and building them and selling them. I’m writing to share some of my experiences in case they can help you make better software.

Our focus on helping developers explains why one of our early products, CityDesk, flopped: it had nothing to do with software developers. And it explains why another of our products, Fog Creek Copilot, only found a market in the niche of software developers doing tech support.

So, here you go, the new tagline: “We help the world’s best developers make better software.”

Going through this exercise made it easy to figure out what belongs in future versions of FogBugz and what doesn’t. In particular, we’re adding source control and code review features to FogBugz, using Mercurial, the best open-source distributed version control system. Everything that helps developers make better software belongs in FogBugz: project planning, project management, bug tracking, and customer service.

It took almost ten years, but I think we finally got the mission for the next ten nailed.

Optional Advertainment: If you’ve got a moment, check out this 4½ minute trailer for Make Better Software, a new video training series we’ve been working on for more than a year. It’s the video edition of Joel on Software and fits perfectly with our agenda of helping developers make great software.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

It is amazing how easy it is to sail through a Computer Science degree from a top university without ever learning the basic tools of software developers, without ever working on a team, and without ever taking a course for which you don’t get an automatic F for collaborating. Many CS departments are trapped in the 1980s, teaching the same old curriculum that has by now become completely divorced from the reality of modern software development.

Where are students supposed to learn about version control, bug tracking, working on teams, scheduling, estimating, debugging, usability testing, and documentation? Where do they learn to write a program longer than 20 lines?

Many universities have managed to convince themselves that the more irrelevant the curriculum is to the real world, the more elite they are. It’s the liberal arts way. Leave it to the technical vocational institutes, the red-brick universities, and the lesser schools endowed with many compass points (“University of Northern Southwest Florida”) to actually produce programmers. The Ivy Leagues of the world want to teach linear algebra and theories of computation and Haskell programming, and all the striver CS departments trying to raise their standards are doing so by eliminating anything practical from the curriculum in favor of more theory.

Now, don’t get me wrong, this isn’t necessarily a bad thing. At least they’re replacing Java with Scheme, if only because “that’s what MIT does.” (Too late!) And they are teaching students to think a certain way. And given how much the average CS professor knows about real-world software engineering, I think I’d rather have kids learn that stuff at an internship at Fog Creek.

Greg Wilson, an assistant professor at the University of Toronto, gave a talk at the StackOverflow DevDay conference in Toronto, which was entertaining, informative, and generally just a huge hit. We got to talking, and he told me about his latest brainchild, UCOSP, which stands for “All The Good Names Are Taken.”

It’s a consortium of 15 universities, mostly in Canada, which are organizing joint senior-year capstone projects. They’re setting up teams of a half-dozen undergraduates from assorted universities to collaborate on contributing to an open source project, for credit and for a grade. As soon as I heard about the program I volunteered to sponsor a team to make a contribution to Mercurial. Sponsoring a team consists of offering to pay for a trip to Toronto for all the undergrads to get organized, and providing a programmer to mentor the team.

Browsing around the UCOSP blog, I was reminded of why student projects, while laudatory, frequently fail to deliver anything useful. “One of the points of this course is to give you a chance to find out what it’s like to set and then meet your own goals,” Greg wrote. “The net result is pretty clear at this point: in many cases, students are doing less per week on this course than they would on a more structured course that had exactly the same content.”

College students in their final year have about 16 years of experience doing short projects and leaving everything until the last minute. Until you’re a senior in college, you’re very unlikely to have ever encountered an assignment that can’t be done by staying up all night.

The typical CS assignment expects students to write the “interesting” part of the code (in the academic sense of the word). The other 90% of the work that it takes to bring code up to the level of “useful, real-world code” is never expected from undergrads, because it’s not “interesting” to fix bugs and deal with real-world conditions, and because most CS faculty have never worked in the real world and have almost no idea what it takes to create software that can survive an encounter with users.

Time management is usually to blame. In a group of four students, even if one or two of the students are enterprising enough to try to start early in the term, the other students are likely to drag their heels, because they have more urgent projects from other classes that are due tomorrow. The enterprising student(s) will then have to choose between starting first and doing more than their fair share of the work, or waiting with everyone else until the night before, and guess which wins.

Students have exactly zero experience with long term, team-based schedules. Therefore, they almost always do crappy work when given a term-length project and told to manage their time themselves.

If anything productive is to come out of these kinds of projects, you have to have weekly deadlines, and you have to recognize that ALL the work for the project will be done the night before the weekly deadline. It appears to be a permanent part of the human condition that long term deadlines without short term milestones are rarely met.

This might be a neat opportunity to use Scrum. Once a week, the team gets together, in person or virtually, and reviews the previous week’s work. Then they decide which features and tasks to do over the next week. FogBugz would work great for tracking this: if you’re doing a capstone project and need access to FogBugz, please let us know and we’ll be happy to set you up for free. We can also set you up with beta access to kiln, our hosted version of Mercurial, which includes a code review feature.

I’ve been blaming students, here, for lacking the discipline to do term-length projects throughout the term, instead of procrastinating, but of course, the problem is endemic among non-students as well. It’s taken me a while, but I finally learned that long-term deadlines (or no deadlines at all) just don’t work with professional programmers, either: you need a schedule of regular, frequent deliverables to be productive over the long term. The only reason the real world gets this right where all-student college teams fail is because in the real world there are managers, who can set deadlines, which a team of students who are all peers can’t pull off.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.