Planet Perrygeo

July 27, 2010

Paul Ramsey

FOSS4G Reminder

OK, this is the last time I'm going to remind you: FOSS4G 2010 is coming up fast. September 6-9 in Barcelona, Spain. The presentation call was massive (360 submissions for 120 slots) as was the workshop call. So register soon and make your hotel reservations while you can! That is all.

 

by Paul Ramsey (noreply@blogger.com) at July 27, 2010 04:31 PM

July 22, 2010

Paul Ramsey

On Cable Modems

I had cable guys in my house turning on basic cable, and they noticed that my cable modem was old. When I say "old", I mean "I think I got it when we moved into the house almost 10 years ago". So they said, "here, have a new one". My maximum download speed quadrupled overnight. If you've got an old cable modem, figure an excuse to get a new one.

 

by Paul Ramsey (noreply@blogger.com) at July 22, 2010 04:38 PM

July 21, 2010

Paul Ramsey

Network Walking in PostGIS

One of the new features in PostgreSQL 8.4 was the “WITH RECURSIVE” clause available for queries. It allows you to define a subquery based on a recursive term — fancy language for a function that calls itself. One of the favorite uses of recursion is walking a network. Geospatial applications use networks all the time: electrical grids, stream systems, and storm sewers are all directed networks (they have unidirectional flow).



Here’s an example of network walking using a simple collection of segments. As is common in many GIS applications, the segment are implicitly connected — their end points are coincident with the start points of other segments.



CREATE TABLE network ( segment geometry, id integer primary key );



INSERT INTO network VALUES ('LINESTRING(1 1, 0 0)', 1);

INSERT INTO network VALUES ('LINESTRING(2 1, 1 1)', 2);

INSERT INTO network VALUES ('LINESTRING(1 2, 1 1)', 3);

INSERT INTO network VALUES ('LINESTRING(3 1, 2 1)', 4);

INSERT INTO network VALUES ('LINESTRING(3 2, 2 1)', 5);

INSERT INTO network VALUES ('LINESTRING(2 3, 1 2)', 6);

INSERT INTO network VALUES ('LINESTRING(1 3, 1 2)', 7);

INSERT INTO network VALUES ('LINESTRING(4 2, 3 2)', 8);

INSERT INTO network VALUES ('LINESTRING(3 4, 2 3)', 9);

INSERT INTO network VALUES ('LINESTRING(2 4, 2 3)', 10);

INSERT INTO network VALUES ('LINESTRING(1 4, 1 3)', 11);

INSERT INTO network VALUES ('LINESTRING(4 3, 4 2)', 12);

INSERT INTO network VALUES ('LINESTRING(4 4, 3 4)', 13);



CREATE INDEX network_gix ON network USING GIST (segment);


Visually, the network looks like this:



Network



To walk our network, use a WITH clause that starts with one segment, then repeatedly adds the next downstream segment to the collection. In our case, the “next downstream segment” is defined as a segment whose start point is close to the end point of the current segment. We’ll walk down from segment 6.



WITH RECURSIVE walk_network(id, segment) AS (

SELECT id, segment FROM network WHERE id = 6

UNION ALL

SELECT n.id, n.segment

FROM network n, walk_network w

WHERE ST_DWithin(ST_EndPoint(w.segment),ST_StartPoint(n.segment),0.01)

)

SELECT id

FROM walk_network


Which returns:



 id

----

6

3

1

(3 rows)


From 6 to 3 to 1, correct! Once we have our walker producing the results we want, we can wrap more PostGIS and PostgreSQL functions around the walker to produce a finished product. Here’s a function that takes in an edge identifier and outputs a linestring based on the downstream path.



CREATE OR REPLACE FUNCTION downstream(integer)

RETURNS geometry

LANGUAGE sql

AS '

WITH RECURSIVE walk_network(id, segment) AS (

SELECT id, segment FROM network WHERE id = $1

UNION ALL

SELECT n.id, n.segment

FROM network n, walk_network w

WHERE ST_DWithin(ST_EndPoint(w.segment),ST_StartPoint(n.segment),0.01)

)

SELECT ST_MakeLine(ST_EndPoint(segment))

FROM walk_network

' IMMUTABLE;


And here’s the function in action, generating the downstream path from segment 12.



=# SELECT ST_AsText(Downstream(12));



st_astext

---------------------------------

LINESTRING(4 2,3 2,2 1,1 1,0 0)

(1 row)


Check the generated path against our network picture — looks good!



Path 12

by Paul Ramsey (noreply@blogger.com) at July 21, 2010 09:44 PM

Chris Schmidt

Brave New World: OpenLayers 3.0

OpenLayers is approaching the ‘Brave New World’ of a major API breakage for the first time since OpenLayers 2.0 in October of 2006.

Things that have changed in OpenLayers since then:

  • Support for vector drawing.
  • The existence of ’spherical mercator’
  • reprojection of objects
  • More than a dozen new layer types
  • The existence of TileCache, FeatureServer, and RESTful HTTP APIs for geodata
  • A tile-serving service from OpenStreetMap (also, for OpenStreetMap: 4 major API changes)
  • Over four dozen additional contributors to the source code
  • Over 2000 trac tickets
  • 10 2.x releases

Overall, is it any shock that it’s time for a pretty major change?

In order to facilitate rapid development, we’ve shifted development of OpenLayers ‘3.0′ to github; you can follow along (or fork you own) on the OpenLayers github project. To be honest, git scares the crap out of me; every time I’ve used it, I have consistently lost data as a result of not understanding the tool and using the wrong command. However, I fully realize I am a fuddy duddy who needs to get over his problems at this point. :)

Looking forward to seeing the future of the world where OpenLayers is even more awesome than it already is!

by crschmidt at July 21, 2010 05:56 AM

July 17, 2010

Joel Spolsky

The growing empire of Stack Exchange

We launched three new Stack Exchange sites this week!

We’ll have three more for you next week, too.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at July 17, 2010 01:57 AM

July 13, 2010

Joel Spolsky

Domain Names

“We decided that individually-branded sites felt more authentic and trustworthy. We thought that letting every Stack Exchange site have its own domain name, visual identity, logo, and brand would help the community feel more coherent. After all, nobody wants to say that they live in Housing Block 2938TC. They want to live in Colonial Manor. Never mind the connotation of, well, colonies.”

Domain Names

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at July 13, 2010 02:56 AM

Allan Doyle

Review: Rubbermaid 12-Slot Organizer as a Mac mini server rack

I needed to do something about the Mac minis that were accumulating on the table in my office. Digging around, I found this Rubbermaid organizer on Amazon.

It turns out to be nearly perfect. The unit is very sturdy, was easy to put together, and the shelf height is just right. There’s enough clearance for airflow but not so much that you feel space is being wasted.

I used self-stick cable tie anchors and cable ties to mount the power bricks and used double-stick mounting tape as stops to keep things in place. The old-style minis are heavy enough and are pretty non-slip, so I just put some tape at the front of the shelf to keep them from sliding off. The one new-style mini was pretty slippery so I used the tape to actually stick the base to the shelf.

The unit came with vertical rods that go in the back of each column of shelves to keep them from sliding out the back, but I decided to leave those out. That way I can slide each shelf forward to get DVDs into the mini, or back to get at the connectors.

The weak spot of the minis is the power cord (at least on the pre-2010 models) which comes out quite easily. I tied those down as well and am pretty sure they won’t jiggle their way out. I have four minis in the rack right now along with a Drobo with 10TB of disk. I’m going to be adding a 5th mini with a stackable disk drive, that’s why there’s double-high slot  still open on the mini side of the rack.

Cable management is an issue, mostly because of the power bricks long cables. I may fiddle with how I fold the cables into the shelves a bit more.

The whole thing plus a UPS and monitor/keyboard/mouse sits nicely on some steel shelves in our A/V equipment room at the museum. I still need to time how long the UPS runs. I’m only going to have the public web site minis on it.

by Allan at July 13, 2010 12:18 AM

July 10, 2010

Allan Doyle

Old bits slipping away

I moved this blog from one of the Mac minis in my basement to the other (I’m trying to put everything on the newer one to free the other one up) yesterday. Originally I had been blogging using Plone (from about 2005-2007) and then moved to Wordpress. Moving the Plone part seemed like it was more work than I wanted to put in, so it’s goodbye to those posts.

by Allan at July 10, 2010 01:42 PM

July 07, 2010

Joel Spolsky

Web Applications Stack Exchange now in beta

Want to know how to export mail from Gmail? Or delete your Facebook account? Or send giant files via email?

Well, the new Web Applications Stack Exchange is for you. It’s a part of the Stack Exchange network, so it has the clean, elegant design that made Stack Overflow a phenomenal success.

The newest member of the Stack Exchange Network is the first one to go through the community site-creation process called Area 51. There are more great sites in the pipeline, but they have to demonstrate that they can reach critical mass or we won’t create them.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at July 07, 2010 07:55 PM

Paul Ramsey

SXSW Geo in a Nutshell

From Mapping and Geolocation: Turnkey Approaches You Need to Know [MP3] at 19:55.
I think that, at least for us here [at SXSW] this week, Foursquare and Gowalla have been at the center of the universe for a little while.



And I think that [is because] it is something to do with the fact that it's been a problem we've been trying to solve forever. You know you've been thinking "wouldn't it be great if I knew where my friends are so I could find them and we could hang out more" for years and so now this is finally here.



So not only is it an easy way to do something, and solves a problem that we we've been trying to fix forever, it's the first time that location is kind of useful beyond maps. Everyone understands that you need location in maps, because you have to get from point A to point B … I want to know where I am so I can broadcast it, because it's cool that I go to cool conferences (um, isn't it?), and I want my friends to know where I am, and I want to be able to find where they are.
I'm over the hill and I'm not even 40. Now get the hell off my lawn!

 

by Paul Ramsey (noreply@blogger.com) at July 07, 2010 06:33 PM

July 02, 2010

Allan Doyle

OGC (re)discovers URLs, but let’s tighten up the terminology a bit

I had seen this tidbit that Sean Gillies writes about in the recent OGC newsletter. My thoughts were along the lines of Sean’s. I never understood the big deal behind URNs.

EDIT: Forget the semi-rant, see the comments, and then go read about URI…

But in re-reading Sean’s post and the OGC news coming out of the June 2010 meetings, I think the terminology is a bit imprecise. Too bad the source document, 10-124r1 isn’t available on the OGC web site (promised for mid-July, I see) to see if the issue is in the document or in the news page. Here’s the news page version:

OGC Identifiers – the case for http URIs’

The OGC Members approved release of ‘OGC Identifiers – the case for http URIs’ [OGC 10- 124r1] as an OGC Whitepaper. .According to the current OGC policy either URNs or http URIs may be used in OGC standards. However, the use of http URIs (a) resolves some deployment challenges and (b) provides an opportunity for easier engagement with broader communities. So OGC should now consider taking the next step, and mandate the use of http URIs for persistent identifiers in OGC specifications. This whitepaper canvasses a number of issues around this proposal.

http URI Policy

The OGC Members approved the following as official OGC policy to be included in the OGC Policies related to OGC standards [OGC 06- 135rN]:

  • OGC TC directs the OGC-NA that all new OGC identifiers issued for persistent public OGC resources shall be http URIs, instead of URNs
  • New standards and new major versions of existing standards shall use http URIs for persistent public OGC resources to replace OGC URN identifiers defined in previous standards and versions, unless OGC- NA approves an exception

Operational Implications: OGC should carefully manage (maintain for the long term) the http://www.opengis.net domain and identifiers in this domain

So what’s wrong? Refer to RFC3986 (or the html version). Section 1.1.3 talks about URI, URL, and URN:

A URI can be further classified as a locator, a name, or both. The term “Uniform Resource Locator” (URL) refers to the subset of URIs that, in addition to identifying a resource, provide a means of locating the resource by describing its primary access mechanism (e.g., its network “location”). The term “Uniform Resource Name” (URN) has been used historically to refer to both URIs under the “urn” scheme [RFC2141], which are required to remain globally unique and persistent even when the resource ceases to exist or becomes unavailable, and to any other URI with the properties of a name.

A URN is a kind of URI. What is called an “http URI” is really a “just” a URL in RFC3986. And, a URN need not (or I should say “need no longer”) be something with “urn:” in the scheme. A URL could be a URN based on the last part of the definition above, “any other URI with the properties of a name”

Therefore, an “http URI” (from the OGC wording) can be either a URL or a URN, based on section 1.1.3 of RFC3986. Of course, the URN is really a URL with the additional uniqueness and persistence properties. So let’s just call OGC’s newly mandated URIs URLs.

There are two primary motivations for using RFC2141 URNs. One is as a globally unique name managed by some authority. The other is as a persistent identifier, sometimes used to map onto a URL with a resolver. The trouble with the latter is that URLs really work better in the first place, and I’m guessing that’s what 10-124r1 says.

So here’s what I think they should have said in the TC:

URI Policy

The OGC Members approved the following as official OGC policy to be included in the OGC Policies related to OGC standards [OGC 06- 135rN]:

  • OGC TC directs the OGC-NA that all new OGC identifiers issued for persistent public OGC resources shall be http URLs, instead of RFC2141 URNs
  • New standards and new major versions of existing standards shall use http URLs for persistent public OGC resources to replace OGC RFC2141 URN identifiers defined in previous standards and versions, unless OGC- NA approves an exception

Sorry to be so pedantic. Back in the day, there would have been half a dozen people at any given TC who would have been able to argue the finer points of this for hours….

(And I just figured out what the OGC-NA is. I guess it’s the “Naming Authority”.)

by Allan at July 02, 2010 05:17 PM

June 30, 2010

Paul Ramsey

How to Survive Foo Camp (Introvert Edition)

I just read the 2010 edition of Scott Berkun's regular Foo Camp review. It's very good, and makes me want to amplify a couple points for future, introverted attendees.



First, the social dynamics are much as he describes, although I would certainly not argue as he does that pretense is "eliminated". It is somewhat attenuated. The attending population can be segmented in lots of ways, but here's mine:



Foo RegularsFoo Newbies
ExtrovertsIntroverts


Extroverts should have a grand time, lots of people to talk to. Regular attending introverts can lean on existing acquaintances when burnt out. But if you, like me, are an introvert, and you aren't part of the "in group" of repeat attendees, you may find yourself feeling socially isolated and very very tired, in the midst of a happy group of very energetic people. You might begin to resent them.



Foo is what it is, though. It is an opportunity to connect with some people you'd otherwise never meet, so all you can do is arm yourself before-hand with some tools and expectations to avoid having a negative experience.



Social Fatigue



First of all, bad news, the Cocktail Party dynamic is going to be a major mode of interaction. "So, what do you do?" You're going to have to answer that (and ask that) a lot, and people will filter you based on your answer. It's best to have an answer that is interesting to a wide swathe of people outside your field. The population of askers will be intellectually sophisticated (Michael Arrington excluded) so the answers that generally work can be surprising.



For me the general truth "I work for OpenGeo doing spatial database programming" was not effective at all. But a specific instance "I'm working on spherical geometry algorithms for location data" worked pretty well. (Unfortunately I was too dazed to really parse that fact until after I left.) Anyhow, on review, the second explanation has lots of potential follow-ups from "why?" for people who understand "location data" to "what's that?" for people who don't. The first explanation is too mushy.



Back to the bad news. Because a lot of your interactions are going to be draining first encounters, you're going to run out of gas pretty fast. Find a seat by the fire, get a quiet corner, and don't feel guilty about not talking. You're tired. You can't. Go to bed early, it's OK – the alternative is just going to make you feel worse.
Note to Parents: Your early-to-bed-early-to-rise conditioning comes in handy sometimes. You can get up early, get a coffee, and join the other parents for a quiet, introvert friendly, discussion around the fire pit while all the 20-something future masters of the universe are sleeping off their hangovers.


The "Formal" Program



Sessions are easy. There's structure, you can talk or not. For a first time attendee, I think this piece of wisdom on the Foo wiki ("Find people whose work you have no supposed interest in and go to their talks" -- Chris DiBona) is absolutely wrong.



You can bring the most value to the group, and meet the people who are going to be of most direct value to you in the future by going to sessions that are directly interesting to you, that fall in your area of expertise, that you have passion for. If you're going to Foo over and over again like Chris, then you have the luxury of sampling. I (foolishly) took the advice and missed going to database sessions with folks like Brian Aker, who says something worth writing down (if you're a database geek) about every 30 seconds. Bad, bad decision on my part.
Blowhards: There will be blowhards. Statistically speaking, the larger the session, the more likely there will be a blowhard in your session. Scott Berkun's 2009 post mentions them, and I like his idea of the "anonymous gong" to help the blowhard achieve self-awareness.


Bring a Pen and Paper



Laptops and iPads are heavy and take time to start up and get into data entry mode. Paper is light and fast. There will be names and ideas you want to remember. Precisely when the nuggets fall from the sky is unpredictable, but you probably won't be sitting down with your device at the ready.



Celebrities



Own your inferiority complex. Yes, you're inferior. Suck it up. Unfortunately, this is "do as I say, not as I do" advice, because I wasn't very good at it.



I didn't tell Kathy Sierra that her 2009 Business of Software talk is currently conditioning all my thinking about building the OpenGeo Suite materials. I just glanced at her out of the corner of my eye.



I did, however, have a nice time talking with (well, listening to) Mitch Kapor. Feed them some rope, provide them a context to tell you what's on their mind. A general request for advice on a general topic will usually do. I asked Kapor about economic development. I should have asked Sierra about our own product issues.
Invisible Celebrities: I actually talked to Chad Dickerson, but I never connected the dots until afterwards that he wrote the "CTO Corner" column in InfoWorld, back when it was a paper publication. That column was the only reason I kept receiving the magazine, it made me feel sane (I am not alone!) to be advocating open source, which was very much a minority position at the time. I would have liked to have told him that.


Meeting People You Want To Meet



If you look at the list and see a name you absolutely want to meet, track them down ahead of time, get their contact information and then contact them during the conference and arrange a time.



The venue is deceptively intimate, but 250 people is a lot of people. You may not randomly run into the person you are looking for. You probably will not. I was absolutely looking forward to talking with James Dixon about open source business models. I even staked out the main passage ways, during transition times. I did not see the man! (Here's where having one of those "iPhone" things would have been handy.)
Surprising Social Norms: Either there is a Silicon Valley thing going on that I'm not acclimated to, or this behavior is a form of status advertisement for high status individuals, but a number of times I saw people pull out their smart phones and check messages while other people were talking to them. I had not encountered this before, though I suppose it is just a variation on the equally noxious answering-your-cell-phone-in-the-middle-of-a-conversation. (Both should be grounds for an immediate caning.)


Us and Them

People sometimes ask, "What can I do to be invited back?" and your best bet is to make a (positive) impression by engaging and presenting.

— Foo Camp Wiki
In general, it's going to be hard to make a good impression if you have no psychological energy. And it's difficult to plan for a session or an Ignite talk when you have no idea what the audience is going to find interesting.



However, having seen the audience, I can now (post-facto (arg!)) offer a generic recipe for an interesting topic: something that is foundational in your specialty field but mostly unknown in the larger world. In my case a talk about general principles of geography, like Tobler's Law or the ecological fallacy or some topical examples of how to lie with maps would have gone over well at Ignite.



It's worth noting, given the mix of regular repeat attendees versus new attendees, that the criteria are not strictly meritocratic — obviously you can also get invited back by being intrinsically "important" in some way to the O'Reilly organization. The "engaging and interesting" criterion is only applied to some.



In Conclusion



Like many of life's memorable experiences ("there was this time I hiked to the top of a mountain in the rain…"), there is a strong possibility that, as an introvert in a highly social milieu without a group of friends to lean on, you might find Foo Camp difficult and unenjoyable in the moment. You might want to leave, a lot of the time. But if you stick with it, you'll come out with some good memories and look back on it as an important experience. You'll also meet some very friendly people who will make you feel welcome despite yourself. Give it a shot.
Shout-out: Thanks to the people who made me feel not like a stranger at Foo Camp, despite my acting like one. Selena Deckelmann, Cary Davis, Joe Hughes, Matt Cutts, Nat Torkington, Shel Kaphan, Toby Boudreaux, Andrew McLaughlin, Laurel Ruma, Emily Jacobi and many others.

by Paul Ramsey (noreply@blogger.com) at June 30, 2010 06:04 PM

June 28, 2010

Joel Spolsky

Don Norman at the Business of Software 2009

Neil has posted a video of Don Norman (most famous for his book The Design of Everyday Things) speaking at the Business of Software conference last year in San Francisco.

“Imagine you’re on the first slide of your powerpoint presentation and want to move to the next slide. Your remote control has two buttons. They are unmarked, but one button points up and one button points down.

“Which button do you press?”

It turns out half the people press up, half the people press down, and everybody thinks their choice is obvious. It’s a great talk.

The early bird discount for the Business of Software 2010 (Boston, October 4-6) saves you $400, but it expires this week, so this is the right time to sign up for my favorite conference.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at June 28, 2010 06:55 PM

June 25, 2010

Tom Kralidis

Geoprocessing with OGR and PyWPS

PyWPS is a neat Python package supporting the OGC Web Processing Service standard.  Basic setup and configuration can be found in the documentation, or Tim’s useful post. I’ve been working on a demo to expose the OGR Python bindings for geoprocessing (buffer, centroid, etc.). Here’s an example process to buffer a geometry (input as WKT), [...]

by tomkralidis at June 25, 2010 06:14 PM

June 22, 2010

Joel Spolsky

Area 51 is now in beta

Area 51 is now in beta. This is the promised place where the community comes together to invent new Stack Exchange sites.

Benofsky from Hacker News writes:

Seems overly complicated, I have no idea what's going on when I visit Area 51, I guess this is their strategy for turning away uncommitted users.



Also, how are they going to make money?

I’m glad you asked, benofsky! The answer is simple. Volume.

(transcript)

Well, I’m not one to take Internet chat board comments seriously. After all, the Anonymous Nostradamus’s over at Code Project reacted thus when Stack Overflow itself launched:

I think the UI sucks. I can't imagine this site being around in a year.

We all know how stunningly accurate that prediction was:

Benofsky is onto something, though. Area 51 is not for everyone. If you don’t know what it’s for, or why it’s going to work, or you can’t figure it out, it’s not, actually for you.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at June 22, 2010 02:40 PM

Paul Ramsey

Nerds Redux

I had a chance to re-present my FOSS4G 2009 keynote talk at the Rendezvous OSGeo a Quebec last week, and thanks to the good work of the FOSSLC team, there's now a pretty clean online video of it.





 

by Paul Ramsey (noreply@blogger.com) at June 22, 2010 12:52 AM

June 19, 2010

Paul Ramsey

PostGIS @ FOSS4G 2010

One of the things that tickled me about the presentations selected for FOSS4G 2010 was the number of talks in the list that specifically mention PostGIS:
  • Beyond PostGIS - New developments in Open Source Spatial Databases

  • Introducing PostGIS WKT Raster: Seamless raster/vector operations in a spatial database

  • Introduction of flood evacuation route search system?using QGIS,PostGIS,GRASS and PgRouting

  • Moving from Oracle/ArcGIS to PostGresql/PostGIS

  • PostGIS meets the third dimension

  • PostGIS WKT Raster. An Open Source alternative to Oracle GeoRaster

  • Running long and complex processes with PostGIS

  • The State of PostGIS

  • Tips for the PostGIS Power User
The last two are mine! And one is about not using PostGIS. But still, some interesting talks on the use and future of my favourite spatial database.

 

by Paul Ramsey (noreply@blogger.com) at June 19, 2010 02:00 PM

June 18, 2010

Paul Ramsey

PgSQL on EC2

The theory behind putting a PostgreSQL (and PostGIS) instance on an Amazon EC2 instance with an Elastic Block Store (EBS) file system underneath is pretty straightforward, even for big databases. But when you want those databases to show the kind of properties we have come to expect from our systems, like durability, throughput, and reliability, things get much harder.



This thread on pgsql-general was very illuminating to me. Among the tidbits:
Let's be clear here, physical I/O is at times *terrible*. :)
There's no way we could run this database on a single EBS volume.
We had to fail over to one of our spares twice in the last 1.5 years. Not fun. Both times were due to instance failure.
Basically the assumptions of AWS architecture (virtual instances will be less reliable than real world computers, but that doesn't matter because getting a new one is really easy) don't map well with the requirements of running a classic production database.



There are probably some engineering solutions around for this (GlusterFS, for example, but the core PgSQL would need some serious work and end up looking a lot more like OracleRAC than the currently single-machine set-up.

 

by Paul Ramsey (noreply@blogger.com) at June 18, 2010 06:03 PM

Tom Kralidis

Tips on Finding a Job

Great post by Dave here on his experience and suggestions / ideas on finding a job.  Upbeat, positive and encouraging.  Congratulations and good post Dave! Written from Starbucks in Laval: window.onload=function() { var point494 = new GLatLng(45.568007, -73.727295); var map494 = new GMap2(document.getElementById("map494")); map494.addControl(new GSmallMapControl()); map494.setCenter(point494, 10); var icon494 = new GIcon(); icon494.image = "http://labs.google.com/ridefinder/images/mm_20_red.png"; [...]

by tomkralidis at June 18, 2010 05:27 PM

June 09, 2010

Joel Spolsky

My secret weapon for setting executive compensation

My friend Noam at Harvard Business School does annual surveys of executive compensation in tech companies. The surveys are a great way to figure out how much to pay that VP of Sales you were thinking of hiring, or whether your equity grant as CTO is fair.

You can buy the studies for $999, or you can participate in the survey and get free access to the results when they’re published.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at June 09, 2010 08:05 PM

Matt Perry

kmltree

When the MarineMap team started delving into the Google Earth plugin, it was apparent that it supported the display and rendering of KML files almost as well as the Google Earth desktop application. The missing piece of functionality was the nice tree-style legend that is provided with the desktop app. The plugin lets you add KML for display but gives you no HTML interface to work with it. For simple apps, you can just roll your own html/js form. But that quickly becomes unmanageable if you’re adding KML dynamically and need to create a tree-style legend for any arbitrary KML document.

Enter kmltree.

kmltree is a javascript tree widget that can be used in conjunction with the Google Earth API. It replicates the functionality of the Google Earth desktop client, and is fast, extensible, and stable for use in advanced web applications. It’s built utilizing the earth-api-utility-library and jQuery.

kmltree

Any arbitrary KML can be parsed and represented in a tree-style legend right in the web browser. Try it out.

Kmltree is the brainchild of Chad Burt who developed it as part of the marinemap codebase but had the foresight to realize that this would be useful to a much wider audience and abstracted it into its own javascript library. If you’re building a web mapping application with the Google Earth API, give it a shot!

by perrygeo at June 09, 2010 02:27 PM

June 08, 2010

Joel Spolsky

Paul Kenny’s talk at the Business of Software 2009

Paul Kenny is a regular speaker at the Business of Software. He’s an expert on sales, especially high tech sales, and both Red Gate and Fog Creek hire him all the time to train our sales teams and build our sales organization.

Here’s a video of his talk at last year’s conference:

The next Business of Software conference will be in Boston, October 4th-6th.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at June 08, 2010 04:31 PM

Paul Ramsey

More Stories from the Future of Computing

From the Jobs keynote of yesterday, a slide with a quote from Theo Gray of Wolfram, regarding the popular "Elements" iPad application:
I earned more on sales of The Elements for iPad in the first day than from the past 5 years of Google ads on periodictable.com.
Quoth Jobs, "That's what I like to hear from you guys." Audience whoops.



Right now the walled garden is kicking the jungle's ass, but for how long? It's incredibly interesting, that for the moment the old school revenue model of application sales is actually besting the new school free-with-strings (ads) model that we were told was the Future. Perhaps once HTML5 application quality gets up to the level of fit and finish that the current crop of native apps is providing we will flip back again.



I think, for example, of a stock market application. Would people pay a buck for a really excellent application that "just works" in a clean and uncluttered way for displaying current information, research, blah blah blah, instead of just going to Yahoo! Finance? The information is available for free (just like the periodic table!) but a really excellent encapsulation of that information might be compelling enough to pay for. Walled garden starts to fall apart where it interfaces with the jungle... once the application has to link out to things like company reports, and other non-structure pieces in the raw internet, it re-gains the clunkiness of the old browser experience. So why not start with the browser?



For geo, I think that sites like GeoCommons, which have applied a baseline level of structure to a wide swath of data, are fertile grounds for the "app treatment". An application that provides superior interactive access to their data archives would be an alternative monetization path for leveraging their growing holdings of structured GIS data.



Interesting times!

 

by Paul Ramsey (noreply@blogger.com) at June 08, 2010 12:35 PM

June 04, 2010

Tom Kralidis

Displaying GRIB data with MapServer

I recently had the opportunity to prototype WMS visualization of meteorological data.  MapServer, GDAL and Python to the rescue!  Here are the steps I took to make it happen. The data, (GRIB), is a GDAL supported format, so MapServer can handle processing as a result.  The goal here was to create a LAYER object.  First [...]

by tomkralidis at June 04, 2010 11:02 PM

June 02, 2010

Paul Ramsey

Finding Corrupt PostgreSQL Data Files

While PostgreSQL itself will never create corrupt data files, that doesn't stop other processes or hardware failures for corrupting the files underneath the database, which can cause database crashes. Josh Williams of End Point provides a super rundown of how to track and repair a file corruption.

 

by Paul Ramsey (noreply@blogger.com) at June 02, 2010 01:04 PM

May 27, 2010

Matt Perry

MarineMap wins award for Environmental Conflict Resolution

For the last year or so, I’ve had the pleasure of working with the MarineMap Consortium. We just learned yesterday that the U.S. Institute for Environmental Conflict Resolution awarded MarineMap the “Innovation in Technology and Environmental Conflict Resolution”.

I joined the team after the launch of the South Coast of California site which was already widely recognized as a successful decision-support tool for marine spatial planning. We’ve since been working on version 2 of the MarineMap tool which is deployed currently for the North Coast of California in support of their Marine Life Protection Act (MLPA) process.

It’s been a tremendous challenge to bring a new version of the software to life and have it meet and exceed the standards set by its predecessor. It has also been tremendously rewarding and having our work recognized at this level is a great honor. It’s nice to know that the tools we’ve developed have been so helpful and instrumental in the marine planning process along the coast of California. Looking forward, I see MarineMap growing beyond a tool for a specific purpose (supporting the MLPA Initiative) to a robust framework for developing web-based spatial planning tools for all sorts of environmental applications, both marine and terrestrial. And this award confirms that we are already heading in the right direction. Very exciting news!

by perrygeo at May 27, 2010 11:36 PM

May 18, 2010

Joel Spolsky

Random thoughts for May 18, 2010

FogBugz 7.3 is a pretty huge release, despite the modest version number. It has a bunch of features that almost everyone will find useful on a daily basis.

  • The Case Event Edit Plugin now lets you edit cases (instead of just appending to them).
  • Bulk Reply, so you can respond to multiple emails with the same response. You can even flip through the outgoing responses and customize each one if you want.
  • Width of the case area is now resizable.
  • We now support IMAP (in addition to POP3) for incoming email
  • Project backlog editing (for scrum/agile shops) is now much faster.

What’s new in FogBugz 7.3?

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at May 18, 2010 03:02 PM

Paul Ramsey

Who's Your Dealer?

The mapbutcher thinks it's OK to get hooked on proprietary software:
People are interested in the ‘express’ editions because on the surface of it the marketing works on them, they’re familiar with the brand and are attracted by the ‘free’ carrot dangling from the end of the stick. Open Source software starts from that position – it’s already free so open source projects need another carrot to get us hooked.
To which I can only say "Simon! Look at yourself in the mirror, man! Do you want to end up whoring yourself down on Dalgety Street to pay for your ESRI habit?"



OK, metaphor is tricky and fun, because there's so many ways to approach a metaphor. And for this one, it's easy to get distracted by the "drug" side, but the point of my metaphor is not that addictive drugs put you in a subordinate relationship to the drug (though some do) but that they put you in a subordinate relationship to the dealer.



The reason we can all survive and function in society despite our crippling addiction to oxygen is because oxygen is free and plentiful. The pusherman isn't trying to hook new customers because he believes that drugs are wonderful, he's doing it because he wants their money.



Software is "addictive" to organizations. Once you choose a piece of software and implement it, you're going to be "addicted". It's going to be hard to change. There will be withdrawal symptoms. Given that fact, what kind of software do you want to use? Software that is as free as the air you breath? Or software that is only available on terms dictated by someone else?



Your choice. Your future. Your life.

 

by Paul Ramsey (noreply@blogger.com) at May 18, 2010 03:53 AM

May 17, 2010

Paul Ramsey

Free like...

A favorite bon mot of open source critics is that open source software is "free like a free puppy". Tee hee! Open source advocates should remember to keep the rejoinder handy, that proprietary software is "free like a free hit of crack". Oracle "Express", SQL Server "Express", ESRI educational copies, yes I am looking at you.

 

by Paul Ramsey (noreply@blogger.com) at May 17, 2010 05:14 PM

May 14, 2010

Paul Ramsey

NerdBA

I was blessed (or cursed) to go through my undergraduate years from 1989 to 1993, and at that time, I'm sad to say, there wasn't a lot of nerd chic. I did my degree in Mathematics. The computer science student on my dorm floor ended up gainfully employed programming mainframes at Safeway. It was not, shall we say, a glamourous time for young nerds. On the other hand, the pressure wasn't exactly on.



What do young nerds today make of the examples before them, the Brins and Pages and Zuckerbergs? Is there a sense of performance anxiety? Does the start-up zeitgeist weigh on them?



Reading articles about how Mark Zuckerberg screwed his college employers, I am torn. On the one hand, is the satisfaction that the guy who did the actual work and had the particular, detailed vision of the social network got the reward, not his moneyed waspy employers. On the other hand, he clearly screwed them in a very premeditated fashion. What led to this kind of anti-social behavior? The lure of the Big Score? With Brin and Page in the back of your mind, maybe becoming a 20-something billionaire doesn't seem so far fetched? Maybe it's worth violating some social norms for?



Anyhow, now Zuckerberg is part of the pantheon. And it makes me wonder what the psychology is, today, in the nerdly dorm rooms? Is it still OK to go work for Safeway? Or is everyone secretly hoping they can spend a couple years drinking coke, working in the garage, and making it to the NerdBA?

 

by Paul Ramsey (noreply@blogger.com) at May 14, 2010 01:56 AM

May 10, 2010

Joel Spolsky

Kathy Sierra’s talk at the Business of Software 2009

I’m a long-time Kathy Sierra fan. Her talk at the Business of Software 2009 was amazing, even though, in retrospect, I realize now that she had a slide making fun of Stack Overflow’s terrific new VC-backed business model. She even got the unicorn!

Watch the  whole talk here. The next Business of Software conference will be in Boston, October 4th-6th.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at May 10, 2010 06:53 PM

May 08, 2010

Tom Kralidis

OWSLib CSW Updates and Implementation Thoughts

I’ve had some time to work on CSW support in OWSLib in the last few days.  Some thoughts and updates: FGDC Support Added Some CSW endpoints out there serve up GetRecords responses in FGDC CSDGM format.  This has now been added to trunk (mandatory elements + eainfo).  Note that csw:Record (DMCI + ows:BoundingBox) and ISO [...]

by tomkralidis at May 08, 2010 01:34 AM

May 07, 2010

Matt Perry

Exploring Geometry

I don’t know how I let this gem slip past my radar for so long. It was only via a post by Dr. JTS himself (aka Martin Davis) that I saw a screenshot of JTS TestBuilder and decided to check it out.

I was actually just talking with someone about a tool that could provide simple visualization of WKT geometries; JTS Test Builder does that and much more.

You can input geometries (graphically or by well-known text) and compare two geometries based on spatial predicates:

spatial predicates

Do overlay analyses with the two geometries. Note that you can see the result as WKT below.

overlay

And there are a host of other spatial operations to generate geometries using buffers…

buffers

… convex hulls …

convex hull

This app provides a very nice and user-friendly way to quickly and simply explore and test geometric operations. To try it out, download JTS and unzip the contents somewhere. If you’re on windows, the .bat file is provided. If you’re running anything else, you have to cook up a shell script that will set up the environment and run JTS TestBuilder:

JTS_HOME=/usr/share/java/jts-1.11
CP=$CLASSPATH
for i in $JTS_HOME/lib/*.jar; do CP=$i:$CP; done
java -Xmx256m -cp $CP com.vividsolutions.jtstest.testbuilder.JTSTestBuilder $*

by perrygeo at May 07, 2010 02:35 AM

May 05, 2010

Paul Ramsey

The End of the End of General Purpose Computing

One of the memes that got slung around a log during the launch of the iPad and associated apologetics for the closed nature of iPad (and iPhone) software development was that we were at the "end of general purpose computing", that the closed and controlled environment was a good thing, and in fact required in order to provide the seamless, user friendly experience necessary to bring computing, finally, to the unwashed masses.



In this meme, the limitations of the new platform—no multi-tasking, applications as unitary bundles that don't share files, the lack of a (user visible) hierarchical file system—were all features, not drawbacks, they were in fact the core benefits that make computing understandable for grandma.



How quickly the worm turns. iPhone OS 4 now introduces multi-tasking, file sharing between applications, and all the associated user interface scruft that goes along with managing those concepts. Also, folders for holding the many application icons in your now-totally-crowded phone screens. And on and on.



So, now that bath water (the blessed "simple enough for grandma" experience) is heading over the balcony edge, can we have our baby back (open application development)?

 

by Paul Ramsey (noreply@blogger.com) at May 05, 2010 04:55 PM

May 04, 2010

Joel Spolsky

Random thoughts for May 04, 2010

“And the best answer we could come up with was, let’s make the damn thing free, and get some VC somewhere to pay for it.”

Announcing the Stack Overflow Series A financing

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at May 04, 2010 05:43 PM

May 01, 2010

Paul Ramsey

You know you've arrived...

...when people start discussing the best way to replace you.



Seen amongst the abstracts in the FOSS4G Community Program Review: "Beyond PostGIS - New developments in Open Source Spatial Databases" and "JASPA, an alternative to PostGIS".



Is that a bright shining light? I'm floating above my body...

 

by Paul Ramsey (noreply@blogger.com) at May 01, 2010 02:00 AM

Joel Spolsky

Note from Taco

Dear TechSmith,

Thank you very much for all the pictures of your pets:

Also thanks for the toys, which were great. I especially like the iPhone.

Yours,

Taco.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at May 01, 2010 01:30 AM

April 30, 2010

Paul Ramsey

FOSS4G Community Program Review

It's that time of year again! The Community Program Review for FOSS4G has begun!



If you're going to FOSS4G, or seriously thinking about going, the review is a way to ensure that the program includes topics you are interested in. You scroll through a list of all the abstracts the conference has received, and select the ones that are of interest to you. The conference committee tabulates all the results and uses that data to help build the program and to slot talks into rooms of appropriate size.



The number of abstracts is a good deal larger than the number of slots this year, so taking the time to do the review is a good idea if you're planning to attend this year!

by Paul Ramsey (noreply@blogger.com) at April 30, 2010 01:15 PM

April 28, 2010

Tom Kralidis

Batch Centroid Calculations with Python and OGR

I recently had a question on how to do batch centroid calculations against GIS data. OGR to the rescue again! Using OGR’s Python bindings (GDAL/OGR needs to be built –with-geos=yes), one can process, say, an ESRI Shapefile, and calculate a centroid for each feature. The script below does exactly this, and writes out a new [...]

by tomkralidis at April 28, 2010 02:19 PM

April 22, 2010

Paul Ramsey

Lazyweb: SDE requirements for PostGIS

Oh, lazyweb, I beseach thee!



Can you tell me, for each version of ArcSDE, what version(s) of PostGIS is/are supported? I'm at the Washington GIS conference, and folks are using SDE on PostGIS, and some of them are using old versions of PostGIS, so I need to know how far back we have to apply patches in order to fully support users who have deployed SDE on PostGIS.



Update: The lazyweb responds, with ArcSDE 10 requirements, ArcSDE 9.3.x requirements and recap: PostGIS 1.4.0 for ArcSDE 10, PostGIS 1.3.2 for ArcSDE 9.3/9.3.1



Also, from Cort Daniel of Pierce County I hear that minor releases as late as PostGIS 1.3.7 and PostgreSQL 8.3.5 work (unofficially) for ArcSDE 9.3.



Update2: Cort also said that while PostgreSQL 8.3.5 worked with ArcSDE 9.3, 8.3.7 did not (date fields stopped working right), which is really really odd. Not sure if anyone else could confirm that.

 

by Paul Ramsey (noreply@blogger.com) at April 22, 2010 05:04 PM

April 20, 2010

Paul Ramsey

Surveyor's Sermon

Last month I had an opportunity to give a 5-minute "Ignite" talk at Where 2.0 in San Jose. I chose my topic because I fear one of the things missing in the technologist enthusiasm for geolocation, particularly in the population of technically astute but non-geo people, is a respect for how locations are actually derived, and knowledge of the provenance of the data that undergirds our new mapping tools.





 

by Paul Ramsey (noreply@blogger.com) at April 20, 2010 06:07 AM

April 19, 2010

Joel Spolsky

Sites for experts

Since announcing the new plans for Stack Exchange, there’s been a lot of discussion about what kind of new Q&A sites will work best on this platform.

So far there are 32 informal proposals on meta.stackexchange.com. We’re weeks away from opening a site where these proposals can become real.

From my answer to the Firearms proposal: “The power of the Stack Exchange platform is detailed, expert answers to extremely rare, ‘long-tail,’ highly technical questions. To get expert answers, you need experts. To attract experts, you need a site where people are asking very interesting and hard questions, not the basic questions, so that it’s clear that this is a PRO site, not a consumer/enthusiast site.... and remember, the pro sites WILL attract the enthusiasts, but not the other way around.”

From my answer to the Law proposal: “There are only 200 easy law questions, and they've all been asked 100 times on Mahalo and Yahoo!Answers. But there are 20,000,000 detailed, difficult, long-tail questions that only professionals can answer, and we'd be doing a REAL service to the Internet by creating a place where you can find answers to the 20,000,000 hard questions, not the 200 easy ones.”

Our core mission at Stack Overflow is:

Make the Internet a better place to get expert answers to your questions.

The “expert” part of that mission is important, otherwise we’re just building another place for the same questions that every other Q&A site has. We want the law Stack Exchange to attract lawyers, the movie Stack Exchange to attract filmmakers, and the aviation Stack Exchange to attract pilots. What makes a community great is great people... that is, real experts. And the experts want to hang out with other experts.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at April 19, 2010 03:46 PM

April 17, 2010

Chris Schmidt

N900 vs. Nexus One vs. iPhone

(The following opinions are my own, and not representative of my employer or anyone else.)

A friend asked me via email:

I noticed you’ve had your N900 for a while. I am thinking about upgrading from my iPhone 3G (it’s really slow — lots of lagging on UI) to a Nexus One or a N900. Have you had a chance to compare the two at all? Where did you get your N900? Any “killer apps” or major problems with it?

I don’t know if ‘5 days’ is ‘a while’ yet. :) As for how I got it… well, perhaps you missed my previous post

I think that upgrading from an iPhone — if you generally like the way it works, and are just upset about the UI speed/performance — will be a disappointment. I have an iPod Touch and the n900, and when I want to run an app real quick, the iPhone will win, hands down. The N900 has far more functionality to be excited about, but in general, there are still aspects of the iPhone that win. Basically: If you didn’t jailbreak your iPhone, you’re not looking for the ‘right’ kind of thing for an N900 to be the solution. (I didn’t jailbreak my iPod, but it’s a media player, not a communications device.)

I don’t have experience with the Nexus One. The G1 users in the office have commented repeatedly on how ‘fast’ the N900 response is by comparison: my understanding is that the Nexus One is a significant hardware step up from the G1, so that experience may be irrelevant. (G1 users also have said “If they just came out with a ‘G2′ that just had double the CPU + double the memory, I’d buy it in a heartbeat.”)

Before the recent events, I was considering the N900 or the G1. After playing with a G1 for a while, I realized that it — sort of like the iPhone, though to a lesser extent — is also not a general purpose computer. It’s closer, because it’s more open, but in the end, it’s designed as a ‘platform’, and that’s obvious in many aspects of how it works. The Maemo OS was always developed for Internet Tablets, and that shows in many aspects of the design. (This is also why it makes a somewhat sub par phone; although the OS has grown well into many aspects of the device, the ‘phone’ app is recent enough that it hasn’t had time to mature into that role.)

Overall: If you’re the type of person who wants to run a shell script, and display the results of a shell script on your phone’s desktop, the N900 may well be for you. (Really.) If you want a platform, but for something that’s designed originally as a phone, and with a much broader ecosystem and community — but more limitations ‘out of the box’ — the Nexus One may be for you, though that’s mostly hearsay. (I would consult a G1/Nexus One user for anything resembling a serious opinion here.) And if you want something pretty that isn’t going to require you to pop open a terminal to fix it when something breaks — the Apple way may be the right way for you.

Some cool aspects of the N900: two-way video calling with Google talk (but only when initiated from a computer, sadly). Two way audio-chat over Jabber. Built-in always-on IM clients. Exchange mail push support built in. (Exchange support in general is excellent; this includes both ‘real’ exchange and clones like Zimbra.) ‘apt-get install openssh’. Doom on the phone. High res screen, great for video playback.

Some things it doesn’t do well: Making it trivial to open the actual ‘phone’ part (hardware button) would be great. The lack of Facebook app is disappointing; the iPhone one was so good that I was spoiled by it. (I’m not a huge Facebook user, so this isn’t a deal breaker by any stretch.) The fact that there was not, until recently, a well-supported way of distributing non-open applications, means that there isn’t a lot of non-open application development — so fewer professionally done games, free or pay, than on a platform like the iPhone (and I assume fewer than on Android as well). Ovi Maps, does not compare favorably to Google Maps in many cases, and running apps via the browser just doesn’t feel right. (This is a start, but not really ideal.) Battery life is poor, but that’s probably my own fault. :)

As a final point of comparison, the app community for the other devices doesn’t even compare to Apple’s in quantity. However, overall, the ‘decent, usable, free apps’ communities do seem somewhat larger on N900 than on Apple, in my experience. Finding something free on iTunes that’s worth downloading is hard — but the Maemo garage is full of fun, useful apps, and they’re all free.

I like the N900. It’s certainly not for everyone, but for me, I like having my communications device being a general purpose computer (so long as the phone part keeps working). There’s plenty of room to improve, but having a very open ecosystem reassures me that I may be able to contribute some of that, and overall, it fits the bill for being the most awesome thing to carry around in my pocket.

by crschmidt at April 17, 2010 02:38 AM

April 16, 2010

Paul Ramsey

FOSS4G 2010 Going to be Big

The presentation abstract deadline has passed for FOSS4G 2010 and there have been 354 abstracts submitted! I'm not sure the exact number of session slots in the planned program, but for comparison 2007 had 120 slots and subsequent events have been similar -- that's about how many 30 minute talks you can fit into two days in a five-track format. In 2007, the biggest FOSS4G so far, we had around 240 submissions for our 120 slots. So, with three submissions for every slot, 2010 is looking like it is going to be gangbusters! Everyone wants to go to Barcelona, I guess they heard about the escudella i carn d’olla.

 

by Paul Ramsey (noreply@blogger.com) at April 16, 2010 11:51 PM

April 13, 2010

Joel Spolsky

Stack Exchange 2.0

Like the small-town mayor who suddenly finds herself running an entire state, our ambitions for Stack Overflow keep growing. Our original idea of making the Internet a better place to get expert answers to your programming questions suddenly seemed too small. Programming questions? We asked. Why just programming questions? Why not every question under the sun? And who says we can’t run for Vice President of the United States of America?

We tried making our software available as a hosted white label product called Stack Exchange. We thought that other people would create awesome sites on every imaginable topic. Some people did (yay!), but it wasn’t the flood of high quality sites we were hoping for.

So we’re making a few changes. Briefly:

  1. Stack Exchange will now be free.
  2. We’re changing the way that new Stack Exchange sites are created to move to a more democratic, community process.
  3. The content of these new, community-created Stack Exchange sites will be publically owned under a Creative Commons license, instead of being owned by individuals or businesses.

If you’ve already created a Stack Exchange site, be sure to read the announcement in more detail to hear about our transition plan. Don’t be alarmed; we’d never do anything to mess with Stack Exchange sites that are already working.

Read all the details on the StackExchange Blog.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at April 13, 2010 08:06 PM

April 12, 2010

Allan Doyle

QR Codes in exhibits

We’ve put our first QR Code up at the museum as an experiment. This seems like one way to deal with “nth level” information that might be interesting to some visitors but not to others. And it’s a way to make it easy for visitors to bookmark information for themselves.

Generating the QR Code isn’t hard. Google’s Chart api can do it. Basically, anything you put fullowing the “chl=” part of the URL here will generate a code: http://chart.apis.google.com/chart?cht=qr&chs=350×350&chl=http://think.random-stuff.org/

We’ve set aside a specific URL space for our codes to send people to. Each link will first take you to a short URL which will then redirect you to the actual URL relevant to the exhibit. The advantage of this is that we can also put up short, “human readable” links on the exhibits.  That will let us track how often the links are getting used and what mobile devices people are using. We’ve also put an explanatory page at the top level of that space.

There was a little debate about how much we should try to explain QR Codes on the exhibit itself. You can see what we wound up with in the detail photo. Basically we’ve decided that visitors will either (a) recognize the code and know what to do with it, (b) not recognize it and skip over it, or (c) ask someone. The “c” people can be given a printout of the explanatory page.

We’ve decided to introduce the codes slowly and in a way that hopefully doesn’t get in the way of people who don’t know what they are or don’t choose to use them.

Another question was whether we should format the linked pages in a phone-friendly format. The easy answer (because it requires no additional resources…) was not to do that. Phones are getting pretty good at reading full-blown web pages.

For further reading, here’s a March 2009 article titled QR codes in the museum – problems and opportunities with extended object labels

What QR Code app am I using? Right now on my iPhone 3G, I’ve got Barcodes. It’s got a huge number of one-star, negative comments but it works for me. The critical thing you need to know is that it only works on QR Codes, not regular barcodes, and at least with the 3G, you have to take the photo from about 18″ away and then use the app to zoom it to the right size.

by Allan at April 12, 2010 06:59 PM

April 09, 2010

Chris Schmidt

MetaCarta Acquired by Nokia

As of April 9th, MetaCarta has been acquired by Nokia, and I am now an employee of Nokia working on local search in the Ovi services. (Woohoo!)

by crschmidt at April 09, 2010 02:37 PM

April 07, 2010

Paul Ramsey

On the road to Damascus... GPL to BSD

In response to a Twitter comment I made about my change in attitude towards the BSD and GPL open source licenses, Martin Daly asks:
Blog post on the Damascene (or otherwise) conversion from GPL-ish to BSD-ish please. In your own time.
The reason I was even mentioning the license issue is because I was watching a panel discussion at Where 2.0, in which Steve Coast, in defending the GPL-ish license used for OpenStreetMap, said (paraphrase) that the rationale was keeping a third party from taking the open data, closing it, and working from there. And that rationale pretty closely mirrors how we were thinking when choosing the GPL license for PostGIS, back in 2001.



I have experienced no spectral presence or flashing lights.



However, over the years, I have spent far too much time talking to various corporate folks about how the PostGIS GPL license wouldn't affect their plans to use PostGIS as a database component in their systems. For everyone who came and asked, I am sure many didn't. But in general the license was an impediment to some organizations engaging with the project. So there was a downside to the GPL. And was there an upside? Did the license protect us from privatized forks?



Well, yes, insofar is as it legally prevented them from happening. But it is worth questioning the implicit assumption that such forks are actually harmful to the project.



And here my experience watching the MapServer community (working with the BSD MIT license) was useful. Because over those same years, I watched the MapServer project chug healthily along, even as some third parties did in fact take the code and work on closed forks from time to time.



The lesson I have taken from my observation is that the strength of open source projects resides not in the licenses or the code, but in the communities arrayed about them. Copying the code of MapServer and doing your own thing with it does not stop MapServer developers from working, and in the long run you'll probably be better off working within the community than outside it, to gain from the efforts of others. Trying to maintain a private parallel branch and patch in the changes from the open development will quickly become more effort than it is worth. At the same time, because the BSD license is so permissive, there are no legal impediments for companies to engage with the community.



Look at any healthy open source project and ask yourself: what is more valuable, the code or the community? You could take all the code away from a healthy project, and it would start right up again from scratch and probably do a better job the second time. The value is in the human relationships and the aggregation of cooperating talent.



The GPL tends to keep corporate actors out of the community (for good reasons or bad, that's a separate discussion, but it does). That slows down the development of the project by reducing size of the development pool. Which, counterintuitively, makes forking or ignoring the open project more attractive. Because a slow moving project is easy to beat. A fast moving project is risky to fork, because it is likely that your (relatively understaffed) fork will be left behind by the open project.



So, back to what originally made me delve into license wankery, the question of OpenStreetMap license: the value of the OSM community and philosophy and infrastructure is way higher than the data at this point. And bringing corporate actors into the OSM community would only increase the relative advantage of the open project over any equivalent closed effort.



But changing licenses is a brutally hard thing to do, and it gets harder the longer a license is in place. PostGIS will never change licenses, for example. There are too many developers who contributed in the past under the GPL, and changing license would require the assent of all of them.



But if I ever start a new project, it will definitely be under the BSD.

 

by Paul Ramsey (noreply@blogger.com) at April 07, 2010 03:21 PM

April 05, 2010

Allan Doyle

Ubuntu on Eee Box B202

I’m always on the lookout for low-cost, easily maintained computers to drive displays or do other odd jobs around the museum. The Mac mini is still the gold standard for me. But even with the educational discount, a Mac mini can seem like overkill. We have an exhibit coming up where there’s going to be a projection of a single, still image. Some projectors can take an image from a flash drive, but none of the ones we have laying around can do that. So it was either spend money on a projector that can, or get a computer to drive it. (We could also use a DVD player with a single frame video loop, but that wouldn’t give me full 1024×768 resolution on the projector.) So last week I bought an Eee Box B202 preloaded with Linux.

Initially it was a bust. I hooked it up to a monitor, turned it on, and it booted to a text login prompt. No nice multi-media GUI, nothing. Just login:

Googling around was no help. I couldn’t figure out how to make it start X Windows or anything. The OS that’s on there is called Red Flag Linux. It seemed to have everything it needed, but I don’t have enough Xorg fu to understand how it was supposed to figure out how to configure itself for the monitor I had.

It turns out that was a blessing in disguise. After a little more Googling, I found myself making a bootable USB stick with Ubuntu 9.10 on it. I wound up using the “From Linux” instructions from another Ubuntu box. I debated whether or not to use the netbook distro or the full desktop one. I picked the full one. It turned out to be the right choice.

I did mess with the B202’s BIOS a bit to figure out how to make it boot from the USB stick. I’m not sure whether any of that was strictly necessary, but I had changed a few things to try to get the original software running.

After that it was smooth sailing. Ubuntu booted right up, running from the USB stick. It helpfully presented the option of installing from the stick. Once I made sure it would be able to work with the ethernet and WiFi, I used the installer to reformat the drive and install Ubuntu.

Things went so well, I’m ordering another one. The first one’s going to run our projector. The next one’s going to act as a WiFi to ethernet connection sharing router.

This was my first experience with a LiveCD/LiveUSB/etc. linux. It’s probably unremarkable these days, but I’m impressed with how easy it was to do.

Update: I just got my 2nd Eee Box. This one came with Windows XP pre-installed. Here are the steps needed to install Ubuntu:

  1. When it boots into ExpressGate, click the Exit icon, then hold down DEL to get into the BIOS setup
  2. In BIOS->Tools: Disable ExpressGate, then hit ESC to exit
  3. In BIOS->Boot->Hard Drives: hit + to make 1st drive = USB:SMI USB Disk
  4. Plug in your Ubuntu USB stick
  5. Hit F10 to save and exit

Now it will boot from the USB stick. At this point you can either boot into Ubuntu w/o installing (i.e. run from the stick) or install Ubuntu. If you’re unsure whether you want to go through with this before you try, then choose the first option. Note that it takes a longish time to boot from the stick.

Once it boots, you’ll see an icon in the upper left corner labeled “Install Ubuntu 9.10″. Double-click that to do an installation.

by Allan at April 05, 2010 08:33 PM

March 31, 2010

Joel Spolsky

The Design of Design

There is a new book out by Fred Brooks (author of The Mythical Man Month)!

The Design of Design

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at March 31, 2010 10:15 PM

Matt Perry

Distributed

I’ve been playing around with some distributed version control systems (DVCS) to replace svn.

First, the why: I’ll leave the details up to Joel in his excellent HgInit tutorial. Its mercurial-specific but the general concepts apply to any DVCS. The takeaway message for any project with > 1 developer is this:

Mercurial [ed: DVCS] separates the act of committing new code from the act of inflicting it on everybody else.

Next, the implementation: I’m using git to work on another project (Golden Cheetah) and its been a tough learning curve. Git is no doubt the most powerful DVCS out there. You can do magical things with it like combine commits and mess with history trees. And you can also screw things up pretty badly if you misinterpret the esotric docs for some non-intuitive piece of the workflow.

I just tried mercurial this morning - hg seems to fit my mind well. There is less power but the workflow is very clear and intuitive. And there are docs written for people who don’t want to do an in-depth study of their version control software. It stays out of the way.

Long story short, I’m going to use mercurial/hg for my new projects. Ah what the heck my old/ongoing projects as well. My googlecode repository has been converted over to Mercurial. Svn will stick around but wont be updated.

by perrygeo at March 31, 2010 10:06 PM

Joel Spolsky

Stack Overflow Party

The entire extended Stack Overflow team (including the Stack Exchange team) is meeting in New York in April to do some strategic planning. For example, we need to plan our Rock Band song lists, decide who gets to be on the drums and who is stuck with the USB cowbell, etc.

If you’re going to be in the city on Tuesday, April 6th, we’d like to invite you to join us for a party and a chance to meet the team. This is also a chance to see the new Fog Creek office if you haven’t visited yet.

If you want to come to the party, you must register. Please do not register if you’re not sure that you’re coming, as space is limited. Only people who are on the list will be admitted.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

by Joel Spolsky at March 31, 2010 08:46 PM

March 30, 2010

Allan Doyle

newsyslog on Mac OS X

So, just to finish off what I’ve learned about newsyslog on Mac OS X…

In addition to figuring out how to deal with denyhosts, here’s how to set up Apache and Mailman log rotation.

I’m running standard Apache 2.2 that comes with Mac OS X 10.6. I like to keep all my virtual hosts in one place, so in this example they are all in /Users/web/. Each host gets a directory structure with it’s own name:

# ls -l www.example.org/
drwxrwxr-x   4 _unknown  _unknown   136 Jan 27 16:47 htdocs
drwxrwxr-x  80 _www      _www      2720 Mar 29 00:33 logs
-rw-r--r--   1 adoyle    web       1881 Feb 20 15:44 www.example.org.conf

The logs for each virtual host go into the logs directory for that host (access_log, error_log, rewrite_log). Ownership on the log files turns out to be important. I’ve found it works best for me if they are owned by the www user and group (or _www, they are essentially the same – something I need to understand the reason for someday).

In /etc/newsyslog.d/local.conf, the following lines deal with rotating logs for three virtual hosts. Using the ‘G’ flag lets you use ‘*’ and other shell wildcards in the file names. I think I could probably have collapsed these into a single line if I had used /Users/web/*/logs/*log instead. In this case, folding things up too much makes it less readable, I think.

The _www:_www takes care of preserving the file ownership after the logs are rotated. A count of 30 means keep around up to 30 old logs. $D0 means rotate daily at 0:00. The ‘B’ flag prevents the “Log file was rotated” message. Apache keeps a pidfile in /var/run/httpd.pid. If you send a kill -30 to the pid in that file, it will cause the equivalent of an ‘apachectl graceful’.

If you don’t provide the pidfile and proper signal number, the logs will rotate, new log files get created, but Apache won’t write to them because it’s still trying to write to the old ones.

# logfilename          [owner:group]            mode count  size  when   flags [/pid_file] [sig_num]
/Users/web/lists.example.org/logs/*log _www:_www 664   30     *    $D0     GBJ /var/run/httpd.pid 30
/Users/web/foo.example.org/logs/*log   _www:_www 664   30     *    $D0     GBJ /var/run/httpd.pid 30
/Users/web/www.example.org/logs/*log   _www:_www 664   30     *    $D0     GBJ /var/run/httpd.pid 30
#

With Mailman I had a slight problem. Mailman doesn’t use nice .log or _log names. It just uses names like bounce, error, post, qrunner, etc. I could have made an entry in the local.conf file for each one, but that seemed error-prone. What if later there’s a new version of Mailman that generates different log files?

My initial assumption was that I could use …/logs/* and newsyslog wouldn’t try to rotate logs it had already rotated. Guess again. After two days, my disk had nearly filled up with files ending in .bz2, .bz2.bz2, .bz2.bz2.b2z…, you get the picture. Luckily I noticed it before the disk did fill up. I got suspicious when my backups on the third day were 15GB bigger than the ones on the first day. I have no idea how many files actually got created. ‘ls’ was unable to produce a listing in the amount of time I was willing to wait. Luckily rm -rf did work. It took several hours to delete all the files.

I’m assuming newsyslog had gotten into a recursive loop right away when it ran at midnight and never stopped churning out files until I killed it.

So anyway, the moral of the story is, if you’re going to use wildcards, make sure they don’t match the rotated logs.

# logfilename          [owner:group]            mode count  size  when  flags [/pid_file] [sig_num]
/Users/mailman/logs/*[a-z]         mailman:_www 664   30     *    $D0     GBJ /Users/mailman/data/master-qrunner.pid 1
#

Mailman wants to be hit with a kill -1, so that’s what I used.

Things have been noodling along for a few weeks with my setup, so I think I have the kinks ironed out.

by Allan at March 30, 2010 01:04 AM

March 29, 2010

Tyler Mitchell

Autodesk's Revit, Butterfly and Google Maps

Usually I'm 100% focused on open source software, but couldn't resist talking about this side project. A couple months ago Geoff pointed out the new Autodesk Butterfly project, allowing you to share and collaborate on drawings using only a web browser. They have some geospatial examples on there too. Here are couple notes about my tests on it lately.

http://spatialguru.com/files/butterfly-georef.png

read more

by Tyler Mitchell at March 29, 2010 05:03 PM

Allan Doyle

March 27, 2010

Allan Doyle

Links for 2010-03-26 [del.icio.us]

  • Dereza typeface

    Great looking typeface, targeted at kids' books, but could look very nice in presentations, etc.

March 27, 2010 07:00 AM

March 26, 2010

Allan Doyle

denyhosts on Mac OS X

I just spent some time figuring out how to set up denyhosts on Snow Leopard. I’ve used denyhosts before, but never felt like I had things set up properly for Mac OS. Now I think I have it figured out, so here it is. This is for 10.6, your mileage may vary on earlier versions.

I had three goals – get denyhosts working, get it to start automatically at boot time, and to deal with rotating the logs.

1. Installation

Easiest first – installing denyhosts. Note that you need to be root to do this. Pretty much just follow the directions. These are the three main settings to worry about.

SECURE_LOG = /private/var/log/secure.log
LOCK_FILE = /var/run/denyhosts.pid
DAEMON_LOG = /var/log/denyhosts

Note that you also may need to create the file /etc/hosts.deny:

touch /etc/hosts.deny

Using touch will create a zero-length file if it’s not there. It won’t affect the contents if it is there.

2. Log rotation

Mac OS 10.6 uses newsyslog to rotate some log files (I’m not sure why, but apache logs don’t seem to be dealt with by newsyslog). To add your own to the mix, just put a file into /etc/newsyslog.d/ following the format for newsyslog.conf(5). I called mine local.conf

# logfilename          [owner:group]            mode count size when  flags [/pid_file] [sig_num]
/var/log/denyhosts                              640   5     *    $D0     J
#

The trouble is, this rotated the log just fine, but then denyhosts stopped logging because newsyslog essentially pulls the rug out from under denyhosts by moving the file.

One design difference between newsyslog and logrotate is the way they deal with notifying processes that logs have been rotated. Logrotate uses prerotate and postrotate scripts, which would be ideal for denyhosts. The way you start and stop it is with

daemon-control start
 
daemon-control stop

daemon-control stop actually sends a SIGTERM to the denyhosts process, but that won’t do any good in the newsyslog config file since once stopped, you need a command line to start it up again. So I decided to tweak the daemon-control script to do this. I replaced the start() function with the one here:

def start(*args):
    cmd = "%s --daemon " % DENYHOSTS_BIN
    if args: cmd += ' '.join(args)
 
    print "starting DenyHosts:   ", cmd
 
    while True:
        os.system(cmd)
        time.sleep(5)
 
        while True:
            pid = getpid()
            if pid >= 0:
                time.sleep(300)
            else:
                break

This just keeps daemon-control running rather than letting it exit after it starts denyhosts. The outer loop starts denyhosts running and later restarts it. The inner loop just waits until it sees the pid file go away. That’s a sure sign that denyhosts stopped running, most likely because of the SIGHUP it will get from newsyslog. Now all I needed to do was add the signal info to my /etc/denyhosts.d/local.conf /etc/newsyslog.d/local.conf file:

# logfilename          [owner:group]            mode count size when  flags [/pid_file] [sig_num]
/var/log/denyhosts                              640   30     *  $D0     BJ  /var/run/denyhosts.pid 15
#

I’ve also changed it to keep 30 days of logs, and added the B flag to prevent newsyslog from adding a line to the file saying it’s rotated the logs. Note that I changed the name to daemon-control2 so if I update denyhosts later, my changes don’t get clobbered.

3. Start at boot time

It turns out that modifying daemon-control to never exit is also just the ticket for running it under launchd. Launchd doesn’t work well on scripts that launch daemonized processes. It watches the script and notices that it’s exited, then tries to start it again.

I made a file called /Library/LaunchDaemons/net.hosts.deny.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>net.denyhosts</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/share/denyhosts/daemon-control2</string>
      <string>start</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>ServiceDescription</key>
    <string>Lauch denyhosts</string>
  </dict>
</plist>

Get it started with launchctl:

 launchctl load /Library/LaunchDaemons/net.hosts.deny.plist

My /etc/hosts.deny has about 8500 hosts in it right now. Many of those are probably from the denyhosts synchronization feature pulling in IP addresses from the central server.

Update 2010-03-26: Added some links and clarified some bits.

Update 2010-06-06: Note that /etc/hosts.deny must be present. denyhosts won’t create it.

by Allan at March 26, 2010 12:46 AM

March 25, 2010

Allan Doyle

March 24, 2010

Allan Doyle

March 23, 2010

Paul Ramsey

FOSS4G 2011 Decision Process

The 2011 FOSS4G siting process has begun. After the 2010 process, where there were too many good proposals and not enough slots, we decided to do a couple things to lower the effort level required by bidders in aggregate.



First, we wanted to be more definitive about our regional siting plan, which is to rotate between Europe (2010), North America (2011) and elsewhere (with elsewhere being Asia (2012) in the current rotation plan). For 2011 the regional arrow points to North America, since 2010 is in Europe.



Second, we wanted to lower the bar to submitting a proposal. In 2010 we got four full proposals, and could choose only one. So 75% of the bidders did a large amount of preliminary research and spade-work for no purpose. For the 2011 process, we will start with a "letter of intent" phase, a short document outlining expressions of interest. Hopefully having all the interested parties publicly declared will promote locations coming together and forming joint bids.



The OSGeo conference committee will vote on the letters and only the top two will be asked to prepare full bid documents. That would still leave one potentially disappointed party at the end of the process, but is a big improvement over leaving three, as happened in the 2008 and 2010 processes. And hopefully the letters phase will encourage groups to consolidate bids so we won't have to even run the final round.



The winning site will still have to submit a proposal and budget, so the Board and Conference Committee have confidence that the local team has thought things through, but the kind of research necessary to prepare a full proposal is a necessary precursor to putting on a conference, so the effort will be put to good purpose.



I'm looking forward to seeing the letters!

 

by Paul Ramsey (noreply@blogger.com) at March 23, 2010 11:30 PM