How to produce a JSON tree with nested data from Scrapy

This was an interesting puzzle: creating one single well formed JSON from a hierarchy of web pages. E.g., the sporting goods hierarchy of an e-commerce site could be Categories, Brands, Products. And so you’d like to output JSON like this: [Etc.] As an aside, I like architecting my systems to generate this kind of output…More

Making mistakes: Django startproject and Rails new

I like to see how software reacts when I step off the happy path and make a mistake. Today I found this interesting difference with an unknown (or misspelled) command line option: “–derp”: An interesting difference! Personally, Django is reacting like I’d expect. And so for me it’s following the principle of least astonishment. Rails,…More

The perfect IDE in pictures: Part 1, *it* works for *you*

I don’t think the perfect IDE exists. But all the right ingredients are out there. Allan MacGregor wrote a post about VIM as the “perfect IDE” that started an interesting discussion on But I’m skeptical about the pages-long config file, the dozens of independent plugins, and whether the result is really an “IDE”. This reminded…More

Testing the multi-subdomain Rails app

Writing tests for a multi-tenant, multi-subdomain app turns out to be very tricky to figure out, e.g.: I saw that, and understood the frustration. Integration tests (“request specs” or “feature specs“) are built on a stack of frequently changing libraries and shifting API’s. And the recipe for subdomain-aware testing isn’t documented in any particular tool’s notes.…More