How to produce a JSON tree with nested data from Scrapy

This was an interesting puzzle: creating one single well formed JSON from a hierarchy of web pages. E.g., the sporting goods hierarchy of an e-commerce site could be Categories, Brands, Products. And so you'd like to output JSON like this: https://gist.github.com/dogweather/cb364044df65bd104a0b051f257ae1f6 [Etc.] As an aside, I like architecting my systems to generate this kind of … Continue reading How to produce a JSON tree with nested data from Scrapy

Making mistakes: Django startproject and Rails new

I like to see how software reacts when I step off the happy path and make a mistake. Today I found this interesting difference with an unknown (or misspelled) command line option: "--derp": https://gist.github.com/dogweather/aac2dce97936484e765c04105ef6fa06 https://gist.github.com/dogweather/937420a48708ee36f3d0268161d61b80 An interesting difference! Personally, Django is reacting like I'd expect. And so for me it's following the principle of least … Continue reading Making mistakes: Django startproject and Rails new

The perfect IDE in pictures: Part 1, *it* works for *you*

I don't think the perfect IDE exists. But all the right ingredients are out there. Allan MacGregor wrote a post about VIM as the "perfect IDE" that started an interesting discussion on dev.to. But I'm skeptical about the pages-long config file, the dozens of independent plugins, and whether the result is really an "IDE". This reminded … Continue reading The perfect IDE in pictures: Part 1, *it* works for *you*

Testing the multi-subdomain Rails app

Writing tests for a multi-tenant, multi-subdomain app turns out to be very tricky to figure out, e.g.: I saw that, and understood the frustration. Integration tests ("request specs" or "feature specs") are built on a stack of frequently changing libraries and shifting API's. And the recipe for subdomain-aware testing isn't documented in any particular tool's notes. … Continue reading Testing the multi-subdomain Rails app

The Economics of Programming: Externalized vs. Internalized Costs

Many days I feel like my work as an agile consultant is simply internalizing (externalized) costs. First example that comes to mind: software development done too quickly which creates technical debt as it goes. In the short term, a project like that can seem very successful, exceeding expectations for delivery time and customer satisfaction. And then … Continue reading The Economics of Programming: Externalized vs. Internalized Costs

Python vs. Haskell round 2: Making Me a Better Programmer

Another point for Haskell In Round 1 I described the task: find the number of "Titles" in an HTML file. I started with the Python implementation, and wrote this test: Very simple: I had already downloaded the  web page  and so this function had to do just two things: (1) read in the file, and then (2) parse … Continue reading Python vs. Haskell round 2: Making Me a Better Programmer

Python vs. Haskell round 1: Test Output

The point goes to Haskell On the surface, it may sound silly to compare these two languages because they're about opposite as you could get: Python is interpreted, dynamically typed, and slightly weakly typed as well. Haskell on the other hand, is compiled, statically and strongly typed. But they're both open source, and they both … Continue reading Python vs. Haskell round 1: Test Output