How to produce a JSON tree with nested data from Scrapy

This was an interesting puzzle: creating one single well formed JSON from a hierarchy of web pages. E.g., the sporting goods hierarchy of an e-commerce site could be Categories, Brands, Products. And so you'd like to output JSON like this: https://gist.github.com/dogweather/cb364044df65bd104a0b051f257ae1f6 [Etc.] As an aside, I like architecting my systems to generate this kind of … Continue reading How to produce a JSON tree with nested data from Scrapy

Making mistakes: Django startproject and Rails new

I like to see how software reacts when I step off the happy path and make a mistake. Today I found this interesting difference with an unknown (or misspelled) command line option: "--derp": https://gist.github.com/dogweather/aac2dce97936484e765c04105ef6fa06 https://gist.github.com/dogweather/937420a48708ee36f3d0268161d61b80 An interesting difference! Personally, Django is reacting like I'd expect. And so for me it's following the principle of least … Continue reading Making mistakes: Django startproject and Rails new

Python vs. Haskell round 2: Making Me a Better Programmer

Another point for Haskell In Round 1 I described the task: find the number of "Titles" in an HTML file. I started with the Python implementation, and wrote this test: Very simple: I had already downloaded the  web page  and so this function had to do just two things: (1) read in the file, and then (2) parse … Continue reading Python vs. Haskell round 2: Making Me a Better Programmer

Python vs. Haskell round 1: Test Output

The point goes to Haskell On the surface, it may sound silly to compare these two languages because they're about opposite as you could get: Python is interpreted, dynamically typed, and slightly weakly typed as well. Haskell on the other hand, is compiled, statically and strongly typed. But they're both open source, and they both … Continue reading Python vs. Haskell round 1: Test Output

Goodbye “X for Y”: the cryptic Ruby error is becoming friendlier

Anyone who's used Ruby has seen this message: r.rb:1:in `name': wrong number of arguments (3 for 2) (ArgumentError) This particular error has been driving me nuts for years. It's just so unnecessarily difficult to interpret — especially if Ruby's not the only language you use. I never remember which number is which. Compare to Python: TypeError: … Continue reading Goodbye “X for Y”: the cryptic Ruby error is becoming friendlier

Python project organization, circa early 2015

The best, current guides I've found Python Packaging Authority's tutorial and sample project Cookie Cutter project templating system, specifically cookiecutter-pypackage Open Sourcing a Python Project the Right Way, and Supercharge Your Python Developers by Jeff Knupp It's hard to find current best practices for organizing a reusable python package. I'm starting a new project to get the world's … Continue reading Python project organization, circa early 2015