Tag: Python

How to set Scrapy configuration values

In a nutshell The Public.Law Open-gov spiders set the secrets in production via Zyte's Spider Settings UI. Then this code at the end of settings.py enables local development mode: # # In development mode only, set the sensitive and environment- # dependent configuration values via env variables. On development # machines, set `SCRAPY_DEVELOPMENT_MODE` to make … Continue reading How to set Scrapy configuration values →

How to produce a JSON tree with nested data from Scrapy

This was an interesting puzzle: creating one single well formed JSON from a hierarchy of web pages. E.g., the sporting goods hierarchy of an e-commerce site could be Categories, Brands, Products. And so you'd like to output JSON like this: https://gist.github.com/dogweather/cb364044df65bd104a0b051f257ae1f6 [Etc.] As an aside, I like architecting my systems to generate this kind of … Continue reading How to produce a JSON tree with nested data from Scrapy →

Making mistakes: Django startproject and Rails new

I like to see how software reacts when I step off the happy path and make a mistake. Today I found this interesting difference with an unknown (or misspelled) command line option: "--derp": https://gist.github.com/dogweather/aac2dce97936484e765c04105ef6fa06 https://gist.github.com/dogweather/937420a48708ee36f3d0268161d61b80 An interesting difference! Personally, Django is reacting like I'd expect. And so for me it's following the principle of least … Continue reading Making mistakes: Django startproject and Rails new →

Python vs. Haskell round 2: Making Me a Better Programmer

Another point for Haskell In Round 1 I described the task: find the number of "Titles" in an HTML file. I started with the Python implementation, and wrote this test: Very simple: I had already downloaded the  web page  and so this function had to do just two things: (1) read in the file, and then (2) parse … Continue reading Python vs. Haskell round 2: Making Me a Better Programmer →

Python vs. Haskell round 1: Test Output

The point goes to Haskell On the surface, it may sound silly to compare these two languages because they're about opposite as you could get: Python is interpreted, dynamically typed, and slightly weakly typed as well. Haskell on the other hand, is compiled, statically and strongly typed. But they're both open source, and they both … Continue reading Python vs. Haskell round 1: Test Output →

Goodbye “X for Y”: the cryptic Ruby error is becoming friendlier

Anyone who's used Ruby has seen this message: r.rb:1:in `name': wrong number of arguments (3 for 2) (ArgumentError) This particular error has been driving me nuts for years. It's just so unnecessarily difficult to interpret — especially if Ruby's not the only language you use. I never remember which number is which. Compare to Python: TypeError: … Continue reading Goodbye “X for Y”: the cryptic Ruby error is becoming friendlier →

Python project organization, circa early 2015

The best, current guides I've found Python Packaging Authority's tutorial and sample project Cookie Cutter project templating system, specifically cookiecutter-pypackage Open Sourcing a Python Project the Right Way, and Supercharge Your Python Developers by Jeff Knupp It's hard to find current best practices for organizing a reusable python package. I'm starting a new project to get the world's … Continue reading Python project organization, circa early 2015 →