In a nutshell The Public.Law Open-gov spiders set the secrets in production via Zyte's Spider Settings UI. Then this code at the end of settings.py enables local development mode: # # In development mode only, set the sensitive and environment- # dependent configuration values via env variables. On development # machines, set `SCRAPY_DEVELOPMENT_MODE` to make … Continue reading How to set Scrapy configuration values
Tag: Python
How to produce a JSON tree with nested data from Scrapy
This was an interesting puzzle: creating one single well formed JSON from a hierarchy of web pages. E.g., the sporting goods hierarchy of an e-commerce site could be Categories, Brands, Products. And so you'd like to output JSON like this: https://gist.github.com/dogweather/cb364044df65bd104a0b051f257ae1f6 [Etc.] As an aside, I like architecting my systems to generate this kind of … Continue reading How to produce a JSON tree with nested data from Scrapy
Making mistakes: Django startproject and Rails new
I like to see how software reacts when I step off the happy path and make a mistake. Today I found this interesting difference with an unknown (or misspelled) command line option: "--derp": https://gist.github.com/dogweather/aac2dce97936484e765c04105ef6fa06 https://gist.github.com/dogweather/937420a48708ee36f3d0268161d61b80 An interesting difference! Personally, Django is reacting like I'd expect. And so for me it's following the principle of least … Continue reading Making mistakes: Django startproject and Rails new
Python vs. Haskell round 2: Making Me a Better Programmer
Another point for Haskell In Round 1 I described the task: find the number of "Titles" in an HTML file. I started with the Python implementation, and wrote this test: Very simple: I had already downloaded the web page and so this function had to do just two things: (1) read in the file, and then (2) parse … Continue reading Python vs. Haskell round 2: Making Me a Better Programmer
Python vs. Haskell round 1: Test Output
The point goes to Haskell On the surface, it may sound silly to compare these two languages because they're about opposite as you could get: Python is interpreted, dynamically typed, and slightly weakly typed as well. Haskell on the other hand, is compiled, statically and strongly typed. But they're both open source, and they both … Continue reading Python vs. Haskell round 1: Test Output
Goodbye “X for Y”: the cryptic Ruby error is becoming friendlier
Anyone who's used Ruby has seen this message: r.rb:1:in `name': wrong number of arguments (3 for 2) (ArgumentError) This particular error has been driving me nuts for years. It's just so unnecessarily difficult to interpret — especially if Ruby's not the only language you use. I never remember which number is which. Compare to Python: TypeError: … Continue reading Goodbye “X for Y”: the cryptic Ruby error is becoming friendlier
The problem with packaging in Python
Excellent article. We got to this situation because of the changing nature of writing software, and how we deploy. Our standards have risen, and packaging (and tooling in general) takes on ever greater importance. But we don't have to theorize and invent paradigms from scratch. A few other languages have solved this. Ruby is the … Continue reading The problem with packaging in Python
Python project organization, circa early 2015
The best, current guides I've found Python Packaging Authority's tutorial and sample project Cookie Cutter project templating system, specifically cookiecutter-pypackage Open Sourcing a Python Project the Right Way, and Supercharge Your Python Developers by Jeff Knupp It's hard to find current best practices for organizing a reusable python package. I'm starting a new project to get the world's … Continue reading Python project organization, circa early 2015
mctop – a top-like app for monitoring memcached
I couldn't find a simple script for monitoring memcached's efficiency, so I started this project.