How to set Scrapy configuration values

In a nutshell The Public.Law Open-gov spiders set the secrets in production via Zyte's Spider Settings UI. Then this code at the end of settings.py enables local development mode: # # In development mode only, set the sensitive and environment- # dependent configuration values via env variables. On development # machines, set `SCRAPY_DEVELOPMENT_MODE` to make … Continue reading How to set Scrapy configuration values →

The Iterate-and-Mutate Programming Anti-Pattern

Take a look at this small, typical code snippet: let car_ids = []; for (let i = 1; i <= 5; i++) { const car_id = `car-${i}`; if (car_is_empty(car_id)) { car_ids.push(car_id); } } return car_ids; I call this the iterate-and-mutate pattern. It comes up all the time in code. But every time we see it, … Continue reading The Iterate-and-Mutate Programming Anti-Pattern →

Elixir can be a lot easier to learn than Python

I stumbled on a surprising instance of Elixir being much easier to learn and use than Python: I was in each one’s REPL, comparing how to do some typical programming task. (I forget exactly, but it might have been JSON parsing.) The function in each language was easy to call. And in Elixir, the REPL … Continue reading Elixir can be a lot easier to learn than Python →

Ruby RSpec or Minitest on new projects?

I've used both for years. But I use RSpec on my new projects: RSpec is an application, but Minitest is only a library. This cannot be overstated. RSpec has libraries too, of course, but the user experience is via a mature CLI program. (Bisect is wicked, and a life-saver!) It has incredibly useful options and … Continue reading Ruby RSpec or Minitest on new projects? →

The “MacOS Feel” that Developers Love

IMO, it comes down to a: Controlled UI, Consistent user experience, and Dev-friendly choices. Windows controls the desktop, but only has so-so consistency (many windowing toolkits in evidence in the OS). And the choices aren’t dev-friendly (more below). Linux has nearly zero control over its own UI: The Chrome browser, for example, refuses to follow … Continue reading The “MacOS Feel” that Developers Love →

How to produce a JSON tree with nested data from Scrapy

This was an interesting puzzle: creating one single well formed JSON from a hierarchy of web pages. E.g., the sporting goods hierarchy of an e-commerce site could be Categories, Brands, Products. And so you'd like to output JSON like this: https://gist.github.com/dogweather/cb364044df65bd104a0b051f257ae1f6 [Etc.] As an aside, I like architecting my systems to generate this kind of … Continue reading How to produce a JSON tree with nested data from Scrapy →

Making mistakes: Django startproject and Rails new

I like to see how software reacts when I step off the happy path and make a mistake. Today I found this interesting difference with an unknown (or misspelled) command line option: "--derp": https://gist.github.com/dogweather/aac2dce97936484e765c04105ef6fa06 https://gist.github.com/dogweather/937420a48708ee36f3d0268161d61b80 An interesting difference! Personally, Django is reacting like I'd expect. And so for me it's following the principle of least … Continue reading Making mistakes: Django startproject and Rails new →

The perfect IDE in pictures: Part 1, it works for you

I don't think the perfect IDE exists. But all the right ingredients are out there. Allan MacGregor wrote a post about VIM as the "perfect IDE" that started an interesting discussion on dev.to. But I'm skeptical about the pages-long config file, the dozens of independent plugins, and whether the result is really an "IDE". This reminded … Continue reading The perfect IDE in pictures: Part 1, *it* works for *you* →

Testing the multi-subdomain Rails app

Writing tests for a multi-tenant, multi-subdomain app turns out to be very tricky to figure out, e.g.: I saw that, and understood the frustration. Integration tests ("request specs" or "feature specs") are built on a stack of frequently changing libraries and shifting API's. And the recipe for subdomain-aware testing isn't documented in any particular tool's notes. … Continue reading Testing the multi-subdomain Rails app →

What is a Design Pattern? (Answered.)

I recently answered that question: The entire concept of Design Patterns for software was popularized and maybe invented by the Gang of Four. This is the book. It's an amazing artifact, organized differently from any other book I've seen. It's worth checking it out from a library and reading at least the introduction and a … Continue reading What is a Design Pattern? (Answered.) →