Skip to content
Past Localhost
M01Local is not productionLesson 4 of 4

M01 summary and postmortem: Knight Capital

The module's core ideas, review questions and the deploy that lost 460 million dollars in 45 minutes.

What to take from this module

Section titled “What to take from this module”
  • A production request passes through DNS, TLS, a CDN, a load balancer, the app, the pool and the database. Each hop costs time and can fail.
  • The network usually costs more than the code. Location and the number of round trips matter more than micro-optimizations.
  • Environments differ in data, scale, network, configuration and permissions. Remove the accidental differences and validate configuration at startup.
  • Build once and promote the same artifact. The code shouldn’t ask which environment it’s in.
  • The local environment hides concurrency, latency, volume and partial failures. Test each one on purpose.
  • Rules that must always hold (uniqueness, counters) belong in the database, the only place that sees every request.

Postmortem

Section titled “Postmortem”

Review questions

Section titled “Review questions”
An endpoint takes 20 ms locally and 800 ms in production, with no errors. What are the first two hypotheses you’d test?

Many database round trips per request (N+1), which cost almost nothing locally, and network distance: server and database in different regions, or users far from the server. To tell them apart, compare the database step in Server-Timing with the total time measured by curl -w.

Why is “check if it exists, then insert” a bug even if it passes all your tests?

Because the gap between the check and the insert lets another request do the same thing. With one user, that gap is never exercised. The fix is a UNIQUE constraint in the database, plus handling the violation error (23505 in Postgres).

What’s wrong with having if (process.env.NODE_ENV === ‘production’) in business logic?

That code path only runs in production, so it has never been tested. Ideally the configuration says what to use (which key, which URL) and the code is the same in every environment.

What does staging prove, and what doesn’t it prove?

It proves the build starts, the migration runs, the configuration is complete and the integrations respond. It doesn’t prove that the system can take production load or handle real production data.

Go deeper

Materials in English unless noted.

Written by About this guide