Running Trading Algos in Production: The Kill Switch
Feed watchdogs, silent socket death, gap guards, roll automation and a kill line set from the backtest's drawdown distribution before the algo goes live.
Part 20 of 20 · How to Algo · Previous: Backtest vs live
The backtest assumes a perfect world: every tick arrives, every order is acknowledged, the clock is right, the contract is the front month, and the program is running. Production is the work of making each of those assumptions true, detecting when it is not, and deciding in advance what to do about it.
The failures that have cost us the most were not crashes. A crash is loud. The expensive failures are quiet: a strategy that stops trading without an error, a position the system believes it does not have, a price jump booked as a single session’s profit. This last part of the series is about the machinery that catches those, and about the hardest production decision of all: when to stop.

Silence is the default failure
A feed that dies without closing. In June 2026 our algos service booked zero trades for hours. The upstream tick connection had died silently after a redeploy: no close event, no error, simply no more messages. The reconnect logic only ran on a close event, so it never ran. The agents never captured the daily close and never traded. The fix is a watchdog: if no message arrives for 90 seconds, force a reconnect. It covers silent drops and sockets stuck while connecting.
A global health check that hides a partial outage. Later our gateway lost the subscriptions for several products after an internal component restarted during the cash-open burst. ES, NQ, gold and crude printed nothing for 47 minutes while other products kept flowing, and those other products kept the global “data is arriving” clock alive. Nothing alarmed. We now keep per-symbol liveness clocks for the products that matter: 45 seconds of silence triggers a resubscribe, longer silence declares the session dead.
Detection plus recovery time is the real gap. Two earlier stalls at the cash close produced 13-minute holes: three minutes to detect, plus a long fixed wait before reconnecting. A two-stage response (resubscribe on the live session first, reconnect only if that fails) cut the expected gap to a few minutes.
After every restart we run an empty-minute scan, one query that lists every regular-session minute with no prints per symbol. It would have shown us, on the same day, a 20-minute hole caused by one of our own restarts that we only noticed five days later through a stale price elsewhere.
Guards against impossible bookings
The gap guard. Daily agents compute P&L from the previous close to the current one. If an agent is offline for weeks, or freshly launched against an old reference, the “previous close” can be months old. One of our agents would have booked a phantom of about +$90,000 as a single session’s move against a stale seed price. The guard: if the previous bar is more than five days old, re-anchor with zero P&L instead of booking the jump.
The late-start guard. An intraday agent that reads the market at the open must not fabricate an open it did not see. If it starts mid-session, after a deploy or a crash, it must not reconstruct the opening read from later prices. Our agents take a signal only if they captured the opening price within the first two minutes. The guard fired correctly on the day it was deployed.
The quantity guard. The execution engine refuses any target position above a hard maximum, regardless of what the signal asks for.

Position truth comes from the broker
The engine’s belief about its position and the account’s actual position can drift apart. On a paper account our engine’s resync step tried to flatten a leftover position before the order route was usable, the flatten failed, and the code assumed success anyway. The next reconcile added to the position: the account was long two contracts while the engine believed it held one.
The fix has two halves. The resync retries the flatten until the broker accepts it and only then asserts that the account is flat, and if it cannot get there it refuses to trade at all. And every cycle reconciles against the broker’s reported position, never against the engine’s own memory.
Rolls, sessions and the clock
Rolls as events. Our engine follows the gateway’s current front contract rather than computing its own, and when the front changes it flattens both the new front and the contract that just rolled off before trading again. A restart across a roll therefore cannot strand a position in an expiring contract. Part 4 covers how the front is chosen.
The session clock. Prop-firm accounts must be flat before the daily close. Our engine flattens every account and refuses new entries from 16:55 to 18:05 New York, in real New York time including daylight saving. Why 16:55 and not 16:59? Because one loop cycle can stretch to about 150 seconds behind a slow broker connection, and a flatten that starts at 16:59 can land after 17:00. Accounts with a ready session are processed first, so a dead account can only delay itself. Part 3 explains why every session gate must use exchange time.
Do not deploy into an open session. Deploys restart things, restarts create gaps and missed opens. We schedule updates for the maintenance window and treat a deploy during the session as an incident.
The kill switch
Our execution engine reads one configuration document on every loop. A single flag in it decides whether the engine trades live or in dry-run. Switching it off stops new orders on the next cycle; switching it on arms the engine, which drops and rebuilds its broker sessions cleanly on that transition. No restart is involved, and the flag survives redeploys. An optional contract pin lives in the same place, so a symbol can be pinned without a code change.
Properties worth copying:
- One control, read every loop. Not an environment variable that needs a restart, not a file on one machine.
- Arming is a deliberate transition. The engine treats “off to on” as an event and starts from a clean state.
- Usable by someone else. The person who notices a problem at 03:00 may not be the author.
When to stop: set the line before the first trade
Every live strategy will have a losing stretch. The question is whether a given stretch means the edge is gone or is normal variance. Answer it before going live, when you have no position to defend.
Start from how much a working strategy loses. In a simulation of daily returns for a strategy with a true Sharpe of 1, the maximum drawdown within one year was about 0.78 times the annual volatility at the median, 1.55 times at the 95th percentile and 1.99 times at the 99th. Over three years the 95th percentile rose to about 2.16 times. (Simulated, normal returns; real tails are fatter.)
And losing periods are not rare even when everything works:
| True Sharpe | Down after 6 months | Down after 1 year | Down after 2 years |
|---|---|---|---|
| 0.5 | 36% | 31% | 24% |
| 1.0 | 24% | 16% | 8% |
| 2.0 | 8% | 2% | 0.2% |
So a pre-committed rule looks like this: compute the drawdown distribution for the sizing you will run, using the backtest’s own trades (resampled) and the more pessimistic of the backtest and walk-forward Sharpe. Set a review line around the 95th percentile and a kill line at or beyond the 99th. Write both down with the date. When the kill line is hit, stop and investigate; do not resize to stay in.
The chart at the top is that procedure on a simulated strategy: 5,000 resampled paths of 700 backtest trades averaging 60 USD. The worst drawdown within 300 trades was 6.9k at the median, 13.3k at the 95th percentile and 17.6k at the 99th, which became the kill line. A live run losing 40 USD a trade crosses it at trade 182, long before its average would look convincing on its own.
Retirement can also come from outside the P&L. When an audit found that two of our narrow-range breakout sleeves had been validated on artefact fills, we retired them even though their live records were too short to say anything either way. Evidence that the backtest was wrong outranks a live record that has not yet had time to disagree.
Alerts that someone reads
Every alert should go somewhere a person will see it, and it should be rare enough that they still read it. Ours report entries, exits, a daily wrap after the close, and the operational alarms above. One honest gap remains in our own setup: a nightly timestamp check writes to a log and does not page anyone, which means it only helps if someone looks. Every monitor should be checked for that property: who finds out, and how fast?
Checklist
- Add a message watchdog to every data connection; reconnect on silence, not only on close.
- Keep per-symbol liveness clocks for the instruments you trade.
- Run an empty-minute scan after every restart.
- Guard against stale references, late starts and oversized targets.
- Reconcile positions with the broker every cycle; refuse to trade if flat cannot be confirmed.
- Treat rolls as events; flatten old and new contracts before trading again.
- Gate sessions in exchange time; leave margin for slow cycles before hard deadlines.
- Never deploy into an open session.
- Build a kill switch read every loop, usable by someone else.
- Set review and kill lines from the drawdown distribution before the first trade, and follow them.
That closes the series. The order of the twenty parts is the order in which we wish we had learned them: start with a hypothesis you can kill, and every part after that is a way to kill it more honestly. For the whole path on one page, the roadmap for coding a trading algorithm ties the twenty parts together. The tick and depth data behind all of our tests is described in the data docs, and the strategies that survived are tracked live on our algos page.
Frequently asked questions
What is a kill switch in algorithmic trading?
A single control that stops new orders and, if configured, flattens positions. It should be read on every loop and survive restarts and redeploys. Ours is a flag in a configuration store: switching it off stops the engine on the next cycle, and arming it again is a flag flip, not a restart. It must be usable in seconds by someone who did not write the code.
When should you stop trading a strategy that is losing?
When it breaches a line you set before it went live, based on the drawdowns the strategy produces when it works. In a simulation of a strategy with a true Sharpe of 1, the one-year maximum drawdown exceeded 1.55 times the annual volatility in 5% of years and 1.99 times in 1%. A drawdown beyond the 99th percentile is evidence the strategy changed; one inside it is the strategy behaving normally.
How do you set a maximum drawdown limit from a backtest?
Resample the backtest's own trades into thousands of paths at the size you will run, and read off the distribution of worst drawdowns. In our simulated example, 5,000 paths of 300 trades gave 6.9k USD at the median, 13.3k at the 95th percentile and 17.6k at the 99th; the 99th became the kill line. Write it down with the date before the first live trade.
What happens if the market data feed silently disconnects?
Often nothing visible: no error, no new prices, no trades. Our algos once booked zero trades for hours after a redeploy because the upstream socket died without a close event, and the reconnect logic only ran on a close. A watchdog that forces a reconnect after 90 seconds without a message fixed it.
How long does it take to know whether a live strategy has stopped working?
Longer than most traders wait. Even with a true Sharpe of 1, a strategy is down after six months about 24% of the time and down after a full year about 16% of the time; at a Sharpe of 0.5 it is down after a year about 31% of the time. Short losing stretches are expected and carry little information on their own.