Feeder: architecture
The full design behind the ten-minute version. The reference implementation in this repository runs every part described here on a synthetic network in one DuckDB file; the production notes say what changes against a real utility's systems. Known gaps lists what the synthetic world cannot show.
1. Sources
| Fact | System | Freshness |
|---|---|---|
| Last gasp, power restore | Head-end system of the smart meters (AMI) | seconds |
| Daily import, export, interval reads received | Meter data management (MDM) | daily, after the overnight run |
| Feeder head load, end-of-line voltage | SCADA historian | 15 minutes |
| Transformer loading (monitored units) | SCADA historian | hourly |
| Outage tickets | Outage management system (OMS) | seconds |
| Work orders, crew movements | Workforce management (WFM) | seconds |
| Feeders, transformers, tie points, meter connectivity | GIS network model | nightly |
| Accounts, billing day, priority register, solar registrations | Customer information system | hourly |
| Calls | Contact centre | seconds |
| Booked planned work | Works planning | hourly |
The reference world holds these as tables (feeder/world/schema.sql); the data map in systems and fact_sources says which system owns which column.
2. The definitions
Each term is a YAML file in feeder/dictionary/definitions/: a title, owner, version history, parameters, SQL templated on {now} and the parameters, and tests. A change to a parameter is appended to a history file and bumps the version of every variant that uses it; the ledger records the version each decision ran on.
Detector terms return one row per incident with a subject (transformer, feeder, meter, work order, planned work), a start time and an evidence object. The ten detectors:
| Term | Cadence | Subject | What it sees |
|---|---|---|---|
| outage_cluster | every tick | transformer or feeder | meters dark three minutes or more with no ticket or planned work covering them |
| priority_customer_in_outage | every tick | meter | a priority-register premise dark in an unplanned outage |
| crew_to_restored_area | every tick | work order | a crew still driving to a ticket whose meters are eight in ten back |
| voltage_out_of_band | every tick | feeder | three readings in a row outside 0.94 to 1.06 per unit |
| planned_work_collision | every tick | planned work | booked work within twelve hours on a feeder whose own or tie feeder has an open fault |
| transformer_overload | hourly | transformer | two hours or more above 120 % of the rating |
| wrong_feeder_meter | hourly | meter | lost and regained supply with another feeder's cluster while its own stayed on |
| zero_consumption | daily 06:00 | meter | three complete days of zero at an occupied premise that used power before |
| reverse_flow_unregistered | daily 06:00 | meter | two days of export with no solar registered |
| estimated_bill | daily 06:00 | meter | three days with no reads and the billing close within five days |
Sector terms that Ask answers: outage, sustained outage (two meanings: regulator, over three minutes; network operations, over five), restored, affected customers, priority customer, feeder load.
3. Decisions
3.1 From incident to proposal
The runner deduplicates incidents (same kind and subject within fifteen minutes, already in the queue, already overridden by a person, or still inside the last action's cooldown) and asks a worker for one proposal each. The worker uses the fixed rule when the runbook leaves one action, or when the policy says rules decide that kind; otherwise it may ask a model, validates the answer against the policy and the evidence, and falls back to the rule if the answer is unusable.
3.2 The controller
For each proposal, in order: the action exists, is not on the never list, and is allowed for this kind; its arguments are known, well-formed and name only ids that the evidence names; its written conditions hold (a crew is only recalled when eight in ten of the ticket's meters are back); an ask-first rule does not apply (a notice promising more than twelve hours); the tier is not 2; no cooldown is running; the feeder's hourly budget, the network's per-minute limit, or the metering desk's own limit is not used up. The first failing check decides the outcome: execute, queue for a person, wait for the next minute, or decline, each with a plain-English reason.
3.3 Runbooks that chain
A new outage ticket is followed by the notice to its customers, with an estimated restoration time from a fixed table: two hours for one transformer, three for a section, one more for a hundred customers or more, plus time for the open tickets across the network.
3.4 Ledger and undo
Every outcome is a ledger row; approvals, rejections and rollbacks are new rows pointing at the row they act on. Each executed tool records the rows it changed and how to neutralise the ones it created (a ticket is withdrawn, a notice cancelled, a crew stand-down reversed by a compensating event). A rollback refuses if somebody changed the same row since.
Anatomy of a ledger row: time, kind (detection, proposal, action, approval, rejection, rollback, housekeeping), subject type and id, action and endpoint, tier, status, the human involved (if any), a note, the parent row it acts on, and the evidence (definition key and version, the rows that matched, and a stake note in customers). Undo adds a new row whose parent is the original.
3.5 People
The approval queue holds tier-2 items. In a replay, a scripted control-room shift (control engineers, a planner, a crew lead; synthetic, and labelled so on every row they write) approves, rejects or reverts by written rules after a realistic delay. Items older than two hours are decided by the shift's backlog rules, so the queue does not grow for days.
4. Scale
The reference world has 200,000 meters on 1,200 feeders from 90 substations, about 200,000 meter events over 90 days, 920,000 feeder readings, 2.8 million daily meter rows, and around 1,170 planted faults. A replay of the storm evening on a 30-second tick, after six days on a five-minute tick, takes a few minutes on a laptop with rules only. Detector queries stay in tens of milliseconds because each reads a bounded time window.
In production the realtime detectors move to stream jobs on the head-end and OMS feeds (the definitions compile to both), daily ones stay in batch on the MDM, and the ledger lives in a relational store with the same append-only rules.
5. Measurement and rollout
- Shadow mode (
FEEDER_SHADOW=1): every tier 0 and 1 action is written as "would do" and nothing changes. - Accuracy: precision and recall per kind against the answer key, the share handled without a person, reverted actions, time to resolve, decided by rules or model, and a drift alarm on model answers.
- Labels in production: the control room marks a sample of "would do" rows right or wrong; that sample is the answer key a real network does not come with.
- Gates: a kind moves from shadow to tier 0 only after its precision on the labelled sample holds for two weeks; tier 2 kinds stay with people.
6. Failure modes
- A detector that cannot run is logged as a failed row once per window; the rest keep running.
- A model that fails or answers outside the policy falls back to the rule, and the row says so.
- A tool that finds the world changed (supply back before the ticket, a crew already on site) changes nothing and says why.
- Budgets stop a runaway: one feeder cannot take more than twelve automatic actions an hour.
7. The map and play mode: routes
GET /: the map screen;?at=pins a moment of the recorded run.GET /api/map/base: static geometry (bounds, substations, feeders, transformers with meter counts, 200,000-meter density cells).GET /api/map/state?at=(recorded run) or?run=live: counters, dark clusters, crews and the receipts rail at that moment.GET /api/map/timeline?run=: the storm strip series (tickets and meters dark over time).GET /api/playandPOST /api/play/{start|pause|speed|stop}(?speed=60|180|360): state and control of the live run on a fresh copy of the world underout/play; the operator acts through the same controller, tools and ledger.