Service provider
Taking control of equipment the vendor never meant you to automate
A service provider had to keep thousands of customer records in step across a handful of network appliances whose vendor offered no way to drive them by software, with a regulatory deadline coming. An outside contractor had spent most of a year on it without shipping. We worked out how the appliances could be driven, built one record of how things should be and a service that keeps the appliances matching it, and shipped in about two months.
- Shipped in
- about two months, after a stalled year
- Running
- years, never changed for a fault of its own
- Service affected
- never
What was wrong
Thousands of customer records had to live correctly on a handful of appliances at the edge of the provider's network, and a regulatory deadline meant that from then on they would change constantly. The appliances had no supported way to be driven by software; the vendor expected a person at a screen. A contractor had spent most of a year on the admin tool and had not shipped. The admin tool was never the hard part. Reliable control of the appliances was.
What we did
We watched what the appliance's own screens sent when a person clicked, and reproduced every operation from software against a test unit until all of it was reliable. Then, instead of a script that pushes changes, we built one record of how everything should be and a service that keeps every appliance matching it: it looks for drift, applies a change all at once or not at all, and never takes more than one appliance offline at a time. The same change applied twice does nothing, so a half-finished update heals on the next pass.
What it got them
Shipped in about two months after a year of waiting. Years in production since, and in that time it never once had to change to fix a fault of its own; when things around it broke, it was the systems upstream. Staff configure, the appliances follow, and service was never affected.
A component you never have to open again is worth more than one that never technically failed.
Ryan Luccioni
Principal
Anonymized. A service provider; name, sector and equipment withheld. Engagement led by Ryan Luccioni, principal.