--- title: "The Infrastructure Invisibility Problem" description: "Plant personnel ask for a simple flat network and internet on the floor in the same breath. The hidden distribution-layer complexity that makes β€œsimple” impossible. And the dependency it quietly locks plants into." canonical: "https://rivercaudle.com/writing/the-infrastructure-invisibility-problem/" author: "River Caudle" date: "2025-07-11" --- # The Infrastructure Invisibility Problem *2025-07-11 Β· networks, architecture* ## The Simple Network Contradiction Generally, a simple OT network to plant personnel is a flat one, logically. From their perspective, they want everything to operate on its default settings, where they can set an address and subnet and move on. In the same breath, though, they also need Internet access on the plant floor for their computers and phones. And in reality, this flat network is dispersed, spread out through the facility. And their computers need to access the plant network from their office. But not all the time. And they want to be able to remote in and troubleshoot from home. So what is described as a simple network, in reality involves: 1. **Shared infrastructure with IT**, because it's more efficient and already in place. Meaning... 2. **VLAN segmentation has to exist**, just invisibly to the user 3. **Multiple plant VLANs are required**, at least two here, one for computer and one for control 4. **Common infrastructure means QoS is needed** 5. **Internet access/Corp access requirements mean there needs to be stringent logical segmentation** on switch ports and more To add to the above, the requirement to granularly traverse those layers means **central firewalling and routing is required**. Spanning tree is often here, and should be. We can call this infrastructure **the distribution layer**. It includes the management backplane, shared switches, VLANs, QoS rules, and routing required to make the control system networks layer simple. ## When the Invisible Becomes Critical When something breaks, they are in the dark about why and how, because the break often exists in the transport realm beyond their understanding and control. It's rarely on their side, because the complexity is heavily offloaded on the distribution side. Imagine an issue where a spanning tree misconfiguration on a distribution switch controlled by IT has shut down a port that links two sections of the same line. So what happens is they call the helpdesk and say the plant network is down. And that's usually it. If IT is monitoring at the plant level (and they usually aren't), maybe they are aware that something is down and can begin troubleshooting. Most likely, the ticket gets punted back for more information. Once they field it, most internal help desks don't actually have internal SLAs... Or if they do, they are often IT-aligned. Meaning that there may be a several hour delay before they are fielded because the perceived impact is relatively low. So by the time the tech even gets the correct information to act on, it may be several hours out before a network admin actually logs into a switch. Maybe they fix the issue right away once they are in. But what seems to inevitably happen is they... reboot the switch. Or shut down another port. Or otherwise begin troubleshooting as if it's an IT network. It sounds like a damn comedy, but this is an example of a more functional relationship, they at least have understood shared infrastructure, a dedicated IT team, a ticket system for accountability, and a network admin willing to investigate. **That's a lofty goal for the majority of us manufacturing.** ## The Dairy Plant Reality Check Eventually they'll whip out the unmanaged switches and take the problematic stuff out of the equation. But never has a plant diagnosed a spanning tree issue on their own. Normally that's where I got involved, because the next call is to the system integrator that sold them the switches. This call is usually urgent, and results in a high bill. I go out, fix the issue, move on. Great way to spend $200/hr. In the dairy story, they didn't bypass. They dealt with hour-long intermittent outages for several months, once every few weeks, before engaging me. They got the contractor's IT folks involved, as well as several engineers. At that point they were resigned, anyone internal had already looked at it. No one even thought about spanning tree or knew anything about it. The reaction was relief and astonishment that a simple setting difference could cause such chaos. And frustration it wasn't set up right to begin with. I explained in simple terms what was wrong, so they did understand. Like many projects, **IT and OT had never even met, much less coordinated config.** ## The Dependency Business Model When plants create these "simple" networks that actually require shared infrastructure, VLANs, QoS, spanning tree, firewalls, who typically ends up responsible for all that invisible complexity? It's definitely IT, if it's created by them. What gets difficult are the many scenarios where that shared infrastructure was put in by a third party who is gone. Like the integrator who integrated several lines with different subnets, set up the infrastructure, linked it to Enterprise, and then left. Without educating anyone, IT or not, or documenting, leaving passwords, etc. **I can't help but think it's by design.** They pay the integrator. It's definitely a tactic to keep the integrator integrating. Create artificial complexity or fail to hand over the keys to design. Whether that's knowledge, passwords, licenses, even physical hardware. And in a lot of cases, these system integrators are charging up to $300 an hour for someone who is likely inexperienced in plant environments anyway. Of course, the integrator is disincentivized from training or doing knowledge transfer to their customers. Or for making it easy. Each issue, each complexity whether real or implemented, is a tie back to the possibility of recurring revenue. If OT owned that infrastructure, and had the agency and education to effectively administrate it, there would be no recurring revenue potential. And in fact, this system integrator's influence would be diluted, because OT networking is considered a very special niche profitable expensive area. ## The Gentle Fucking Reality When plants try to break free, integrators do all of those things. They push back. They make it sound impossibly complex. They warn about warranty issues or support problems. The barrier is, **who will liberate them and teach them to rule their own?** Not the distributor, or the integrator... Why? Sure, they do have training. From experience, it's expensive, unnecessarily complex, hard to get into (registration minimums), and truly, boring. **So they understand they are being fucked, it's just a matter of who fucks them most gently.** ## The Infrastructure Independence Solution That transformation would involve: **πŸ”₯ OT-owned firewall** - Firewalling OT from the rest of the world with an OT owned and located firewall **πŸ”₯ OT-owned distribution switches** - Ownership of the distribution switches transfers to the plant, and they are carrying IT's network, not the other way around **πŸ”₯ Plant-set priorities** - Network priorities set by operational requirements, not IT convenience **πŸ”₯ Plant-maintained documentation** - Documentation maintained by the people who actually use and depend on the network **πŸ”₯ True ownership** - Real infrastructure ownership by the plant, not dependency disguised as service **πŸ”₯ Training + tribal knowledge** - Training is crucial, but eventually becomes bolstered by tribal and community knowledge **And Riverman can god damn show them how to break free.** --- *The infrastructure invisibility problem isn't just a technical challenge, it's an industry-wide dependency scam. The question isn't whether plants can achieve infrastructure independence. The question is who will teach them they don't have to accept being gently fucked anymore.*