Refineries that have been through a failed software implementation share a common experience: the vendor knew how to build software. They just didn't know how refineries work.
The gap between general software competence and domain-specific operational understanding is wider in downstream industrial environments than almost anywhere else. The stakes, including process safety, regulatory compliance, unplanned downtime, and environmental exposure, are high enough that a vendor who underestimates that gap creates problems the refinery has to absorb. Evaluating software vendors in this environment isn't just a procurement exercise. It's a risk management decision.
Here's what that evaluation should actually look like.
Domain Fluency, Not Just Industry Experience
There's a meaningful difference between a vendor who has sold to refineries and a vendor who understands refinery operations. The first category is common. The second is the one that matters.
Domain fluency shows up in specific ways. A vendor who understands your environment will speak your language without prompting. They know what a permit to work actually governs. They know why LOTO procedures are non-negotiable. They understand how process safety management documentation fits into your compliance obligations, and why a scheduled turnaround isn't just a maintenance event but a finite operational window where dozens of competing priorities converge.
A vendor who doesn't have that fluency will ask you to explain it. That's a yellow flag. It doesn't disqualify them, because good vendors learn, but it tells you how much translation work you're going to be doing and how much implementation risk that translation creates.
When evaluating vendors, walk them through a real operational challenge and watch how they respond. Do they ask follow-up questions that reveal operational understanding, or do they pivot to a product demo that may or may not be relevant? The quality of their questions is more informative than the quality of their presentations.
Integration Capability That Matches Your Reality
Refineries don't start with a blank slate. They have DCS systems, historians, CMMS platforms, ERP instances, laboratory information systems, and process data infrastructure that varies by unit, by vintage, and sometimes by vendor. Any software partner that expects to work in isolation from that environment, or that can't articulate a clear integration strategy, is proposing to add complexity rather than reduce it.
The right question isn't "can you integrate with our systems?" Every vendor will say yes. The right questions are more specific:
- What protocols do you work with, and which ones have you actually deployed in a refinery context?
- How do you handle historian data extraction from legacy systems without affecting control layer performance?
- What does your integration architecture look like at an existing site, and can we talk to that customer?
- When something breaks on the integration side, who owns the fix?
Vendors who have built real integrations in operating facilities will answer these questions with specifics. Vendors who haven't will answer with generalities. That distinction is worth pressing for.
A Delivery Model Built for Operating Environments
Software implementations in refineries are not like software implementations in corporate environments. Access is controlled. Schedules are constrained. The ability to push a quick fix doesn't exist when you're working near live process units. Deployment windows are narrow, change management is formal, and the tolerance for disruption is close to zero.
A software partner who hasn't worked in operating facilities will underestimate all of this. Their project timelines won't account for site access protocols. Their testing procedures won't anticipate the constraints of working near hazardous areas. Their deployment plans won't align with your change management process.
What you're looking for is a vendor who treats your operational constraints as inputs to their delivery model, not obstacles to work around. That means a project approach that builds in site-specific requirements from the start, a team that understands how to work in a managed access environment, and a communication cadence that connects to the people actually running the facility, not just the IT department.
Honesty About What They Build Versus What They Configure
There's an important distinction between software vendors who build what you need and vendors who configure what they have. Neither is inherently better. The right answer depends on what you need. But the distinction matters enormously for total cost of ownership, long-term flexibility, and who you're dependent on when things need to change.
Configurable platforms can be deployed faster and carry lower initial cost, but they constrain what's possible to what the platform was designed to support. When your operation has requirements that fall outside that envelope, and refinery operations frequently do, you hit a ceiling that's expensive to work around.
Custom-built platforms take longer and require a vendor you trust to maintain, but they're built around your process rather than a platform's assumptions. The right vendor will be honest with you about which approach fits your situation, including when the answer is a hybrid: a configurable foundation with custom-built components for the parts of your operation that don't fit the standard mold.
Be skeptical of vendors who tell you their platform does everything you need before they've spent enough time understanding what you actually need.
What Does the Research Say About Building Versus Buying?
MIT's NANDA initiative reported that AI tools bought from specialized vendors or built with partners reached deployment about 67% of the time, while internal builds succeeded about a third as often. The study looked at generative AI broadly, not industrial work specifically, and critics argue its headline numbers oversimplify. Treat it as a signal, not a verdict.
The lesson applies to software too: the demo is rarely the hard part. Support, security, and proof usually are.
Should You Build, Buy, or Do Both?
| Option | Works best when | Watch for |
|---|---|---|
| Buy off the shelf | The need is common across industries | Fit to your jobs, sites, and rules |
| Build custom | The work is unique to your operation | Year two support and ownership |
| Partner on a build | You need your process encoded fast | Clear ownership and exit terms |
| Mix | Common parts are bought, unique parts built | Integration between the pieces |
Five Questions to Answer Before You Build or Buy
This applies to custom software, integrations, and AI agents, also called digital workers: AI that performs a defined job, such as supporting a JSA, reconciling contractor hours, or following up on missing paperwork.
1. Does Your Team Know the Work Well Enough to Encode It?
Software and AI agents both have to understand what a job, a site, and a job type mean. Maintenance, emergency work, turnarounds, and capital projects follow different rules. Someone has to define those rules and keep them current. If that knowledge lives in a few experienced heads, building means pulling them off the work they do best.
2. Who Owns It in Year Two?
Most internal builds start with enthusiasm and one capable person. Ask what happens when that person changes roles, when a model or a connected system changes, or when a site needs a new job type. A tool without an owner becomes shelfware.
3. What Security Questions Apply to Software and AI Agents?
Whoever builds it has to decide what data it can see, how personal information is detected and handled, and how long recordings, photos, transcripts, and records are kept. For AI, add whether your data is used to train models. Incident evidence raises its own retention questions. These are decisions, not features, and your legal and IT teams will want them in writing before any pilot.
Our Trust Center covers how we approach this.
4. Who Is Responsible for Governance?
This is usually the first question IT asks. Does the software or agent act under its own identity or a user's credentials? What can it read, and what can it change? Can you trace every action to a rule or a person?
OWASP, a nonprofit security standards group, lists agent goal hijack, where poisoned emails or documents redirect an AI agent, as the top risk for agentic applications. A system that can change records needs limits on what it can change and a record of what it did.
Human oversight belongs in the design. Routine, rules based steps can run on their own. Judgment steps should surface a finding, a recommendation, or a confidence level to the right person. In industrial work, that is a feature, not a limitation.
5. How Fast Can You Show a Measured Result?
Pick one problem, record a baseline, and set a short window. Idle days, admin hours, hazards identified, or time to payment are all measurable. If a build cannot show movement on one of them quickly, it will compete with every other priority for budget.
What Should Owner Operators and Contractors Consider?
Your hardest questions are integration and approval. Systems like ERP matter, and your IT and security teams will want answers on data, identity, and audit before a pilot begins. Ask every builder or vendor for those answers early.
Another constraint is usually time. For example, admin work such as aligning hours to work orders and chasing staffing details, task IDs, badge IDs, and expired documents uses the same hours your supervisors need in the field. Judge any option by how quickly it takes that work off your people, and by whether it fits how your customers already require you to work.
See how Valorian works with Owners and Contractors.
References From Comparable Environments
The most efficient way to evaluate a software partner is to talk to someone who has worked with them in a similar context. Not a reference call that the vendor arranges with their happiest customer, but a real conversation with an operations or technology leader at a facility that looks like yours, someone who can speak honestly about what the implementation was like when things got hard.
Ask the vendor for references from environments similar in scale and complexity to your facility. If they don't have them, that tells you something. If they do, make the call. What you learn about how a vendor behaves when a project hits a complication is worth more than anything you'll learn during a sales cycle.
The Long-Term Relationship Question
Software in an operating facility isn't a transaction. You're choosing a partner who will be part of your operational environment for years. That means thinking about what the relationship looks like after go-live: how they handle support requests, how they manage product updates in a live environment, whether they're genuinely responsive to feedback from the people actually using the system day to day.
The best industrial software partners treat the operational outcomes of their clients as their own accountability. That orientation shapes how they build, how they deliver, and how they show up when something isn't working. It's the characteristic most worth looking for, and the hardest to fake in a real conversation.
Valorian builds custom software and digital workers for refinery and industrial contractor work, and our team has worked inside these facilities. Some of what we do is live. Some is in pilot. If you ask which is which, we will tell you plainly. Valorian is SOC 2 Type II certified, and we are glad to walk through our approach to security and governance.

