AI tools are quickly becoming part of everyday engineering work. Whether it’s summarizing documents, reviewing requirements, or helping write code, they can absolutely speed things up, especially when you’re dealing with large amounts of information. But there’s a gap between what these tools can do and how they’re often perceived. They’re powerful, yes, but they’re not inherently reliable. They don’t truly understand what they’re analyzing, and when they get something wrong, it’s not always obvious.
We were reminded of that recently through a real-world experience…
One of our engineers was reviewing a medical device software requirements document and decided to use AI tooling to help analyze a very specific section. It seems simple enough in theory – point the AI to a section, ask for analysis, save some time.
The AI assistant responded quickly and confidently.
The problem was, the response didn’t line up with the document. There were references to things that didn’t seem to exist. The terminology felt off. The structure didn’t match what the engineer was seeing.
It wasn’t obviously nonsense, but it also wasn’t right.
That’s where experience kicked in. Instead of assuming the tool was correct, the engineer paused and started digging. While reviewing the document more closely, the engineer noticed the table of contents was inaccurate. Sections were misaligned and not properly indexed which could easily throw off an AI Tool trying to navigate the document. So, the engineer fixed the table of contents and ran the same query again. This time, the tool gave a different answer.
Still wrong.
At that point, it was clear something else was going on. After a bit more investigation, the likely cause came into focus. Instead of analyzing the intended document, the AI had pulled from a different, but similarly named, file stored in the same SharePoint environment.
It wasn’t just misinterpreting content, it was analyzing the wrong source entirely and presenting it as if it were correct.
AI tools don’t just make mistakes, they make them convincingly. The output sounds polished, structured, and authoritative. If you’re not paying close attention, it’s easy to accept it at face value.
In a low-stakes situation, that might just waste some time. In medical device development, it’s a different story.
If that output had been trusted and acted upon, it could have introduced incorrect assumptions into requirements, validation, or documentation. That’s not just a technical issue, that’s a compliance and safety concern.
The only reason this didn’t turn into a bigger problem is because an experienced engineer was in the loop. They didn’t just read the output, they questioned it. They recognized when something felt off. They validated it against the source material. That instinct doesn’t come from a tool. It comes from years of experience working with requirements, understanding how documents are structured, and knowing what “right” is supposed to look like.
AI doesn’t know when it’s wrong. Engineers do.
There’s a tendency right now to frame AI as something that can replace parts of engineering work. Experiences like this show why that’s an oversimplification. AI is great at speeding things up. It can help scan large documents, highlight patterns, and reduce some of the manual effort. But it lacks context, it lacks judgment.
And it has no accountability for being wrong.
That’s why it works best as an assistant, not a decision maker – the responsibility still sits with the engineer.
A Practical Takeaway for Teams Using AI
If your team is using AI tools, and most are at this point, there are a few practical lessons here:
- Don’t assume the output is correct just because it looks good
- Always validate against the actual source material. Also, validate your source material.
- Be aware that tools can pull from the wrong context or dataset even when given specific instructions.
- Keep experienced people actively involved in the review process
Used the right way, AI can absolutely save time. Used blindly, it can create new risks just as fast.
A More Realistic View of AI in Engineering
AI isn’t going anywhere, and it shouldn’t. It’s already proving to be a valuable tool.
But it’s important to stay grounded in what it actually is, and what it isn’t.
It’s not a replacement for engineering expertise. It doesn’t understand risk, context, or consequences. It generates outputs based on patterns, not judgment.
That’s why the “human in the loop” isn’t just a nice idea, it is essential.
At IQ, Inc., we see AI as something that augments engineers, not to replace them. The goal is to move faster and work smarter, without compromising accuracy or quality.
Because at the end of the day, especially in environments like medical devices, getting it right isn’t optional.
Connect with us at https://iq-inc.com/connect-with-us/ or info@iq-inc.com to start the conversation.
#ArtificialIntelligence #AIinEngineering #EngineeringDiscipline #SoftwareEngineering #SystemsEngineering #EmbeddedSystems #IndustrialAutomation #EngineeringLeadership #TechLeadership #AIAdoption #ResponsibleAI #DigitalTransformation #Innovation #EngineeringCulture #QualityEngineering #SafetyCritical #Automation #FutureOfWork #TechStrategy