OT Artificial Intelligence Test and Assurance Engineer (ACE AI Project)

OT Artificial Intelligence | Test and Assurance Engineer

Position Overview

We are seeking an OT Artificial Intelligence Test and Assurance Engineer to independently verify that AI-enabled solutions are safe, secure, reliable, and fit for use in industrial operations. This role develops and executes risk-based assurance strategies across models, data, applications, integrations, edge platforms, and operational workflows before solutions enter service and throughout their production lifecycle.

The position provides independent testing, verification, validation, and assurance support for industrial AI use cases. The engineer establishes reusable test frameworks, evidence requirements, acceptance criteria, and release gates for machine learning, computer vision, generative AI, agentic systems, and AI-enabled applications deployed across edge, on-premises, and authorized cloud environments.

The successful candidate will collaborate with industrial technology specialists, product owners, AI and data engineers, solution architects, cybersecurity professionals, safety specialists, and operations personnel while maintaining appropriate independence from solution delivery. The focus is on producing objective evidence that AI solutions perform as intended, fail safely, withstand foreseeable misuse, and remain dependable under real-world operating conditions.

High-Level Job Description / Scope

This role focuses on test and assurance engineering for industrial AI solutions. Responsibilities include assurance planning, requirements traceability, data and model validation, system and integration testing, safety and cybersecurity verification, human-factors evaluation, performance and resilience testing, operational acceptance, release evidence, and post-deployment assurance.

The engineer evaluates complete AI-enabled systems—not just model accuracy—under normal, degraded, abnormal, and recovery conditions.

Key Responsibilities

1. Develop risk-based verification, validation, and assurance strategies for industrial AI solutions from concept through retirement.

2. Translate operational, safety, cybersecurity, regulatory, and business requirements into measurable test objectives, acceptance criteria, and evidence plans.

3. Maintain bidirectional traceability among requirements, hazards, controls, datasets, models, test cases, defects, approvals, and production monitoring.

4. Independently evaluate training, validation, and test data for provenance, representativeness, quality, leakage, imbalance, labeling accuracy, and coverage of rare events.

5. Test AI models for accuracy, precision, recall, false alarms, missed detections, calibration, robustness, explainability, bias, and fitness for their intended operational use.

6. Design scenario-based, boundary, negative, fault-injection, adversarial, and misuse tests for machine learning, computer vision, generative AI, and agentic systems.

7. Verify end-to-end behavior across data pipelines, models, applications, interfaces, edge devices, networks, human workflows, and downstream industrial systems.

8. Validate performance under representative load, latency, bandwidth, lighting, weather, process-state, data-quality, and disconnected-operation conditions.

9. Verify human oversight, alerting, confidence presentation, fallback, fail-safe, rollback, recovery, and manual-override mechanisms.

10. Assess cybersecurity controls, attack surfaces, identity, access, secrets, software supply chains, network segmentation, logging, and auditability in collaboration with relevant security specialists.

11. Automate regression, integration, performance, and model-evaluation tests within CI/CD and MLOps pipelines while preserving independent approval gates.

12. Document defects, residual risks, test limitations, deviations, and release recommendations; verify corrective actions before closure.

13. Define post-deployment assurance checks for drift, degradation, incidents, changes, retraining, and continued operational fitness.

Required Qualifications

1. Bachelor's degree in Computer Science, Software Engineering, Systems Engineering, Electrical Engineering, or a related field, or equivalent practical experience.

2. Five or more years of experience in software, systems, automation, or quality engineering, including production verification and validation responsibilities.

3. Experience developing risk-based test strategies, acceptance criteria, test cases, traceability, and objective release evidence for complex systems.

4. Experience testing machine learning, computer vision, analytics, or other data-driven systems using appropriate performance and error metrics.

5. Experience evaluating datasets for quality, representativeness, leakage, imbalance, labeling accuracy, lineage, and version control.

6. Experience with automated test frameworks, Python, APIs, data validation, source control, CI/CD, and defect-management tools.

7. Experience with system, integration, performance, resilience, cybersecurity, and user-acceptance testing.

8. Understanding of AI limitations, uncertainty, drift, model monitoring, human oversight, and lifecycle controls.

9. Understanding of industrial cybersecurity, network segmentation, constrained connectivity, high availability, and controlled change in industrial environments.

10. Ability to communicate test results, uncertainty, defects, and residual risk clearly to engineers, operators, security reviewers, and decision-makers.

11. Demonstrated independence, professional skepticism, and willingness to recommend against release when the available evidence does not support approval.

Preferred Qualifications

1. Master's degree in Systems Engineering, Computer Science, Artificial Intelligence, Safety Engineering, or a related field.

2. Experience assuring generative AI, retrieval-augmented generation (RAG), agentic systems, or multimodal AI, including grounding, hallucination, tool-use, and prompt-injection testing.

3. Experience validating industrial computer vision across variations in lighting, weather, motion, occlusion, viewpoint, background, and equipment state.

4. Experience with model robustness, explainability, fairness, adversarial testing, red teaming, or responsible AI assessment.

5. Experience with safety cases, assurance cases, hazard analysis, failure modes and effects analysis (FMEA), bow-tie analysis, or similar risk-assessment methods.

6. Experience with IEC 61508, IEC 61511, IEC 62443, the NIST AI Risk Management Framework, or other relevant assurance standards and guidance.

7. Experience with simulation, digital twins, hardware-in-the-loop, software-in-the-loop, or synthetic-data testing environments.

8. Experience integrating automated assurance checks into Azure Machine Learning, Microsoft Foundry, Azure DevOps, GitHub Actions, or equivalent platforms.

9. Experience supporting incident investigations, model-performance reviews, and production requalification after material changes.

10. Experience in upstream, midstream, or downstream industrial environments and the ability to work directly with asset and operations personnel.

Critical Selection Criteria

1. Test and Assurance Capability: Demonstrated ability to independently plan and execute risk-based verification and validation of AI-enabled systems and produce traceable, decision-quality evidence.

2. Operational and Safety Focus: Ability to test normal, degraded, abnormal, misuse, and recovery scenarios and verify effective human oversight, fallback, fail-safe, and rollback controls.

3. Industrial Technology, Security, and AI Understanding: Knowledge of industrial operations, cybersecurity, AI behavior, and the complete data, model, application, edge, and human system that determines operational risk.

4. Independence and Communication: Ability to challenge assumptions constructively, explain evidence and residual risk clearly, and make defensible release recommendations to technical and nontechnical stakeholders.