AI Workflow Automation for Continuous Model Improvement in Automotive Imaging

Customer Overview

A US-based automotive photography giant offered an app that enabled auto dealers to capture professional-grade interior and exterior car photos using 37 standardized angles. These photos were polished through a mix of automated tools and manual editing to deliver high-definition 4K images. Processing over 70,000 images daily, the client initially relied on a third-party tool for background replacement and shadow generation. To reduce costs and gain more control, they partnered with TenUp to develop an in-house AI-powered solution for image background replacement, along with shadow and tinted effects generation.

Project Overview

We developed an AI-based automotive image background removal and replacement tool in Phase 1 and enhanced it with capabilities for generating realistic shadows and window effects in Phase 2. Our client found that around 5-7% of images processed daily by the AI-based tool were unsatisfactory, which increased manual editing effort and costly man-hours. They understood that to address this challenge, the AI model’s performance can be improved over time by continually retraining it on erroneous outputs. So, they wanted to build workflows for output feedback and continuous AI model improvement, while ensuring complete visibility into the end-to-end process. Meaning, the proposed solution must orchestrate workflows across editors, annotators, and MLOps tools.

Challenges

Building a solution to orchestrate and automate the end-to-end process of continuous AI model improvement, while enabling multiple tools and teams to work in coordination and giving our client complete visibility into the overall process.

• Designing a feedback process where editors could flag erroneous outputs and automating its availability in the annotation tool for the annotation team was a challenge.
• Automating the collection of the required number of annotated images for training an AI model and preprocessing them required enabling coordination between different tools.
• Ensuring access to scalable and cost-efficient GPU resources for large-scale model training was a key constraint.
• An iterative model performance comparison required that the training results be made available in a user-friendly manner.
• The deployment of the best-performing model after comparison must also be orchestrated and automated.
• The client wanted visibility into whether flagged images were actually processed and if they improved AI model performance.

Solution

Developed a workflow orchestration and automation solution to coordinate all tools and processes for continuous AI improvement. It maintains an audit trail of images across annotation and training and tracks all model training experiments for performance comparison.

• Integrated an API with our client’s system, where the editors flagged ~500–1000 erroneous images daily with structured feedback on issues (background, shadows, tire grounding, and tinting) and input this data into our workflow orchestration platform.
• Orchestrated an annotation workflow where our platform creates jobs in CVAT.ai using flagged images and assigns them to our team of 15 annotators every day, ensuring steady availability of jobs for the annotator team. It re-imports completed datasets the next day.
• Enabled preprocessing of annotated datasets in the workflow orchestration platform. Implemented a rule-based process for the platform to collect sufficient annotated datasets before triggering model training and evaluation workflows.
• Integrated the system with Vast.ai to utilize its GPUs for model training with prepared datasets.
• For MLOps, we integrated Wandb.ai at every stage, from preprocessing datasets and selecting base models to training and testing, to capture detailed logs and metrics for each model and compare their training and evaluation results.
• Orchestrated the model training feedback loop by importing Wandb.ai results into the workflow orchestration platform to compare image editing outputs of current and previous models and identify the best-performing one.
• Used AWS SageMaker to automatically deploy the best-performing model after improvements post-training are validated within the system.
• Created a UI in the workflow orchestration platform to track status updates for every stage (annotation, training readiness, model performance, deployment), enabling our client to see how flagged images were processed and contributed to AI improvement.

Benefits

The AI workflow automation solution we developed provided the following advantages to our client:

• Reduced costly manual editing effort by 25–30%, saving hundreds of man-hours weekly.
• Improved AI tool performance with structured feedback and retraining, lowering error rates by 15–20%.
• Automated the entire workflow from annotation to deployment, ensuring 100% of flagged images were processed on time without manual hand-offs.
• Enabled side-by-side model comparison, improving deployment decisions and boosting model accuracy by an additional 8–10%.

Technology

• FastAPI
• ReactJS
• CVAT.ai
• Vast.ai
• Wandb.ai
• AWS Sagemaker

Industry

• Automobile/Automotive