Projects / AI
Intelligent Sign-Off
AI-Powered Packaging Compliance Validation System.

Project Overview
Intelligent Sign-Off (DSO) is an enterprise-grade AI-powered packaging compliance validation system built for Mattel. A quality reviewer uploads a .ai or .pdf artwork file together with product context (brand, age grade, language/market code), and the system validates it against structured compliance requirements stored in a BigQuery knowledge graph — returning PASS, FAIL, NEED_VERIFICATION, or NOT_APPLICABLE for every applicable requirement.
Problem Statement
Manual packaging verification involves checking digital artwork files against an extensive list of branding and regulatory requirements that differ per country and language market. This process:
- Is time-consuming and repetitive across dozens of SKUs and markets.
- Is error-prone, especially with small compliance marks and multi-language legal text.
- Does not scale across diverse product lines and international markets.
An automated pipeline was needed to extract both symbols and text from packaging artwork and validate them against structured, machine-readable requirement rules.
System Architecture
The system is composed of three independent services that run in parallel:
| Service | Runtime | Role |
|---|---|---|
dso-frontend | ASP.NET Core (.NET 10) | Web UI — artwork upload, product context form, compliance report viewer |
dso-detection | Python / FastAPI | Symbol detection (Gemini VLM) + text detection (Cloud Vision OCR) |
dso-bridge | Python / FastAPI | Orchestrates the ADK Agent Engine session and injects detection results |
Workflow
- The reviewer uploads an artwork file and fills in brand, age grade, language/market code, and any special feature.
- The frontend sends the file to both backend services in parallel:
- Detection Service → extracts compliance symbols via Gemini VLM (from
.aivector layers) and text via Cloud Vision API → returns annotated page images and structured JSON. - ADK Bridge → uploads the file to GCS, injects detection results as
[PRE_DETECTED_JSON]into the agent message, opens a Vertex AI Agent Engine session, and calls the compliance agent.
- Detection Service → extracts compliance symbols via Gemini VLM (from
- Inside the Agent Engine, a root agent routes artwork submissions to the
knowledge_graph_agent, which:- Loads applicable requirements from BigQuery (filtered by language/market code).
- Checks each requirement against the detection results.
- Queries the Vertex AI RAG corpus for QSOP documentation when needed.
- Returns a structured Markdown compliance report.
- The frontend parses the report into PASS / FAIL / NEED_VERIFICATION / NOT_APPLICABLE check items and renders them alongside an annotated artwork viewer.
Detection Pipeline
Symbol Detection
The Detection Service processes .ai files through a multi-stage pipeline:
- PyMuPDF opens the
.aias a PDF and reads Illustrator layers (OCGs). - Selects the
"Copy"layer (Mattel convention for compliance marks); falls back to all layers. - Size filter — drops paths smaller than 1.5 mm or larger than 60 mm.
- Union-find clustering — groups nearby paths by centroid distance (6 mm threshold). A 20 mm cap prevents merging symbols from different regions.
- NMS — removes overlapping duplicates; scale-exempt to preserve nested symbols.
- Gemini VLM classifies each symbol crop concurrently (8 threads) against a few-shot reference library stored in GCS.
Text Detection
Cloud Vision API extracts all text blocks from a high-DPI rasterized page render, returning word-level bounding boxes and confidence scores.
Compliance Verdict Logic
Severity in the knowledge graph controls how a requirement is validated:
| Severity | Who validates | Possible outcomes |
|---|---|---|
error | AI (from detection results) | PASS or FAIL (confidence ≥ 0.5); NEED_VERIFICATION if confidence < 0.5 |
warning | Human (position, proximity, layout) | Always NEED_VERIFICATION |
info | Human (advisory/ordering rules) | Always NEED_VERIFICATION |
NOT_APPLICABLE is assigned only when the requirement's applies_when conditions (language codes, product types) do not match the uploaded product context — never from detection evidence.
Technologies & Tools
| Component | Technology |
|---|---|
| Symbol Detection | Gemini VLM (Gemini 3.5 Flash via Vertex AI) |
| Text Detection | Cloud Vision API (OCR) |
| Agent Framework | Google ADK (Agent Development Kit) |
| Agent Deployment | Vertex AI Agent Engine |
| Knowledge Graph | BigQuery |
| RAG Corpus | Vertex AI RAG Engine (QSOP documentation) |
| Backend Services | Python / FastAPI + Hypercorn |
| Frontend | ASP.NET Core (.NET 10) — Razor Pages + Vanilla ES Modules |
| Containerization | Docker / Google Cloud Run |
| File Storage | Google Cloud Storage (GCS) |
| Observability | Arize (OpenInference tracing) |
Multi-Agent Design
The Agent Engine hosts a two-level agent hierarchy:
- Root agent — receives the artwork submission message (with pre-detected JSON injected) and routes it to the appropriate sub-agent.
knowledge_graph_agent— loads applicable requirements from BigQuery (query_knowledge_graphtool), retrieves supporting QSOP documentation from the RAG corpus (retrieve_rag_documentationtool), and produces the structured compliance report.
This separation allows the root agent to be extended with additional sub-agents (e.g., for historical audit lookups) without changing the compliance logic.
Infrastructure
All three services are deployed on Google Cloud Run with --use-http2 enabled, since artwork files can reach 60–100 MB and Cloud Load Balancer caps HTTP/1.1 request bodies at 32 MB.
- GCP Project:
mt-ptmi-quality-nonprod - Agent Engine region:
us-east1 - Gemini VLM region:
us-central1(Gemini 3.5 Flash not available inus-east1)
Conclusion
Intelligent Sign-Off transforms a manual, error-prone QA process into an automated, cloud-native compliance pipeline. The three-service microservices architecture — with parallel detection, a multi-agent RAG system, and a structured knowledge graph — enables independent scaling of each component and straightforward integration into existing manufacturing quality control workflows.