Skip to content

Current section: Work

Projects / AI

Intelligent Sign-Off

AI-Powered Packaging Compliance Validation System.

Intelligent Sign-Off preview

Project Overview

Intelligent Sign-Off (DSO) is an enterprise-grade AI-powered packaging compliance validation system built for Mattel. A quality reviewer uploads a .ai or .pdf artwork file together with product context (brand, age grade, language/market code), and the system validates it against structured compliance requirements stored in a BigQuery knowledge graph — returning PASS, FAIL, NEED_VERIFICATION, or NOT_APPLICABLE for every applicable requirement.


Problem Statement

Manual packaging verification involves checking digital artwork files against an extensive list of branding and regulatory requirements that differ per country and language market. This process:

  • Is time-consuming and repetitive across dozens of SKUs and markets.
  • Is error-prone, especially with small compliance marks and multi-language legal text.
  • Does not scale across diverse product lines and international markets.

An automated pipeline was needed to extract both symbols and text from packaging artwork and validate them against structured, machine-readable requirement rules.


System Architecture

The system is composed of three independent services that run in parallel:

ServiceRuntimeRole
dso-frontendASP.NET Core (.NET 10)Web UI — artwork upload, product context form, compliance report viewer
dso-detectionPython / FastAPISymbol detection (Gemini VLM) + text detection (Cloud Vision OCR)
dso-bridgePython / FastAPIOrchestrates the ADK Agent Engine session and injects detection results

Workflow

  1. The reviewer uploads an artwork file and fills in brand, age grade, language/market code, and any special feature.
  2. The frontend sends the file to both backend services in parallel:
    • Detection Service → extracts compliance symbols via Gemini VLM (from .ai vector layers) and text via Cloud Vision API → returns annotated page images and structured JSON.
    • ADK Bridge → uploads the file to GCS, injects detection results as [PRE_DETECTED_JSON] into the agent message, opens a Vertex AI Agent Engine session, and calls the compliance agent.
  3. Inside the Agent Engine, a root agent routes artwork submissions to the knowledge_graph_agent, which:
    • Loads applicable requirements from BigQuery (filtered by language/market code).
    • Checks each requirement against the detection results.
    • Queries the Vertex AI RAG corpus for QSOP documentation when needed.
    • Returns a structured Markdown compliance report.
  4. The frontend parses the report into PASS / FAIL / NEED_VERIFICATION / NOT_APPLICABLE check items and renders them alongside an annotated artwork viewer.

Detection Pipeline

Symbol Detection

The Detection Service processes .ai files through a multi-stage pipeline:

  1. PyMuPDF opens the .ai as a PDF and reads Illustrator layers (OCGs).
  2. Selects the "Copy" layer (Mattel convention for compliance marks); falls back to all layers.
  3. Size filter — drops paths smaller than 1.5 mm or larger than 60 mm.
  4. Union-find clustering — groups nearby paths by centroid distance (6 mm threshold). A 20 mm cap prevents merging symbols from different regions.
  5. NMS — removes overlapping duplicates; scale-exempt to preserve nested symbols.
  6. Gemini VLM classifies each symbol crop concurrently (8 threads) against a few-shot reference library stored in GCS.

Text Detection

Cloud Vision API extracts all text blocks from a high-DPI rasterized page render, returning word-level bounding boxes and confidence scores.


Compliance Verdict Logic

Severity in the knowledge graph controls how a requirement is validated:

SeverityWho validatesPossible outcomes
errorAI (from detection results)PASS or FAIL (confidence ≥ 0.5); NEED_VERIFICATION if confidence < 0.5
warningHuman (position, proximity, layout)Always NEED_VERIFICATION
infoHuman (advisory/ordering rules)Always NEED_VERIFICATION

NOT_APPLICABLE is assigned only when the requirement's applies_when conditions (language codes, product types) do not match the uploaded product context — never from detection evidence.


Technologies & Tools

ComponentTechnology
Symbol DetectionGemini VLM (Gemini 3.5 Flash via Vertex AI)
Text DetectionCloud Vision API (OCR)
Agent FrameworkGoogle ADK (Agent Development Kit)
Agent DeploymentVertex AI Agent Engine
Knowledge GraphBigQuery
RAG CorpusVertex AI RAG Engine (QSOP documentation)
Backend ServicesPython / FastAPI + Hypercorn
FrontendASP.NET Core (.NET 10) — Razor Pages + Vanilla ES Modules
ContainerizationDocker / Google Cloud Run
File StorageGoogle Cloud Storage (GCS)
ObservabilityArize (OpenInference tracing)

Multi-Agent Design

The Agent Engine hosts a two-level agent hierarchy:

  • Root agent — receives the artwork submission message (with pre-detected JSON injected) and routes it to the appropriate sub-agent.
  • knowledge_graph_agent — loads applicable requirements from BigQuery (query_knowledge_graph tool), retrieves supporting QSOP documentation from the RAG corpus (retrieve_rag_documentation tool), and produces the structured compliance report.

This separation allows the root agent to be extended with additional sub-agents (e.g., for historical audit lookups) without changing the compliance logic.


Infrastructure

All three services are deployed on Google Cloud Run with --use-http2 enabled, since artwork files can reach 60–100 MB and Cloud Load Balancer caps HTTP/1.1 request bodies at 32 MB.

  • GCP Project: mt-ptmi-quality-nonprod
  • Agent Engine region: us-east1
  • Gemini VLM region: us-central1 (Gemini 3.5 Flash not available in us-east1)

Conclusion

Intelligent Sign-Off transforms a manual, error-prone QA process into an automated, cloud-native compliance pipeline. The three-service microservices architecture — with parallel detection, a multi-agent RAG system, and a structured knowledge graph — enables independent scaling of each component and straightforward integration into existing manufacturing quality control workflows.