AWS Bedrock AgentCore: Observability Architecture for AI-Driven Merchant Banking Application Intake

Monitoring agentic tool performance, compliance attribution, and operational insights

The Research That Sparked This Architecture

A recent paper caught my attention: “AgentSHAP: Interpreting LLM Agent Tool Importance with Monte Carlo Shapley Value Estimation” (arXiv:2512.12597). It addresses a blind spot in AI agent development that most teams overlook until regulators ask questions.

The problem: AI agents call tools, logs show the calls, but nobody can prove which tools actually influenced the response. The paper applies Shapley values from cooperative game theory—running agents with different tool subsets, measuring how responses change, and computing fair attribution scores.

For consumer applications, this is interesting. For regulated financial services, this is essential.

When an examiner asks “How do you know your OFAC screening actually influenced this onboarding decision?”, the answer can’t be “we logged the API call.” You need provable attribution.

This got me thinking about a use case where this architecture would deliver immediate value: merchant banking application intake.

The Merchant Application Problem: A Story Repeated Daily

Let me paint the picture of what happens thousands of times daily at merchant acquiring banks.

9:15 AM — Maria owns a food truck. She visits a bank’s website looking for payment processing. She finds a “Request a Callback” form.

9:18 AM — Maria fills out her name, phone number, and business type. She submits and sees “A specialist will contact you within 24 hours.”

Next day, 2:30 PM — A sales rep calls Maria. She’s in the middle of the lunch rush. “Can you call back in an hour?”

3:45 PM — The rep calls back. Maria is now available. The rep starts collecting information:

  • Legal business name
  • DBA name
  • Tax ID (Maria has to look this up)
  • Business address
  • Years in operation
  • Estimated monthly volume
  • Average transaction size

4:10 PM — Now they move to owner verification:

  • Maria’s full legal name
  • Social Security Number
  • Date of birth
  • Home address
  • Ownership percentage

4:25 PM — The rep explains Maria needs to provide:

  • Copy of driver’s license
  • Voided check or bank letter
  • Business license
  • Articles of incorporation (Maria doesn’t know where this is)

4:30 PM — Maria says she’ll email the documents later. The call ends.

Three days later — Maria hasn’t sent the documents. Life got busy. A follow-up email sits in her inbox.

One week later — Another follow-up. Maria finally gathers documents, emails them as attachments.

Day 10 — Underwriting reviews the application. The EIN doesn’t match the business name in IRS records. The application goes to a hold queue.

Day 12 — A different rep calls Maria to resolve the discrepancy. It was a typo in the original call.

Day 15 — Application finally approved. Equipment ships.

Total elapsed time: 15 days Total bank cost: Approximately $200 (multiple rep calls, data entry, follow-ups, underwriting rework) Maria’s experience: Frustrating

Now multiply this by thousands of applications monthly.

The Numbers Behind the Pain

Industry experience suggests these ranges are common:

Metric Typical Range What’s Actually Happening
60-70% abandonment Merchants start applications but never finish. Forms are complex. They don’t have their EIN handy. They get interrupted. They find a competitor who onboards in 5 minutes.  
$150-300 internal cost per application The bank’s cost: sales rep time, system usage, follow-up calls, data entry labor, underwriting review, error correction.  
15-20% rework rate Applications delayed due to: TIN/business name mismatch with IRS, unverifiable addresses, failed identity verification, incorrect MCC classification, missing required documents.  
5-10 business days From first contact to underwriting decision. Fintechs do this in minutes.  

The data quality issues deserve specific attention:

  • EIN doesn’t match business name: IRS TIN matching returns “no match” because of typos, abbreviations, or using DBA instead of legal name
  • Address validation fails: USPS can’t standardize the address, or it’s flagged as non-deliverable
  • Identity verification fails: Owner’s SSN doesn’t match name/DOB in credit bureau records
  • Business not found: State Secretary of State has no record of the entity
  • MCC misclassification: Business described incorrectly, assigned wrong category, triggers incorrect risk treatment
  • Missing documents: Required documents for that business type or state weren’t collected

Every one of these issues is discoverable in real-time during the application conversation. Instead, they’re discovered days later in underwriting.

Why an AI Agent Fits This Problem

Merchant application intake isn’t a simple form—it’s a branching conversation:

                    ┌─────────────────────┐
                    │  What type of       │
                    │  business?          │
                    └──────────┬──────────┘
                               │
        ┌──────────────────────┼──────────────────────┐
        ▼                      ▼                      ▼
   Restaurant             E-commerce            Professional
        │                      │                  Services
        ▼                      ▼                      ▼
   Recommend POS          Recommend             Recommend
   terminal + tips        payment gateway       virtual terminal
        │                      │                      │
        ▼                      ▼                      ▼
   Need tableside?        Shopping cart?        Invoice or
   Counter? Both?         platform?             key-in?
        │                      │                      │
        └──────────────────────┼──────────────────────┘
                               │
                    ┌──────────┴──────────┐
                    │  Expected monthly   │
                    │  volume?            │
                    └──────────┬──────────┘
                               │
        ┌──────────────────────┼──────────────────────┐
        ▼                      ▼                      ▼
   <$10K/month            $10K-$50K              >$50K
   Standard rates        Volume discount        Custom pricing
        │                      │                      │
        └──────────────────────┼──────────────────────┘
                               │
                               ▼
                    Risk profile determines
                    documentation requirements...

A rigid web form can’t navigate this. A human rep can, but at $150-300 per application with inconsistent quality.

An AI agent with the right tools can:

  • Ask the right questions based on business type
  • Recommend appropriate products
  • Validate data in real-time (catch the EIN typo during conversation)
  • Run compliance checks in the background
  • Collect documents via chat with instant OCR extraction
  • Submit complete, validated applications

The transformation:

  • 5-8 minutes instead of 45
  • $15-30 instead of $150-300
  • 24/7 availability
  • Near-perfect data quality
  • Instant compliance verification

But here’s the challenge that brings us back to the AgentSHAP research: you need to prove the agent is doing what it should.

AWS Bedrock AgentCore: Configuration Deep-Dive

Bedrock AgentCore provides the managed infrastructure for production AI agents. Rather than listing features, let me walk through the specific configurations that matter for this implementation.

Agent Definition

yaml
Agent:
  FoundationModel: anthropic.claude-3-5-sonnet-20241022-v2:0
  IdleSessionTTL: 1800  # 30 minutes - merchants may need to locate documents
  
  InstructionConfiguration:
    Instruction: |
      You are a merchant services application specialist. Guide business 
      owners through payment processing applications.
      
      CRITICAL BEHAVIORS:
      - Validate all data in real-time before proceeding
      - Never skip compliance checks regardless of merchant requests
      - If verification fails, explain clearly and offer alternatives
      - Escalate to human review for: high-risk MCCs, verification 
        failures, complex multi-location setups
      
      CONVERSATION FLOW:
      1. Understand business type and needs → recommend product
      2. Collect and validate business information
      3. Verify owner/principal identity (KYC)
      4. Run compliance screening (OFAC, PEP)
      5. Collect and process required documents
      6. Generate pricing and submit application
      
      BEFORE COLLECTING DOCUMENTS:
      - Always check if documents are already on file
      - If exists, compare for material changes before requesting new
      
      BEFORE STARTING APPLICATION:
      - Check for existing pending or recent applications
      - Offer to check status or continue existing if found

The IdleSessionTTL of 30 minutes is intentional. Merchants often need to step away to find documents. Too short loses progress; too long wastes resources.

Action Groups: Tool Organization

Tools are organized into Action Groups. This grouping affects how the model reasons about tool selection.

Product Selection Group

yaml
ActionGroup: ProductSelection
Tools:
  - Name: RecommendProduct
    Description: Recommend payment processing products based on business needs
    Parameters:
      - business_type: string (restaurant, retail, ecommerce, services, etc.)
      - monthly_volume: number
      - sales_channel: string (in-person, online, phone, mixed)
      - mobility_needs: boolean
    Returns: Ranked list of product recommendations with rationale
    Backend: Lambda → DynamoDB product catalog + rules engine
    
  - Name: GetProductDetails
    Description: Get specifications and pricing for a specific product
    Parameters:
      - product_id: string
    Returns: Features, specifications, pricing tiers, compatibility info
    Backend: Lambda → DynamoDB + S3 (product documentation)
    
  - Name: CalculatePricing
    Description: Estimate monthly costs based on transaction patterns
    Parameters:
      - product_id: string
      - monthly_volume: number
      - average_ticket: number
      - transaction_count: number
    Returns: Itemized monthly cost breakdown
    Backend: Lambda → Pricing rules engine

Business Verification Group

yaml
ActionGroup: BusinessVerification
Tools:
  - Name: ValidateTIN
    Description: Verify EIN or SSN format and match against business name
    Parameters:
      - tin_type: string (EIN, SSN)
      - tin_value: string
      - legal_business_name: string
    Returns: Validity status, match result, suggested corrections
    Backend: Lambda → IRS TIN Matching API
    
  - Name: VerifyBusinessRegistration
    Description: Check business entity registration with state
    Parameters:
      - business_name: string
      - state: string
      - entity_type: string (LLC, Corp, Sole Prop, etc.)
    Returns: Registration status, formation date, standing, registered agent
    Backend: Lambda → State Secretary of State APIs (varies by state)
    
  - Name: LookupMCC
    Description: Determine Merchant Category Code based on business description
    Parameters:
      - business_description: string
      - products_services: string
      - sales_channel: string
    Returns: Recommended MCC, description, risk tier, required documentation
    Backend: Lambda → MCC classification model
    
  - Name: ValidateAddress
    Description: Standardize and verify business address
    Parameters:
      - street: string
      - city: string
      - state: string
      - zip: string
    Returns: Standardized address, deliverability score, corrections applied
    Backend: Lambda → USPS Address Validation API

KYC and Compliance Group

This is where regulatory requirements live. KYC (Know Your Customer) typically integrates with third-party identity verification services.

yaml
ActionGroup: KYCCompliance
Tools:
  - Name: VerifyIdentity
    Description: KYC identity verification for business owner/principal
    Parameters:
      - full_name: string
      - ssn: string
      - date_of_birth: string
      - address: object
    Returns: Verification status, confidence score, flags, verification source
    Backend: Lambda → Third-party KYC provider (Jumio, Onfido, Socure, etc.)
    Note: Returns verification result, not raw identity data
    
  - Name: ScreenOFAC
    Description: Screen individual and business against OFAC sanctions lists
    Parameters:
      - individual_name: string
      - business_name: string
      - country: string
    Returns: Match found (boolean), match details, lists checked, screening ID
    Backend: Lambda → Treasury OFAC API or sanctions screening service
    Compliance: Federal regulatory requirement - cannot be skipped
    
  - Name: CheckPEP
    Description: Screen for Politically Exposed Persons
    Parameters:
      - full_name: string
      - date_of_birth: string
      - country: string
    Returns: PEP status, position details, risk level
    Backend: Lambda → PEP screening service
    
  - Name: AssessRisk
    Description: Calculate preliminary risk score for underwriting
    Parameters:
      - mcc: string
      - monthly_volume: number
      - business_age_years: number
      - owner_credit_indicator: string
    Returns: Risk tier, required documentation, manual review flag
    Backend: Lambda → Risk scoring model

Document Group

Document handling requires more sophistication than simple upload and OCR. The agent should check existing documents first.

yaml
ActionGroup: Documents
Tools:
  - Name: CheckExistingDocuments
    Description: Check if required document is already on file for this entity
    Parameters:
      - application_id: string
      - document_type: string (drivers_license, voided_check, articles, etc.)
    Returns: Exists (boolean), document_id, upload_date, expiration_status
    Backend: Lambda → DynamoDB document registry
    Purpose: Avoid re-collecting documents already on file
    
  - Name: CompareDocuments
    Description: Compare newly uploaded document against existing version
    Parameters:
      - application_id: string
      - document_type: string
      - new_document_s3_uri: string
    Returns: Changes detected, change summary, recommendation (use_new/keep_existing)
    Backend: Lambda → Textract + comparison logic
    Purpose: Determine if new document has material changes worth updating
    
  - Name: ProcessDocument
    Description: Extract structured data from uploaded document image
    Parameters:
      - document_s3_uri: string
      - document_type: string
      - expected_fields: array
    Returns: Extracted fields, confidence scores, validation status
    Backend: Lambda → Amazon Textract (AnalyzeDocument API)
    
  - Name: ValidateDocumentData
    Description: Compare extracted document data against application data
    Parameters:
      - extracted_fields: object
      - application_data: object
    Returns: Match status, discrepancies found, fields requiring confirmation
    Backend: Lambda → Field matching logic

Application Management Group

Application handling must account for existing applications to prevent duplicates and enable session recovery.

yaml
ActionGroup: ApplicationManagement
Tools:
  - Name: CheckExistingApplications
    Description: Find any existing applications for this business/owner
    Parameters:
      - tin: string
      - owner_ssn_last4: string
      - business_name: string
    Returns: Applications found (array), status of each, recommendation
    Backend: Lambda → DynamoDB applications table
    Purpose: Detect pending, approved, or recently declined applications
    
  - Name: SaveApplicationProgress
    Description: Persist current application state for session recovery
    Parameters:
      - application_id: string
      - current_phase: string
      - collected_data: object
    Returns: Save confirmation, resume_token
    Backend: Lambda → DynamoDB
    
  - Name: ResumeApplication
    Description: Retrieve saved application state
    Parameters:
      - resume_token: string
      # OR
      - tin: string
      - owner_verification: object  # Light verification to resume
    Returns: Application state, next required fields, conversation context
    Backend: Lambda → DynamoDB
    
  - Name: SubmitApplication
    Description: Submit completed application to underwriting queue
    Parameters:
      - application_id: string
    Returns: Submission confirmation, reference number, expected timeline
    Backend: Lambda → SQS (underwriting queue) + DynamoDB status update
    Prerequisites: All required fields validated, compliance checks passed
    
  - Name: ScheduleCallback
    Description: Escalate to human representative
    Parameters:
      - application_id: string
      - reason: string
      - preferred_time: string
      - phone_number: string
    Returns: Callback scheduled confirmation, reference number
    Backend: Lambda → CRM integration or callback queue

Bedrock Guardrails: Safety Configuration

Guardrails provide declarative safety controls—critical for financial services.

yaml
Guardrail:
  Name: MerchantIntakeGuardrail
  
  ContentFilters:
    - Type: HATE
      Strength: HIGH
    - Type: VIOLENCE
      Strength: HIGH
    - Type: SEXUAL
      Strength: HIGH
    - Type: MISCONDUCT
      Strength: HIGH
  
  SensitiveInformationPolicy:
    PIIEntities:
      - Type: SSN
        Action: ANONYMIZE      # Replace with [SSN] in all responses
      - Type: CREDIT_CARD_NUMBER
        Action: BLOCK          # Never include under any circumstances
      - Type: BANK_ACCOUNT_NUMBER
        Action: ANONYMIZE
      - Type: DRIVER_LICENSE
        Action: ANONYMIZE
    
    RegexPatterns:
      - Name: EIN_PATTERN
        Pattern: '\b\d{2}-\d{7}\b'
        Action: ANONYMIZE
      - Name: ROUTING_NUMBER
        Pattern: '\b\d{9}\b'
        Action: ANONYMIZE

  TopicPolicy:
    DeniedTopics:
      - Name: COMPETITOR_DISCUSSION
        Definition: "Discussion of competitor products, pricing, or recommendations"
        Examples:
          - "What about Square or Stripe?"
          - "Is PayPal better?"
        Action: BLOCK
      
      - Name: INVESTMENT_ADVICE
        Definition: "Financial planning or investment recommendations"
        Action: BLOCK
      
      - Name: UNRELATED_BANKING
        Definition: "Personal loans, mortgages, or products unrelated to merchant services"
        Action: BLOCK

  WordPolicy:
    ManagedWordLists:
      - Type: PROFANITY
        Action: BLOCK
    
    CustomWordLists:
      - Words: 
          - "guaranteed approval"
          - "instant approval" 
          - "no credit check required"
        Action: BLOCK  # These phrases create compliance risk

The SensitiveInformationPolicy is critical. The agent collects SSNs, EINs, and bank account numbers during the application. These must never appear in agent responses—not even when confirming information back to the merchant. Guardrails enforce this at the platform level.

Knowledge Base Configuration

The Knowledge Base provides RAG-powered retrieval for product information, pricing rules, and compliance requirements.

yaml
KnowledgeBase:
  Name: MerchantServicesKB
  
  DataSource:
    Type: S3
    BucketArn: arn:aws:s3:::merchant-kb-documents
    InclusionPrefixes:
      - "products/"              # Product specifications and features
      - "pricing/"               # Pricing tiers and rules
      - "compliance/"            # State-specific requirements
      - "mcc-codes/"             # MCC descriptions and risk mappings
      - "faqs/"                  # Common questions and answers
      - "prohibited-businesses/" # Industries not serviced
    
  ChunkingConfiguration:
    Strategy: HIERARCHICAL
    ParentChunkSize: 1500    # Broader context
    ChildChunkSize: 300      # Precise retrieval
    OverlapPercentage: 20
  
  EmbeddingModel: amazon.titan-embed-text-v2
  
  VectorStore:
    Type: OPENSEARCH_SERVERLESS
    CollectionArn: arn:aws:aoss:us-east-1:xxx:collection/merchant-kb
    
  RetrievalConfiguration:
    NumberOfResults: 5
    SearchType: HYBRID  # Combines semantic and keyword search

The HIERARCHICAL chunking strategy matters. Parent chunks provide context (full product description), while child chunks enable precise retrieval (specific feature or pricing detail).

The Observability Architecture: Where AgentSHAP Meets Production

Now we arrive at the monitoring architecture that makes tool attribution practical.

Design Principles

  1. Log everything, but hash PII — Complete tool invocation records without compliance risk
  2. Capture context — What prompted the tool call, not just that it happened
  3. Enable attribution analysis — Structure data for Shapley-style importance computation
  4. Real-time and historical — CloudWatch for immediate visibility, Athena/QuickSight for analysis

Tool Invocation Logging

Every tool call generates a structured log entry:

json
{
  "timestamp": "2025-01-13T14:32:18.445Z",
  "trace_id": "1-65a2b3c4-abc123def456",
  "session_id": "ses_abc123def456",
  "application_id": "APP-2025-00892",
  "conversation_turn": 12,
  "phase": "kyc_verification",
  
  "tool_invocation": {
    "action_group": "KYCCompliance",
    "tool_name": "ScreenOFAC",
    "invocation_id": "inv_xyz789",
    "input_hash": "sha256:a1b2c3d4e5f6...",
    "input_field_names": ["individual_name", "business_name", "country"],
    "execution_start_ms": 1705156338445,
    "execution_end_ms": 1705156339337,
    "execution_duration_ms": 892,
    "output_summary": {
      "match_found": false,
      "confidence": 1.0,
      "lists_checked": ["SDN", "CONS", "PLC"],
      "screening_id": "scr_abc123"
    },
    "output_hash": "sha256:f6e5d4c3b2a1..."
  },
  
  "context": {
    "preceding_user_message_type": "ssn_provided",
    "agent_reasoning": "User provided owner SSN, initiating compliance screening",
    "tools_called_this_turn": ["VerifyIdentity", "ScreenOFAC", "CheckPEP"],
    "tool_sequence_position": 2
  },
  
  "application_state": {
    "fields_collected": 18,
    "fields_remaining": 4,
    "compliance_checks_completed": ["tin_validation", "business_registration"],
    "documents_collected": ["drivers_license"]
  }
}

Key design decisions:

  • input_hash instead of actual input — PII never lands in analytics
  • context.agent_reasoning — Captures why the tool was called (crucial for attribution)
  • tool_sequence_position — Enables analysis of tool ordering effects
  • application_state — Allows correlation with application outcomes

Analytics Pipeline Architecture

┌─────────────────────────────────────────────────────────────────────────────────┐
│                         OBSERVABILITY PIPELINE                                  │
├─────────────────────────────────────────────────────────────────────────────────┤
│                                                                                 │
│  BEDROCK AGENT                                                                  │
│       │                                                                         │
│       ▼                                                                         │
│  CloudWatch Logs (/aws/bedrock/merchant-intake-agent)                          │
│       │                                                                         │
│       ├──────────────────────────────────┐                                     │
│       │                                  │                                     │
│       ▼                                  ▼                                     │
│  CloudWatch Metrics               Subscription Filter                          │
│  (Real-time dashboards)                  │                                     │
│       │                                  ▼                                     │
│       │                           Kinesis Firehose                             │
│       │                                  │                                     │
│       │                                  ▼                                     │
│       │                           Lambda Transform                             │
│       │                           (Enrich + Parquet)                           │
│       │                                  │                                     │
│       │                                  ▼                                     │
│       │                           S3 Data Lake                                 │
│       │                           (Partitioned by date)                        │
│       │                                  │                                     │
│       │                                  ▼                                     │
│       │                           Glue Data Catalog                            │
│       │                                  │                                     │
│       │                                  ▼                                     │
│       │                              Athena                                    │
│       │                                  │                                     │
│       └──────────────────────────────────┼──────────────────────────────────┐ │
│                                          ▼                                  ▼ │
│                                     QuickSight                      EventBridge│
│                                     (Dashboards)                    (Alerts)   │
│                                                                                 │
└─────────────────────────────────────────────────────────────────────────────────┘

Kinesis Firehose Configuration

yaml
DeliveryStream:
  Name: merchant-intake-tool-logs
  
  Source:
    Type: CloudWatchLogsSubscription
    LogGroup: /aws/bedrock/merchant-intake-agent
    FilterPattern: "{ $.tool_invocation.tool_name = * }"
  
  Processing:
    Processors:
      - Type: Lambda
        LambdaArn: arn:aws:lambda:us-east-1:xxx:function:EnrichToolLogs
        BufferSizeInMBs: 1
        BufferIntervalInSeconds: 60
  
  Destination:
    Type: S3
    BucketArn: arn:aws:s3:::merchant-analytics-data-lake
    Prefix: "tool-logs/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/"
    ErrorOutputPrefix: "errors/"
    
    BufferingHints:
      SizeInMBs: 64
      IntervalInSeconds: 300
    
    DataFormatConversion:
      Enabled: true
      InputFormat: JSON
      OutputFormat: PARQUET
      SchemaConfiguration:
        DatabaseName: merchant_analytics
        TableName: tool_invocations
        Region: us-east-1
        CatalogId: !Ref AWS::AccountId

The enrichment Lambda adds application outcomes when known (joins with DynamoDB) and computes derived metrics.

Athena Table Definition

sql
CREATE EXTERNAL TABLE merchant_analytics.tool_invocations (
  timestamp timestamp,
  trace_id string,
  session_id string,
  application_id string,
  conversation_turn int,
  phase string,
  
  action_group string,
  tool_name string,
  invocation_id string,
  execution_duration_ms int,
  
  input_hash string,
  output_hash string,
  output_summary map<string, string>,
  
  preceding_message_type string,
  agent_reasoning string,
  tool_sequence_position int,
  
  -- Enriched fields (added by Lambda)
  application_outcome string,
  time_to_completion_hours decimal(10,2),
  data_quality_score decimal(3,2)
)
PARTITIONED BY (year string, month string, day string)
STORED AS PARQUET
LOCATION 's3://merchant-analytics-data-lake/tool-logs/'
TBLPROPERTIES ('parquet.compression'='SNAPPY');

Analytics Queries for Tool Attribution

Query 1: Tool Importance by Phase

sql
-- Which tools are called in each phase, and how do they correlate with success?
SELECT 
  phase,
  tool_name,
  COUNT(*) as invocation_count,
  COUNT(DISTINCT application_id) as applications_using_tool,
  
  -- Success rate when this tool is used in this phase
  AVG(CASE WHEN application_outcome = 'APPROVED' THEN 1.0 ELSE 0.0 END) as approval_rate,
  
  -- Average execution time
  AVG(execution_duration_ms) as avg_latency_ms,
  APPROX_PERCENTILE(execution_duration_ms, 0.95) as p95_latency_ms
  
FROM merchant_analytics.tool_invocations
WHERE year = '2025' AND month = '01'
  AND application_outcome IS NOT NULL
GROUP BY phase, tool_name
ORDER BY phase, invocation_count DESC;

Query 2: Compliance Tool Attribution

 
 
sql
-- Critical: Are compliance tools showing appropriate importance?
-- Alert if OFAC or KYC tools have low correlation with outcomes

WITH compliance_tools AS (
  SELECT 
    application_id,
    MAX(CASE WHEN tool_name = 'ScreenOFAC' THEN 1 ELSE 0 END) as ofac_called,
    MAX(CASE WHEN tool_name = 'VerifyIdentity' THEN 1 ELSE 0 END) as kyc_called,
    MAX(CASE WHEN tool_name = 'CheckPEP' THEN 1 ELSE 0 END) as pep_called
  FROM merchant_analytics.tool_invocations
  WHERE phase = 'kyc_verification'
    AND year = '2025' AND month = '01'
  GROUP BY application_id
),
outcomes AS (
  SELECT DISTINCT application_id, application_outcome
  FROM merchant_analytics.tool_invocations
  WHERE application_outcome IS NOT NULL
)
SELECT 
  'ScreenOFAC' as tool,
  SUM(ofac_called) as times_called,
  COUNT(*) as total_applications,
  CAST(SUM(ofac_called) AS DECIMAL(10,2)) / COUNT(*) as call_rate,
  -- This should be very close to 1.0 - if not, investigate
  CASE 
    WHEN CAST(SUM(ofac_called) AS DECIMAL(10,2)) / COUNT(*) < 0.95 
    THEN 'ALERT: OFAC not called in >5% of applications'
    ELSE 'OK'
  END as status
FROM compliance_tools c
JOIN outcomes o ON c.application_id = o.application_id

UNION ALL

SELECT 
  'VerifyIdentity' as tool,
  SUM(kyc_called),
  COUNT(*),
  CAST(SUM(kyc_called) AS DECIMAL(10,2)) / COUNT(*),
  CASE 
    WHEN CAST(SUM(kyc_called) AS DECIMAL(10,2)) / COUNT(*) < 0.95 
    THEN 'ALERT: KYC not called in >5% of applications'
    ELSE 'OK'
  END
FROM compliance_tools c
JOIN outcomes o ON c.application_id = o.application_id;

Query 3: Tool Latency Impact on Abandonment

sql
-- Do slow tools cause abandonment?
WITH session_metrics AS (
  SELECT 
    session_id,
    application_id,
    MAX(conversation_turn) as total_turns,
    SUM(execution_duration_ms) as total_tool_latency_ms,
    AVG(execution_duration_ms) as avg_tool_latency_ms,
    MAX(application_outcome) as outcome
  FROM merchant_analytics.tool_invocations
  WHERE year = '2025' AND month = '01'
  GROUP BY session_id, application_id
)
SELECT 
  CASE 
    WHEN avg_tool_latency_ms < 500 THEN '0-500ms'
    WHEN avg_tool_latency_ms < 1000 THEN '500-1000ms'
    WHEN avg_tool_latency_ms < 2000 THEN '1000-2000ms'
    ELSE '>2000ms'
  END as latency_bucket,
  COUNT(*) as sessions,
  AVG(CASE WHEN outcome IS NULL THEN 1.0 ELSE 0.0 END) as abandonment_rate,
  AVG(total_turns) as avg_conversation_length
FROM session_metrics
GROUP BY 1
ORDER BY 1;

Query 4: AgentSHAP-Style Importance Estimation

This query approximates Shapley-style importance by measuring outcome correlation:

sql
-- Simplified importance scoring
-- Full Shapley computation runs as separate batch job

WITH tool_presence AS (
  SELECT 
    application_id,
    tool_name,
    1 as tool_used
  FROM merchant_analytics.tool_invocations
  WHERE year = '2025' AND month = '01'
  GROUP BY application_id, tool_name
),
all_apps AS (
  SELECT DISTINCT application_id, application_outcome
  FROM merchant_analytics.tool_invocations
  WHERE application_outcome IS NOT NULL
    AND year = '2025' AND month = '01'
),
tool_outcomes AS (
  SELECT 
    t.tool_name,
    a.application_id,
    a.application_outcome,
    COALESCE(tp.tool_used, 0) as tool_used
  FROM all_apps a
  CROSS JOIN (SELECT DISTINCT tool_name FROM tool_presence) t
  LEFT JOIN tool_presence tp 
    ON a.application_id = tp.application_id 
    AND t.tool_name = tp.tool_name
)
SELECT 
  tool_name,
  
  -- Approval rate when tool IS used
  AVG(CASE WHEN tool_used = 1 AND application_outcome = 'APPROVED' THEN 1.0
           WHEN tool_used = 1 THEN 0.0 
           ELSE NULL END) as approval_rate_with_tool,
  
  -- Approval rate when tool is NOT used
  AVG(CASE WHEN tool_used = 0 AND application_outcome = 'APPROVED' THEN 1.0
           WHEN tool_used = 0 THEN 0.0
           ELSE NULL END) as approval_rate_without_tool,
  
  -- Importance proxy: difference in approval rates
  AVG(CASE WHEN tool_used = 1 AND application_outcome = 'APPROVED' THEN 1.0
           WHEN tool_used = 1 THEN 0.0 
           ELSE NULL END) -
  AVG(CASE WHEN tool_used = 0 AND application_outcome = 'APPROVED' THEN 1.0
           WHEN tool_used = 0 THEN 0.0
           ELSE NULL END) as importance_proxy

FROM tool_outcomes
GROUP BY tool_name
HAVING COUNT(CASE WHEN tool_used = 1 THEN 1 END) > 10  -- Minimum sample size
ORDER BY importance_proxy DESC;

QuickSight Dashboard Design

Dashboard 1: Application Funnel

Visual Type Metrics
Applications Over Time Line chart Started, Completed, Approved by day
Abandonment by Phase Funnel Drop-off count and rate per phase
Completion Time Distribution Histogram Minutes to complete (p50, p90)
Abandonment Reasons Pie chart Phase where session ended

Dashboard 2: Tool Performance

Visual Type Metrics
Tool Invocation Heatmap Heatmap Tools × Phases, colored by frequency
Latency by Tool Box plot p50, p75, p95, p99 per tool
Error Rate Trend Line chart Tool errors over time
Slow Tool Alert KPI Tools exceeding latency threshold

Dashboard 3: Compliance Monitoring

Visual Type Metrics
OFAC Completion Rate Gauge Should be 100%
KYC Verification Outcomes Stacked bar Verified, Failed, Manual Review
Compliance Check Coverage Table Applications missing required checks
Risk Tier Distribution Pie chart Low, Medium, High risk applications

Dashboard 4: Tool Attribution (AgentSHAP-Inspired)

Visual Type Metrics
Tool Importance by Phase Heatmap Tools × Phases, colored by importance score
Importance Trend Line chart Tool importance scores over time
Attribution Alerts Table Tools with unexpected importance patterns
Optimization Candidates Ranked list Low-importance, high-frequency tools

EventBridge Alerting

yaml
Rule:
  Name: ComplianceToolAlert
  Description: Alert when compliance tools show low usage or importance
  
  EventPattern:
    source:
      - "custom.merchant-analytics"
    detail-type:
      - "ComplianceMetricBreach"
    detail:
      metric_name:
        - "ofac_call_rate"
        - "kyc_call_rate"
        - "compliance_tool_importance"
      breach_type:
        - "below_threshold"
  
  Targets:
    - Id: SNSAlert
      Arn: arn:aws:sns:us-east-1:xxx:compliance-alerts
    - Id: PagerDuty
      Arn: arn:aws:events:us-east-1:xxx:api-destination/pagerduty

A Lambda function runs hourly, queries Athena for compliance metrics, and publishes events when thresholds are breached.

Conversation Patterns: Handling Real-World Complexity

The agent must handle edge cases gracefully. Here’s how tool attribution helps verify correct behavior.

Scenario 1: Existing Application Found

Merchant: "I'd like to apply for a payment terminal"

Agent: [Calls CheckExistingApplications]
       [Returns: 1 pending application from 5 days ago]

Agent: "I found an existing application for Tasty Tacos LLC that you 
        submitted on January 8th. It's currently in underwriting review.
        
        Would you like me to:
        • Check the status of that application, or
        • Start a new application for a different location or product?"

Attribution Check: CheckExistingApplications should show HIGH importance early in conversation. If merchants frequently proceed to create duplicates, either the tool isn’t being called or its output isn’t influencing the agent.

Scenario 2: Document Already on File

Agent: "I'll need a copy of your driver's license for identity verification."

Merchant: [Uploads driver's license image]

Agent: [Calls CheckExistingDocuments → found existing from prior app]
       [Calls CompareDocuments → detects address change]

Agent: "I see we have a driver's license on file from your previous 
        application, but the address is different. Your new license 
        shows 456 Oak Street instead of 123 Main Street.
        
        Should I update your business address to match?"

Attribution Check: CheckExistingDocuments and CompareDocuments should both show importance during document collection. If documents are being re-collected unnecessarily, these tools may not be invoked.

Scenario 3: KYC Verification Fails

Agent: [Calls VerifyIdentity → returns FAILED, confidence 0.3]

Agent: "I wasn't able to verify your identity with the information 
        provided. This sometimes happens when:
        
        • The Social Security Number has a typo
        • Your name doesn't match exactly (maiden name, suffix, etc.)
        • Your address doesn't match credit bureau records
        
        Would you like to:
        • Re-enter your information, or
        • Speak with a specialist who can help verify manually?
        
        I can schedule a callback if that's easier."

Attribution Check: When VerifyIdentity fails, ScheduleCallback should show increased importance (escalation path). If applications with failed KYC proceed without escalation, there’s a compliance gap.


Implementation Considerations

Cold Start Latency

Lambda functions backing Action Groups experience cold starts. For merchant-facing tools where latency matters:

  • Configure provisioned concurrency for VerifyIdentity, ScreenOFAC, ValidateTIN
  • Accept cold starts for less time-sensitive tools like SaveApplicationProgress
  • Consider Lambda SnapStart for Java-based tools

Third-Party API Resilience

External APIs (IRS TIN matching, state SOS, KYC providers) fail. Implement:

  • Circuit breakers (AWS App Config feature flags)
  • Retry with exponential backoff
  • Graceful degradation: set manual_review_required flag and proceed
  • Fallback providers where available

Session Recovery

Merchants abandon sessions to find documents. Enable recovery:

  • Call SaveApplicationProgress after each phase completion
  • Generate resume tokens with reasonable expiration (7 days)
  • Allow light re-authentication (TIN + owner DOB) to resume
  • Send email/SMS with resume link if contact info collected

Compliance Audit Trail

For regulatory examination, maintain immutable records:

  • Every compliance tool invocation with full request/response
  • Input parameters (hashed for PII, but recoverable with key)
  • Timestamp, duration, outcome
  • Agent’s interpretation and subsequent action
  • Store in separate compliance log with extended retention

Architecture Summary

┌─────────────────────────────────────────────────────────────────────────────────┐
│                    MERCHANT INTAKE AGENT - COMPLETE ARCHITECTURE                │
├─────────────────────────────────────────────────────────────────────────────────┤
│                                                                                 │
│  PRESENTATION                                                                   │
│  ┌─────────────────────────────────────────────────────────────────────────┐   │
│  │  CloudFront ──▶ S3 (React SPA) ──▶ API Gateway (WebSocket)             │   │
│  └─────────────────────────────────────────────────────────────────────────┘   │
│                                          │                                      │
│                                          ▼                                      │
│  AGENT LAYER                                                                    │
│  ┌─────────────────────────────────────────────────────────────────────────┐   │
│  │                      AWS BEDROCK AGENTCORE                              │   │
│  │  ┌───────────────────────────────────────────────────────────────────┐ │   │
│  │  │  Agent: Claude 3.5 Sonnet                                         │ │   │
│  │  │  Knowledge Base: S3 → Titan Embeddings → OpenSearch Serverless    │ │   │
│  │  │  Guardrails: PII filtering, topic blocking, word policies         │ │   │
│  │  │  Session Memory: DynamoDB-backed                                  │ │   │
│  │  └───────────────────────────────────────────────────────────────────┘ │   │
│  │                                  │                                      │   │
│  │  ┌───────────────────────────────┼───────────────────────────────────┐ │   │
│  │  │                    ACTION GROUPS                                  │ │   │
│  │  │                                                                   │ │   │
│  │  │  ProductSelection │ BusinessVerify │ KYCCompliance │ Documents   │ │   │
│  │  │  ApplicationMgmt  │                                              │ │   │
│  │  │                                                                   │ │   │
│  │  └───────────────────────────────┼───────────────────────────────────┘ │   │
│  └──────────────────────────────────┼──────────────────────────────────────┘   │
│                                     │                                          │
│  COMPUTE LAYER                      ▼                                          │
│  ┌─────────────────────────────────────────────────────────────────────────┐   │
│  │                         AWS LAMBDA FUNCTIONS                            │   │
│  │                    (Tool implementations + integrations)                │   │
│  └─────────────────────────────────────────────────────────────────────────┘   │
│                    │                │                │                         │
│  DATA LAYER        ▼                ▼                ▼                         │
│  ┌──────────────────────────────────────────────────────────────────────────┐  │
│  │  DynamoDB          S3                Secrets Manager    External APIs    │  │
│  │  • Sessions        • Documents       • API credentials  • IRS TIN        │  │
│  │  • Applications    • KB source       • KYC provider     • State SOS      │  │
│  │  • Tool logs       • Analytics       • OFAC service     • KYC/IDV        │  │
│  │                                                         • OFAC           │  │
│  └──────────────────────────────────────────────────────────────────────────┘  │
│                                                                                 │
│  OBSERVABILITY LAYER                                                           │
│  ┌──────────────────────────────────────────────────────────────────────────┐  │
│  │                                                                          │  │
│  │  CloudWatch ──▶ Kinesis Firehose ──▶ S3 (Parquet) ──▶ Athena            │  │
│  │       │                                                   │              │  │
│  │       │                     ┌─────────────────────────────┤              │  │
│  │       │                     │                             │              │  │
│  │       ▼                     ▼                             ▼              │  │
│  │  CloudWatch            QuickSight                   EventBridge          │  │
│  │  Dashboards            Dashboards                   Alerts               │  │
│  │  (Real-time)           (Analytics)                  (Compliance)         │  │
│  │                                                                          │  │
│  └──────────────────────────────────────────────────────────────────────────┘  │
│                                                                                 │
└─────────────────────────────────────────────────────────────────────────────────┘

Measuring Success

Beyond standard operational metrics, apply AgentSHAP-inspired attribution:

Expected Tool Importance by Phase

Phase Expected HIGH Importance Alert If LOW
Qualification RecommendProduct
Business Info ValidateTIN, VerifyBusinessRegistration ValidateTIN
KYC/Compliance VerifyIdentity, ScreenOFAC Either one
Documents CheckExistingDocuments, ProcessDocument CheckExistingDocuments
Submission CheckExistingApplications, SubmitApplication CheckExistingApplications

If OFAC screening shows low importance during KYC phase, investigate immediately. Either the tool isn’t being called, or its output isn’t influencing agent behavior. Both are compliance failures.

Optimization Candidates

After sufficient volume (1000+ applications), identify:

  • High frequency, low importance: Cost reduction candidates
  • High latency, high importance: Performance optimization priority
  • Low frequency, high importance: Reliability critical—ensure redundancy

Conclusion

AWS Bedrock AgentCore provides the managed infrastructure for production AI agents. The combination of Action Groups, Knowledge Bases, and Guardrails creates a foundation suitable for regulated financial services.

The observability architecture—CloudWatch through Athena to QuickSight—enables the tool attribution analysis that AgentSHAP research describes. This transforms compliance from “we logged the call” to “we can prove the tool influenced the outcome.”

For merchant banking application intake, this architecture addresses both the business problem (cost, abandonment, data quality) and the regulatory requirement (auditable compliance). The patterns described here—tool organization, guardrail configuration, logging design, and attribution analysis—apply to any domain requiring explainable AI agents.

References:

  • Horovicz, M. (2025). AgentSHAP: Interpreting LLM Agent Tool Importance with Monte Carlo Shapley Value Estimation. arXiv:2512.12597
  • TokenSHAP Library: github.com/GenAISHAP/TokenSHAP