Monitoring agentic tool performance, compliance attribution, and operational insights
The Research That Sparked This Architecture
A recent paper caught my attention: “AgentSHAP: Interpreting LLM Agent Tool Importance with Monte Carlo Shapley Value Estimation” (arXiv:2512.12597). It addresses a blind spot in AI agent development that most teams overlook until regulators ask questions.
The problem: AI agents call tools, logs show the calls, but nobody can prove which tools actually influenced the response. The paper applies Shapley values from cooperative game theory—running agents with different tool subsets, measuring how responses change, and computing fair attribution scores.
For consumer applications, this is interesting. For regulated financial services, this is essential.
When an examiner asks “How do you know your OFAC screening actually influenced this onboarding decision?”, the answer can’t be “we logged the API call.” You need provable attribution.
This got me thinking about a use case where this architecture would deliver immediate value: merchant banking application intake.
The Merchant Application Problem: A Story Repeated Daily
Let me paint the picture of what happens thousands of times daily at merchant acquiring banks.
9:15 AM — Maria owns a food truck. She visits a bank’s website looking for payment processing. She finds a “Request a Callback” form.
9:18 AM — Maria fills out her name, phone number, and business type. She submits and sees “A specialist will contact you within 24 hours.”
Next day, 2:30 PM — A sales rep calls Maria. She’s in the middle of the lunch rush. “Can you call back in an hour?”
3:45 PM — The rep calls back. Maria is now available. The rep starts collecting information:
- Legal business name
- DBA name
- Tax ID (Maria has to look this up)
- Business address
- Years in operation
- Estimated monthly volume
- Average transaction size
4:10 PM — Now they move to owner verification:
- Maria’s full legal name
- Social Security Number
- Date of birth
- Home address
- Ownership percentage
4:25 PM — The rep explains Maria needs to provide:
- Copy of driver’s license
- Voided check or bank letter
- Business license
- Articles of incorporation (Maria doesn’t know where this is)
4:30 PM — Maria says she’ll email the documents later. The call ends.
Three days later — Maria hasn’t sent the documents. Life got busy. A follow-up email sits in her inbox.
One week later — Another follow-up. Maria finally gathers documents, emails them as attachments.
Day 10 — Underwriting reviews the application. The EIN doesn’t match the business name in IRS records. The application goes to a hold queue.
Day 12 — A different rep calls Maria to resolve the discrepancy. It was a typo in the original call.
Day 15 — Application finally approved. Equipment ships.
Total elapsed time: 15 days Total bank cost: Approximately $200 (multiple rep calls, data entry, follow-ups, underwriting rework) Maria’s experience: Frustrating
Now multiply this by thousands of applications monthly.
The Numbers Behind the Pain
Industry experience suggests these ranges are common:
| Metric | Typical Range | What’s Actually Happening |
|---|---|---|
| 60-70% abandonment | Merchants start applications but never finish. Forms are complex. They don’t have their EIN handy. They get interrupted. They find a competitor who onboards in 5 minutes. | |
| $150-300 internal cost per application | The bank’s cost: sales rep time, system usage, follow-up calls, data entry labor, underwriting review, error correction. | |
| 15-20% rework rate | Applications delayed due to: TIN/business name mismatch with IRS, unverifiable addresses, failed identity verification, incorrect MCC classification, missing required documents. | |
| 5-10 business days | From first contact to underwriting decision. Fintechs do this in minutes. |
The data quality issues deserve specific attention:
- EIN doesn’t match business name: IRS TIN matching returns “no match” because of typos, abbreviations, or using DBA instead of legal name
- Address validation fails: USPS can’t standardize the address, or it’s flagged as non-deliverable
- Identity verification fails: Owner’s SSN doesn’t match name/DOB in credit bureau records
- Business not found: State Secretary of State has no record of the entity
- MCC misclassification: Business described incorrectly, assigned wrong category, triggers incorrect risk treatment
- Missing documents: Required documents for that business type or state weren’t collected
Every one of these issues is discoverable in real-time during the application conversation. Instead, they’re discovered days later in underwriting.
Why an AI Agent Fits This Problem
Merchant application intake isn’t a simple form—it’s a branching conversation:
┌─────────────────────┐
│ What type of │
│ business? │
└──────────┬──────────┘
│
┌──────────────────────┼──────────────────────┐
▼ ▼ ▼
Restaurant E-commerce Professional
│ │ Services
▼ ▼ ▼
Recommend POS Recommend Recommend
terminal + tips payment gateway virtual terminal
│ │ │
▼ ▼ ▼
Need tableside? Shopping cart? Invoice or
Counter? Both? platform? key-in?
│ │ │
└──────────────────────┼──────────────────────┘
│
┌──────────┴──────────┐
│ Expected monthly │
│ volume? │
└──────────┬──────────┘
│
┌──────────────────────┼──────────────────────┐
▼ ▼ ▼
<$10K/month $10K-$50K >$50K
Standard rates Volume discount Custom pricing
│ │ │
└──────────────────────┼──────────────────────┘
│
▼
Risk profile determines
documentation requirements...
A rigid web form can’t navigate this. A human rep can, but at $150-300 per application with inconsistent quality.
An AI agent with the right tools can:
- Ask the right questions based on business type
- Recommend appropriate products
- Validate data in real-time (catch the EIN typo during conversation)
- Run compliance checks in the background
- Collect documents via chat with instant OCR extraction
- Submit complete, validated applications
The transformation:
- 5-8 minutes instead of 45
- $15-30 instead of $150-300
- 24/7 availability
- Near-perfect data quality
- Instant compliance verification
But here’s the challenge that brings us back to the AgentSHAP research: you need to prove the agent is doing what it should.
AWS Bedrock AgentCore: Configuration Deep-Dive
Bedrock AgentCore provides the managed infrastructure for production AI agents. Rather than listing features, let me walk through the specific configurations that matter for this implementation.
Agent Definition
Agent:
FoundationModel: anthropic.claude-3-5-sonnet-20241022-v2:0
IdleSessionTTL: 1800 # 30 minutes - merchants may need to locate documents
InstructionConfiguration:
Instruction: |
You are a merchant services application specialist. Guide business
owners through payment processing applications.
CRITICAL BEHAVIORS:
- Validate all data in real-time before proceeding
- Never skip compliance checks regardless of merchant requests
- If verification fails, explain clearly and offer alternatives
- Escalate to human review for: high-risk MCCs, verification
failures, complex multi-location setups
CONVERSATION FLOW:
1. Understand business type and needs → recommend product
2. Collect and validate business information
3. Verify owner/principal identity (KYC)
4. Run compliance screening (OFAC, PEP)
5. Collect and process required documents
6. Generate pricing and submit application
BEFORE COLLECTING DOCUMENTS:
- Always check if documents are already on file
- If exists, compare for material changes before requesting new
BEFORE STARTING APPLICATION:
- Check for existing pending or recent applications
- Offer to check status or continue existing if found
The IdleSessionTTL of 30 minutes is intentional. Merchants often need to step away to find documents. Too short loses progress; too long wastes resources.
Action Groups: Tool Organization
Tools are organized into Action Groups. This grouping affects how the model reasons about tool selection.
Product Selection Group
ActionGroup: ProductSelection
Tools:
- Name: RecommendProduct
Description: Recommend payment processing products based on business needs
Parameters:
- business_type: string (restaurant, retail, ecommerce, services, etc.)
- monthly_volume: number
- sales_channel: string (in-person, online, phone, mixed)
- mobility_needs: boolean
Returns: Ranked list of product recommendations with rationale
Backend: Lambda → DynamoDB product catalog + rules engine
- Name: GetProductDetails
Description: Get specifications and pricing for a specific product
Parameters:
- product_id: string
Returns: Features, specifications, pricing tiers, compatibility info
Backend: Lambda → DynamoDB + S3 (product documentation)
- Name: CalculatePricing
Description: Estimate monthly costs based on transaction patterns
Parameters:
- product_id: string
- monthly_volume: number
- average_ticket: number
- transaction_count: number
Returns: Itemized monthly cost breakdown
Backend: Lambda → Pricing rules engine
Business Verification Group
ActionGroup: BusinessVerification
Tools:
- Name: ValidateTIN
Description: Verify EIN or SSN format and match against business name
Parameters:
- tin_type: string (EIN, SSN)
- tin_value: string
- legal_business_name: string
Returns: Validity status, match result, suggested corrections
Backend: Lambda → IRS TIN Matching API
- Name: VerifyBusinessRegistration
Description: Check business entity registration with state
Parameters:
- business_name: string
- state: string
- entity_type: string (LLC, Corp, Sole Prop, etc.)
Returns: Registration status, formation date, standing, registered agent
Backend: Lambda → State Secretary of State APIs (varies by state)
- Name: LookupMCC
Description: Determine Merchant Category Code based on business description
Parameters:
- business_description: string
- products_services: string
- sales_channel: string
Returns: Recommended MCC, description, risk tier, required documentation
Backend: Lambda → MCC classification model
- Name: ValidateAddress
Description: Standardize and verify business address
Parameters:
- street: string
- city: string
- state: string
- zip: string
Returns: Standardized address, deliverability score, corrections applied
Backend: Lambda → USPS Address Validation API
KYC and Compliance Group
This is where regulatory requirements live. KYC (Know Your Customer) typically integrates with third-party identity verification services.
ActionGroup: KYCCompliance
Tools:
- Name: VerifyIdentity
Description: KYC identity verification for business owner/principal
Parameters:
- full_name: string
- ssn: string
- date_of_birth: string
- address: object
Returns: Verification status, confidence score, flags, verification source
Backend: Lambda → Third-party KYC provider (Jumio, Onfido, Socure, etc.)
Note: Returns verification result, not raw identity data
- Name: ScreenOFAC
Description: Screen individual and business against OFAC sanctions lists
Parameters:
- individual_name: string
- business_name: string
- country: string
Returns: Match found (boolean), match details, lists checked, screening ID
Backend: Lambda → Treasury OFAC API or sanctions screening service
Compliance: Federal regulatory requirement - cannot be skipped
- Name: CheckPEP
Description: Screen for Politically Exposed Persons
Parameters:
- full_name: string
- date_of_birth: string
- country: string
Returns: PEP status, position details, risk level
Backend: Lambda → PEP screening service
- Name: AssessRisk
Description: Calculate preliminary risk score for underwriting
Parameters:
- mcc: string
- monthly_volume: number
- business_age_years: number
- owner_credit_indicator: string
Returns: Risk tier, required documentation, manual review flag
Backend: Lambda → Risk scoring model
Document Group
Document handling requires more sophistication than simple upload and OCR. The agent should check existing documents first.
ActionGroup: Documents
Tools:
- Name: CheckExistingDocuments
Description: Check if required document is already on file for this entity
Parameters:
- application_id: string
- document_type: string (drivers_license, voided_check, articles, etc.)
Returns: Exists (boolean), document_id, upload_date, expiration_status
Backend: Lambda → DynamoDB document registry
Purpose: Avoid re-collecting documents already on file
- Name: CompareDocuments
Description: Compare newly uploaded document against existing version
Parameters:
- application_id: string
- document_type: string
- new_document_s3_uri: string
Returns: Changes detected, change summary, recommendation (use_new/keep_existing)
Backend: Lambda → Textract + comparison logic
Purpose: Determine if new document has material changes worth updating
- Name: ProcessDocument
Description: Extract structured data from uploaded document image
Parameters:
- document_s3_uri: string
- document_type: string
- expected_fields: array
Returns: Extracted fields, confidence scores, validation status
Backend: Lambda → Amazon Textract (AnalyzeDocument API)
- Name: ValidateDocumentData
Description: Compare extracted document data against application data
Parameters:
- extracted_fields: object
- application_data: object
Returns: Match status, discrepancies found, fields requiring confirmation
Backend: Lambda → Field matching logic
Application Management Group
Application handling must account for existing applications to prevent duplicates and enable session recovery.
ActionGroup: ApplicationManagement
Tools:
- Name: CheckExistingApplications
Description: Find any existing applications for this business/owner
Parameters:
- tin: string
- owner_ssn_last4: string
- business_name: string
Returns: Applications found (array), status of each, recommendation
Backend: Lambda → DynamoDB applications table
Purpose: Detect pending, approved, or recently declined applications
- Name: SaveApplicationProgress
Description: Persist current application state for session recovery
Parameters:
- application_id: string
- current_phase: string
- collected_data: object
Returns: Save confirmation, resume_token
Backend: Lambda → DynamoDB
- Name: ResumeApplication
Description: Retrieve saved application state
Parameters:
- resume_token: string
# OR
- tin: string
- owner_verification: object # Light verification to resume
Returns: Application state, next required fields, conversation context
Backend: Lambda → DynamoDB
- Name: SubmitApplication
Description: Submit completed application to underwriting queue
Parameters:
- application_id: string
Returns: Submission confirmation, reference number, expected timeline
Backend: Lambda → SQS (underwriting queue) + DynamoDB status update
Prerequisites: All required fields validated, compliance checks passed
- Name: ScheduleCallback
Description: Escalate to human representative
Parameters:
- application_id: string
- reason: string
- preferred_time: string
- phone_number: string
Returns: Callback scheduled confirmation, reference number
Backend: Lambda → CRM integration or callback queue
Bedrock Guardrails: Safety Configuration
Guardrails provide declarative safety controls—critical for financial services.
Guardrail:
Name: MerchantIntakeGuardrail
ContentFilters:
- Type: HATE
Strength: HIGH
- Type: VIOLENCE
Strength: HIGH
- Type: SEXUAL
Strength: HIGH
- Type: MISCONDUCT
Strength: HIGH
SensitiveInformationPolicy:
PIIEntities:
- Type: SSN
Action: ANONYMIZE # Replace with [SSN] in all responses
- Type: CREDIT_CARD_NUMBER
Action: BLOCK # Never include under any circumstances
- Type: BANK_ACCOUNT_NUMBER
Action: ANONYMIZE
- Type: DRIVER_LICENSE
Action: ANONYMIZE
RegexPatterns:
- Name: EIN_PATTERN
Pattern: '\b\d{2}-\d{7}\b'
Action: ANONYMIZE
- Name: ROUTING_NUMBER
Pattern: '\b\d{9}\b'
Action: ANONYMIZE
TopicPolicy:
DeniedTopics:
- Name: COMPETITOR_DISCUSSION
Definition: "Discussion of competitor products, pricing, or recommendations"
Examples:
- "What about Square or Stripe?"
- "Is PayPal better?"
Action: BLOCK
- Name: INVESTMENT_ADVICE
Definition: "Financial planning or investment recommendations"
Action: BLOCK
- Name: UNRELATED_BANKING
Definition: "Personal loans, mortgages, or products unrelated to merchant services"
Action: BLOCK
WordPolicy:
ManagedWordLists:
- Type: PROFANITY
Action: BLOCK
CustomWordLists:
- Words:
- "guaranteed approval"
- "instant approval"
- "no credit check required"
Action: BLOCK # These phrases create compliance risk
The SensitiveInformationPolicy is critical. The agent collects SSNs, EINs, and bank account numbers during the application. These must never appear in agent responses—not even when confirming information back to the merchant. Guardrails enforce this at the platform level.
Knowledge Base Configuration
The Knowledge Base provides RAG-powered retrieval for product information, pricing rules, and compliance requirements.
KnowledgeBase:
Name: MerchantServicesKB
DataSource:
Type: S3
BucketArn: arn:aws:s3:::merchant-kb-documents
InclusionPrefixes:
- "products/" # Product specifications and features
- "pricing/" # Pricing tiers and rules
- "compliance/" # State-specific requirements
- "mcc-codes/" # MCC descriptions and risk mappings
- "faqs/" # Common questions and answers
- "prohibited-businesses/" # Industries not serviced
ChunkingConfiguration:
Strategy: HIERARCHICAL
ParentChunkSize: 1500 # Broader context
ChildChunkSize: 300 # Precise retrieval
OverlapPercentage: 20
EmbeddingModel: amazon.titan-embed-text-v2
VectorStore:
Type: OPENSEARCH_SERVERLESS
CollectionArn: arn:aws:aoss:us-east-1:xxx:collection/merchant-kb
RetrievalConfiguration:
NumberOfResults: 5
SearchType: HYBRID # Combines semantic and keyword search
The HIERARCHICAL chunking strategy matters. Parent chunks provide context (full product description), while child chunks enable precise retrieval (specific feature or pricing detail).
The Observability Architecture: Where AgentSHAP Meets Production
Now we arrive at the monitoring architecture that makes tool attribution practical.
Design Principles
- Log everything, but hash PII — Complete tool invocation records without compliance risk
- Capture context — What prompted the tool call, not just that it happened
- Enable attribution analysis — Structure data for Shapley-style importance computation
- Real-time and historical — CloudWatch for immediate visibility, Athena/QuickSight for analysis
Tool Invocation Logging
Every tool call generates a structured log entry:
{
"timestamp": "2025-01-13T14:32:18.445Z",
"trace_id": "1-65a2b3c4-abc123def456",
"session_id": "ses_abc123def456",
"application_id": "APP-2025-00892",
"conversation_turn": 12,
"phase": "kyc_verification",
"tool_invocation": {
"action_group": "KYCCompliance",
"tool_name": "ScreenOFAC",
"invocation_id": "inv_xyz789",
"input_hash": "sha256:a1b2c3d4e5f6...",
"input_field_names": ["individual_name", "business_name", "country"],
"execution_start_ms": 1705156338445,
"execution_end_ms": 1705156339337,
"execution_duration_ms": 892,
"output_summary": {
"match_found": false,
"confidence": 1.0,
"lists_checked": ["SDN", "CONS", "PLC"],
"screening_id": "scr_abc123"
},
"output_hash": "sha256:f6e5d4c3b2a1..."
},
"context": {
"preceding_user_message_type": "ssn_provided",
"agent_reasoning": "User provided owner SSN, initiating compliance screening",
"tools_called_this_turn": ["VerifyIdentity", "ScreenOFAC", "CheckPEP"],
"tool_sequence_position": 2
},
"application_state": {
"fields_collected": 18,
"fields_remaining": 4,
"compliance_checks_completed": ["tin_validation", "business_registration"],
"documents_collected": ["drivers_license"]
}
}
Key design decisions:
input_hashinstead of actual input — PII never lands in analyticscontext.agent_reasoning— Captures why the tool was called (crucial for attribution)tool_sequence_position— Enables analysis of tool ordering effectsapplication_state— Allows correlation with application outcomes
Analytics Pipeline Architecture
┌─────────────────────────────────────────────────────────────────────────────────┐
│ OBSERVABILITY PIPELINE │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ BEDROCK AGENT │
│ │ │
│ ▼ │
│ CloudWatch Logs (/aws/bedrock/merchant-intake-agent) │
│ │ │
│ ├──────────────────────────────────┐ │
│ │ │ │
│ ▼ ▼ │
│ CloudWatch Metrics Subscription Filter │
│ (Real-time dashboards) │ │
│ │ ▼ │
│ │ Kinesis Firehose │
│ │ │ │
│ │ ▼ │
│ │ Lambda Transform │
│ │ (Enrich + Parquet) │
│ │ │ │
│ │ ▼ │
│ │ S3 Data Lake │
│ │ (Partitioned by date) │
│ │ │ │
│ │ ▼ │
│ │ Glue Data Catalog │
│ │ │ │
│ │ ▼ │
│ │ Athena │
│ │ │ │
│ └──────────────────────────────────┼──────────────────────────────────┐ │
│ ▼ ▼ │
│ QuickSight EventBridge│
│ (Dashboards) (Alerts) │
│ │
└─────────────────────────────────────────────────────────────────────────────────┘
Kinesis Firehose Configuration
DeliveryStream:
Name: merchant-intake-tool-logs
Source:
Type: CloudWatchLogsSubscription
LogGroup: /aws/bedrock/merchant-intake-agent
FilterPattern: "{ $.tool_invocation.tool_name = * }"
Processing:
Processors:
- Type: Lambda
LambdaArn: arn:aws:lambda:us-east-1:xxx:function:EnrichToolLogs
BufferSizeInMBs: 1
BufferIntervalInSeconds: 60
Destination:
Type: S3
BucketArn: arn:aws:s3:::merchant-analytics-data-lake
Prefix: "tool-logs/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/"
ErrorOutputPrefix: "errors/"
BufferingHints:
SizeInMBs: 64
IntervalInSeconds: 300
DataFormatConversion:
Enabled: true
InputFormat: JSON
OutputFormat: PARQUET
SchemaConfiguration:
DatabaseName: merchant_analytics
TableName: tool_invocations
Region: us-east-1
CatalogId: !Ref AWS::AccountId
The enrichment Lambda adds application outcomes when known (joins with DynamoDB) and computes derived metrics.
Athena Table Definition
CREATE EXTERNAL TABLE merchant_analytics.tool_invocations (
timestamp timestamp,
trace_id string,
session_id string,
application_id string,
conversation_turn int,
phase string,
action_group string,
tool_name string,
invocation_id string,
execution_duration_ms int,
input_hash string,
output_hash string,
output_summary map<string, string>,
preceding_message_type string,
agent_reasoning string,
tool_sequence_position int,
-- Enriched fields (added by Lambda)
application_outcome string,
time_to_completion_hours decimal(10,2),
data_quality_score decimal(3,2)
)
PARTITIONED BY (year string, month string, day string)
STORED AS PARQUET
LOCATION 's3://merchant-analytics-data-lake/tool-logs/'
TBLPROPERTIES ('parquet.compression'='SNAPPY');
Analytics Queries for Tool Attribution
Query 1: Tool Importance by Phase
-- Which tools are called in each phase, and how do they correlate with success?
SELECT
phase,
tool_name,
COUNT(*) as invocation_count,
COUNT(DISTINCT application_id) as applications_using_tool,
-- Success rate when this tool is used in this phase
AVG(CASE WHEN application_outcome = 'APPROVED' THEN 1.0 ELSE 0.0 END) as approval_rate,
-- Average execution time
AVG(execution_duration_ms) as avg_latency_ms,
APPROX_PERCENTILE(execution_duration_ms, 0.95) as p95_latency_ms
FROM merchant_analytics.tool_invocations
WHERE year = '2025' AND month = '01'
AND application_outcome IS NOT NULL
GROUP BY phase, tool_name
ORDER BY phase, invocation_count DESC;
Query 2: Compliance Tool Attribution
-- Critical: Are compliance tools showing appropriate importance?
-- Alert if OFAC or KYC tools have low correlation with outcomes
WITH compliance_tools AS (
SELECT
application_id,
MAX(CASE WHEN tool_name = 'ScreenOFAC' THEN 1 ELSE 0 END) as ofac_called,
MAX(CASE WHEN tool_name = 'VerifyIdentity' THEN 1 ELSE 0 END) as kyc_called,
MAX(CASE WHEN tool_name = 'CheckPEP' THEN 1 ELSE 0 END) as pep_called
FROM merchant_analytics.tool_invocations
WHERE phase = 'kyc_verification'
AND year = '2025' AND month = '01'
GROUP BY application_id
),
outcomes AS (
SELECT DISTINCT application_id, application_outcome
FROM merchant_analytics.tool_invocations
WHERE application_outcome IS NOT NULL
)
SELECT
'ScreenOFAC' as tool,
SUM(ofac_called) as times_called,
COUNT(*) as total_applications,
CAST(SUM(ofac_called) AS DECIMAL(10,2)) / COUNT(*) as call_rate,
-- This should be very close to 1.0 - if not, investigate
CASE
WHEN CAST(SUM(ofac_called) AS DECIMAL(10,2)) / COUNT(*) < 0.95
THEN 'ALERT: OFAC not called in >5% of applications'
ELSE 'OK'
END as status
FROM compliance_tools c
JOIN outcomes o ON c.application_id = o.application_id
UNION ALL
SELECT
'VerifyIdentity' as tool,
SUM(kyc_called),
COUNT(*),
CAST(SUM(kyc_called) AS DECIMAL(10,2)) / COUNT(*),
CASE
WHEN CAST(SUM(kyc_called) AS DECIMAL(10,2)) / COUNT(*) < 0.95
THEN 'ALERT: KYC not called in >5% of applications'
ELSE 'OK'
END
FROM compliance_tools c
JOIN outcomes o ON c.application_id = o.application_id;
Query 3: Tool Latency Impact on Abandonment
-- Do slow tools cause abandonment?
WITH session_metrics AS (
SELECT
session_id,
application_id,
MAX(conversation_turn) as total_turns,
SUM(execution_duration_ms) as total_tool_latency_ms,
AVG(execution_duration_ms) as avg_tool_latency_ms,
MAX(application_outcome) as outcome
FROM merchant_analytics.tool_invocations
WHERE year = '2025' AND month = '01'
GROUP BY session_id, application_id
)
SELECT
CASE
WHEN avg_tool_latency_ms < 500 THEN '0-500ms'
WHEN avg_tool_latency_ms < 1000 THEN '500-1000ms'
WHEN avg_tool_latency_ms < 2000 THEN '1000-2000ms'
ELSE '>2000ms'
END as latency_bucket,
COUNT(*) as sessions,
AVG(CASE WHEN outcome IS NULL THEN 1.0 ELSE 0.0 END) as abandonment_rate,
AVG(total_turns) as avg_conversation_length
FROM session_metrics
GROUP BY 1
ORDER BY 1;
Query 4: AgentSHAP-Style Importance Estimation
This query approximates Shapley-style importance by measuring outcome correlation:
-- Simplified importance scoring
-- Full Shapley computation runs as separate batch job
WITH tool_presence AS (
SELECT
application_id,
tool_name,
1 as tool_used
FROM merchant_analytics.tool_invocations
WHERE year = '2025' AND month = '01'
GROUP BY application_id, tool_name
),
all_apps AS (
SELECT DISTINCT application_id, application_outcome
FROM merchant_analytics.tool_invocations
WHERE application_outcome IS NOT NULL
AND year = '2025' AND month = '01'
),
tool_outcomes AS (
SELECT
t.tool_name,
a.application_id,
a.application_outcome,
COALESCE(tp.tool_used, 0) as tool_used
FROM all_apps a
CROSS JOIN (SELECT DISTINCT tool_name FROM tool_presence) t
LEFT JOIN tool_presence tp
ON a.application_id = tp.application_id
AND t.tool_name = tp.tool_name
)
SELECT
tool_name,
-- Approval rate when tool IS used
AVG(CASE WHEN tool_used = 1 AND application_outcome = 'APPROVED' THEN 1.0
WHEN tool_used = 1 THEN 0.0
ELSE NULL END) as approval_rate_with_tool,
-- Approval rate when tool is NOT used
AVG(CASE WHEN tool_used = 0 AND application_outcome = 'APPROVED' THEN 1.0
WHEN tool_used = 0 THEN 0.0
ELSE NULL END) as approval_rate_without_tool,
-- Importance proxy: difference in approval rates
AVG(CASE WHEN tool_used = 1 AND application_outcome = 'APPROVED' THEN 1.0
WHEN tool_used = 1 THEN 0.0
ELSE NULL END) -
AVG(CASE WHEN tool_used = 0 AND application_outcome = 'APPROVED' THEN 1.0
WHEN tool_used = 0 THEN 0.0
ELSE NULL END) as importance_proxy
FROM tool_outcomes
GROUP BY tool_name
HAVING COUNT(CASE WHEN tool_used = 1 THEN 1 END) > 10 -- Minimum sample size
ORDER BY importance_proxy DESC;
QuickSight Dashboard Design
Dashboard 1: Application Funnel
| Visual | Type | Metrics |
|---|---|---|
| Applications Over Time | Line chart | Started, Completed, Approved by day |
| Abandonment by Phase | Funnel | Drop-off count and rate per phase |
| Completion Time Distribution | Histogram | Minutes to complete (p50, p90) |
| Abandonment Reasons | Pie chart | Phase where session ended |
Dashboard 2: Tool Performance
| Visual | Type | Metrics |
|---|---|---|
| Tool Invocation Heatmap | Heatmap | Tools × Phases, colored by frequency |
| Latency by Tool | Box plot | p50, p75, p95, p99 per tool |
| Error Rate Trend | Line chart | Tool errors over time |
| Slow Tool Alert | KPI | Tools exceeding latency threshold |
Dashboard 3: Compliance Monitoring
| Visual | Type | Metrics |
|---|---|---|
| OFAC Completion Rate | Gauge | Should be 100% |
| KYC Verification Outcomes | Stacked bar | Verified, Failed, Manual Review |
| Compliance Check Coverage | Table | Applications missing required checks |
| Risk Tier Distribution | Pie chart | Low, Medium, High risk applications |
Dashboard 4: Tool Attribution (AgentSHAP-Inspired)
| Visual | Type | Metrics |
|---|---|---|
| Tool Importance by Phase | Heatmap | Tools × Phases, colored by importance score |
| Importance Trend | Line chart | Tool importance scores over time |
| Attribution Alerts | Table | Tools with unexpected importance patterns |
| Optimization Candidates | Ranked list | Low-importance, high-frequency tools |
EventBridge Alerting
Rule:
Name: ComplianceToolAlert
Description: Alert when compliance tools show low usage or importance
EventPattern:
source:
- "custom.merchant-analytics"
detail-type:
- "ComplianceMetricBreach"
detail:
metric_name:
- "ofac_call_rate"
- "kyc_call_rate"
- "compliance_tool_importance"
breach_type:
- "below_threshold"
Targets:
- Id: SNSAlert
Arn: arn:aws:sns:us-east-1:xxx:compliance-alerts
- Id: PagerDuty
Arn: arn:aws:events:us-east-1:xxx:api-destination/pagerduty
A Lambda function runs hourly, queries Athena for compliance metrics, and publishes events when thresholds are breached.
Conversation Patterns: Handling Real-World Complexity
The agent must handle edge cases gracefully. Here’s how tool attribution helps verify correct behavior.
Scenario 1: Existing Application Found
Merchant: "I'd like to apply for a payment terminal"
Agent: [Calls CheckExistingApplications]
[Returns: 1 pending application from 5 days ago]
Agent: "I found an existing application for Tasty Tacos LLC that you
submitted on January 8th. It's currently in underwriting review.
Would you like me to:
• Check the status of that application, or
• Start a new application for a different location or product?"
Attribution Check: CheckExistingApplications should show HIGH importance early in conversation. If merchants frequently proceed to create duplicates, either the tool isn’t being called or its output isn’t influencing the agent.
Scenario 2: Document Already on File
Agent: "I'll need a copy of your driver's license for identity verification."
Merchant: [Uploads driver's license image]
Agent: [Calls CheckExistingDocuments → found existing from prior app]
[Calls CompareDocuments → detects address change]
Agent: "I see we have a driver's license on file from your previous
application, but the address is different. Your new license
shows 456 Oak Street instead of 123 Main Street.
Should I update your business address to match?"
Attribution Check: CheckExistingDocuments and CompareDocuments should both show importance during document collection. If documents are being re-collected unnecessarily, these tools may not be invoked.
Scenario 3: KYC Verification Fails
Agent: [Calls VerifyIdentity → returns FAILED, confidence 0.3]
Agent: "I wasn't able to verify your identity with the information
provided. This sometimes happens when:
• The Social Security Number has a typo
• Your name doesn't match exactly (maiden name, suffix, etc.)
• Your address doesn't match credit bureau records
Would you like to:
• Re-enter your information, or
• Speak with a specialist who can help verify manually?
I can schedule a callback if that's easier."
Attribution Check: When VerifyIdentity fails, ScheduleCallback should show increased importance (escalation path). If applications with failed KYC proceed without escalation, there’s a compliance gap.
Implementation Considerations
Cold Start Latency
Lambda functions backing Action Groups experience cold starts. For merchant-facing tools where latency matters:
- Configure provisioned concurrency for
VerifyIdentity,ScreenOFAC,ValidateTIN - Accept cold starts for less time-sensitive tools like
SaveApplicationProgress - Consider Lambda SnapStart for Java-based tools
Third-Party API Resilience
External APIs (IRS TIN matching, state SOS, KYC providers) fail. Implement:
- Circuit breakers (AWS App Config feature flags)
- Retry with exponential backoff
- Graceful degradation: set
manual_review_requiredflag and proceed - Fallback providers where available
Session Recovery
Merchants abandon sessions to find documents. Enable recovery:
- Call
SaveApplicationProgressafter each phase completion - Generate resume tokens with reasonable expiration (7 days)
- Allow light re-authentication (TIN + owner DOB) to resume
- Send email/SMS with resume link if contact info collected
Compliance Audit Trail
For regulatory examination, maintain immutable records:
- Every compliance tool invocation with full request/response
- Input parameters (hashed for PII, but recoverable with key)
- Timestamp, duration, outcome
- Agent’s interpretation and subsequent action
- Store in separate compliance log with extended retention
Architecture Summary
┌─────────────────────────────────────────────────────────────────────────────────┐
│ MERCHANT INTAKE AGENT - COMPLETE ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ PRESENTATION │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ CloudFront ──▶ S3 (React SPA) ──▶ API Gateway (WebSocket) │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ AGENT LAYER │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ AWS BEDROCK AGENTCORE │ │
│ │ ┌───────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Agent: Claude 3.5 Sonnet │ │ │
│ │ │ Knowledge Base: S3 → Titan Embeddings → OpenSearch Serverless │ │ │
│ │ │ Guardrails: PII filtering, topic blocking, word policies │ │ │
│ │ │ Session Memory: DynamoDB-backed │ │ │
│ │ └───────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ┌───────────────────────────────┼───────────────────────────────────┐ │ │
│ │ │ ACTION GROUPS │ │ │
│ │ │ │ │ │
│ │ │ ProductSelection │ BusinessVerify │ KYCCompliance │ Documents │ │ │
│ │ │ ApplicationMgmt │ │ │ │
│ │ │ │ │ │
│ │ └───────────────────────────────┼───────────────────────────────────┘ │ │
│ └──────────────────────────────────┼──────────────────────────────────────┘ │
│ │ │
│ COMPUTE LAYER ▼ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ AWS LAMBDA FUNCTIONS │ │
│ │ (Tool implementations + integrations) │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │ │ │ │
│ DATA LAYER ▼ ▼ ▼ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ DynamoDB S3 Secrets Manager External APIs │ │
│ │ • Sessions • Documents • API credentials • IRS TIN │ │
│ │ • Applications • KB source • KYC provider • State SOS │ │
│ │ • Tool logs • Analytics • OFAC service • KYC/IDV │ │
│ │ • OFAC │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
│ │
│ OBSERVABILITY LAYER │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ CloudWatch ──▶ Kinesis Firehose ──▶ S3 (Parquet) ──▶ Athena │ │
│ │ │ │ │ │
│ │ │ ┌─────────────────────────────┤ │ │
│ │ │ │ │ │ │
│ │ ▼ ▼ ▼ │ │
│ │ CloudWatch QuickSight EventBridge │ │
│ │ Dashboards Dashboards Alerts │ │
│ │ (Real-time) (Analytics) (Compliance) │ │
│ │ │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────────┘
Measuring Success
Beyond standard operational metrics, apply AgentSHAP-inspired attribution:
Expected Tool Importance by Phase
| Phase | Expected HIGH Importance | Alert If LOW |
|---|---|---|
| Qualification | RecommendProduct | – |
| Business Info | ValidateTIN, VerifyBusinessRegistration | ValidateTIN |
| KYC/Compliance | VerifyIdentity, ScreenOFAC | Either one |
| Documents | CheckExistingDocuments, ProcessDocument | CheckExistingDocuments |
| Submission | CheckExistingApplications, SubmitApplication | CheckExistingApplications |
If OFAC screening shows low importance during KYC phase, investigate immediately. Either the tool isn’t being called, or its output isn’t influencing agent behavior. Both are compliance failures.
Optimization Candidates
After sufficient volume (1000+ applications), identify:
- High frequency, low importance: Cost reduction candidates
- High latency, high importance: Performance optimization priority
- Low frequency, high importance: Reliability critical—ensure redundancy
Conclusion
AWS Bedrock AgentCore provides the managed infrastructure for production AI agents. The combination of Action Groups, Knowledge Bases, and Guardrails creates a foundation suitable for regulated financial services.
The observability architecture—CloudWatch through Athena to QuickSight—enables the tool attribution analysis that AgentSHAP research describes. This transforms compliance from “we logged the call” to “we can prove the tool influenced the outcome.”
For merchant banking application intake, this architecture addresses both the business problem (cost, abandonment, data quality) and the regulatory requirement (auditable compliance). The patterns described here—tool organization, guardrail configuration, logging design, and attribution analysis—apply to any domain requiring explainable AI agents.
References:
- Horovicz, M. (2025). AgentSHAP: Interpreting LLM Agent Tool Importance with Monte Carlo Shapley Value Estimation. arXiv:2512.12597
- TokenSHAP Library: github.com/GenAISHAP/TokenSHAP