Understanding and managing bots, AI agents, and non-human traffic

Overview and how to use this document

This guide consolidates the practical, architectural, and forward-looking considerations required to manage bots across Adobe Analytics (AA), Adobe Experience Platform (AEP), and Customer Journey Analytics (CJA). It is designed for analytics leaders, practitioners, architects, developers, security teams, and data engineers.

Forward-looking statement

Roadmap content in this paper is provided for informational purposes only, is not a guarantee of any future feature or functionality, and is subject to change at any time. Roadmap capabilities must not be promised or committed contractually. Confirm availability, naming, timing, and entitlement with the Adobe account team before customer commitments.

How to use this document

Executive Summary

Digital analytics was built primarily to measure people. Modern digital properties must now account for people, conventional crawlers, operational automation, sophisticated fraud bots, AI crawlers, conversational systems, and autonomous agents. These actors differ materially in intent, technical behavior, business value, and risk.

The correct objective is therefore not to "remove all bots." It is to establish a governed decision system that determines which traffic should be allowed, challenged, blocked, excluded from human reporting, quarantined, or analyzed as a distinct machine-mediated journey.

Key conclusions

• Adobe Analytics provides strong baseline reporting controls through IAB bot filtering, custom bot rules, bot reporting, behavioral analysis, standardized segments, and Virtual Report Suites.
• Reporting exclusion and consumption reduction are different. A hit may be excluded from reporting after the server call has already been generated.
• Consumption reduction requires earlier action, such as WAF/CDN controls, application logic, tag suppression, Web SDK onBeforeEventSend logic, AppMeasurement s.abort, or datastream routing.
• AEP datastream bot detection creates a bot score on matching Edge events. The score is a classification signal and does not itself drop the event.
• Dynamic Datastream Configuration can route bot-scored events to a quarantine dataset, preserving recoverability while keeping that dataset outside a CJA connection when appropriate.
• CJA supports centralized reporting governance through Data Views and flexible classification through fields, datasets, segments, and derived logic.
• Data Distiller supports SQL thresholds and machine-learning workflows that classify traffic from behavior rather than relying only on static signatures.
• Roadmap themes such as Consent Navigator, Filtering in Connections, Conversation Insights, and MCP reflect a broader transition from bot removal to machine-traffic intelligence.

A five-layer operating model

The Modern Non-Human Traffic Landscape

2.1 Classification before exclusion

A binary human-versus-bot model is no longer sufficient. Classification should reflect business intent, technical behavior, and confidence. A useful baseline taxonomy follows.

2.2 The JavaScript divide

For client-side analytics, the most important technical distinction is whether automation executes JavaScript. Non-JavaScript crawlers may be invisible to client-side collection, even though they consume site infrastructure and matter for AI-discovery analysis. JavaScript-executing bots can trigger analytics tags, create sessions, enter experiments, qualify for audiences, and generate billable activity.

2.3 Signals are evidence, not proof

A user agent, IP range, cloud ASN, missing referrer, high velocity, unfamiliar geography, or single-visit behavior may be suspicious, but no single signal proves bot activity. Corroborate multiple indicators and compare them with known baselines, authentication state, approved integrations, and server/CDN evidence.

Business and Measurement Impact

3.1 The full cost picture

The business case should include more than analytics licensing. Consider infrastructure, paid media, experimentation velocity, analyst and engineering effort, storage and processing, fraud response, and the opportunity cost of decisions made from contaminated data.

3.2 Critical distinction: reporting quality versus consumption

Diagnose and Quantify Bot Exposure

4.1 Establish the factual baseline

1. Map collection paths: AppMeasurement, Web SDK, Mobile SDK, Edge Network API, server-side APIs, batch flows, and hybrids.
2. Inventory report suites, datastreams, datasets, CJA Connections, Data Views, tags, APIs, and owners.
3. Establish daily and weekly usage baselines by report suite, dataset, and Connection where available.
4. Collect a representative evidence window from analytics, CDN, WAF, application, authentication, and fraud systems.
5. Reconcile approved procurement, marketplace, accessibility, monitoring, QA, load-test, and internal automation sources.

4.2 Evidence package

• IP addresses, CIDRs, ASNs, user agents, request headers, paths, query strings, timestamps, rates, and authentication state.
• ECIDs or other approved pseudonymous identifiers, with privacy and governance review.
• Evidence that suspicious requests actually generate an Adobe collection request.
• Bot Name and Bot Page reporting, custom-rule matches, and clean-versus-unfiltered comparisons.
• WAF/CDN decisions, challenge outcomes, application errors, customer support cases, and integration-health indicators.

4.3 Investigative patterns

4.4 Validation plan

Before enforcing any block, create a testable hypothesis, capture baseline metrics, run the rule in observe-only or quarantine mode where possible, review false positives, verify customer and integration health, and define rollback criteria. Changes to AA bot rules or collection suppression generally affect future data; reporting segments and CJA Data Views can change analysis without rewriting source data.

Adobe Analytics: Native Controls and Governance

5.1 IAB bot filtering

Adobe Analytics report-suite Bot Rules can enable the IAB International Spiders & Bots List. Incoming traffic is compared against enabled bot rules; recognized bot traffic is separated from standard reporting and is available through bot-specific reporting. Adobe recommends enabling the IAB option at minimum for most customers.

5.2 Custom bot rules

Custom rules can match user agents, individual IP addresses, or IP ranges, and can be uploaded via CSV. Use them for confirmed signatures and assign an owner, evidence source, creation date, review date, and retirement criteria. Static rules require maintenance and can be evaded by user-agent spoofing and IP rotation.

5.3 Behavioral analysis and clean views

• Use Analysis Workspace to investigate high velocity, missing referrers, unusual devices, malformed paths, extreme navigation, and other anomalies.
• Use Data Warehouse exports to study ECIDs or visitor populations identified by behavioral segments.
• Where appropriate, standardize exclusion logic through approved segments or Virtual Report Suites so teams do not create conflicting definitions.
• Use annotations to explain known bot incidents and changes in filtering logic.

5.4 Limitations

• Known-list and custom-rule matching does not reliably identify sophisticated headless browsers or agents that mimic normal browser behavior.
• AA filtering occurs after the collection request has been generated, so reporting exclusion does not by itself reduce server-call consumption.
• Rule changes are generally prospective; historical comparability must be managed deliberately.
• Many non-JavaScript crawlers are invisible to client-side analytics and require CDN/server data for visibility.

Beyond Filtering: Preventing Server Calls at the Source

When evidence supports a high-confidence decision, collection-side prevention can stop unwanted events before they are sent. This can reduce analytics consumption but should be implemented narrowly because false positives may remove legitimate customer behavior and investigative evidence.

6.1 AEP Web SDK: onBeforeEventSend

For Web SDK implementations, returning false from onBeforeEventSend prevents the sendEvent command from generating a server call for the affected event.

if (iDontWantToCollectThisData === true) {
return false;
}

6.2 AppMeasurement: s.abort

For AppMeasurement implementations, set the abort variable to true before the request is sent. A common placement is doPlugins.

s.doPlugins = function() {
if (iDontWantToCollectThisData === true) {
s.abort = true;
}
};

6.3 Adobe Data Collection Tags rule conditions

A Custom Code condition can prevent a rule's actions from running when the condition returns false. This can suppress Web SDK or AppMeasurement actions, but the condition may need to be consistently applied to many rules. Centralized interception points are generally easier to govern than duplicated conditions.

6.4 User-agent identification

Browser-side JavaScript can inspect navigator.userAgent. An organization may suppress confirmed patterns, but user-agent matching alone is easy to evade and broad substring rules can create false positives.

const ua = navigator.userAgent || "";
if (ua.includes("Googlebot") || ua.includes("spider") || ua.includes("adbot")) {
return false;
}

6.5 Temporary multi-suite user-agent discovery experiment

1. Create a dedicated investigation report suite that will never be used as the production reporting source.
2. Copy relevant production settings. Keep IAB filtering enabled in production and disable it only in the investigation suite.
3. Temporarily capture navigator.userAgent into a dedicated eVar in both suites.

   content.data.__adobe.analytics.eVar10 = navigator.userAgent;

4. Temporarily multi-suite tag a controlled period, understanding that additional primary or secondary server calls may be generated.
5. Compare user-agent values. Values present only in the unfiltered suite are candidates for analysis, not automatic blocking.
6. Reverse the temporary collection change quickly, validate findings, and convert only high-confidence patterns into governed controls.

6.6 IP and CIDR-based prevention

Client-side JavaScript does not natively expose a visitor IP address in the same way it exposes the user agent. An IP-based client suppression design therefore requires server-side or third-party support, legal and privacy review, secure handling, and careful validation. CIDR notation can simplify large range lists.

window.ipAddressRanges = [
"128.0.0.0/26",
"8.8.4.0/24",
"8.8.8.0/24",
"174.162.149.228"
];

If the visitor address falls within an approved block range, the implementation can return false from onBeforeEventSend or set s.abort=true. Prefer server/CDN enforcement where possible because it operates earlier and avoids exposing networking logic to the browser.

Behavioral Rate Limiting: The Hit Governor Pattern

The Hit Governor is a community/consulting pattern for Web SDK and browser-based Tags implementations. It identifies a device that sends at least X Adobe events within Y seconds and suppresses additional events for Z days. The default example uses a 60-hit, 60-second, 60-day threshold, but thresholds must be calibrated to the customer's legitimate behavior.

7.1 Why it is useful

• Detects spam-like velocity without requiring a known bot identity.
• Can catch runaway implementations, repeated scraping, and unknown automated behavior.
• Uses browser behavior as a signal rather than relying only on user agent or IP.

7.2 Architecture

1. Load a global hitGovernor function at Tags library load.
2. Call the function after Web SDK sendEvent completion so the local counter is updated.
3. Store the counter and exclusion flag in localStorage.
4. In onBeforeEventSend, return false when the active flag indicates the device is suppressed.

7.3 Configuration example

hitGovernor(); // default: 60 hits / 60 seconds / 60 days
hitGovernor(100, 80, 30); // example: 100 hits / 80 seconds / 30 days

if (localStorage.getItem("hg_flag") === "1") {
return false;
}

7.4 Limitations and safeguards

• Web/JavaScript only; the described pattern does not apply directly to Mobile SDK.
• localStorage is scoped by origin, so a flag does not automatically follow the same device across other subdomains.
• Users can clear storage, and privacy or browser settings may limit persistence.
• A threshold that is too low can suppress legitimate rapid interactions, accessibility tools, testing, kiosk use, or application bursts.
• Treat it as a rate-limiting safety control, not definitive bot identity proof.

CJA and AEP: Classify, Route, Govern, and Analyze

8.1 Edge datastream bot detection

AEP datastream bot detection can use the standard IAB list plus custom IP, IP-range, and request-header rules. When a request matches, the XDM payload is enriched with botDetection.score = 1 and the event continues through the configured datastream services.

{
"botDetection": {
"score": 1
}
}

• Add the Bot Detection Information field group to the schema.
• Detection applies to unauthenticated requests to edge.adobedc.net; authenticated requests to server.adobedc.net are not evaluated by this service.
• Rule updates can take up to 15 minutes to propagate across the Edge Network according to product documentation.
• Scoring is not suppression. Routing, filtering, or downstream service decisions are required to act on the score.

8.2 Data View governance

CJA Data Views provide a reusable analytical control layer. Organizations can define approved human, bot, suspected, operational, AI-crawler, and AI-referred-human views without forcing every analyst to reproduce the logic in every project. Data View filtering is report-time governance and does not by itself reduce ingestion or reportable-row usage.

8.3 Derived and modeled classifications

CJA can expose classification fields generated upstream or through available modeling logic. Useful dimensions include bot class, detection source, confidence, reason code, validation status, allowed/blocked/quarantined decision, AI source, and authentication state. Keep original signals available for auditability.

Dynamic Datastream Configuration and Bot Quarantine

Dynamic Datastream Configuration (DDC) can route events based on datastream conditions. A common bot-governance pattern is to route events with botDetection.score = 1 into a separate quarantine dataset, then keep that dataset out of the CJA Connection used for standard business reporting.

9.1 Why quarantine instead of immediate deletion

• Preserves a recovery and investigation path when rules are wrong or incomplete.
• Supports false-positive review before moving to hard suppression.
• Allows retention to be managed independently from the primary analytical dataset.
• Can prevent quarantined events from counting in a CJA Connection when the quarantine dataset is not included in that Connection.

9.2 Configuration flow

1. Create a quarantine event dataset using the same schema as the primary event dataset.
2. Add the quarantine dataset to the datastream's AEP service configuration, without making it primary.
3. Open Configure DDC and create a routing configuration.
4. Build a rule where Bot Detection Score equals 1.
5. Select the quarantine dataset as the destination and save the configuration.
6. Validate routing, fallback behavior, client-side datastream overrides, and downstream services before production enforcement.

9.3 Coverage and economics

DDC quarantine is prospective and does not alter previously collected data. The IAB list is not exhaustive, so organizations should continue investigating residual automation and refining custom rules. A strong pattern is to minimize the volume left for report-time segments while retaining quarantine data long enough to support review and recovery.

A third-party article notes that quarantined events kept outside CJA no longer count toward that CJA Connection's reportable-row limit, while still consuming AEP Data Lake ingestion and storage. Customers should confirm contractual metrics and apply an appropriate retention policy.

Bot Filtering with Statistics and Machine Learning

Data Distiller provides SQL and machine-learning capabilities for identifying bot-like activity in AEP datasets. The documented workflow progresses from explicit activity thresholds to feature engineering, supervised classification, model evaluation, prediction, and model lifecycle management.

10.1 Core attributes and behavioral concept

• ECID or another approved identity key groups behavior by visitor or device.
• Timestamp enables event-rate analysis across defined windows.
• Page or content attributes add context to repetitive behavior.
• The output label, such as isBot, enables supervised model training.

10.2 SQL threshold example

The following pattern excludes ECIDs that exceed 60 events in a one-minute bucket. Thresholds are examples and must be calibrated to the property.

SELECT *
FROM analytics_events_table
WHERE enduserids._experience.ecid NOT IN (
SELECT enduserids._experience.ecid
FROM analytics_events_table
GROUP BY unix_timestamp(timestamp) / 60,
enduserids._experience.ecid
HAVING count(*) > 60
);

10.3 Multiple-window feature set

The Adobe example creates event-count features across one-minute, five-minute, and thirty-minute intervals, combines them with identity and webpage context, and creates an isBot label. Multi-window features help distinguish brief bursts from sustained automated behavior.

10.4 Model creation

The documented example uses imputers, indexers, discretizers, a vector assembler, min-max scaling, and a decision-tree classifier. The final transformation in the TRANSFORM clause becomes the feature column used by the model.

CREATE MODEL bot_filtering_model
TRANSFORM (
numeric_imputer(count_per_id.one_minute, 'mean') imputed_one_minute,
numeric_imputer(count_per_id.five_minute, 'mode') imputed_five_minute,
numeric_imputer(count_per_id.thirty_minute) imputed_thirty_minute,
string_imputer(id, 'unknown') imputed_id,
string_indexer(imputed_id) si_id,
quantile_discretizer(imputed_five_minute) buckets_five,
string_indexer(web.webpagedetails.name) si_name,
quantile_discretizer(imputed_thirty_minute) buckets_thirty,
vector_assembler(array(si_id, imputed_one_minute, buckets_five, si_name, buckets_thirty)) features,
min_max_scaler(features) scaled_features
)
OPTIONS (model_type='decision_tree_classifier', max_depth=4, label='isBot')
AS SELECT count_per_id, isBot, web, id
FROM analytics_events_clicks_count_criteria;

10.5 Evaluate, predict, and manage

SELECT *
FROM model_evaluate(
bot_filtering_model, 1,
SELECT count_per_id, isBot, web, id
FROM analytics_events_clicks_count_criteria
);

Evaluation returns classification metrics such as AUC-ROC, accuracy, precision, and recall. Do not treat performance on the training dataset as proof of production quality. Use separate test data, inspect class balance, validate false positives and false negatives, and monitor drift.

SELECT *
FROM model_predict(
bot_filtering_model, 1,
SELECT count_per_id, web, id
FROM analytics_events_clicks_count_criteria
);

SHOW MODELS;
DROP MODEL bot_filtering_model;

10.6 Production governance requirements

• Document label-generation methodology and avoid training a model only on the same threshold it is intended to replace.
• Evaluate on unseen data and by important segments, devices, geographies, authenticated states, and business journeys.
• Track precision to limit human false positives and recall to understand missed automation.
• Version models, features, thresholds, training windows, approvals, and rollback logic.
• Automate retraining only after drift monitoring and governance are established.
• Use model output as a score or classification signal first; move to suppression only after controlled validation.

Third-Party Bot Management: Complementary, Not Either/Or

Dedicated bot-management platforms operate at the network edge before a request reaches the page, origin, or analytics implementation. Adobe-native controls focus on analytics classification, reporting, and data routing. Mature strategies often use both.

11.1 Common third-party signals

• Browser and device fingerprinting, including rendering and API characteristics.
• Behavioral signals such as mouse, keyboard, scroll, timing, and navigation patterns.
• Network signals such as IP reputation, ASN, proxy/VPN intelligence, and TLS fingerprints.
• Challenges that help distinguish full browsers and automated clients.
• Cross-customer machine-learning intelligence.

11.2 Integration patterns

11.3 Vendor evaluation questions

• Can the platform block or challenge before origin and before the analytics tag fires?
• Can it pass a score, confidence, reason code, and policy decision into Adobe systems?
• How does it distinguish valuable AI agents and approved integrations from harmful automation?
• How are false-positive rates measured, investigated, tuned, and rolled back?
• Does it support the customer's AppMeasurement, Web SDK, mobile, server-side, and hybrid architecture?

CJA Governance and Historical Analysis

12.1 One definition, applied consistently

CJA can centralize classification and filtering through Connections, Data Views, reusable components, and modeled fields. This reduces the risk that merchandising, media, product, and executive reporting each use different bot definitions.

12.2 Historical flexibility

CJA reporting logic can be changed without destroying the underlying source dataset. This enables analysts to reclassify historical analysis when a Data View or derived-field definition changes, subject to the availability and retention of the necessary source attributes. It is more accurate to describe this as flexible report-time reinterpretation than as automatic correction of every upstream dataset.

12.3 Data Mirror and continuous correction

Internal positioning materials describe Data Mirror as a way to replicate event data into AEP and propagate updated classifications into CJA without rebuilding bespoke backfill pipelines. Because product behavior and availability may vary, confirm the current Data Mirror capability, source support, and entitlement before presenting it as a customer solution.

Measuring AI Crawlers, Agents, and Conversations

13.1 AI crawlers versus AI-referred humans

Separate direct crawler requests from human visits that arrive through an AI interface. Direct crawlers often do not execute JavaScript and are better measured with CDN or server logs. AI-referred human visits may appear in client-side analytics through referrer or campaign parameters. An agentic browser may execute JavaScript and behave more like a customer session.

13.2 Recommended dimensions

• Traffic actor: human, crawler, browser agent, API integration, unknown.
• AI source or agent family, where reliably observed.
• Detection source: IAB, custom UA, IP/CIDR, header, vendor score, rate model, manual review.
• Confidence and reason code.
• Business treatment: allow, monitor, exclude from human KPIs, quarantine, challenge, block.
• Human-intent linkage: direct crawler, AI referral, authenticated user, agent-assisted transaction.

13.3 Financial reporting and behavioral reporting

A transaction completed by an approved agent may represent real revenue while still distorting human-centered metrics such as duration, navigation, experimentation, or funnel behavior. Preserve separate analytical views so financial outcomes can include valid transactions while experience analysis can distinguish machine-mediated behavior.

Roadmap Themes

14.1 Consent Navigator

Roadmap materials describe a capability that augments CJA with CDN log data. A Global Request Identifier is intended to support deduplication and join-time stitching between CDN events and Web SDK or AppMeasurement events. The roadmap also describes enrichment with bot detection, geo, device, VPN/proxy, privacy opt-out signals, and threat intelligence.

• Capture bots, AI agents, ad-blocked sessions, opted-out visitors, and other JavaScript-invisible traffic alongside analytics-visible activity.
• Identify AI crawlers and analyze content performance in AI-mediated discovery.
• Deduplicate CDN and JavaScript event streams through a shared request identifier.
• Carry consent and opt-out signals across collection paths.

14.2 Filtering in Connections

Roadmap materials describe the ability to apply filters when adding data to a CJA Connection, with the intent of avoiding ingestion of unwanted data such as bot or test traffic. Confirm exact supported filters, timing, entitlement, and effect on contractual usage before positioning.

14.3 Conversation Insights

Roadmap materials describe collection of prompts, responses, and feedback from LLMs or human agents; automated extraction of intent, sentiment, tone, and keywords; analysis in Workspace; and connection of conversational signals to outcomes such as retention and conversion.

14.4 MCP Server for CJA and AA

Internal release summaries describe MCP support that enables LLMs and AI agents to run analytics reports through natural-language interactions. This is distinct from bot detection: it concerns AI agents as consumers of analytics and supports agentic workflows. Confirm current release status and supported operations before customer use.

14.5 Emerging transport-level fingerprinting

Internal materials also discuss JA4-style connection fingerprinting as a possible future layer. The concept evaluates how a client connects rather than trusting only a self-reported user agent. Treat this as roadmap material, not a currently committed capability, unless confirmed through current product documentation and the account team.

Recommended Operating Model

15.1 Decision matrix

15.2 Change-control requirements

• Rule identifier, owner, evidence, scope, confidence, and business justification.
• Affected properties, collection paths, report suites, datastreams, datasets, and Data Views.
• Expected change to KPIs and historical comparability.
• Test cases, allowlists, false-positive threshold, monitoring window, and rollback.
• Review date, expiration, and retirement decision.

Phased Action Plan

Phase 1: Immediate containment and visibility

• Verify IAB filtering and bot reporting in relevant AA report suites.
• Create a fact-based baseline for server calls, bot volume, suspicious sources, and affected journeys.
• Document approved automation and establish allowlists.
• Standardize clean reporting views and annotate incidents.
• Use WAF/CDN challenge or narrow blocking for confirmed malicious sources.

Phase 2: Collection and routing controls

• Implement Web SDK onBeforeEventSend, AppMeasurement s.abort, or Tags conditions only for validated high-confidence traffic.
• Pilot rate-limiting patterns such as Hit Governor in observe-first mode.
• Enable Edge bot scoring where architecture permits.
• Create a quarantine dataset and pilot DDC routing before hard suppression.

Phase 3: Advanced classification

• Create a governed feature dataset for velocity, navigation, identity, authentication, and content context.
• Pilot Data Distiller SQL thresholds and supervised classification.
• Evaluate models on unseen data and use precision/recall plus business impact.
• Feed approved classifications into CJA Data Views and operational controls.

Phase 4: Machine-traffic intelligence

• Separate AI crawlers, AI referrals, agentic sessions, and human direct activity.
• Measure content consumed by AI systems and downstream outcomes where signals exist.
• Evaluate Consent Navigator, Filtering in Connections, Conversation Insights, and MCP as roadmap capabilities become available and relevant.

Success measures

AA and CJA Capability Comparison

Frequently Asked Questions

Technical Implementation Checklist

Architecture

• Map all client, mobile, Edge, server-side, API, and batch paths.
• Confirm report suites, datastreams, schemas, datasets, Connections, Data Views, and downstream services.
• Document client-side datastream overrides and routing behavior.

AA controls

• Verify IAB and custom bot rules.
• Review Bot Name and Bot Pages reporting.
• Standardize behavior segments and Virtual Report Suites.
• Communicate KPI shifts before changes.

Collection suppression

• Implement only high-confidence conditions.
• Use non-production validation and staged rollout.
• Monitor server calls, conversion, errors, checkout, login, search, and integrations.
• Maintain rollback.

AEP and CJA

• Add Bot Detection Information field group.
• Enable and validate datastream detection.
• Create quarantine dataset and retention policy.
• Configure and test DDC routing.
• Keep quarantine outside the business CJA Connection when appropriate.
• Create governed Data Views.

Data Distiller

• Create features and labels.
• Use training/test separation.
• Evaluate accuracy, precision, recall, and AUC-ROC.
• Inspect false positives and drift.
• Version and govern models.

Reference Code

if (confirmedBot === true) return false;

if (confirmedBot === true) s.abort = true;

content.data.__adobe.analytics.eVar10 = navigator.userAgent;

if (localStorage.getItem("hg_flag") === "1") return false;

SHOW MODELS;
DROP MODEL bot_filtering_model;

Resources

Source and Claim Notes

This whitepaper integrates customer-provided content, official Adobe documentation, Adobe Experience League community guidance, internal planning materials, and a third-party implementation article. Official documentation is the authority for current product behavior. Community and third-party patterns should be independently validated. Forward-looking content is labeled and subject to the roadmap disclaimer.