So
Agentic Web Data HubAI Agent data infrastructure/real-time structured search and knowledge base · V42
Connecting to the live product service
AI Agent data infrastructure/real-time structured search and knowledge base

Turn public web pages into traceable, searchable, and structured callable Agent data sources

Users can convert websites into searchable knowledge bases in batches, extract structured Internet data with sources based on conditions, and generate auditable competitive product intelligence records.

  1. 01
    Configure data boundariesFill in the project name, website scope, query conditions, target company, retention period and compliance exclusion instructions.
  2. 02
    Collect public sourcesCrawl website pages or retrieve public web pages based on action type and log access, failure, and exclusion reasons.
  3. 03
    Structuring and indexingWrite web content into a searchable index, or extract structured records by field.
  4. 04
    Review and releaseUser review sources, missing fields, risk alerts, and report status.
Number of indexed pagesNo indexed pages yetCount the sum of crawl_job.pages_indexed of the current project
Number of failed pagesNo failure record yetCount the sum of crawl_job.pages_failed of the current project
Number of structured recordsNo structured records yetCount the number of structured_records in the current project
source coverageNo source coverage can be calculated yetNumber of structured records with source_url / Number of total structured records
Number of information to be reviewedNo information yet to be reviewedreview_status is the number of competitor_intel_reports to be reviewed
BUSINESS INPUT

New project

Each project defines an Internet data acquisition workspace, including crawl scope, structured conditions, target companies, compliance preferences, and credential configuration.

Loading product-specific business fields.

Not saved
TRACEABLE OUTPUT

Data task results

Results, sources, failures, approvals, and rollbacks are saved by project.

0 live runs
∅

No business results yet

There are no knowledge base tasks yet. Please enter the public website address and set the crawl range to start.

PRIVACY

Data use boundary

Items may include search service keys, target company lists, enterprise query criteria, compliance exclusion instructions, and crawl logs. The API Key must be saved in encrypted text, and the page will no longer be echoed in plain text; query conditions and target lists are only used for current project tasks and audits, and should not be used for external marketing or unrelated training. Current 10 role review additions: Incorporate query_conditions, target_companies, schema_fields, field_values, evidence_snippet, finding, audit_notes, compliance_notes, crawl logs, export reports, and third-party retrieval request fields into sensitivity policies. ;The API Key must be saved in server-side ciphertext. After saving, only the tail number or ciphertext identification will be displayed. It shall not be entered into logs, exports, error responses, model prompts, or front-end plaintext. ;Users must be able to test connections, rotate, delete credentials, and view task errors after credentials expire. ; Collect only the pages, fields and evidence fragments required to complete the current task; prompt the sensitive range and perform desensitization before exporting. ; Delete or archive queryable data on this site by retention_days; deletion does not affect external public web pages or local files exported by users. ; Disclose to users the fields and uses that may be received by external search services; No online execution without valid consent.

RETENTION

Data retention

Indexes, reports and audit logs are saved according to retention_days, and will be deleted or archived from the queryable data of this site after expiration; deletion will not affect local files that have been exported by users, nor will it delete external public web pages.

SAFETY

Human responsibility and rollback

This product is used for retrieval, indexing, structured extraction and intelligence compilation of public Internet data. Results may be affected by web page timeliness, source reliability, crawling limitations, and model extraction errors; users should check the original source before all business decisions. Products do not bypass access controls, crawl restricted content, or confirm undisclosed facts on behalf of users.

Rollback only means undoing the task status, report release status, index record or review mark in this site and restoring to the previous version; it does not mean or guarantee the rollback of any unconnected external production system, external website or third-party service.