Indexing and AI Crawlers
The site is open to search engines and AI crawlers, with public access to crawlable files. Actual indexing submissions are performed after site verification in each search engine's webmaster platform.
Search Engine Directory
109Home
Includes official submission portal
12Home
AI Crawler Directory
160one
Cover Region
16one
sitemap.xml Site MapDeclares all indexable pages on the site for batch crawling by search engines.llms.txt Site DescriptionSite theme, keywords, and content navigation guide for AI assistants.robots.txt AllowlistExplicitly allow major search engines and AI crawlers to index the entire site.
Search Engine Directory
Together109Home
Search engineRegionTypeSubmitDescription
BaiduChinaGlobal SearchSubmit to Webmaster ToolsAfter verifying your site in the Webmaster Platform, you can submit links and sitemaps.
360 Search (Haoso)ChinaGlobal SearchSubmit to Webmaster ToolsVerify site on Webmaster Tools and submit sitemap
Shenma SearchChinaMobile SearchSubmit to Webmaster ToolsMobile search: Submit your site via the Webmaster Platform
Bing ChinaChinaGlobal SearchSubmit to Webmaster ToolsBing Webmaster Tools supports URL submission and sitemaps.
China SearchChinaGlobal SearchVerify in Search ConsoleNational comprehensive search: submit after verification in the Webmaster Platform.
Youdao SearchChinaGlobal SearchVerify in Search ConsoleSubmit after verifying in the Webmaster Tools
WeChat SearchChinaContent SearchSubmit to Webmaster ToolsWeChat Official Account content is indexed after being published publicly via the backend.
Taobao SearchChinaE-commerce SearchVerify in Search ConsoleProduct inclusion is determined by the platform's store system.
JD SearchChinaE-commerce SearchVerify in Search ConsoleProduct inclusion is determined by the platform's store system.
Pinduoduo searchChinaE-commerce SearchVerify in Search ConsoleProduct inclusion is determined by the platform's store system.
Search DouyinChinaContent SearchVerify in Search ConsoleSubmit content after completing qualification on the platform's developer portal.
Kuaishou SearchChinaContent SearchVerify in Search ConsoleSubmit content after completing qualification on the platform's developer portal.
Xiaohongshu SearchChinaContent SearchVerify in Search ConsoleSubmit content after completing qualification on the platform's developer portal.
Bilibili SearchChinaContent SearchVerify in Search ConsoleSubmit content after completing qualification on the platform's developer portal.
Zhihu SearchChinaQ&A SearchVerify in Search ConsoleContent inclusion is determined by the platform's system.
Weibo SearchChinaContent SearchVerify in Search ConsoleContent inclusion is determined by the platform's system.
Douban SearchChinaContent SearchVerify in Search ConsoleContent inclusion is determined by the platform's system.
Dianping SearchChinaLocal SearchVerify in Search ConsoleStore information must be maintained after claiming on the platform.
Amap SearchChinaMap SearchVerify in Search ConsoleStore locations must be claimed on the map platform before they can be updated.
Baidu Maps SearchChinaMap SearchVerify in Search ConsoleStore locations must be claimed on the map platform before they can be updated.
Ctrip SearchChinaLife SearchVerify in Search ConsoleContent inclusion is determined by the platform's system.
58 Local SearchChinaCategory SearchVerify in Search ConsoleInformation inclusion is determined by the platform system.
AI Crawler Directory
All listed crawlers are explicitly permitted in robots.txt. Site content is understandable and citable per llms.txt guidelines.
Crawler NameManufacturerUser-Agent IdentifierPurpose of scrapingStatus
GPTBotOpenAI
GPTBotFor OpenAI model training and content understandingCrawling allowedOAI SearchBotOpenAI
OAI-SearchBotSearch Index for ChatGPTCrawling allowedChatGPT UserOpenAI
ChatGPT-UserReal-time web access triggered by userCrawling allowedGPT-UserOpenAI
GPT-UserReal-time scraping of plugins and browsing capabilitiesCrawling allowedClaudeBotAnthropic
ClaudeBotClaude Model Training and IndexingCrawling allowedClaude-UserAnthropic
Claude-UserUser-triggered page accessCrawling allowedClaude Search BotAnthropic
Claude-SearchBotIndex for Claude's search capabilitiesCrawling allowedAnthropic AIAnthropic
anthropic-aiHistorical Training Capture FlagCrawling allowedPerplexityBotPerplexity
PerplexityBotIndex for Perplexity SearchCrawling allowedPerplexity UserPerplexity
Perplexity-UserReal-time web access triggered by userCrawling allowedGoogle-ExtendedGoogle
Google-ExtendedGemini Training and GroundingCrawling allowedGoogle Cloud Vertex BotGoogle
Google-CloudVertexBotVertex AI Customer-Specified CrawlingCrawling allowedApplebotApple
ApplebotSiri and Spotlight IndexingCrawling allowedApplebot-ExtendedApple
Applebot-ExtendedApple Intelligence Generative UseCrawling allowedBytespiderByteDance
BytespiderDoubao and ByteDance AI Training ScrapingCrawling allowedPetalBotHuawei
PetalBotPetal Search and Xiaoyi Content IndexCrawling allowedYisouSpiderAlibaba (Quark)
YisouSpiderQuark Search Content IndexCrawling allowedBaiduspiderBaidu
BaiduspiderBaidu Search Index Related to ERNIECrawling allowedBaiduspider-renderBaidu
Baiduspider-renderBaidu Rendering and CrawlingCrawling allowedSogou web crawlerSogou
Sogou web spiderSogou Search Content IndexCrawling allowed36Spider360
360Spider360 Search Content IndexCrawling allowedbingbotMicrosoft
bingbotBing Search Index and Copilot CitationsCrawling allowedBing PreviewMicrosoft
BingPreviewPage preview renderingCrawling allowedadidxbotMicrosoft
adidxbotAd Relevance ExtractionCrawling allowedDuckDuckBotDuckDuckGo
DuckDuckBotDuckDuckGo IndexCrawling allowedDuckAssistBotDuckDuckGo
DuckAssistBotDuckDuckGo AI Assistant Real-Time ScrapingCrawling allowedYandexBotYandex
YandexBotYandex Search IndexCrawling allowedYetiNaver
YetiNaver Search IndexCrawling allowedAmazon BotAmazon
AmazonbotUsing Alexa with Search and Q&ACrawling allowedMeta-External AgentMeta
Meta-ExternalAgentMeta AI Training and Product CapabilitiesCrawling allowedMeta External FetcherMeta
Meta-ExternalFetcherMeta AI user triggerCrawling allowedFacebook External HitMeta
facebookexternalhitLink Preview ScrapeCrawling allowedcohere-aiCohere
cohere-aiCohere Model Training ScrapingCrawling allowedcohere-training-data-crawlerCohere
cohere-training-data-crawlerCohere Training Data ScrapingCrawling allowedAI2BotAllen AI
AI2BotOpen Corpus and Model ResearchCrawling allowedAi2Bot-DolmaAllen AI
Ai2Bot-DolmaDolma Open Corpus CrawlerCrawling allowedYouBotYou.com
YouBotAI Search IndexCrawling allowedMistralAI UserMistral AI
MistralAI-UserReal-time scraping triggered by userCrawling allowedCCBotCommon Crawl
CCBotOpen Corpus CrawlingCrawling allowedDiffbotDiffbot
DiffbotKnowledge Graphs and Structured ExtractionCrawling allowedOmgilibotWebz.io
OmgilibotMedia Data CollectionCrawling allowedWebzio-ExtendedWebz.io
Webzio-ExtendedGenerative Corpus CollectionCrawling allowedImagesift BotImagesift
ImagesiftBotImage IndexCrawling allowedTimpibotVast/Timpi
TimpibotSearch Indexing & AI CorpusCrawling allowedKangaroo BotKangaroo
Kangaroo BotAI Corpus CollectionCrawling allowedBLEXBotWebMeUp
BLEXBotBacklink & Content AnalysisCrawling allowedSemrushBotSemrush
SemrushBotSEO Data CollectionCrawling allowedAhrefsBotAhrefs
AhrefsBotSEO Data CollectionCrawling allowedMJ12botMajestic
MJ12botOutbound Link Data CollectionCrawling allowedDotBotMoz
DotBotSEO Data CollectionCrawling allowedDataForSeoBotDataForSEO
DataForSeoBotSearch Data CollectionCrawling allowedZoomInfo BotZoomInfo
ZoominfoBotEnterprise Information CollectionCrawling allowedPiplBotPipl
PiplBotIdentity Information CollectionCrawling allowedSeekrSeekr
SeekrTrusted Content IndexCrawling allowedScrapyOpen Source Project
ScrapyOpen-source crawler framework identifierCrawling allowedFirecrawl AgentFirecrawl
FirecrawlAgentScrape pages for AI applicationsCrawling allowedVercel BotVercel
VercelbotDeploy & Preview CrawlCrawling allowedInternet MeasurementInternet Measurement
InternetMeasurementNetwork MeasurementCrawling allowedChatGPT OperatorOpenAI
ChatGPT-OperatorReal-time access for operational agentsCrawling allowedGoogle AgentGoogle
Google-AgentAgent Real-Time AccessCrawling allowedAliyunBotAlibaba Cloud
AliyunBotCloud Product Documentation and Content IndexCrawling allowedQwen-UserAlibaba (Tongyi)
Qwen-UserReal-time access triggered by userCrawling allowedQuarkBotAlibaba (Quark)
QuarkBotQuark AI Search Content IndexCrawling allowedTencentBotTencent
TencentBotTencent Content Index and Yuanbao Q&A CitationCrawling allowedYuanbao-UserTencent (Yuanbao)
Yuanbao-UserReal-time access to user-triggered yuanbaoCrawling allowedWeChat BotTencent (WeChat)
WeChatBotWeChat Official Account & Mini Program Content IndexCrawling allowediFlytekBotiFLYTEK
iFlytekBotSpark Model Content IndexCrawling allowedSpark UseriFLYTEK (Spark)
Spark-UserUser-triggered Spark real-time accessCrawling allowedZhipuBotZhipu AI
ZhipuBotZhipu Model Content IndexCrawling allowedGLM-UserZhipu AI (GLM)
GLM-UserGLM real-time access triggered by userCrawling allowedMoonshotBotMoonshot AI
MoonshotBotKimi Content IndexCrawling allowedKimi-UserMoonshot AI (Kimi)
Kimi-UserKimi real-time access triggered by userCrawling allowedMiniMaxBotMiniMax
MiniMaxBotMiniMax Content IndexCrawling allowedDeepSeek BotDeepSeek
DeepSeekBotDeepSeek Content IndexCrawling allowedDeepSeek UserDeepSeek
DeepSeek-UserDeepSeek real-time access triggered by userCrawling allowedSenseBotSenseTime
SenseBotSenseTime Content IndexCrawling allowedJDSpiderJD.com
JDSpiderProduct and Content IndexCrawling allowedMeituan SpiderMeituan
MeituanSpiderLocal Life Content IndexCrawling allowedXHS-SpiderXiaohongshu
XHS-SpiderNote Content IndexCrawling allowedZhihuBotZhihu
ZhihuBotQ&A Content IndexCrawling allowedWeiboBotWeibo
WeiboBotWeibo Content IndexCrawling allowedKuaishou SpiderKuaishou
KuaishouSpiderShort-form Video Content IndexCrawling allowedBilibiliBotBilibili
BilibiliBotVideo Content IndexCrawling allowedDouyin SpiderByteDance (TikTok)
DouyinSpiderTikTok Content IndexCrawling allowedToutiao SpiderByteDance
ToutiaoSpiderHeadline Content IndexCrawling allowedByteDance CrawlerByteDance
ByteDanceCrawlerByte-series content scrapingCrawling allowedHuawei CrawlerHuawei
HuaweiCrawlerHuawei Content IndexCrawling allowedXiaoyi UserHuawei (Xiaoyi)
Xiaoyi-UserXiaoyi real-time access triggered by userCrawling allowedXiaomiBotXiaomi
XiaomiBotXiaomi Content IndexCrawling allowedOppoBotOPPO
OppoBotTerminal Assistant Content IndexCrawling allowedVivoBotvivo
VivoBotTerminal Assistant Content IndexCrawling allowedBaiduspider-imageBaidu
Baiduspider-imageBaidu Image IndexCrawling allowedBaiduspider-videoBaidu
Baiduspider-videoBaidu Video IndexCrawling allowedBaiduspider-newsBaidu
Baiduspider-newsBaidu News IndexCrawling allowedBaiduspider-favoBaidu
Baiduspider-favoBaidu Collects ScrapingCrawling allowedBaiduspider-cproBaidu
Baiduspider-cproBaidu Partner CrawlerCrawling allowedBaidu Cloud MonitorBaidu
Baidu-YunGuanCeBaidu Cloud ObservabilityCrawling allowedBaiduspider-mobileBaidu
Baiduspider-mobileMobile IndexCrawling allowedSogou News CrawlerSogou
Sogou news spiderSogou News IndexCrawling allowedSogou Image CrawlerSogou
Sogou Pic SpiderSogou Image IndexCrawling allowedSogou Video SpiderSogou
Sogou Video SpiderSogou Video IndexCrawling allowed360Spider-Image360
360Spider-Image360 Image IndexCrawling allowed360Spider-Video360
360Spider-Video360 Video IndexCrawling allowedYisouSpider-MobileWhat
YisouSpider-MobileShenma Mobile IndexCrawling allowedBingbot NewsMicrosoft
bingbot-newsBing News IndexCrawling allowedmsnbotMicrosoft
msnbotBing Historical Index IDCrawling allowedMSNBot-MediaMicrosoft
MSNBot-MediaBing Media IndexCrawling allowedMicrosoft PreviewMicrosoft
MicrosoftPreviewPage preview renderingCrawling allowedCopilot-UserMicrosoft (Copilot)
Copilot-UserCopilot real-time access triggered by userCrawling allowedGoogle OtherGoogle
GoogleOtherGoogle Universal CrawlCrawling allowedGoogle Other - ImageGoogle
GoogleOther-ImageGoogle Image GrabbingCrawling allowedGoogle Other - VideoGoogle
GoogleOther-VideoGoogle Video ScraperCrawling allowedGoogle Inspection ToolGoogle
Google-InspectionToolSearch Console Crawl DetectionCrawling allowedGoogle Read AloudGoogle
Google-Read-AloudRead-aloud Service ScrapingCrawling allowedGoogle SafetyGoogle
Google-SafetySecurity scanCrawling allowedGoogle Site VerificationGoogle
Google-Site-VerificationSite Verification CrawlCrawling allowedStorebot-GoogleGoogle
Storebot-GoogleShopping content scrapingCrawling allowedFeed Fetcher - GoogleGoogle
FeedFetcher-GoogleSubscribe to feedCrawling allowedMedia Partners - GoogleGoogle
Mediapartners-GoogleAd Relevance ExtractionCrawling allowedClaude-WebAnthropic
Claude-WebClaude Web ReaderCrawling allowedAnthropic AI BotAnthropic
anthropic-ai-botAnthropic Training ScrapeCrawling allowedPerplexityBot NewsPerplexity
PerplexityBot-NewsPerplexity News IndexCrawling allowedYouBot NewsYou.com
YouBot-NewsYou.com News IndexCrawling allowedPhindBotPhind
PhindBotDeveloper Q&A IndexCrawling allowedPoeBotQuora (Poe)
PoeBotPoe Content IndexCrawling allowedCharacterBotCharacter.AI
CharacterBotCharacter Content IndexCrawling allowedHugging Face BotHugging Face
HuggingFaceBotDataset and Model IndexCrawling allowedTogetherBotTogether AI
TogetherBotModel Service Content ScrapingCrawling allowedGroqBotGroq
GroqBotInference Service Content CrawlingCrawling allowedFireworksBotFireworks AI
FireworksBotModel Service Content ScrapingCrawling allowedxAI BotxAI
xAI-BotGrok Content IndexCrawling allowedGrok UserxAI (Grok)
Grok-UserGrok real-time access triggered by userCrawling allowedNVIDIA BotNVIDIA
NVIDIABotContent scraping in model researchCrawling allowedIBMBotIBM
IBMBotEnterprise AI Content IndexCrawling allowedSalesforceBotSalesforce
SalesforceBotEnterprise Content IndexCrawling allowedAdobeBotAdobe
AdobeBotContent and Asset IndexCrawling allowedCanvaBotCanva
CanvaBotDesign Asset IndexCrawling allowedNotionBotNotion
NotionBotLink Preview ScrapeCrawling allowedSlackbotSlack
SlackbotLink Preview ScrapeCrawling allowedDiscord BotDiscord
DiscordbotLink Preview ScrapeCrawling allowedTelegram BotTelegram
TelegramBotLink Preview ScrapeCrawling allowedWhatsAppMeta (WhatsApp)
WhatsAppLink Preview ScrapeCrawling allowedLinkedIn BotLinkedIn
LinkedInBotLink Preview ScrapeCrawling allowedPinterest BotPinterest
PinterestbotImage IndexCrawling allowedReddit BotReddit
RedditBotLink Preview ScrapeCrawling allowedQuora BotQuora
Quora-BotQ&A Content IndexCrawling allowedExabotDassault
ExabotComprehensive IndexCrawling allowedGigabotGigablast
GigabotOpen-source indexingCrawling allowedBarkrowlerBabbar
BarkrowlerSEO Data CollectionCrawling allowedBrightbotBright Data
BrightbotCompliant Data CollectionCrawling allowedAwarioBotAwario
AwarioBotSocial Media MonitoringCrawling allowedAwario RSS BotAwario
AwarioRssBotRSS MonitoringCrawling allowedAwario Smart BotAwario
AwarioSmartBotSmart MonitoringCrawling allowedScreaming Frog SEO SpiderScreaming Frog
Screaming Frog SEO SpiderSite Audit CrawlCrawling allowedSitebulbSitebulb
SitebulbSite Audit CrawlCrawling allowedOnCrawlOnCrawl
OnCrawlLogs & Crawl AnalysisCrawling allowedSerpstatBotSerpstat
SerpstatBotSearch Data CollectionCrawling allowedSeobilityBotSeobility
SeobilityBotSEO Data CollectionCrawling allowedSISTRIXSistrix
SISTRIXSearch Data CollectionCrawling allowedUptimeRobotUptimeRobot
UptimeRobotAvailability MonitoringCrawling allowedSubmission Guidelines
- 1. Register an account and verify site ownership in the target search engine's webmaster platform (most support file verification, HTML tag verification, or DNS verification).
- 2. Submit the site map at `/sitemap.xml` to help search engines discover all public pages in bulk.
- 3. Submit key pages one by one via the Search Console's "URL Submission" or "Fast Indexing" tool to reduce discovery time.
- 4. Keep the page publicly crawlable (do not apply noindex to public pages) and maintain allow declarations in `robots.txt` and `llms.txt`.
Note: Submission for indexing must be completed after verifying your site on each search engine's webmaster platform. This site provides a genuine directory and entry links. Indexing and ranking are determined solely by the search engines and AI platforms; we ensure only that your content is crawlable, understandable, and citable.
View all site keywords and long-tail keywords