Susirahayu99.blogspot.com|| Google has provided further clarification regarding the documentation for its "Google-Extended" web crawler ~ This recent update addresses changes in product naming and aims to clarify its impact on search results, which may concern those who wish to block the crawler. The revised documentation provides clearer instructions on managing content access for the purpose of AI model training.
Introduced on September 28, 2023, the Google-Extended user agent allows web publishers to control how their websites are crawled. Publishers can choose to allow or disallow the Google-Extended user agent through the Robots Exclusion Protocol, providing them with a means to opt out of having their content scraped and utilized in AI training datasets.
Although Google refers to Google-Extended as a "standalone product token," this term deviates from the standard understanding of User Agents among publishers. The original announcement described Google-Extended as a tool for web publishers to manage whether their sites contribute to improving Bard and Vertex AI generative APIs, which power products like Gemini Apps.
Blocking Google-Extended is accomplished using the "Google-Extended" User Agent. Google maintains a changelog of significant updates made to guidance and communication with web publishers and the search marketing community. The recent change in the Google-Extended documentation follows the renaming of Bard to Gemini Apps, specifying that Google-Extended's indexing now contributes to Gemini Apps and Vertex AI generative APIs.
The revised wording reassures publishers that opting out of Google-Extended AI data collection does not affect Google Search. Google's changelog clarifies that Google-Extended crawling is exclusive to Gemini Apps and has no impact on Google Search results.
The updated guidance no longer mentions the Bard brand name, replacing it with Gemini. Additionally, it explicitly states that "Google-Extended does not impact a site's inclusion or ranking in Google Search."
For more information, you can refer to Google's updated crawler overview titled "Overview of Google crawlers and fetchers (user agents)."
Posting Komentar