Popular on s4story
- Sycamore Dove Releases "Loving Every Moment" - 251
- Arizona Pest Control Surpasses $200,000 in Contributions to Southern Arizona High School Sports and Education Programs - 128
- North Carolina Author Carrie Sybert Celebrates Three Published Books - 123
- The Super Awesome Activity Book - 108
- New Children's Picture Book "Adam's Apple" Celebrates Childhood Curiosity and the Funny Misundersta - 108
- RAS AP Consulting Releases Educational Resource to Help Organizations Prepare for IRS FIRE-to-IRIS Transition - 105
- New Children's Picture Book "Tippy Tay" Explores the Dangers of Peer Pressure and the Importance of - 104
- Good Organics Now at Pop Up Grocer NYC With October 30 Sampling Event - 103
- Integris Composites Joins International Sea Power Conference Aboard HMS Prince of Wales - 103
- The YA Fantasy Novel of the Year.... See You in Your Sleep by R.T. Tita
Similar on s4story
- Iglesia.com Positioned in Multimillion-Dollar Domain Market with $10 Million Asking Price
- ViewDNS.info Launches Monitors, Alerting Teams the Moment Something Changes on Their Domains
- Tampa Web Design Launches New Website for Entrepreneur and Capital Markets Advisor Martin Sumichrast
- AeroNet Deploys netElastic vBNG and CGNAT with Tarana Wireless to Deliver Scalable, Fiber-Class Broadband Across Puerto Rico
- Nextlink Internet Deploys netElastic to Modernize Its Hybrid Broadband Network
- BESTBUYSTORE opens its dropshipping model to independent merchants
- Clearnetworx Deploys netElastic vBNG, CGNAT And netVision To Power Fiber Expansion Across Colorado's Western Slope
- Coinfiliate Launches Platform to Helps Shoppers Earn Cashback and Creators Earn Referral Rewards
- netElastic Announces vBNG And CGNAT 2.0 General Availability
- 3ptechies Makes Its Articles Audio-Ready With Instaread Voice Technology
Robots.txt one line AI training. Anthropic, Microsoft, Google and OpenAI back the concept
S For Story/10708569
How Lab451 Generator handles it: an opt-in training preference for robots.txt. No preference is the default; choose to opt out of the whole site or all but some paths. When you opt out, the per-crawler training groups are written to match.
VICTORIA, British Columbia - s4story -- Lab451 Helps Website Owners Get Ready for the New AI-Training Signal in robots.txt Backed by Microsoft, OpenAI, Google and Anthropic
Free tool generates, verifies and monitors robots.txt, sitemap.xml, llms.txt and llms-full.txt — now with support for the IETF draft '__usage_ai-training' token
One line instead of a growing block list
Until now, a publisher who did not want their content used to train AI models had to add a separate robots.txt rule for every AI crawler, and keep that list current as new crawlers appeared. In late October, the four companies announced they would honor a single shared signal instead: a policy token, '__usage_ai-training', placed in the User-Agent line of an ordinary robots.txt group. The proposal was written by Kevin Kelley, Anthropic's technical lead for standards, and submitted to the Internet Engineering Task Force (IETF) as an Internet-Draft.
More on S For Story
A site that wants to opt out of AI training would add a group such as:
User-Agent: __usage_ai-training
Disallow: /
AI Training, Generate, Verify, Monitor
Availability
More on S For Story
The Lab451 generator and checker are free at https://lab451.org. Monitoring is included in the Pro and Agency plans.
Media contact
Roger Remacle
hello@lab451.org
https://lab451.org
Free tool generates, verifies and monitors robots.txt, sitemap.xml, llms.txt and llms-full.txt — now with support for the IETF draft '__usage_ai-training' token
One line instead of a growing block list
Until now, a publisher who did not want their content used to train AI models had to add a separate robots.txt rule for every AI crawler, and keep that list current as new crawlers appeared. In late October, the four companies announced they would honor a single shared signal instead: a policy token, '__usage_ai-training', placed in the User-Agent line of an ordinary robots.txt group. The proposal was written by Kevin Kelley, Anthropic's technical lead for standards, and submitted to the Internet Engineering Task Force (IETF) as an Internet-Draft.
More on S For Story
- Record Q3 Revenue and Gross Profit Delivered: Revenue Surges 40%, Gross Margin Soars to 36% & Adjusted EBITDA Loss Narrows 64% for Cycurion, Inc $CYCU
- Inaugural 12 Trees of Christmas gala raises $525,000 for DMG research and family support
- BitTitan Expands MigrationWiz with SharePoint Sites Migration and Simplified Collaboration Licensing
- Olympic champion Katie Moon to compete at Pole Vault in Paradise in Key West
- Sri Aurobindo's Vedic Glossary — Long Out-of-print Guide To The Rig Veda Now Available As Ebook
A site that wants to opt out of AI training would add a group such as:
User-Agent: __usage_ai-training
Disallow: /
AI Training, Generate, Verify, Monitor
- AI Training (new): Lab451's generator offers the __usage_ai-training group as an opt-in option clearly labelled "IETF draft". The checker recognizes and validates the token in existing robots.txt files, monitoring alerts when it is added, removed or malformed, and a short explainer page walks site owners through what it means.
- Generate: Build four files from one crawl of a site — robots.txt, sitemap.xml, llms.txt and llms-full.txt — with robots.txt as the entry point that points search engines and AI agents to the other three.
- Verify: Every generated file is checked for dead links and hostnames, unregistered package names, and text hidden from human readers before it is published. The free Checker runs the same validation on files a site already publishes.
- Monitor: Pro and Agency plans watch published files six times a day, send a weekly report by email, and notify the owner immediately when a critical problem appears — such as a robots.txt change that accidentally blocks search engines or drops the AI-training preference.
Availability
More on S For Story
- Smith Strong Merges With Gravis Law, Bringing Gravis Law to Virginia
- Schizophrenia Groups Recommend Just Published Thirty Year Caregiving Struggle
- Independent Scholar Yogesh Patel Proposes 'Spacenow': A Fifth Dimension Reinterpreting Spacetime
- Pale Face Receives Distinguished Author Award from International Impact Book Awards
- Warren Pulley Groups Practitioner Security Books into Four New Series
The Lab451 generator and checker are free at https://lab451.org. Monitoring is included in the Pro and Agency plans.
Media contact
Roger Remacle
hello@lab451.org
https://lab451.org
Source: Lab451
0 Comments
Latest on S For Story
- Children's Series Adds a Holiday Tale!
- LCH2 Awarded Additional $1 Million to Advance Lewis County Clean Energy Campus
- New Book - In the Weeds: Notes on Motherhood, Gardening, and Other Chaos
- Security Planning Tips For Your Holiday Event From Access Patrol Services
- Contracting Resources Group Receives 2026 HIRE Vets Platinum Medallion Award
- NursingColleges.com Announces Jesutofunmi Oluwaleke as 2026 Scholarship Winner
- Business News & Info Goes Inside Paragon FX Group's Hunt for Original Film Props
- Stephen Thomas Releases Book Two In His Four Lights series - In the Day of the Lion
- Kitchen Table Economics: An Invitation to Think Differently
- New publisher Reed & Alden launches Mastery Lessons with Times Square billboard campaign
- Gabby Press Earns High Praise From BookLife For Book About the Dangers of Corporate AI
- Win a $5,000 VIP Family Experience at the 100th Thanksgiving Day Parade in New York City
- CB Stuffer Announces Private Sale to Secure Future of Brand Growth
- The Writer Who Works With Ai Around The Clock Opens His Workshop To The Public
- ReciScan Adds Website Cookbook Builder Alongside Mobile App
- Pyrrhic Press Publishing Welcomes Ten-Book Author Kiminou Knox
- Emanuel Rose Launches Second Edition of The Agentic Pivot to Help Business Leaders Scale AI Adoption
- Joel Rodriguez Tirado Honors Veterans Day With a Message of Service, Sacrifice, and Legacy
- The Fifth Annual "Bamboo Lantern Festival" to Illuminate Tokyo Mizumachi and Sumida River Walk®
- Gem Publishers Highlights Professional Book Publishing Services for Authors