OpenAI Launches Web Crawler ‘GPTBot’
“GPTBot” is a new web crawling tool from artificial intelligence company OpenAI that the company claims may be used to enhance future ChatGPT models.
According to a new blog post by OpenAI, “web pages crawled with the GPTBot user agent may potentially be used to improve future models,” adding that it might increase the accuracy and capabilities of subsequent iterations.
A bot that indexes the content of webpages on the internet is known as a web crawler, sometimes known as a web spider. They are used by search engines like Google and Bing so that the websites appear in search results.
OpenAI has developed a web crawler that collects publicly available data from the internet but filters out sources with paywalled content, personally identifiable information, or violated policies. By including a “disallow” command in a common server file, website owners can prevent the crawler from accessing their pages. The crawler comes three weeks after OpenAI filed a trademark application for the anticipated successor to the current GPT-4 model.
(With inputs from Shikha Singh)
You need to login in order to Like
