SlideShare uma empresa Scribd logo

Introduction to robots.txt

•
•
madamseo
madamseo
TecnologiaNegócios
1 de 6
Baixar para ler offline
INTRODUCTIONTO ROBOTS.TXT
• Robots.txt is a plain text file that you upload to the root directory of your site. Once the web spiders (ants, boots, indexers) that index your webpages reach your site, they first look at that text file and process it. Put differently, robots.txt says to the spider which pages to crawl.
THE SIMPLESTVERSION OF ROBOTS.TXT FILE IS: USER-AGENT:* DISALLOW: THE FIRST LINE INDICATESTHE USER AGENT- ASTERISK INDICATES THATTHE FOLLOWING LINES APPLYTO ALL AGENTS. SPACE AFTER "DISALLOW:" MEANSTHAT NOTHING IS LIMITED. THIS ROBOTS.TXT FILE DOES NOTHING- IT ALLOWS USER AGENTSTO SEE EVERYTHING ONTHE SITE.
NOW LET'S MAKE IT A LITTLE MORE COMPLICATED-THISTIME WE DO NOT WANT SPIDERSTO CRAWL IN OUR /FAQ DIRECTORY: USER-AGENT:* DISALLOW: /FAQ/ IT IS RELATIVELY EASY. SLASH INDICATESTHATTHIS IS A DIRECTORY. IFYOU DON'T PUT A SLASH, NOT ONLY/FAQ DIRECTORY, BUT ALSO EACH FILE WHICH NAME BEGINS WITH "FAQ" WILL BE BANNED. ALTERNATIVELY, YOU CAN ADD MORE DIRECTORIESTOYOUR BANNED LIST.
VIEWTHE FULL PRESENTATION AT WWW.MADAMSEO.COM OR SUBSCRIBETO MY PAID COURSE ON WWW.UDEMY.COM

Recomendados

Difference between php and node
Difference between php and nodeDifference between php and node
Difference between php and nodeNguyễn Ngọc Phương
 
Navegacao de Robótica Autônoma em Ambientes Dinâmicos
Navegacao de Robótica Autônoma em Ambientes DinâmicosNavegacao de Robótica Autônoma em Ambientes Dinâmicos
Navegacao de Robótica Autônoma em Ambientes DinâmicosLCoN Mackenzie
 
Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...
Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...
Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...Brussels, Belgium
 
Pitching With Passion, Make Your Startup Stand Out
Pitching With Passion, Make Your Startup Stand OutPitching With Passion, Make Your Startup Stand Out
Pitching With Passion, Make Your Startup Stand OutDarren Menabney
 
The Real Threats to Your Presentation Success
The Real Threats to Your Presentation SuccessThe Real Threats to Your Presentation Success
The Real Threats to Your Presentation SuccessMichelle Mazur
 
The "Why Should You Talk" Talk
The "Why Should You Talk" TalkThe "Why Should You Talk" Talk
The "Why Should You Talk" TalkChristian Grobmeier
 
Everything You're Doing Wrong With Your Pitch & How To Fix It
Everything You're Doing Wrong With Your Pitch & How To Fix ItEverything You're Doing Wrong With Your Pitch & How To Fix It
Everything You're Doing Wrong With Your Pitch & How To Fix ItClark Van Der Beken
 
Public Speaking: The Anatomy of a Speech
Public Speaking: The Anatomy of a SpeechPublic Speaking: The Anatomy of a Speech
Public Speaking: The Anatomy of a SpeechNandini Doreswamy
 

Recomendados

Difference between php and node
Difference between php and nodeDifference between php and node
Difference between php and nodeNguyễn Ngọc Phương
 
Navegacao de Robótica Autônoma em Ambientes Dinâmicos
Navegacao de Robótica Autônoma em Ambientes DinâmicosNavegacao de Robótica Autônoma em Ambientes Dinâmicos
Navegacao de Robótica Autônoma em Ambientes DinâmicosLCoN Mackenzie
 
Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...
Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...
Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...Brussels, Belgium
 
Pitching With Passion, Make Your Startup Stand Out
Pitching With Passion, Make Your Startup Stand OutPitching With Passion, Make Your Startup Stand Out
Pitching With Passion, Make Your Startup Stand OutDarren Menabney
 
The Real Threats to Your Presentation Success
The Real Threats to Your Presentation SuccessThe Real Threats to Your Presentation Success
The Real Threats to Your Presentation SuccessMichelle Mazur
 
The "Why Should You Talk" Talk
The "Why Should You Talk" TalkThe "Why Should You Talk" Talk
The "Why Should You Talk" TalkChristian Grobmeier
 
Everything You're Doing Wrong With Your Pitch & How To Fix It
Everything You're Doing Wrong With Your Pitch & How To Fix ItEverything You're Doing Wrong With Your Pitch & How To Fix It
Everything You're Doing Wrong With Your Pitch & How To Fix ItClark Van Der Beken
 
Public Speaking: The Anatomy of a Speech
Public Speaking: The Anatomy of a SpeechPublic Speaking: The Anatomy of a Speech
Public Speaking: The Anatomy of a SpeechNandini Doreswamy
 
Formula for a Winning Presentation Design
Formula for a Winning Presentation DesignFormula for a Winning Presentation Design
Formula for a Winning Presentation DesignPGi
 
10 Easy to Follow Tips for Presenting to Senior Executives
10 Easy to Follow Tips for Presenting to Senior Executives10 Easy to Follow Tips for Presenting to Senior Executives
10 Easy to Follow Tips for Presenting to Senior ExecutivesSketchBubble
 
The Secrets of Delivering Impacftul Presentations #ImpactfulPrez
The Secrets of Delivering Impacftul Presentations #ImpactfulPrezThe Secrets of Delivering Impacftul Presentations #ImpactfulPrez
The Secrets of Delivering Impacftul Presentations #ImpactfulPrezHavain
 
Visual Silence can help your PowerPoint slides
Visual Silence can help your PowerPoint slidesVisual Silence can help your PowerPoint slides
Visual Silence can help your PowerPoint slidesSlides | Presentation Design Agency
 
8 Successful Ways to Open Your Speech
8 Successful Ways to Open Your Speech8 Successful Ways to Open Your Speech
8 Successful Ways to Open Your SpeechSketchBubble
 
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...Empowered Presentations
 
Epic Presentation Quotes to Fire You Up!
Epic Presentation Quotes to Fire You Up!Epic Presentation Quotes to Fire You Up!
Epic Presentation Quotes to Fire You Up!Anuj Malhotra
 
Como Fazer Referências Bibliográficas
Como Fazer Referências BibliográficasComo Fazer Referências Bibliográficas
Como Fazer Referências Bibliográficasbela610
 
13 Alternatives to Using Bullet Points in Presentations
13 Alternatives to Using Bullet Points in Presentations13 Alternatives to Using Bullet Points in Presentations
13 Alternatives to Using Bullet Points in PresentationsMelissa Milloway, MSIT
 
THE PRESENTATION DESIGN CRASH COURSE
THE PRESENTATION DESIGN CRASH COURSETHE PRESENTATION DESIGN CRASH COURSE
THE PRESENTATION DESIGN CRASH COURSEJoseph Gelman
 
7 Myths When Answering Tough Questions During Presentations
7 Myths When Answering Tough Questions During Presentations7 Myths When Answering Tough Questions During Presentations
7 Myths When Answering Tough Questions During Presentations24Slides
 
21 Tips for Creating a Boring Presentation
21 Tips for Creating a Boring Presentation21 Tips for Creating a Boring Presentation
21 Tips for Creating a Boring PresentationSketchBubble
 
Secrets to impactful presentations volume 2 - 8 tools
Secrets to impactful presentations volume 2 - 8 toolsSecrets to impactful presentations volume 2 - 8 tools
Secrets to impactful presentations volume 2 - 8 toolsHavain
 
Screw You Bullet Points! [Rest in Peace]
Screw You Bullet Points! [Rest in Peace]Screw You Bullet Points! [Rest in Peace]
Screw You Bullet Points! [Rest in Peace]Ayman Sadiq
 
How to Make your Slides More Memorable?
How to Make your Slides More Memorable?How to Make your Slides More Memorable?
How to Make your Slides More Memorable?SketchBubble
 
Content Marketing Strategy Workshop
Content Marketing Strategy WorkshopContent Marketing Strategy Workshop
Content Marketing Strategy WorkshopMichael Brenner
 
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...David Nihill
 
Robots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml CreationRobots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml CreationJahid Hasan
 
Article19
Article19Article19
Article19egrowtech
 
SEO Robots txt FILE
SEO Robots txt FILESEO Robots txt FILE
SEO Robots txt FILEPriyanka Bhatia
 
Vorlesung "Web-Technologies"
Vorlesung "Web-Technologies" Vorlesung "Web-Technologies"
Vorlesung "Web-Technologies" Wolfgang Wiese
 
From zero to almost rails in about a million slides...
From zero to almost rails in about a million slides...From zero to almost rails in about a million slides...
From zero to almost rails in about a million slides...david_e_worth
 

Mais conteúdo relacionado

Destaque

Formula for a Winning Presentation Design
Formula for a Winning Presentation DesignFormula for a Winning Presentation Design
Formula for a Winning Presentation DesignPGi
 
10 Easy to Follow Tips for Presenting to Senior Executives
10 Easy to Follow Tips for Presenting to Senior Executives10 Easy to Follow Tips for Presenting to Senior Executives
10 Easy to Follow Tips for Presenting to Senior ExecutivesSketchBubble
 
The Secrets of Delivering Impacftul Presentations #ImpactfulPrez
The Secrets of Delivering Impacftul Presentations #ImpactfulPrezThe Secrets of Delivering Impacftul Presentations #ImpactfulPrez
The Secrets of Delivering Impacftul Presentations #ImpactfulPrezHavain
 
8 Successful Ways to Open Your Speech
8 Successful Ways to Open Your Speech8 Successful Ways to Open Your Speech
8 Successful Ways to Open Your SpeechSketchBubble
 
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...Empowered Presentations
 
Epic Presentation Quotes to Fire You Up!
Epic Presentation Quotes to Fire You Up!Epic Presentation Quotes to Fire You Up!
Epic Presentation Quotes to Fire You Up!Anuj Malhotra
 
Como Fazer Referências Bibliográficas
Como Fazer Referências BibliográficasComo Fazer Referências Bibliográficas
Como Fazer Referências Bibliográficasbela610
 
13 Alternatives to Using Bullet Points in Presentations
13 Alternatives to Using Bullet Points in Presentations13 Alternatives to Using Bullet Points in Presentations
13 Alternatives to Using Bullet Points in PresentationsMelissa Milloway, MSIT
 
THE PRESENTATION DESIGN CRASH COURSE
THE PRESENTATION DESIGN CRASH COURSETHE PRESENTATION DESIGN CRASH COURSE
THE PRESENTATION DESIGN CRASH COURSEJoseph Gelman
 
7 Myths When Answering Tough Questions During Presentations
7 Myths When Answering Tough Questions During Presentations7 Myths When Answering Tough Questions During Presentations
7 Myths When Answering Tough Questions During Presentations24Slides
 
21 Tips for Creating a Boring Presentation
21 Tips for Creating a Boring Presentation21 Tips for Creating a Boring Presentation
21 Tips for Creating a Boring PresentationSketchBubble
 
Secrets to impactful presentations volume 2 - 8 tools
Secrets to impactful presentations volume 2 - 8 toolsSecrets to impactful presentations volume 2 - 8 tools
Secrets to impactful presentations volume 2 - 8 toolsHavain
 
Screw You Bullet Points! [Rest in Peace]
Screw You Bullet Points! [Rest in Peace]Screw You Bullet Points! [Rest in Peace]
Screw You Bullet Points! [Rest in Peace]Ayman Sadiq
 
How to Make your Slides More Memorable?
How to Make your Slides More Memorable?How to Make your Slides More Memorable?
How to Make your Slides More Memorable?SketchBubble
 
Content Marketing Strategy Workshop
Content Marketing Strategy WorkshopContent Marketing Strategy Workshop
Content Marketing Strategy WorkshopMichael Brenner
 
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...David Nihill
 

Destaque (17)

Formula for a Winning Presentation Design
Formula for a Winning Presentation DesignFormula for a Winning Presentation Design
Formula for a Winning Presentation Design
 
10 Easy to Follow Tips for Presenting to Senior Executives
10 Easy to Follow Tips for Presenting to Senior Executives10 Easy to Follow Tips for Presenting to Senior Executives
10 Easy to Follow Tips for Presenting to Senior Executives
 
The Secrets of Delivering Impacftul Presentations #ImpactfulPrez
The Secrets of Delivering Impacftul Presentations #ImpactfulPrezThe Secrets of Delivering Impacftul Presentations #ImpactfulPrez
The Secrets of Delivering Impacftul Presentations #ImpactfulPrez
 
Visual Silence can help your PowerPoint slides
Visual Silence can help your PowerPoint slidesVisual Silence can help your PowerPoint slides
Visual Silence can help your PowerPoint slides
 
8 Successful Ways to Open Your Speech
8 Successful Ways to Open Your Speech8 Successful Ways to Open Your Speech
8 Successful Ways to Open Your Speech
 
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
 
Epic Presentation Quotes to Fire You Up!
Epic Presentation Quotes to Fire You Up!Epic Presentation Quotes to Fire You Up!
Epic Presentation Quotes to Fire You Up!
 
Como Fazer Referências Bibliográficas
Como Fazer Referências BibliográficasComo Fazer Referências Bibliográficas
Como Fazer Referências Bibliográficas
 
13 Alternatives to Using Bullet Points in Presentations
13 Alternatives to Using Bullet Points in Presentations13 Alternatives to Using Bullet Points in Presentations
13 Alternatives to Using Bullet Points in Presentations
 
THE PRESENTATION DESIGN CRASH COURSE
THE PRESENTATION DESIGN CRASH COURSETHE PRESENTATION DESIGN CRASH COURSE
THE PRESENTATION DESIGN CRASH COURSE
 
7 Myths When Answering Tough Questions During Presentations
7 Myths When Answering Tough Questions During Presentations7 Myths When Answering Tough Questions During Presentations
7 Myths When Answering Tough Questions During Presentations
 
21 Tips for Creating a Boring Presentation
21 Tips for Creating a Boring Presentation21 Tips for Creating a Boring Presentation
21 Tips for Creating a Boring Presentation
 
Secrets to impactful presentations volume 2 - 8 tools
Secrets to impactful presentations volume 2 - 8 toolsSecrets to impactful presentations volume 2 - 8 tools
Secrets to impactful presentations volume 2 - 8 tools
 
Screw You Bullet Points! [Rest in Peace]
Screw You Bullet Points! [Rest in Peace]Screw You Bullet Points! [Rest in Peace]
Screw You Bullet Points! [Rest in Peace]
 
How to Make your Slides More Memorable?
How to Make your Slides More Memorable?How to Make your Slides More Memorable?
How to Make your Slides More Memorable?
 
Content Marketing Strategy Workshop
Content Marketing Strategy WorkshopContent Marketing Strategy Workshop
Content Marketing Strategy Workshop
 
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
 

Semelhante a Introduction to robots.txt

Robots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml CreationRobots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml CreationJahid Hasan
 
Vorlesung "Web-Technologies"
Vorlesung "Web-Technologies" Vorlesung "Web-Technologies"
Vorlesung "Web-Technologies" Wolfgang Wiese
 
From zero to almost rails in about a million slides...
From zero to almost rails in about a million slides...From zero to almost rails in about a million slides...
From zero to almost rails in about a million slides...david_e_worth
 
Difference between robots txt file, meta robots, X-robots tag
Difference between robots txt file, meta robots, X-robots tagDifference between robots txt file, meta robots, X-robots tag
Difference between robots txt file, meta robots, X-robots tagParidhi Infotech
 
E-commerce Primer For Newbies (Not Dummies)
E-commerce Primer For Newbies (Not Dummies)E-commerce Primer For Newbies (Not Dummies)
E-commerce Primer For Newbies (Not Dummies)webhostingguy
 
What is a Robot txt file?
What is a Robot txt file?What is a Robot txt file?
What is a Robot txt file?Abhishek Mitra
 
E-commerce Primer For Newbies
E-commerce Primer For NewbiesE-commerce Primer For Newbies
E-commerce Primer For NewbiesPeter Fisher
 
Integrating RSS Into Your Web Site
Integrating RSS Into Your Web SiteIntegrating RSS Into Your Web Site
Integrating RSS Into Your Web SiteMichael Sauers
 
Robot operating system [ROS]
Robot operating system [ROS]Robot operating system [ROS]
Robot operating system [ROS]Abrar Mohamed
 
Determining Bias to Search Engines from Robots.txt
Determining Bias to Search Engines from Robots.txtDetermining Bias to Search Engines from Robots.txt
Determining Bias to Search Engines from Robots.txtnitchmarketing
 
XML Sitemap and Robots.TXT Guide for SEO Beginners
XML Sitemap and Robots.TXT Guide for SEO BeginnersXML Sitemap and Robots.TXT Guide for SEO Beginners
XML Sitemap and Robots.TXT Guide for SEO BeginnersAditya Todawal
 
Introduction to "robots.txt
Introduction to "robots.txtIntroduction to "robots.txt
Introduction to "robots.txtIshan Mishra
 
Robots.txt - Control What Crawler Can See
Robots.txt - Control What Crawler Can SeeRobots.txt - Control What Crawler Can See
Robots.txt - Control What Crawler Can SeeLets Get Digital
 
Internert workshop
Internert workshop Internert workshop
Internert workshop sofiag000414
 
Integrating RSS into Your Web site
Integrating RSS into Your Web siteIntegrating RSS into Your Web site
Integrating RSS into Your Web siteMichael Sauers
 

Semelhante a Introduction to robots.txt (20)

Robots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml CreationRobots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml Creation
 
Article19
Article19Article19
Article19
 
SEO Robots txt FILE
SEO Robots txt FILESEO Robots txt FILE
SEO Robots txt FILE
 
Vorlesung "Web-Technologies"
Vorlesung "Web-Technologies" Vorlesung "Web-Technologies"
Vorlesung "Web-Technologies"
 
From zero to almost rails in about a million slides...
From zero to almost rails in about a million slides...From zero to almost rails in about a million slides...
From zero to almost rails in about a million slides...
 
Difference between robots txt file, meta robots, X-robots tag
Difference between robots txt file, meta robots, X-robots tagDifference between robots txt file, meta robots, X-robots tag
Difference between robots txt file, meta robots, X-robots tag
 
E-commerce Primer For Newbies (Not Dummies)
E-commerce Primer For Newbies (Not Dummies)E-commerce Primer For Newbies (Not Dummies)
E-commerce Primer For Newbies (Not Dummies)
 
What is a Robot txt file?
What is a Robot txt file?What is a Robot txt file?
What is a Robot txt file?
 
Helping Things to REST
Helping Things to RESTHelping Things to REST
Helping Things to REST
 
E-commerce Primer For Newbies
E-commerce Primer For NewbiesE-commerce Primer For Newbies
E-commerce Primer For Newbies
 
Integrating RSS Into Your Web Site
Integrating RSS Into Your Web SiteIntegrating RSS Into Your Web Site
Integrating RSS Into Your Web Site
 
Technical SEO
Technical SEOTechnical SEO
Technical SEO
 
Integrating RSS Into Your Web Site
Integrating RSS Into Your Web SiteIntegrating RSS Into Your Web Site
Integrating RSS Into Your Web Site
 
Robot operating system [ROS]
Robot operating system [ROS]Robot operating system [ROS]
Robot operating system [ROS]
 
Determining Bias to Search Engines from Robots.txt
Determining Bias to Search Engines from Robots.txtDetermining Bias to Search Engines from Robots.txt
Determining Bias to Search Engines from Robots.txt
 
XML Sitemap and Robots.TXT Guide for SEO Beginners
XML Sitemap and Robots.TXT Guide for SEO BeginnersXML Sitemap and Robots.TXT Guide for SEO Beginners
XML Sitemap and Robots.TXT Guide for SEO Beginners
 
Introduction to "robots.txt
Introduction to "robots.txtIntroduction to "robots.txt
Introduction to "robots.txt
 
Robots.txt - Control What Crawler Can See
Robots.txt - Control What Crawler Can SeeRobots.txt - Control What Crawler Can See
Robots.txt - Control What Crawler Can See
 
Internert workshop
Internert workshop Internert workshop
Internert workshop
 
Integrating RSS into Your Web site
Integrating RSS into Your Web siteIntegrating RSS into Your Web site
Integrating RSS into Your Web site
 

Mais de madamseo

Meta Tags Optimization (Extended)
Meta Tags Optimization (Extended)Meta Tags Optimization (Extended)
Meta Tags Optimization (Extended)madamseo
 
Advert Funny
Advert Funny Advert Funny
Advert Funny madamseo
 
Meta Tags Optimization
Meta Tags OptimizationMeta Tags Optimization
Meta Tags Optimizationmadamseo
 
Beginners Guide to Search Engine Marketing
Beginners Guide to Search Engine MarketingBeginners Guide to Search Engine Marketing
Beginners Guide to Search Engine Marketingmadamseo
 
Social Media From Scratch
Social Media From ScratchSocial Media From Scratch
Social Media From Scratchmadamseo
 
Social Media Marketing - Today and Tomorrow...
Social Media Marketing - Today and Tomorrow...Social Media Marketing - Today and Tomorrow...
Social Media Marketing - Today and Tomorrow...madamseo
 
How does Google...?
How does Google...?How does Google...?
How does Google...?madamseo
 
Internet Marketing with WordPress
Internet Marketing with WordPressInternet Marketing with WordPress
Internet Marketing with WordPressmadamseo
 
Digital Marketing Communication
Digital Marketing CommunicationDigital Marketing Communication
Digital Marketing Communicationmadamseo
 
Presentation
PresentationPresentation
Presentationmadamseo
 

Mais de madamseo (10)

Meta Tags Optimization (Extended)
Meta Tags Optimization (Extended)Meta Tags Optimization (Extended)
Meta Tags Optimization (Extended)
 
Advert Funny
Advert Funny Advert Funny
Advert Funny
 
Meta Tags Optimization
Meta Tags OptimizationMeta Tags Optimization
Meta Tags Optimization
 
Beginners Guide to Search Engine Marketing
Beginners Guide to Search Engine MarketingBeginners Guide to Search Engine Marketing
Beginners Guide to Search Engine Marketing
 
Social Media From Scratch
Social Media From ScratchSocial Media From Scratch
Social Media From Scratch
 
Social Media Marketing - Today and Tomorrow...
Social Media Marketing - Today and Tomorrow...Social Media Marketing - Today and Tomorrow...
Social Media Marketing - Today and Tomorrow...
 
How does Google...?
How does Google...?How does Google...?
How does Google...?
 
Internet Marketing with WordPress
Internet Marketing with WordPressInternet Marketing with WordPress
Internet Marketing with WordPress
 
Digital Marketing Communication
Digital Marketing CommunicationDigital Marketing Communication
Digital Marketing Communication
 
Presentation
PresentationPresentation
Presentation
 

Último

Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?Antenna Manufacturer Coco
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsJoaquim Jorge
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Servicegiselly40
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc
 

Último (20)

Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 

Introduction to robots.txt

  • 2. • Robots.txt is a plain text file that you upload to the root directory of your site. Once the web spiders (ants, boots, indexers) that index your webpages reach your site, they first look at that text file and process it. Put differently, robots.txt says to the spider which pages to crawl.
  • 3.
  • 4. THE SIMPLESTVERSION OF ROBOTS.TXT FILE IS: USER-AGENT:* DISALLOW: THE FIRST LINE INDICATESTHE USER AGENT- ASTERISK INDICATES THATTHE FOLLOWING LINES APPLYTO ALL AGENTS. SPACE AFTER "DISALLOW:" MEANSTHAT NOTHING IS LIMITED. THIS ROBOTS.TXT FILE DOES NOTHING- IT ALLOWS USER AGENTSTO SEE EVERYTHING ONTHE SITE.
  • 5. NOW LET'S MAKE IT A LITTLE MORE COMPLICATED-THISTIME WE DO NOT WANT SPIDERSTO CRAWL IN OUR /FAQ DIRECTORY: USER-AGENT:* DISALLOW: /FAQ/ IT IS RELATIVELY EASY. SLASH INDICATESTHATTHIS IS A DIRECTORY. IFYOU DON'T PUT A SLASH, NOT ONLY/FAQ DIRECTORY, BUT ALSO EACH FILE WHICH NAME BEGINS WITH "FAQ" WILL BE BANNED. ALTERNATIVELY, YOU CAN ADD MORE DIRECTORIESTOYOUR BANNED LIST.
  • 6. VIEWTHE FULL PRESENTATION AT WWW.MADAMSEO.COM OR SUBSCRIBETO MY PAID COURSE ON WWW.UDEMY.COM