SlideShare uma empresa Scribd logo

Introduction to robots.txt

•
•
madamseo
madamseo
TecnologiaNegócios
1 de 6
Baixar para ler offline
INTRODUCTIONTO ROBOTS.TXT
• Robots.txt is a plain text file that you upload to the root directory of your site. Once the web spiders (ants, boots, indexers) that index your webpages reach your site, they first look at that text file and process it. Put differently, robots.txt says to the spider which pages to crawl.
THE SIMPLESTVERSION OF ROBOTS.TXT FILE IS: USER-AGENT:* DISALLOW: THE FIRST LINE INDICATESTHE USER AGENT- ASTERISK INDICATES THATTHE FOLLOWING LINES APPLYTO ALL AGENTS. SPACE AFTER "DISALLOW:" MEANSTHAT NOTHING IS LIMITED. THIS ROBOTS.TXT FILE DOES NOTHING- IT ALLOWS USER AGENTSTO SEE EVERYTHING ONTHE SITE.
NOW LET'S MAKE IT A LITTLE MORE COMPLICATED-THISTIME WE DO NOT WANT SPIDERSTO CRAWL IN OUR /FAQ DIRECTORY: USER-AGENT:* DISALLOW: /FAQ/ IT IS RELATIVELY EASY. SLASH INDICATESTHATTHIS IS A DIRECTORY. IFYOU DON'T PUT A SLASH, NOT ONLY/FAQ DIRECTORY, BUT ALSO EACH FILE WHICH NAME BEGINS WITH "FAQ" WILL BE BANNED. ALTERNATIVELY, YOU CAN ADD MORE DIRECTORIESTOYOUR BANNED LIST.
VIEWTHE FULL PRESENTATION AT WWW.MADAMSEO.COM OR SUBSCRIBETO MY PAID COURSE ON WWW.UDEMY.COM

Recomendados

Difference between php and node
Difference between php and nodeDifference between php and node
Difference between php and nodeNguyễn Ngọc Phương
 
Navegacao de Robótica Autônoma em Ambientes Dinâmicos
Navegacao de Robótica Autônoma em Ambientes DinâmicosNavegacao de Robótica Autônoma em Ambientes Dinâmicos
Navegacao de Robótica Autônoma em Ambientes DinâmicosLCoN Mackenzie
 
Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...
Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...
Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...Brussels, Belgium
 
Pitching With Passion, Make Your Startup Stand Out
Pitching With Passion, Make Your Startup Stand OutPitching With Passion, Make Your Startup Stand Out
Pitching With Passion, Make Your Startup Stand OutDarren Menabney
 
The Real Threats to Your Presentation Success
The Real Threats to Your Presentation SuccessThe Real Threats to Your Presentation Success
The Real Threats to Your Presentation SuccessMichelle Mazur
 
The "Why Should You Talk" Talk
The "Why Should You Talk" TalkThe "Why Should You Talk" Talk
The "Why Should You Talk" TalkChristian Grobmeier
 
Everything You're Doing Wrong With Your Pitch & How To Fix It
Everything You're Doing Wrong With Your Pitch & How To Fix ItEverything You're Doing Wrong With Your Pitch & How To Fix It
Everything You're Doing Wrong With Your Pitch & How To Fix ItClark Van Der Beken
 
Public Speaking: The Anatomy of a Speech
Public Speaking: The Anatomy of a SpeechPublic Speaking: The Anatomy of a Speech
Public Speaking: The Anatomy of a SpeechNandini Doreswamy
 

Recomendados

Difference between php and node
Difference between php and nodeDifference between php and node
Difference between php and nodeNguyễn Ngọc Phương
 
Navegacao de Robótica Autônoma em Ambientes Dinâmicos
Navegacao de Robótica Autônoma em Ambientes DinâmicosNavegacao de Robótica Autônoma em Ambientes Dinâmicos
Navegacao de Robótica Autônoma em Ambientes DinâmicosLCoN Mackenzie
 
Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...
Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...
Scientix 9th SPNE Brussels 6 November 2015: Karel - Autonomous Robot for Enha...Brussels, Belgium
 
Pitching With Passion, Make Your Startup Stand Out
Pitching With Passion, Make Your Startup Stand OutPitching With Passion, Make Your Startup Stand Out
Pitching With Passion, Make Your Startup Stand OutDarren Menabney
 
The Real Threats to Your Presentation Success
The Real Threats to Your Presentation SuccessThe Real Threats to Your Presentation Success
The Real Threats to Your Presentation SuccessMichelle Mazur
 
The "Why Should You Talk" Talk
The "Why Should You Talk" TalkThe "Why Should You Talk" Talk
The "Why Should You Talk" TalkChristian Grobmeier
 
Everything You're Doing Wrong With Your Pitch & How To Fix It
Everything You're Doing Wrong With Your Pitch & How To Fix ItEverything You're Doing Wrong With Your Pitch & How To Fix It
Everything You're Doing Wrong With Your Pitch & How To Fix ItClark Van Der Beken
 
Public Speaking: The Anatomy of a Speech
Public Speaking: The Anatomy of a SpeechPublic Speaking: The Anatomy of a Speech
Public Speaking: The Anatomy of a SpeechNandini Doreswamy
 
Formula for a Winning Presentation Design
Formula for a Winning Presentation DesignFormula for a Winning Presentation Design
Formula for a Winning Presentation DesignPGi
 
10 Easy to Follow Tips for Presenting to Senior Executives
10 Easy to Follow Tips for Presenting to Senior Executives10 Easy to Follow Tips for Presenting to Senior Executives
10 Easy to Follow Tips for Presenting to Senior ExecutivesSketchBubble
 
The Secrets of Delivering Impacftul Presentations #ImpactfulPrez
The Secrets of Delivering Impacftul Presentations #ImpactfulPrezThe Secrets of Delivering Impacftul Presentations #ImpactfulPrez
The Secrets of Delivering Impacftul Presentations #ImpactfulPrezHavain
 
Visual Silence can help your PowerPoint slides
Visual Silence can help your PowerPoint slidesVisual Silence can help your PowerPoint slides
Visual Silence can help your PowerPoint slidesSlides | Presentation Design Agency
 
8 Successful Ways to Open Your Speech
8 Successful Ways to Open Your Speech8 Successful Ways to Open Your Speech
8 Successful Ways to Open Your SpeechSketchBubble
 
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...Empowered Presentations
 
Epic Presentation Quotes to Fire You Up!
Epic Presentation Quotes to Fire You Up!Epic Presentation Quotes to Fire You Up!
Epic Presentation Quotes to Fire You Up!Anuj Malhotra
 
Como Fazer Referências Bibliográficas
Como Fazer Referências BibliográficasComo Fazer Referências Bibliográficas
Como Fazer Referências Bibliográficasbela610
 
13 Alternatives to Using Bullet Points in Presentations
13 Alternatives to Using Bullet Points in Presentations13 Alternatives to Using Bullet Points in Presentations
13 Alternatives to Using Bullet Points in PresentationsMelissa Milloway, MSIT
 
THE PRESENTATION DESIGN CRASH COURSE
THE PRESENTATION DESIGN CRASH COURSETHE PRESENTATION DESIGN CRASH COURSE
THE PRESENTATION DESIGN CRASH COURSEJoseph Gelman
 
7 Myths When Answering Tough Questions During Presentations
7 Myths When Answering Tough Questions During Presentations7 Myths When Answering Tough Questions During Presentations
7 Myths When Answering Tough Questions During Presentations24Slides
 
21 Tips for Creating a Boring Presentation
21 Tips for Creating a Boring Presentation21 Tips for Creating a Boring Presentation
21 Tips for Creating a Boring PresentationSketchBubble
 
Secrets to impactful presentations volume 2 - 8 tools
Secrets to impactful presentations volume 2 - 8 toolsSecrets to impactful presentations volume 2 - 8 tools
Secrets to impactful presentations volume 2 - 8 toolsHavain
 
Screw You Bullet Points! [Rest in Peace]
Screw You Bullet Points! [Rest in Peace]Screw You Bullet Points! [Rest in Peace]
Screw You Bullet Points! [Rest in Peace]Ayman Sadiq
 
How to Make your Slides More Memorable?
How to Make your Slides More Memorable?How to Make your Slides More Memorable?
How to Make your Slides More Memorable?SketchBubble
 
Content Marketing Strategy Workshop
Content Marketing Strategy WorkshopContent Marketing Strategy Workshop
Content Marketing Strategy WorkshopMichael Brenner
 
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...David Nihill
 
Robots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml CreationRobots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml CreationJahid Hasan
 
Article19
Article19Article19
Article19egrowtech
 
SEO Robots txt FILE
SEO Robots txt FILESEO Robots txt FILE
SEO Robots txt FILEPriyanka Bhatia
 
Vorlesung "Web-Technologies"
Vorlesung "Web-Technologies" Vorlesung "Web-Technologies"
Vorlesung "Web-Technologies" Wolfgang Wiese
 
From zero to almost rails in about a million slides...
From zero to almost rails in about a million slides...From zero to almost rails in about a million slides...
From zero to almost rails in about a million slides...david_e_worth
 

Mais conteúdo relacionado

Destaque

Formula for a Winning Presentation Design
Formula for a Winning Presentation DesignFormula for a Winning Presentation Design
Formula for a Winning Presentation DesignPGi
 
10 Easy to Follow Tips for Presenting to Senior Executives
10 Easy to Follow Tips for Presenting to Senior Executives10 Easy to Follow Tips for Presenting to Senior Executives
10 Easy to Follow Tips for Presenting to Senior ExecutivesSketchBubble
 
The Secrets of Delivering Impacftul Presentations #ImpactfulPrez
The Secrets of Delivering Impacftul Presentations #ImpactfulPrezThe Secrets of Delivering Impacftul Presentations #ImpactfulPrez
The Secrets of Delivering Impacftul Presentations #ImpactfulPrezHavain
 
8 Successful Ways to Open Your Speech
8 Successful Ways to Open Your Speech8 Successful Ways to Open Your Speech
8 Successful Ways to Open Your SpeechSketchBubble
 
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...Empowered Presentations
 
Epic Presentation Quotes to Fire You Up!
Epic Presentation Quotes to Fire You Up!Epic Presentation Quotes to Fire You Up!
Epic Presentation Quotes to Fire You Up!Anuj Malhotra
 
Como Fazer Referências Bibliográficas
Como Fazer Referências BibliográficasComo Fazer Referências Bibliográficas
Como Fazer Referências Bibliográficasbela610
 
13 Alternatives to Using Bullet Points in Presentations
13 Alternatives to Using Bullet Points in Presentations13 Alternatives to Using Bullet Points in Presentations
13 Alternatives to Using Bullet Points in PresentationsMelissa Milloway, MSIT
 
THE PRESENTATION DESIGN CRASH COURSE
THE PRESENTATION DESIGN CRASH COURSETHE PRESENTATION DESIGN CRASH COURSE
THE PRESENTATION DESIGN CRASH COURSEJoseph Gelman
 
7 Myths When Answering Tough Questions During Presentations
7 Myths When Answering Tough Questions During Presentations7 Myths When Answering Tough Questions During Presentations
7 Myths When Answering Tough Questions During Presentations24Slides
 
21 Tips for Creating a Boring Presentation
21 Tips for Creating a Boring Presentation21 Tips for Creating a Boring Presentation
21 Tips for Creating a Boring PresentationSketchBubble
 
Secrets to impactful presentations volume 2 - 8 tools
Secrets to impactful presentations volume 2 - 8 toolsSecrets to impactful presentations volume 2 - 8 tools
Secrets to impactful presentations volume 2 - 8 toolsHavain
 
Screw You Bullet Points! [Rest in Peace]
Screw You Bullet Points! [Rest in Peace]Screw You Bullet Points! [Rest in Peace]
Screw You Bullet Points! [Rest in Peace]Ayman Sadiq
 
How to Make your Slides More Memorable?
How to Make your Slides More Memorable?How to Make your Slides More Memorable?
How to Make your Slides More Memorable?SketchBubble
 
Content Marketing Strategy Workshop
Content Marketing Strategy WorkshopContent Marketing Strategy Workshop
Content Marketing Strategy WorkshopMichael Brenner
 
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...David Nihill
 

Destaque (17)

Formula for a Winning Presentation Design
Formula for a Winning Presentation DesignFormula for a Winning Presentation Design
Formula for a Winning Presentation Design
 
10 Easy to Follow Tips for Presenting to Senior Executives
10 Easy to Follow Tips for Presenting to Senior Executives10 Easy to Follow Tips for Presenting to Senior Executives
10 Easy to Follow Tips for Presenting to Senior Executives
 
The Secrets of Delivering Impacftul Presentations #ImpactfulPrez
The Secrets of Delivering Impacftul Presentations #ImpactfulPrezThe Secrets of Delivering Impacftul Presentations #ImpactfulPrez
The Secrets of Delivering Impacftul Presentations #ImpactfulPrez
 
Visual Silence can help your PowerPoint slides
Visual Silence can help your PowerPoint slidesVisual Silence can help your PowerPoint slides
Visual Silence can help your PowerPoint slides
 
8 Successful Ways to Open Your Speech
8 Successful Ways to Open Your Speech8 Successful Ways to Open Your Speech
8 Successful Ways to Open Your Speech
 
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
Prior to Your Pitch - 4 things to consider before presenting your #Startups t...
 
Epic Presentation Quotes to Fire You Up!
Epic Presentation Quotes to Fire You Up!Epic Presentation Quotes to Fire You Up!
Epic Presentation Quotes to Fire You Up!
 
Como Fazer Referências Bibliográficas
Como Fazer Referências BibliográficasComo Fazer Referências Bibliográficas
Como Fazer Referências Bibliográficas
 
13 Alternatives to Using Bullet Points in Presentations
13 Alternatives to Using Bullet Points in Presentations13 Alternatives to Using Bullet Points in Presentations
13 Alternatives to Using Bullet Points in Presentations
 
THE PRESENTATION DESIGN CRASH COURSE
THE PRESENTATION DESIGN CRASH COURSETHE PRESENTATION DESIGN CRASH COURSE
THE PRESENTATION DESIGN CRASH COURSE
 
7 Myths When Answering Tough Questions During Presentations
7 Myths When Answering Tough Questions During Presentations7 Myths When Answering Tough Questions During Presentations
7 Myths When Answering Tough Questions During Presentations
 
21 Tips for Creating a Boring Presentation
21 Tips for Creating a Boring Presentation21 Tips for Creating a Boring Presentation
21 Tips for Creating a Boring Presentation
 
Secrets to impactful presentations volume 2 - 8 tools
Secrets to impactful presentations volume 2 - 8 toolsSecrets to impactful presentations volume 2 - 8 tools
Secrets to impactful presentations volume 2 - 8 tools
 
Screw You Bullet Points! [Rest in Peace]
Screw You Bullet Points! [Rest in Peace]Screw You Bullet Points! [Rest in Peace]
Screw You Bullet Points! [Rest in Peace]
 
How to Make your Slides More Memorable?
How to Make your Slides More Memorable?How to Make your Slides More Memorable?
How to Make your Slides More Memorable?
 
Content Marketing Strategy Workshop
Content Marketing Strategy WorkshopContent Marketing Strategy Workshop
Content Marketing Strategy Workshop
 
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
23 Tips From Comedians to Be Funnier in Your Next Presentation (via the book ...
 

Semelhante a Introduction to robots.txt

Robots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml CreationRobots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml CreationJahid Hasan
 
Vorlesung "Web-Technologies"
Vorlesung "Web-Technologies" Vorlesung "Web-Technologies"
Vorlesung "Web-Technologies" Wolfgang Wiese
 
From zero to almost rails in about a million slides...
From zero to almost rails in about a million slides...From zero to almost rails in about a million slides...
From zero to almost rails in about a million slides...david_e_worth
 
Difference between robots txt file, meta robots, X-robots tag
Difference between robots txt file, meta robots, X-robots tagDifference between robots txt file, meta robots, X-robots tag
Difference between robots txt file, meta robots, X-robots tagParidhi Infotech
 
E-commerce Primer For Newbies (Not Dummies)
E-commerce Primer For Newbies (Not Dummies)E-commerce Primer For Newbies (Not Dummies)
E-commerce Primer For Newbies (Not Dummies)webhostingguy
 
What is a Robot txt file?
What is a Robot txt file?What is a Robot txt file?
What is a Robot txt file?Abhishek Mitra
 
E-commerce Primer For Newbies
E-commerce Primer For NewbiesE-commerce Primer For Newbies
E-commerce Primer For NewbiesPeter Fisher
 
Integrating RSS Into Your Web Site
Integrating RSS Into Your Web SiteIntegrating RSS Into Your Web Site
Integrating RSS Into Your Web SiteMichael Sauers
 
Robot operating system [ROS]
Robot operating system [ROS]Robot operating system [ROS]
Robot operating system [ROS]Abrar Mohamed
 
Determining Bias to Search Engines from Robots.txt
Determining Bias to Search Engines from Robots.txtDetermining Bias to Search Engines from Robots.txt
Determining Bias to Search Engines from Robots.txtnitchmarketing
 
XML Sitemap and Robots.TXT Guide for SEO Beginners
XML Sitemap and Robots.TXT Guide for SEO BeginnersXML Sitemap and Robots.TXT Guide for SEO Beginners
XML Sitemap and Robots.TXT Guide for SEO BeginnersAditya Todawal
 
Introduction to "robots.txt
Introduction to "robots.txtIntroduction to "robots.txt
Introduction to "robots.txtIshan Mishra
 
Robots.txt - Control What Crawler Can See
Robots.txt - Control What Crawler Can SeeRobots.txt - Control What Crawler Can See
Robots.txt - Control What Crawler Can SeeLets Get Digital
 
Internert workshop
Internert workshop Internert workshop
Internert workshop sofiag000414
 
Integrating RSS into Your Web site
Integrating RSS into Your Web siteIntegrating RSS into Your Web site
Integrating RSS into Your Web siteMichael Sauers
 

Semelhante a Introduction to robots.txt (20)

Robots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml CreationRobots.txt and Sitemap.xml Creation
Robots.txt and Sitemap.xml Creation
 
Article19
Article19Article19
Article19
 
SEO Robots txt FILE
SEO Robots txt FILESEO Robots txt FILE
SEO Robots txt FILE
 
Vorlesung "Web-Technologies"
Vorlesung "Web-Technologies" Vorlesung "Web-Technologies"
Vorlesung "Web-Technologies"
 
From zero to almost rails in about a million slides...
From zero to almost rails in about a million slides...From zero to almost rails in about a million slides...
From zero to almost rails in about a million slides...
 
Difference between robots txt file, meta robots, X-robots tag
Difference between robots txt file, meta robots, X-robots tagDifference between robots txt file, meta robots, X-robots tag
Difference between robots txt file, meta robots, X-robots tag
 
E-commerce Primer For Newbies (Not Dummies)
E-commerce Primer For Newbies (Not Dummies)E-commerce Primer For Newbies (Not Dummies)
E-commerce Primer For Newbies (Not Dummies)
 
What is a Robot txt file?
What is a Robot txt file?What is a Robot txt file?
What is a Robot txt file?
 
Helping Things to REST
Helping Things to RESTHelping Things to REST
Helping Things to REST
 
E-commerce Primer For Newbies
E-commerce Primer For NewbiesE-commerce Primer For Newbies
E-commerce Primer For Newbies
 
Integrating RSS Into Your Web Site
Integrating RSS Into Your Web SiteIntegrating RSS Into Your Web Site
Integrating RSS Into Your Web Site
 
Technical SEO
Technical SEOTechnical SEO
Technical SEO
 
Integrating RSS Into Your Web Site
Integrating RSS Into Your Web SiteIntegrating RSS Into Your Web Site
Integrating RSS Into Your Web Site
 
Robot operating system [ROS]
Robot operating system [ROS]Robot operating system [ROS]
Robot operating system [ROS]
 
Determining Bias to Search Engines from Robots.txt
Determining Bias to Search Engines from Robots.txtDetermining Bias to Search Engines from Robots.txt
Determining Bias to Search Engines from Robots.txt
 
XML Sitemap and Robots.TXT Guide for SEO Beginners
XML Sitemap and Robots.TXT Guide for SEO BeginnersXML Sitemap and Robots.TXT Guide for SEO Beginners
XML Sitemap and Robots.TXT Guide for SEO Beginners
 
Introduction to "robots.txt
Introduction to "robots.txtIntroduction to "robots.txt
Introduction to "robots.txt
 
Robots.txt - Control What Crawler Can See
Robots.txt - Control What Crawler Can SeeRobots.txt - Control What Crawler Can See
Robots.txt - Control What Crawler Can See
 
Internert workshop
Internert workshop Internert workshop
Internert workshop
 
Integrating RSS into Your Web site
Integrating RSS into Your Web siteIntegrating RSS into Your Web site
Integrating RSS into Your Web site
 

Mais de madamseo

Meta Tags Optimization (Extended)
Meta Tags Optimization (Extended)Meta Tags Optimization (Extended)
Meta Tags Optimization (Extended)madamseo
 
Advert Funny
Advert Funny Advert Funny
Advert Funny madamseo
 
Meta Tags Optimization
Meta Tags OptimizationMeta Tags Optimization
Meta Tags Optimizationmadamseo
 
Beginners Guide to Search Engine Marketing
Beginners Guide to Search Engine MarketingBeginners Guide to Search Engine Marketing
Beginners Guide to Search Engine Marketingmadamseo
 
Social Media From Scratch
Social Media From ScratchSocial Media From Scratch
Social Media From Scratchmadamseo
 
Social Media Marketing - Today and Tomorrow...
Social Media Marketing - Today and Tomorrow...Social Media Marketing - Today and Tomorrow...
Social Media Marketing - Today and Tomorrow...madamseo
 
How does Google...?
How does Google...?How does Google...?
How does Google...?madamseo
 
Internet Marketing with WordPress
Internet Marketing with WordPressInternet Marketing with WordPress
Internet Marketing with WordPressmadamseo
 
Digital Marketing Communication
Digital Marketing CommunicationDigital Marketing Communication
Digital Marketing Communicationmadamseo
 
Presentation
PresentationPresentation
Presentationmadamseo
 

Mais de madamseo (10)

Meta Tags Optimization (Extended)
Meta Tags Optimization (Extended)Meta Tags Optimization (Extended)
Meta Tags Optimization (Extended)
 
Advert Funny
Advert Funny Advert Funny
Advert Funny
 
Meta Tags Optimization
Meta Tags OptimizationMeta Tags Optimization
Meta Tags Optimization
 
Beginners Guide to Search Engine Marketing
Beginners Guide to Search Engine MarketingBeginners Guide to Search Engine Marketing
Beginners Guide to Search Engine Marketing
 
Social Media From Scratch
Social Media From ScratchSocial Media From Scratch
Social Media From Scratch
 
Social Media Marketing - Today and Tomorrow...
Social Media Marketing - Today and Tomorrow...Social Media Marketing - Today and Tomorrow...
Social Media Marketing - Today and Tomorrow...
 
How does Google...?
How does Google...?How does Google...?
How does Google...?
 
Internet Marketing with WordPress
Internet Marketing with WordPressInternet Marketing with WordPress
Internet Marketing with WordPress
 
Digital Marketing Communication
Digital Marketing CommunicationDigital Marketing Communication
Digital Marketing Communication
 
Presentation
PresentationPresentation
Presentation
 

Último

DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024The Digital Insurer
 
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostLeverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostZilliz
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxhariprasad279825
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsMiki Katsuragi
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Wonjun Hwang
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
The Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfThe Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfSeasiaInfotech2
 

Último (20)

DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024
 
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostLeverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptx
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project Setup
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
The Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfThe Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdf
 

Introduction to robots.txt

  • 2. • Robots.txt is a plain text file that you upload to the root directory of your site. Once the web spiders (ants, boots, indexers) that index your webpages reach your site, they first look at that text file and process it. Put differently, robots.txt says to the spider which pages to crawl.
  • 3.
  • 4. THE SIMPLESTVERSION OF ROBOTS.TXT FILE IS: USER-AGENT:* DISALLOW: THE FIRST LINE INDICATESTHE USER AGENT- ASTERISK INDICATES THATTHE FOLLOWING LINES APPLYTO ALL AGENTS. SPACE AFTER "DISALLOW:" MEANSTHAT NOTHING IS LIMITED. THIS ROBOTS.TXT FILE DOES NOTHING- IT ALLOWS USER AGENTSTO SEE EVERYTHING ONTHE SITE.
  • 5. NOW LET'S MAKE IT A LITTLE MORE COMPLICATED-THISTIME WE DO NOT WANT SPIDERSTO CRAWL IN OUR /FAQ DIRECTORY: USER-AGENT:* DISALLOW: /FAQ/ IT IS RELATIVELY EASY. SLASH INDICATESTHATTHIS IS A DIRECTORY. IFYOU DON'T PUT A SLASH, NOT ONLY/FAQ DIRECTORY, BUT ALSO EACH FILE WHICH NAME BEGINS WITH "FAQ" WILL BE BANNED. ALTERNATIVELY, YOU CAN ADD MORE DIRECTORIESTOYOUR BANNED LIST.
  • 6. VIEWTHE FULL PRESENTATION AT WWW.MADAMSEO.COM OR SUBSCRIBETO MY PAID COURSE ON WWW.UDEMY.COM