SlideShare a Scribd company logo
1 of 15
Download to read offline
Introduction to
Programming Bots
DMITRI NESTERUK
TECHNOLOGY ADVOCATE, JETBRАINS
@DNESTERUK
What is a (ro)bot?
• A piece of software (command-line, UI, service)
• Involved in automating some (manual) process
• Usually implemented in order to
• Make money
• Save money or some other resource
• Gather useful information
Some Examples
• Amazon book bots
• Spam bots
• Chat bots
• Trading bots
• Web crawlers/miners
Levels of automation
• Levels of activation
• Manual = you run a console/UI executable/script
• One off tasks
• Fully automatic = service,Windows scheduler
• Regular tasks
• Interaction modes
• Fully automatic = no interaction, only program updates
• Coarse tuning = small level of control to fine tune behavior
Possibly define starting parameters
• Fine tuning = bot with a human front-end
Polling Service
public partial class PollingService : ServiceBase
{
private readonly Thread workerThread;
public PollingService()
{
InitializeComponent();
workerThread = new Thread(DoWork);
workerThread.SetApartmentState(ApartmentState.STA);
}
protected override void OnStart(string[] args)
{ workerThread.Start(); }
protected override void OnStop() { workerThread.Abort(); }
private static void DoWork()
{
while (true)
{
log.Info("Doing work...");
// do some work, then
Thread.Sleep(1000);
}
}
}
Polling Service Usage
var service = new PollingService();
ServiceBase[] servicesToRun = new ServiceBase[] { service };
if (Environment.UserInteractive)
{
Console.CancelKeyPress += (x, y) => service.Stop();
service.Start();
Console.WriteLine("Running service, press a key to stop");
Console.ReadKey();
service.Stop();
Console.WriteLine("Service stopped. Goodbye.");
}
else
{
ServiceBase.Run(servicesToRun);
}
Service Installer Utility
class ServiceInstallerUtility
{
private static readonly ILog log =
LogManager.GetLogger(typeof(Program));
private static readonly string exePath =
Assembly.GetExecutingAssembly().Location;
public static bool Install()
{
try { ManagedInstallerClass.InstallHelper(new[] { exePath }); }
catch { return false; }
return true;
}
public static bool Uninstall()
{
try {ManagedInstallerClass.InstallHelper(new[] { "/u", exePath });}
catch { return false; }
return true;
}
}
Processing Cmd Line Args
if (args != null && args.Length == 1 && args[0].Length > 1
&& (args[0][0] == '-' || args[0][0] == '/'))
{
switch (args[0].Substring(1).ToLower())
{
case "install":
case "i":
if (!ServiceInstallerUtility.InstallMe())
Console.WriteLine("Failed to install service");
break;
case "uninstall":
case "u":
if (!ServiceInstallerUtility.UninstallMe())
Console.WriteLine("Failed to uninstall service");
break;
default:
Console.WriteLine("Unrecognized parameters.");
break;
}
}
Gauging the Environment
• Available API (hooray!)
• Web scraping
• HTML parsing is easy, but…
• Captcha solving – not so much
• Screen scraping
• Taking screenshots and analysing pixels
• Manual input & subsequent analysis
• E.g., game Monte Carlo methods
Web Scraping
• Getting data from the web
• Parsing the data
• Caching/storing data as necessary
• Different approaches to HTML grab
• Grab it raw
• Spin up a ‘headless’ or proper browser
• Parsing HTML
• Not always correct
• HtmlAgilityPack
WatiN
• Web testing framework
• Opens up an actual, physical browser
• You might not need it!
• If you mine static content, just use WebClient etc.
• Real UI! Might want to do it under a separate account
• Throwing UI is nasty
• Some browser choice
WatiN Example
using (var browser = new IE("http://www.pokemon.com"))
{
var doc = new HtmlDocument();
doc.LoadHtml(browser.Body.OuterHtml);
var h1 =
doc.DocumentNode.SelectNodes("//h3").First();
Console.WriteLine(h1.InnerText);
}
Some Random Notes
• Launching IE requires STA
• Watin should be executed in 32-bit
• Polling is not ideal:WatiN and HTML parsing is not
fast!
Market APIs
• APIs are great!
• Standardized (e.g., FIX, FAST, etc.)
• Proprietary (e.g., Plaza2)
That’s It!
• http://bitbucket.org/nesteruk/datagatheringdemos
• Questions?
• @dnesteruk
• skype: dmitri.nesteruk
• dmitrinesteruk@gmail.com
• http://nesteruk.wordpress.com

More Related Content

What's hot

Hourglass Interfaces for C++ APIs - CppCon 2014
Hourglass Interfaces for C++ APIs - CppCon 2014Hourglass Interfaces for C++ APIs - CppCon 2014
Hourglass Interfaces for C++ APIs - CppCon 2014Stefanus Du Toit
 
The Kotlin Programming Language
The Kotlin Programming LanguageThe Kotlin Programming Language
The Kotlin Programming Languageintelliyole
 
Groovy Ast Transformations (greach)
Groovy Ast Transformations (greach)Groovy Ast Transformations (greach)
Groovy Ast Transformations (greach)HamletDRC
 
Iron Languages - NYC CodeCamp 2/19/2011
Iron Languages - NYC CodeCamp 2/19/2011Iron Languages - NYC CodeCamp 2/19/2011
Iron Languages - NYC CodeCamp 2/19/2011Jimmy Schementi
 
Gentle introduction to modern C++
Gentle introduction to modern C++Gentle introduction to modern C++
Gentle introduction to modern C++Mihai Todor
 
C++11: Feel the New Language
C++11: Feel the New LanguageC++11: Feel the New Language
C++11: Feel the New Languagemspline
 
Kotlin advanced - language reference for android developers
Kotlin advanced - language reference for android developersKotlin advanced - language reference for android developers
Kotlin advanced - language reference for android developersBartosz Kosarzycki
 
Modern C++ Explained: Move Semantics (Feb 2018)
Modern C++ Explained: Move Semantics (Feb 2018)Modern C++ Explained: Move Semantics (Feb 2018)
Modern C++ Explained: Move Semantics (Feb 2018)Olve Maudal
 
Reflection in Go
Reflection in GoReflection in Go
Reflection in Gostrikr .
 
STL ALGORITHMS
STL ALGORITHMSSTL ALGORITHMS
STL ALGORITHMSfawzmasood
 
Introduction to Kotlin Language and its application to Android platform
Introduction to Kotlin Language and its application to Android platformIntroduction to Kotlin Language and its application to Android platform
Introduction to Kotlin Language and its application to Android platformEastBanc Tachnologies
 
JS Fest 2018. Douglas Crockford. The Better Parts
JS Fest 2018. Douglas Crockford. The Better PartsJS Fest 2018. Douglas Crockford. The Better Parts
JS Fest 2018. Douglas Crockford. The Better PartsJSFestUA
 
C# What's next? (7.x and 8.0)
C# What's next? (7.x and 8.0)C# What's next? (7.x and 8.0)
C# What's next? (7.x and 8.0)Christian Nagel
 
eMan Dev Meetup: Kotlin - A Language we should know it exists (part 02/03) 18...
eMan Dev Meetup: Kotlin - A Language we should know it exists (part 02/03) 18...eMan Dev Meetup: Kotlin - A Language we should know it exists (part 02/03) 18...
eMan Dev Meetup: Kotlin - A Language we should know it exists (part 02/03) 18...eMan s.r.o.
 

What's hot (20)

Hourglass Interfaces for C++ APIs - CppCon 2014
Hourglass Interfaces for C++ APIs - CppCon 2014Hourglass Interfaces for C++ APIs - CppCon 2014
Hourglass Interfaces for C++ APIs - CppCon 2014
 
The Kotlin Programming Language
The Kotlin Programming LanguageThe Kotlin Programming Language
The Kotlin Programming Language
 
Groovy Ast Transformations (greach)
Groovy Ast Transformations (greach)Groovy Ast Transformations (greach)
Groovy Ast Transformations (greach)
 
Functional go
Functional goFunctional go
Functional go
 
Iron Languages - NYC CodeCamp 2/19/2011
Iron Languages - NYC CodeCamp 2/19/2011Iron Languages - NYC CodeCamp 2/19/2011
Iron Languages - NYC CodeCamp 2/19/2011
 
Gentle introduction to modern C++
Gentle introduction to modern C++Gentle introduction to modern C++
Gentle introduction to modern C++
 
Modern C++
Modern C++Modern C++
Modern C++
 
C++11: Feel the New Language
C++11: Feel the New LanguageC++11: Feel the New Language
C++11: Feel the New Language
 
Kotlin advanced - language reference for android developers
Kotlin advanced - language reference for android developersKotlin advanced - language reference for android developers
Kotlin advanced - language reference for android developers
 
Summary of C++17 features
Summary of C++17 featuresSummary of C++17 features
Summary of C++17 features
 
Modern C++ Explained: Move Semantics (Feb 2018)
Modern C++ Explained: Move Semantics (Feb 2018)Modern C++ Explained: Move Semantics (Feb 2018)
Modern C++ Explained: Move Semantics (Feb 2018)
 
Reflection in Go
Reflection in GoReflection in Go
Reflection in Go
 
The Style of C++ 11
The Style of C++ 11The Style of C++ 11
The Style of C++ 11
 
STL ALGORITHMS
STL ALGORITHMSSTL ALGORITHMS
STL ALGORITHMS
 
Introduction to Kotlin Language and its application to Android platform
Introduction to Kotlin Language and its application to Android platformIntroduction to Kotlin Language and its application to Android platform
Introduction to Kotlin Language and its application to Android platform
 
Introduction to kotlin
Introduction to kotlinIntroduction to kotlin
Introduction to kotlin
 
JS Fest 2018. Douglas Crockford. The Better Parts
JS Fest 2018. Douglas Crockford. The Better PartsJS Fest 2018. Douglas Crockford. The Better Parts
JS Fest 2018. Douglas Crockford. The Better Parts
 
C# What's next? (7.x and 8.0)
C# What's next? (7.x and 8.0)C# What's next? (7.x and 8.0)
C# What's next? (7.x and 8.0)
 
eMan Dev Meetup: Kotlin - A Language we should know it exists (part 02/03) 18...
eMan Dev Meetup: Kotlin - A Language we should know it exists (part 02/03) 18...eMan Dev Meetup: Kotlin - A Language we should know it exists (part 02/03) 18...
eMan Dev Meetup: Kotlin - A Language we should know it exists (part 02/03) 18...
 
What's New in C++ 11?
What's New in C++ 11?What's New in C++ 11?
What's New in C++ 11?
 

Similar to Introduction to Programming Bots

Monitoring and Debugging your Live Applications
Monitoring and Debugging your Live ApplicationsMonitoring and Debugging your Live Applications
Monitoring and Debugging your Live ApplicationsRobert Coup
 
E yantra robot abstractions
E yantra robot abstractionsE yantra robot abstractions
E yantra robot abstractionsAkshar Desai
 
IOT Firmware: Best Pratices
IOT Firmware:  Best PraticesIOT Firmware:  Best Pratices
IOT Firmware: Best Praticesfarmckon
 
The fundamental problems of GUI applications and why people choose React
The fundamental problems of GUI applications and why people choose ReactThe fundamental problems of GUI applications and why people choose React
The fundamental problems of GUI applications and why people choose ReactOliver N
 
El camino a las Cloud Native Apps - Introduction
El camino a las Cloud Native Apps - IntroductionEl camino a las Cloud Native Apps - Introduction
El camino a las Cloud Native Apps - IntroductionPlain Concepts
 
Medium TechTalk — iOS
Medium TechTalk — iOSMedium TechTalk — iOS
Medium TechTalk — iOSjimmyatmedium
 
System insight without Interference
System insight without InterferenceSystem insight without Interference
System insight without InterferenceTony Tam
 
Node js presentation
Node js presentationNode js presentation
Node js presentationmartincabrera
 
Building Hermetic Systems (without Docker)
Building Hermetic Systems (without Docker)Building Hermetic Systems (without Docker)
Building Hermetic Systems (without Docker)William Farrell
 
The ten commandments for an Agile Developer
The ten commandments for an Agile DeveloperThe ten commandments for an Agile Developer
The ten commandments for an Agile DeveloperSowmya Karmali
 
Simple Pure Java
Simple Pure JavaSimple Pure Java
Simple Pure JavaAnton Keks
 
Essay On Cmit 350 Final Project
Essay On Cmit 350 Final ProjectEssay On Cmit 350 Final Project
Essay On Cmit 350 Final ProjectNancy Jarjis
 
Testing ASP.NET - Progressive.NET
Testing ASP.NET - Progressive.NETTesting ASP.NET - Progressive.NET
Testing ASP.NET - Progressive.NETBen Hall
 
History of asynchronous in .NET
History of asynchronous in .NETHistory of asynchronous in .NET
History of asynchronous in .NETMarcin Tyborowski
 
Unit Testing JavaScript Applications
Unit Testing JavaScript ApplicationsUnit Testing JavaScript Applications
Unit Testing JavaScript ApplicationsYnon Perek
 

Similar to Introduction to Programming Bots (20)

Monitoring and Debugging your Live Applications
Monitoring and Debugging your Live ApplicationsMonitoring and Debugging your Live Applications
Monitoring and Debugging your Live Applications
 
E yantra robot abstractions
E yantra robot abstractionsE yantra robot abstractions
E yantra robot abstractions
 
IOT Firmware: Best Pratices
IOT Firmware:  Best PraticesIOT Firmware:  Best Pratices
IOT Firmware: Best Pratices
 
Node azure
Node azureNode azure
Node azure
 
The fundamental problems of GUI applications and why people choose React
The fundamental problems of GUI applications and why people choose ReactThe fundamental problems of GUI applications and why people choose React
The fundamental problems of GUI applications and why people choose React
 
El camino a las Cloud Native Apps - Introduction
El camino a las Cloud Native Apps - IntroductionEl camino a las Cloud Native Apps - Introduction
El camino a las Cloud Native Apps - Introduction
 
RobotStudiopp.ppt
RobotStudiopp.pptRobotStudiopp.ppt
RobotStudiopp.ppt
 
ie450RobotStudio.ppt
ie450RobotStudio.pptie450RobotStudio.ppt
ie450RobotStudio.ppt
 
Medium TechTalk — iOS
Medium TechTalk — iOSMedium TechTalk — iOS
Medium TechTalk — iOS
 
System insight without Interference
System insight without InterferenceSystem insight without Interference
System insight without Interference
 
Node js presentation
Node js presentationNode js presentation
Node js presentation
 
Building Hermetic Systems (without Docker)
Building Hermetic Systems (without Docker)Building Hermetic Systems (without Docker)
Building Hermetic Systems (without Docker)
 
The ten commandments for an Agile Developer
The ten commandments for an Agile DeveloperThe ten commandments for an Agile Developer
The ten commandments for an Agile Developer
 
Simple Pure Java
Simple Pure JavaSimple Pure Java
Simple Pure Java
 
Essay On Cmit 350 Final Project
Essay On Cmit 350 Final ProjectEssay On Cmit 350 Final Project
Essay On Cmit 350 Final Project
 
Testing ASP.NET - Progressive.NET
Testing ASP.NET - Progressive.NETTesting ASP.NET - Progressive.NET
Testing ASP.NET - Progressive.NET
 
Golang dot-testing-lite
Golang dot-testing-liteGolang dot-testing-lite
Golang dot-testing-lite
 
History of asynchronous in .NET
History of asynchronous in .NETHistory of asynchronous in .NET
History of asynchronous in .NET
 
Unit Testing JavaScript Applications
Unit Testing JavaScript ApplicationsUnit Testing JavaScript Applications
Unit Testing JavaScript Applications
 
Responsive interfaces
Responsive interfacesResponsive interfaces
Responsive interfaces
 

More from Dmitri Nesteruk

Good Ideas in Programming Languages
Good Ideas in Programming LanguagesGood Ideas in Programming Languages
Good Ideas in Programming LanguagesDmitri Nesteruk
 
Design Pattern Observations
Design Pattern ObservationsDesign Pattern Observations
Design Pattern ObservationsDmitri Nesteruk
 
CallSharp: Automatic Input/Output Matching in .NET
CallSharp: Automatic Input/Output Matching in .NETCallSharp: Automatic Input/Output Matching in .NET
CallSharp: Automatic Input/Output Matching in .NETDmitri Nesteruk
 
Converting Managed Languages to C++
Converting Managed Languages to C++Converting Managed Languages to C++
Converting Managed Languages to C++Dmitri Nesteruk
 
YouTrack: Not Just an Issue Tracker
YouTrack: Not Just an Issue TrackerYouTrack: Not Just an Issue Tracker
YouTrack: Not Just an Issue TrackerDmitri Nesteruk
 
Victor CG Erofeev - Metro UI
Victor CG Erofeev - Metro UIVictor CG Erofeev - Metro UI
Victor CG Erofeev - Metro UIDmitri Nesteruk
 
Dynamics CRM Data Integration
Dynamics CRM Data IntegrationDynamics CRM Data Integration
Dynamics CRM Data IntegrationDmitri Nesteruk
 
ReSharper Presentation for NUGs
ReSharper Presentation for NUGsReSharper Presentation for NUGs
ReSharper Presentation for NUGsDmitri Nesteruk
 
ReSharper Architecture & Extensions
ReSharper Architecture & ExtensionsReSharper Architecture & Extensions
ReSharper Architecture & ExtensionsDmitri Nesteruk
 

More from Dmitri Nesteruk (20)

Good Ideas in Programming Languages
Good Ideas in Programming LanguagesGood Ideas in Programming Languages
Good Ideas in Programming Languages
 
Design Pattern Observations
Design Pattern ObservationsDesign Pattern Observations
Design Pattern Observations
 
CallSharp: Automatic Input/Output Matching in .NET
CallSharp: Automatic Input/Output Matching in .NETCallSharp: Automatic Input/Output Matching in .NET
CallSharp: Automatic Input/Output Matching in .NET
 
C# Tricks
C# TricksC# Tricks
C# Tricks
 
Converting Managed Languages to C++
Converting Managed Languages to C++Converting Managed Languages to C++
Converting Managed Languages to C++
 
Tpl DataFlow
Tpl DataFlowTpl DataFlow
Tpl DataFlow
 
YouTrack: Not Just an Issue Tracker
YouTrack: Not Just an Issue TrackerYouTrack: Not Just an Issue Tracker
YouTrack: Not Just an Issue Tracker
 
Проект X2C
Проект X2CПроект X2C
Проект X2C
 
Domain Transformations
Domain TransformationsDomain Transformations
Domain Transformations
 
Victor CG Erofeev - Metro UI
Victor CG Erofeev - Metro UIVictor CG Erofeev - Metro UI
Victor CG Erofeev - Metro UI
 
Developer Efficiency
Developer EfficiencyDeveloper Efficiency
Developer Efficiency
 
Distributed Development
Distributed DevelopmentDistributed Development
Distributed Development
 
Dynamics CRM Data Integration
Dynamics CRM Data IntegrationDynamics CRM Data Integration
Dynamics CRM Data Integration
 
ReSharper Presentation for NUGs
ReSharper Presentation for NUGsReSharper Presentation for NUGs
ReSharper Presentation for NUGs
 
ReSharper Architecture & Extensions
ReSharper Architecture & ExtensionsReSharper Architecture & Extensions
ReSharper Architecture & Extensions
 
Web mining
Web miningWeb mining
Web mining
 
Data mapping tutorial
Data mapping tutorialData mapping tutorial
Data mapping tutorial
 
Reactive Extensions
Reactive ExtensionsReactive Extensions
Reactive Extensions
 
Design Patterns in .Net
Design Patterns in .NetDesign Patterns in .Net
Design Patterns in .Net
 
Metaprogramming
MetaprogrammingMetaprogramming
Metaprogramming
 

Recently uploaded

How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Juan Carlos Gonzalez
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideIEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideHironori Washizaki
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdfPaige Cruz
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 

Recently uploaded (20)

How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideIEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 

Introduction to Programming Bots

  • 1. Introduction to Programming Bots DMITRI NESTERUK TECHNOLOGY ADVOCATE, JETBRАINS @DNESTERUK
  • 2. What is a (ro)bot? • A piece of software (command-line, UI, service) • Involved in automating some (manual) process • Usually implemented in order to • Make money • Save money or some other resource • Gather useful information
  • 3. Some Examples • Amazon book bots • Spam bots • Chat bots • Trading bots • Web crawlers/miners
  • 4. Levels of automation • Levels of activation • Manual = you run a console/UI executable/script • One off tasks • Fully automatic = service,Windows scheduler • Regular tasks • Interaction modes • Fully automatic = no interaction, only program updates • Coarse tuning = small level of control to fine tune behavior Possibly define starting parameters • Fine tuning = bot with a human front-end
  • 5. Polling Service public partial class PollingService : ServiceBase { private readonly Thread workerThread; public PollingService() { InitializeComponent(); workerThread = new Thread(DoWork); workerThread.SetApartmentState(ApartmentState.STA); } protected override void OnStart(string[] args) { workerThread.Start(); } protected override void OnStop() { workerThread.Abort(); } private static void DoWork() { while (true) { log.Info("Doing work..."); // do some work, then Thread.Sleep(1000); } } }
  • 6. Polling Service Usage var service = new PollingService(); ServiceBase[] servicesToRun = new ServiceBase[] { service }; if (Environment.UserInteractive) { Console.CancelKeyPress += (x, y) => service.Stop(); service.Start(); Console.WriteLine("Running service, press a key to stop"); Console.ReadKey(); service.Stop(); Console.WriteLine("Service stopped. Goodbye."); } else { ServiceBase.Run(servicesToRun); }
  • 7. Service Installer Utility class ServiceInstallerUtility { private static readonly ILog log = LogManager.GetLogger(typeof(Program)); private static readonly string exePath = Assembly.GetExecutingAssembly().Location; public static bool Install() { try { ManagedInstallerClass.InstallHelper(new[] { exePath }); } catch { return false; } return true; } public static bool Uninstall() { try {ManagedInstallerClass.InstallHelper(new[] { "/u", exePath });} catch { return false; } return true; } }
  • 8. Processing Cmd Line Args if (args != null && args.Length == 1 && args[0].Length > 1 && (args[0][0] == '-' || args[0][0] == '/')) { switch (args[0].Substring(1).ToLower()) { case "install": case "i": if (!ServiceInstallerUtility.InstallMe()) Console.WriteLine("Failed to install service"); break; case "uninstall": case "u": if (!ServiceInstallerUtility.UninstallMe()) Console.WriteLine("Failed to uninstall service"); break; default: Console.WriteLine("Unrecognized parameters."); break; } }
  • 9. Gauging the Environment • Available API (hooray!) • Web scraping • HTML parsing is easy, but… • Captcha solving – not so much • Screen scraping • Taking screenshots and analysing pixels • Manual input & subsequent analysis • E.g., game Monte Carlo methods
  • 10. Web Scraping • Getting data from the web • Parsing the data • Caching/storing data as necessary • Different approaches to HTML grab • Grab it raw • Spin up a ‘headless’ or proper browser • Parsing HTML • Not always correct • HtmlAgilityPack
  • 11. WatiN • Web testing framework • Opens up an actual, physical browser • You might not need it! • If you mine static content, just use WebClient etc. • Real UI! Might want to do it under a separate account • Throwing UI is nasty • Some browser choice
  • 12. WatiN Example using (var browser = new IE("http://www.pokemon.com")) { var doc = new HtmlDocument(); doc.LoadHtml(browser.Body.OuterHtml); var h1 = doc.DocumentNode.SelectNodes("//h3").First(); Console.WriteLine(h1.InnerText); }
  • 13. Some Random Notes • Launching IE requires STA • Watin should be executed in 32-bit • Polling is not ideal:WatiN and HTML parsing is not fast!
  • 14. Market APIs • APIs are great! • Standardized (e.g., FIX, FAST, etc.) • Proprietary (e.g., Plaza2)
  • 15. That’s It! • http://bitbucket.org/nesteruk/datagatheringdemos • Questions? • @dnesteruk • skype: dmitri.nesteruk • dmitrinesteruk@gmail.com • http://nesteruk.wordpress.com