Scraping a website from a windows service

Question

Hi there.  I have a windows forms application that scrapes a website to retrieve some data.  I would like to implement the same functionality as a windows service.  The reason for this is to allow the program to run 24/7 without  having a user signed in. 

To that end, my current version of the program uses a web browser control (system.windows.forms.webbrowser) to navigate the pages, click the buttons, allow scripts to do their thing, etc.  I cannot figure out a way to do the same without the web browser control, but the web browser control cannot be instantiated in a windows service (because there is no user interface in a web service).

Does anyone have any brilliant ideas on how to get around this?

Thank you very much!

Answers

Hi Andy,

There is a tool which could let you manipulate anything you want on the website. This agile HTML parser builds a read/write DOM and supports plain XPATH or XSLT. It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams). More information, please check:

http://htmlagilitypack.codeplex.com/

Have a nice day.

Best regards

All replies

You are not telling if you are using a .NET Express edition or not

You are not telling which Framework

You are not realy saying what data you are getting from the web site.

So

I made an example of service that work on any Studio edition (including the Express)

to install it, I supposed that you have at least the Framework2, so you will use something similar to:

    %SystemRoot%\Microsoft.NET\Framework\v2.0.50727\installutil /i C:\Test\MyWindowService\MyWindowService\bin\Release\MyWindowService.exe

In the example, I supposed that you are downloading some file from the site

You will need a reference to Windows.Form for the timer


Imports System.ServiceProcess

Imports System.Configuration.Install

Public Class WindowsService : Inherits ServiceBase

  Private Minute As Integer = 60000

  Private WithEvents Timer As New Timer With {.Interval = 30 * Minute, .Enabled = True}

  Public Sub New()

    Me.ServiceName = "MyService"

    Me.EventLog.Log = "Application"

    Me.CanHandlePowerEvent = True

    Me.CanHandleSessionChangeEvent = True

    Me.CanPauseAndContinue = True

    Me.CanShutdown = True

    Me.CanStop = True

  End Sub


  Private Sub Timer_Tick(ByVal sender As Object, ByVal e As System.EventArgs) Handles Timer.Tick

    If IO.File.Exists("C:\MyPath.Data") Then IO.File.Delete("C:\MyPath.Data")

    My.Computer.Network.DownloadFile("http://MyURL.com", "C:\MyPath.Data", "MyUserName", "MyPassword")

    'Do Something with the data downloaded

  End Sub

End Class

<Microsoft.VisualBasic.HideModuleName()> _

Module MainModule

  Public TheServiceName As String

  Public Sub main()

    Dim TheServiceApplication As New WindowsService

    TheServiceName = TheServiceApplication.ServiceName

    ServiceBase.Run(TheServiceApplication)

  End Sub

End Module

<System.ComponentModel.RunInstaller(True)> _

Public Class WindowsServiceInstaller : Inherits Installer

  Public Sub New()

    Dim serviceProcessInstaller As ServiceProcessInstaller = New ServiceProcessInstaller()

    Dim serviceInstaller As ServiceInstaller = New ServiceInstaller()

    serviceProcessInstaller.Account = ServiceAccount.LocalSystem

    serviceProcessInstaller.Username = Nothing

    serviceProcessInstaller.Password = Nothing

    serviceInstaller.DisplayName = "My Windows Service"

    serviceInstaller.StartType = ServiceStartMode.Automatic

    serviceInstaller.ServiceName = TheServiceName

    Me.Installers.Add(serviceProcessInstaller)

    Me.Installers.Add(serviceInstaller)

  End Sub

End Class



 Hello Andy,

Thanks for your post.

What do you want to scrape from the page? HttpWebRequest class ans WebClient class may be what you need. More information, please check:

The HttpWebRequest class provides support for the properties and methods defined in WebRequest and for additional properties and methods that enable the user to interact directly with servers using HTTP.

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx

The WebClient class provides common methods for sending data to or receiving data from any local, intranet, or Internet resource identified by a URI

http://msdn.microsoft.com/en-us/library/system.net.webclient.aspx

If you have any concenrs, please feel free to follow up.

Best regards



Hi Andy,

What about this problem on your side now? If you have any concerns, please feel free to follow up.

Have a nice day.

Best regards



Hi Andy,

When you come back, if you need further assistance about this issue, please feel free to let us know. We will continue to work with this issue.

Have a nice day.

Best regards



Thank you for the reply. Sorry it has taken me so long to respond.  I did not receive any notification that someone had replied!

I am using Visual Studio 2010 Ultimate Edition and the .NET framework 4.0.  Actually, I am upgrading some old code written in VB 6.0, but I can use the latest and greatest thats available.

The application uses a browser control to go to the page, fill in values, click on UI elements, read the HTML that returns, etc.  The purpose of the application is to collection useful information regularily/automatically.

I know how to create a web service, but using the web control in such a service is problematic because the web browser control was meant to be placed on a windows form.  I am not able to create a new instance of it in a project designated as a windows service.



Andy

Thank you for the reply. Sorry it has taken me so long to respond.  I did not receive any notification that someone had replied!

I thought a web request was for web services (retrieving information from them).  I am trying to retreive useful information from a website designed for interaction by a human, such as selecting items from lists and clicking buttons.   I currently use a web browser control to programmatically do what a person would do and get the pages back which in turn get parsed.

Andy



Hi Andy,

There is a tool which could let you manipulate anything you want on the website. This agile HTML parser builds a read/write DOM and supports plain XPATH or XSLT. It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams). More information, please check:

http://htmlagilitypack.codeplex.com/

Have a nice day.

Best regards



Thanks for the suggestion.  I will go to that link and see if it will work.  I will update this post with what I find.

I am writing to check the status of the issue on your side. Would you mind letting us know the result of the suggestions? If you have any concerns, please feel free to follow up.

Have a nice day.



Best regards

Hi Liliane

Thank for the follow up reply.  I don't have an answer as of yet.  Implementing this is going to take time and I haven't been given the go-ahead by my boss to spend the time to pursue it.



Hi Andy,

Never minde. You could have a try when you feel free. If you have any further questions about this issue, please feel free to let us know. We will continue to work with you on this issue.

Have a nice day.

Best regards

Source: https://social.msdn.microsoft.com/Forums/vstudio/en-US/f5d565b1-236b-43c2-90c7-f5cc3b2c341b/scraping-a-website-from-a-windows-service

SEO No No! Scraping & Splogging – Content Theft!

Until recently, you could as well as might possibly not have acknowledged how you can perform the earlier mentioned. Even so, the following element could be the really cool element.

Several. Get back to ScrapeBox Add-Ons and also down load your ScrapeBox Blog Analyzer add-on. Open it upwards, and transfer the actual .txt record you merely rescued. Struck start.

ScrapeBox goes through almost every back link you merely scraped and look these phones determine if these are your site that will ScrapeBox presently facilitates placing comments in. If it is, that turns environmentally friendly. If it isn’t, that turns reddish. Soon after it really is concluded, it is possible to “clean” the list insurance agencies the idea remove unsupported websites.

Just what you’re destined to be left with is ALL of the sites the competitor has back-links via, and most importantly, they all are capable of being mentioned in employing ScrapeBox!!

Help save that will “clean” listing with a report, import it this list involving websites you wish to touch upon, and then keep to the exact same steps you’d probably typically follow for you to touch upon websites. Inside of Ten mins you’ll have got all the comps website backlinks (which may be blocked by Public relations if you’d just like) along with you’ll be able to reply to every one of them inside a 20 min (because the list most likely won’t end up being Large).

Desire to force this specific even more?? Obviously you are doing, you’re in BHW

Each step is the same as over with the exception of one tiny issue as well as the addition of an extra step.

Instead of just employing a single foot print inside your first bounty (both from SB’s regular gui after which also the back link checker add-on) you’re likely to be using a A lot of open all of them. Here is what you do to consider this particular to a whole new amount.

Initial, you’re going to pick each of the URLs via AOL, Aol, Ask & Search engines using this footprint:

site:domainyourcompetingwith.org

That will go back ALL the at present found web pages in the area. Remove copy Web addresses along with save that will with a .TXT report.

Now, you’re planning to create the subsequent right in front of each of these URLs:

hyperlink:

Right now follow all of the steps while outlined above. Exactly what this may is actually obtain each of the backlinks to every single site of the rivals web site.

Because Google Back link Checker is simply capable of getting the first 1k Web addresses through Aol (while that’s all Google allows you to view) you could have missed out on a decent amount associated with website inbound links if they had been at night 1st 1k final results. Consequently performing the aforementioned further methods ensures that every brand new web site in the website anyone pay attention to backlinks implies a fresh and other pair of a listing of back-links that is possibly 1k back links long.

Now you understand how to locate, filtration along with take your competitors back links, stop looking at and also move and take action!

Source: https://freescrapeboxlist19.wordpress.com/

What are the ethics of web scraping?

Someone recently asked: "Is web scraping an ethical concept?" I believe that web scraping is absolutely an ethical concept. Web scraping (or screen scraping) is a mechanism to have a computer read a website. There is absolutely no technical difference between an automated computer viewing a website and a human-driven computer viewing a website. Furthermore, if done correctly, scraping can provide many benefits to all involved.

There are a bunch of great uses for web scraping. First, services like Instapaper, which allow saving content for reading on the go, use screen scraping to save a copy of the website to your phone. Second, services like Mint.com, an app which tells you where and how you are spending your money, uses screen scraping to access your bank's website (all with your permission). This is useful because banks do not provide many ways for programmers to access your financial data, even if you want them to. By getting access to your data, programmers can provide really interesting visualizations and insight into your spending habits, which can help you save money.

That said, web scraping can veer into unethical territory. This can take the form of reading websites much quicker than a human could, which can cause difficulty for the servers to handle it. This can cause degraded performance in the website. Malicious hackers use this tactic in what’s known as a "Denial of Service" attack.

Another aspect of unethical web scraping comes in what you do with that data. Some people will scrape the contents of a website and post it as their own, in effect stealing this content. This is a big no-no for the same reasons that taking someone else's book and putting your name on it is a bad idea. Intellectual property, copyright and trademark laws still apply on the internet and your legal recourse is much the same. People engaging in web scraping should make every effort to comply with the stated terms of service for a website. Even when in compliance with those terms, you should take special care in ensuring your activity doesn't affect other users of a website.

One of the downsides to screen scraping is it can be a brittle process. Minor changes to the backing website can often leave a scraper completely broken. Herein lies the mechanism for prevention: making changes to the structure of the code of your website can wreak havoc on a screen scraper's ability to extract information. Periodically making changes that are invisible to the user but affect the content of the code being returned is the most effective mechanism to thwart screen scrapers. That said, this is only a set-back. Authors of screen scrapers can always update them and, as there is no technical difference between a computer-backed browser and a human-backed browser, there's no way to 100% prevent access.

Going forward, I expect screen scraping to increase. One of the main reasons for screen scraping is that the underlying website doesn't have a way for programmers to get access to the data they want. As the number of programmers (and the need for programmers) increases over time, so too will the need for data sources. It is unreasonable to expect every company to dedicate the resources to build a programmer-friendly access point. Screen scraping puts the onus of data extraction on the programmer, not the company with the data, which can work out well for all involved.

Source: https://quickleft.com/blog/is-web-scraping-ethical/

How to Build Data Warehouses using Web Scraping

Businesses all over the world are facing an avalanche of information which needs to be collated, organized, analyzed and utilized in an appropriate fashion. Moreover, with each increasing year there is a perceived shortening of the turnaround time for businesses to take decisions based on information they have assimilated. Data Extractors, therefore, have evolved with a more significant role in modern day businesses than just mere collectors or scrapers of unstructured data. They cleanse structure and store contextual data in veritable warehouses, so as to make it available for transformation into useable information as and when the business requires. Data warehouses, therefore, are the curators of information which businesses seek to treasure and to use.

Understanding Data Warehouses

 Traditionally, Data Warehouses have been premised on the concept of getting easy access to readily available data. Modern day usage has helped it to evolve as a rich repository to store current and historical data that can be used to conduct data analysis and generate reports. As it also stores historical data, Data Warehouses are used to generate trending reports to help businesses foresee their prospects. In other words, data warehouses are the modern day crystal balls which businesses zealously pore over to foretell their future in the Industry.

Scraping Web Data for Creating Warehouses

The Web, as we know it, is a rich repository of a whole host of information. However, it is not always easy to access this information for the benefit of our businesses through manual processes. The data extractor tools, therefore, have been built to quickly and easily, scrape, cleanse and structure and store it in Data Warehouses so as to be readily available in a useable format.

Web Scraping tools are variously designed to help both programmers as well as non-programmers to retain their comfort zone while collecting data to create the data warehouses. There are several tools with point and click interfaces that ease out the process considerably. You can simply define the type of data you want and the tool will take care of the rest. Also, most tools such as these are able to store the data in the cloud and therefore do not need to maintain costly hardware or whole teams of developers to manage the repository.

Moreover, as most tools use a browser rendering technology, it helps to simulate the web viewing experience of humans thereby easing the usability aspect among business users facilitating the data extraction and storage process further.

Conclusion

The internet as we know it is stocked with valuable data most of which are not always easy to access. Web Data extraction tools have therefore gained popularity among businesses as they browse, search, navigate simulating your experience of web browsing and finally extract data fields specific to your industry and appropriate to your needs. These are stored in repositories for analysis and generation of reports. Thus evolves the need and utility of Data warehouses. As the process of data collection and organization from unstructured to structured form is automated, there is an assurance of accuracy built into the process which enhances the value and credibility of data warehouses. Web Data scraping is no doubt the value enhancers for Data warehouses in the current scenario.

Source: http://scraping-solutions.blogspot.in/2014/09/how-to-build-data-warehouses-using-web.html

What is HTML Scraping and how it works

There are many reasons why there may be a requirement to pull data or information from other sites, and usually the process begins after checking whether the site has an official API. There are very few people who are aware about the presence of structured data that is supported by every website automatically. We are basically talking about pulling data right from the HTML, also referred to as HTML scraping. This is an awesome way of gleaning data and information from third party websites.

Any webpage content that can be viewed can be scraped without any trouble. If there is any way provided by the website to the browser of the visitor to download content and use the same in a highly structured manner, in that case, accessing of the content programmatically is possible. HTML scraping works in an amazing manner.

Before indulging in HTML scraping, one can inspect the browser for network traffic. Site owners have a couple of tricks up their sleeve to thwart this access, but majority of them can be worked around.

Before moving on to how HTML scraping works, we must understand the reasons behind the same. Why is scraping needed? Once you get a satisfactory answer to this question, you can start looking for RSS or API feeds or various other traditional structured data forms. It is significant to understand that when compared with APIs, websites are more significant.

The most important advantage of the same is the maintenance of their websites where a lot of visitors visit rather than safeguarding structured data feeds. With Tweeter, the same has been publicly seen when it clamps down on the developer ecosystem. Many times, API feeds change or move without any prior warning. Many times, it can also be a deliberate attempt, but mostly, such issues or problems erupt as there is no authority or an organization that maintains or takes care of the structured data. It is rarely noticed, if the same gets severely mangled or goes offline. In case the website has certain issues or the website no longer works, the problem is more in the form of a ball in your court requiring dealing with the same without losing any time. api-comic-image

Rate limiting is another factor that needs a lot of thinking and in case of public websites, it virtually doesn’t exist. Besides some occasional sign up pages or captchas, many business websites fail to create and built defenses against any unwarranted automated access. Many times, a single website can be scraped for four hours straight without anyone noticing. There are chances that you would not be viewed under DDOS attack unless concurrent requests are being made by you. You will be seen just as an avid visitor or an enthusiast in the logs, that too, in case anyone is looking.

Another factor in HTML scraping is that one can easily access any website anonymously. Behavior tracking can be done with a few ways by the administrator of the website and this turns out to be beneficial if you want to privately gather the data. Many times, registration is imperative with APIs in order to get key and with any request being sent, this key also needs to be sent. But, in case of simple and straightforward HTTP requests, the visitor can stay anonymous besides cookies and IP address, which can again be spoofed.

The availability of HTML scraping is universal and there is no need to wait for the opening of the site for an API or for contacting anyone in the organization. One simply needs to spend some time and browse websites at a leisurely pace until the data you want is available and then find out the basic patterns to access the same.

Now you need to don a hat of a professional scraper and simply dive in. Initially, it may take some time to work up figuring out the way the data have been structured and the way it can be accessed just as we read APIs. If there is no documentation unlike APIs, you need to be a little more smart about it and use clever tricks.

Some of the most used tricks are

Data Fetching

The first thing that is required is data fetching. Find endpoints to begin with, that is the URLs that can help in returning the data that is required. If you are pretty sure about the data and the way it should be structured so as to match your requirements, you will require a particular subset for the same and later you can indulge in site browsing using the navigation tools.

GET Parameter

The URLs must be paid attention to and see the way it changes as you indulge in clicking between the sections and the way they divide into various subsections. Before starting, the other option that can be used is to straight away go to the search functionality of the site. Certain terms can be typed and the URL needs to be focused again for watching the changes on the basis of what is being searched. A GET parameter will be probably seen like q which changes on the basis of the search term used by you. Other GET parameters that are not being used can be removed from the URL until only the ones that are needed are left for data loading. Before a query string, there must always be a “?” beginning.

Now the time has come when you would have started to come across the data that you would like to see and want to access, but sometimes, there may be certain pagination issues that require to be dealt with. Due to these issues, you may not be able to see the data in its entirety. Single requests are kept away by many APIs as well from database slamming. Many times, clicking the next page can add some offset parameter that helps in data visibility on the page. All these steps will help you succeed in HTML scraping.

Source: https://www.promptcloud.com/blog/what-is-html-scraping-and-how-it-works/

Web Data Extraction- The most convenient and easy way to extract data from the internet

Web data extraction is the most proficient technique that will help you find the pertaining data for your existing business or any personal use. Many times, we find that experts’ copy and paste information manually from web pages or download the entire website which is a waste of time and effort.

Now with the new technique of Web data extraction you can crawl through loads and loads of web pages in order to extract particular data and at the same time save this data in the following manner

•     CSV FILE
•     XML FILE or
•     Any other custom format for future use.

Below given are some instances of Web data extraction processes:

•     Take a government portal, extracting names of citizens for a survey
•     Search for competitor websites for product pricing and feature information
•     Utilize web scraping to download images from a stock photography site for website design

How can Web Data Extraction serve you?

 You can extract data from any kind of websites like

Extract Data from any kind of Websites: Directories, Classified Websites, News, Websites, Blogs, Articles, and Job Portals, Search Engines, eCommerce Websites, Social Media Websites and any kind of websites whose content can be accessible. Extract Emails, Contacts, Price/Rate, Features, Contact Names, Contact Details, Full Text, Live updates, ASINs, Meta Tags, Address, Phone, Fax, Latitude & Longitude, Images, Links, Reviews, Ratings, etc. Help in Data Collection, Competitor Analysis, Research, Business Intelligence, Social Media Trend analysis, Brand Monitoring, Lead Data Collection, Website & Competitor Web Monitoring, etc. Deliver Data in any Database, Excel, CSV, Access, Text, My SQL, SQL, Oracle, etc. and in any format Custom Services of Web Data Extraction as per client need one time Data Delivery or Continued/Scheduled Data Delivery

The next is Website Data Scraping:

 Web site Data Scraping is the process of extracting data from a website by using a particular software program available from proven website only.

This extracted data can be utilized by any person and for any purposes as per their needs and wants; data extracted can be used in different industries. There are many companies providing best Website data scraping services.

It is one such field which has active developments and also shares a common objective that needs a breakthrough in the following:

•     Text Processing
•     Semantic Understanding
•     Artificial Intelligence
•     Human Computer Interactions

There are many users or end users, companies and experts that need information or data that is accessible in some or the other format. In such cases Web Data Extraction can tailor the need of extracting data from any proven source and preserve the data on a particular destination.

The source platform contains:

•     Excel
•     CSV
•     MySQL and
•     Others

Moreover, the technique of Web data extractor can also extract information from various websites like Google, Amazon, LinkedIn, EBay and many others.

It can also extract data from eCommerce shopping websites, or other social networking websites, any public websites, classifieds websites, job portal websites and any other search engine websites.

Websitedatascraping.com is enough capable to web data scraping, website data scraping, web scraping services, website scraping services, data scraping services, product information scraping and yellowpages data scraping.

Diamond Mines And Mining Techniques

Diamonds remain to this day a mystical gem with a somewhat checkered past. The story of the Hope diamond is based on a long legend of misfortunes supposedly befalling its various and colorful owners. Segregation and mistreatment of blacks in diamond mines in Africa has long been a terrible mark on humanity in that part of the world. Although early white miners were treated better the working conditions endured by all diamond miners were less than humane. Fortunately most of today's mines while still depending on human labor have most of the heavy work done with machinery.

Contrary to popular belief diamonds are mined in areas other than Africa. Some of the better known diamond mines are these.

* Argye one of the Rio Tinto company mines is located in Western Australia.

* Diavik another Rio Tinto mine is located in Canada.

* Ekati owned by BHP Billiton and located in the Northwest Territories of Canada.

* Baken owned by Trans Hexis is located in South Africa.

* Merlin owned by Striker Resources is located in Australia.

* Orapa owned by a partnership between DeBeers and the government of Botswana is in Botswana.

* Premier owned by the De Beers Company is located in South Africa.

Diamond ore is extracted from these mines using basically four mining techniques which are based on the type of geology in which the diamond bearing material is located.

Marine Mining is the most recent development introduced about 1990. This technique is similar to deep water oil drilling. A large shaft is bored into the seabed to a depth where diamond bearing soil is located and that material is sucked to the surface. Also underwater vehicles called "crawlers" move along the seabed to scoop up diamond bearing gravel and pump it to the surface.

Placer Diamond Mining is a technique that is seen many times in movies of the old West. The diamonds are buried in river banks or mountain sides and water canons are used to wash the material down to be processed.

Hard Rock Diamond Mining is again familiar to movie fans as they watched coal miners or gold and silver miners digging their way into deep underground tunnels. Of course the technique is modern now and makes use of many specialized machines to do the heavy work.

Open-Pit diamond mining is similar to the pit coal mines of West Virginia and some western states. Overburden which is the soil covering the diamond embedded material is moved by machinery and blasting. The diamond bearing material is then moved to processing plants. This technique is common when the diamond bearing material is found close to the surface of if the geology is so unstable that tunneling is not safe or practical.

From this short article it should be apparent that diamonds take long and interesting journeys before they find a cherished spot on your finger or ear lobes.

Source: http://ezinearticles.com/?Diamond-Mines-And-Mining-Techniques&id=4800018