Scraping dynamic data

I am scraping profiles on ask.fm for a research question. The problem is that only the top most recent questions are viewable and I have to click "view more" to see the next 15.

The source code for clicking view more looks like this:

<input class="submit-button-more submit-button-more-active" name="commit" onclick="return Forms.More.allowSubmit(this)" type="submit" value="View more" />

What is an easy way of calling this 4 times before scraping it. I want the most recent 60 posts on the site. Python is preferable.

You could probably use selenium to browse to the website and click on the button/link a few times. You can get that here:

    https://pypi.python.org/pypi/selenium

Or you might be able to do it with mechanize:

    http://wwwsearch.sourceforge.net/mechanize/

I have also heard good things about twill, but never used it myself:

    http://twill.idyll.org/



Source: http://stackoverflow.com/questions/19437782/scraping-dynamic-data

Web Scraping data from different sites

I am looking for a few ideas on how can I solve a design problem I'm going to be faced with building a web scraper to scrape multiple sites. Writing the scraper(s) is not the problem, matching the data from different sites (which may have small differences) is.

For the sake of being generic assume that I am scraping something like this from two or more different sites:

    public class Data {
        public int id;
        public String firstname;
        public String surname;
        ....
    }

If i scrape this from two different sites, I will encounter the situation where I could have the following:

Site A: id=100, firstname=William, surname=Doe

Site B: id=1974, firstname=Bill, surname=Doe

Essentially, I would like to consider these two sets of data the same (they are the same person but with their name slightly different on each site). I am looking for possible design solutions that can handle this.

The only idea I've come up with is scraping the data from a third location and using it as a reference list. Then when I scrape site A or B I can, over time, build up a list of failures and store them in a list for each scraper so that it can know (if i find id=100 then i know that the firstname will be William etc). I can't help but feel this is a rubbish idea!

If you need any more info, or if you think my description is a bit naff, let me know!

Thanks,

DMcB


Source: http://stackoverflow.com/questions/23970057/web-scraping-data-from-different-sites

Scrape Data Point Using Python

I am looking to scrape a data point using Python off of the url http://www.cavirtex.com/orderbook .

The data point I am looking to scrape is the lowest bid offer, which at the current moment looks like this:

<tr>
 <td><b>Jan. 19, 2014, 2:37 a.m.</b></td>
 <td><b>0.0775/0.1146</b></td>
 <td><b>860.00000</b></td>
 <td><b>66.65 CAD</b></td>
</tr>

The relevant point being the 860.00 . I am looking to build this into a script which can send me an email to alert me of certain price differentials compared to other exchanges.

I'm quite noobie so if in your explanations you could offer your thought process on why you've done certain things it would be very much appreciated.

Thank you in advance!

Edit: This is what I have so far which will return me the name of the title correctly, I'm having trouble grabbing the table data though.

import urllib2, sys
from bs4 import BeautifulSoup

site= "http://cavirtex.com/orderbook"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(site,headers=hdr)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)
print soup.title



Here is the code for scraping the lowest bid from the 'Buying BTC' table:

from selenium import webdriver

fp = webdriver.FirefoxProfile()
browser = webdriver.Firefox(firefox_profile=fp)
browser.get('http://www.cavirtex.com/orderbook')

lowest_bid = float('inf')
elements = browser.find_elements_by_xpath('//div[@id="orderbook_buy"]/table/tbody/tr/td')

for element in elements:
    text = element.get_attribute('innerHTML').strip('<b>|</b>')
    try:
        bid = float(text)
        if lowest_bid > bid:
            lowest_bid = bid
    except:
        pass

browser.quit()
print lowest_bid

In order to install Selenium for Python on your Windows-PC, run from a command line:

pip install selenium (or pip install selenium --upgrade if you already have it).

If you want the 'Selling BTC' table instead, then change "orderbook_buy" to "orderbook_sell".

If you want the 'Last Trades' table instead, then change "orderbook_buy" to "orderbook_trades".

Note:

If you consider performance critical, then you can implement the data-scraping via URL-Connection instead of Selenium, and have your program running much faster. However, your code will probably end up being a lot "messier", due to the tedious XML parsing that you'll be obliged to apply...

Here is the code for sending the previous output in an email from yourself to yourself:

import smtplib,ssl

def SendMail(username,password,contents):
    server = Connect(username)
    try:
        server.login(username,password)
        server.sendmail(username,username,contents)
    except smtplib.SMTPException,error:
        Print(error)
    Disconnect(server)

def Connect(username):
    serverName = username[username.index("@")+1:username.index(".")]
    while True:
        try:
            server = smtplib.SMTP(serverDict[serverName])
        except smtplib.SMTPException,error:
            Print(error)
            continue
        try:
            server.ehlo()
            if server.has_extn("starttls"):
                server.starttls()
                server.ehlo()
        except (smtplib.SMTPException,ssl.SSLError),error:
            Print(error)
            Disconnect(server)
            continue
        break
    return server

def Disconnect(server):
    try:
        server.quit()
    except smtplib.SMTPException,error:
        Print(error)

serverDict = {
    "gmail"  :"smtp.gmail.com",
    "hotmail":"smtp.live.com",
    "yahoo"  :"smtp.mail.yahoo.com"
}

SendMail("your_username@your_provider.com","your_password",str(lowest_bid))

The above code should work if your email provider is either gmail or hotmail or yahoo.

Please note that depending on your firewall configuration, it may ask your permission upon the first time you try it...



Source: http://stackoverflow.com/questions/21217034/scrape-data-point-using-python

Is Web Scraping Relevant in Today's Business World?

Different techniques and processes have been created and developed over time to collect and analyze data. Web scraping is one of the processes that have hit the business market recently. It is a great process that offers businesses with vast amounts of data from different sources such as websites and databases.

It is good to clear the air and let people know that data scraping is legal process. The main reason is in this case is because the information or data is already available in the internet. It is important to know that it is not a process of stealing information but rather a process of collecting reliable information. Most people have regarded the technique as unsavory behavior. Their main basis of argument is that with time the process will be over flooded and therefore lead to parity in plagiarism.

We can therefore simply define web scraping as a process of collecting data from a wide variety of different websites and databases. The process can be achieved either manually or by the use of software. The rise of data mining companies has led to more use of the web extraction and web crawling process. Other main functions such companies are to process and analyze the data harvested. One of the important aspects about these companies is that they employ experts. The experts are aware of the viable keywords and also the kind of information which can create usable statistic and also the pages that are worth the effort. Therefore the role of data mining companies is not limited to mining of data but also help their clients be able to identify the various relationships and also build the models.

Some of the common methods of web scraping used include web crawling, text gripping, DOM parsing, and expression matching. The latter process can only be achieved through parsers, HTML pages or even semantic annotation. Therefore there are many different ways of scraping the data but most importantly they work towards the same goal. The main objective of using web scraping service is to retrieve and also compile data contained in databases and websites. This is a must process for a business to remain relevant in the business world.

The main questions asked about web scraping touch on relevance. Is the process relevant in the business world? The answer to this question is yes. The fact that it is employed by large companies in the world and has derived many rewards says it all. It is important to note that many people regarded this technology as a plagiarism tool and others consider it as a useful tool that harvests the data required for the business success.

Using of web scraping process to extract data from the internet for competition analysis is highly recommended. If this is the case, then you must be sure to spot any pattern or trend that can work in a given market.

Source:http://ezinearticles.com/?Is-Web-Scraping-Relevant-in-Todays-Business-World?&id=7091414

SEO Copywriting Training - 3 Reasons to Learn This Lucrative Form of Freelance Writing

Here we're going discuss why SEO copywriting is an excellent career choice for anyone who wants to make good money, work from home and get started almost immediately.

Freelance writing has a reputation for being a low-paying career where jobs are hard to come by. It's almost like some type of professional caste system where some seem to make a lot of money effortlessly, while others scrape by on anything they can get (eg, $3 article writing gigs).

It doesn't have to be this way because there's a new type of writing taking over the web. And, if you know how to do it, you can easily make $100-$250+/day. What is it?

SEO Copywriting.

Following are three excellent reasons to train for this lucrative career.

1. Never Ending Need for Content: Anywhere from 30,000 to 100,000 new websites go live every day. This means the web is an increasingly crowded marketplace. Hence, businesses have to find ways to increase their chances of being found online.

Smart business owners, webmasters and bloggers know that SEO content is one of the best ways to go about it. So they invest in this type of content. And, as they constantly need fresh content to stay ahead of the competition, this means freelance writers who know how to provide it stay busy.

2. It's Flexible: Like freelance writing in general, SEO copywriting is a very flexible career. You can work from a coffee shop in New York, a hotel terrace in Jamaica, or a restaurant in South Beach, Miami. As long as there's an internet connection, you can get your work done.

An added bonus in SEO copywriting is that there's very little client interaction. Once they provide you with their keyword phrases, clients rarely need to be in contact with you. They give you keywords and a deadline, and you write the copy and return it via email. Beautiful!

3. Less Competition: There are many freelance writers -- hundreds of thousands, perhaps even a few million. But, very few of them know how to write SEO content. This means there's less competition for these types of writing assignments.

Just how important is an SEO copywriter to webmasters, bloggers, internet marketers -- and/or anyone who wants to be found online?

ln the SearcheEngineWatch article, The SEO Copywriter: Wordsmithing the WebWillam Flaiz underscores the importance of the SEO copywriter. He writes:

The role of the agency SEO copywriter is unique because. . . . Suddenly, the writer must optimize press releases, craft articles for social news, monitor social network communications, and develop wiki content. . . . a strong SEO copywriter is adaptable and capable of internalizing a lot of new information extremely quickly. These unique individuals have fast become the backbone of many SEO engagements. If content is king, then they [SEO copywriters] are the kingmakers.

If you want a high-paying , work-from-anywhere you please career, get SEO copywriting training -- and be well on your way to running your own successful freelance writing business.

Source: http://ezinearticles.com/?SEO-Copywriting-Training---3-Reasons-to-Learn-This-Lucrative-Form-of-Freelance-Writing&id=2943036

SEO Epiphany of an Offline Marketer Turned Online Marketer

SEO can be maddening for offline marketers such as myself transitioning to the online marketing world. In making this transition I just had an epiphany sparked by a discussion I just had with one of my Skype discussion groups. In offline marketing we are taught to use emotionally charged words depending on our targeted market. Words that evoke emotions such as greed, fear, love, hate, pride etc. are utilized throughout our marketing and is probably equivalent to keywords in online marketing.

When I sold Real Estate for buyers we focused a lot on pride, in my tax business fear was a dominating emotion especially fear of the IRS and fear of audits. In my tax business I specialized in Home Based Business owners and my two biggest client groups were Daycare Providers and Network Marketers. The most effective emotion in the Network Marketing industry was greed.

In offline marketing we send our advertising out through mediums that get us the best results. Newspapers and periodicals are sent to people's homes, flyers are put into people's hands or on their door our car, radio is broadcast to people listening to their radios. We learn how to get a response through our ads with these emotionally charged words.

Then we are taught how to write copy that not only uses these emotionally charged words but we are taught how to use tie downs like wouldn't it, couldn't it, shouldn't it. We are taught how to develop closing sequences that get lots of little yeses leading to the big yes.

Then we come to the online marketing arena and all the rules change. At first the changes are not obvious. We hear familiar terms like call to action and compelling headlines. But unlike in the offline marketing arena it appears that to play in this game you have to contend with this huge monster called Google. And this creates a huge paradigm shift in how one has to develop their marketing tactics. Offline we send our stuff out, online you have to be found. And you have to be found through Google. Bing does not really matter; Yahoo does not really matter only the Google search engine at this time really matters.

Sure there are advertising sites, Google AdWords, Craigslist and Facebook Classified ads and who knows what else is bound to pop up but none of these are true direct response marketing and function more like billboards and posters in high traffic areas. The closes thing to marketing material being sent directly to a person is email marketing which due to spam and individuals emails being overwhelmed with tons of stuff it is becoming increasingly ineffective unless you have a dedicated list looking forward to stuff you are sending them.

It seems that in order to market successfully on line you have to find out two things - what words or phrases that people are using to search for stuff on Google and what words and phrases that Google does not like. Online copywriters don't so much worry about closing sequences as they do Search Engine Optimization often called SEO. That means writing copy in such a way that Google spiders will find you and elevate you to the top of the heap on their clients searched words or phrases.

Besides such little things as meta tags, title tags, h1 tags and making sure your keywords and phrases are at the top of your copy, SEO is about two key elements. One is keywords, you know those words or phrases that people plug into Google to search for stuff and secondly Google perceived relevancy of your copy. Two websites being equal as far as keywords are concerned Google's perception of relevancy will determine who gets the top spots.

The precise keywords and phrases have to be strategically placed in your copy. But you have to be careful because certain words and phrases it seems Google has a hatred for and will band you. Keywords such as riches, wealth, success and money are among these words Google hates and could possibly even penalize you for using by forbidding your site from ever showing in their search results (I have a lot to say about such power being in the hands of one company but not today)

The density or the amount of times your selected keywords or phrases are placed in your copy becomes critical, too often and you will be penalized, not enough and you won't be recognized. Best rule of thumb is to use them were it makes sense in your copy. Trying to figure out percentages is simply not working with today's advanced algorithms. Best to focus on content with the understanding that if your key words or phrases are not in your copy then when people search for it you will not be found so where it makes sense you must use them.

Next is relevancy despite all the stuff we hear about backlinks there is far more to being determined as relevant by Google search engines than backlinks. Simple things such as being listed in the yellow pages can make your site more relevant than another site. Whereas backlinks are still very important you can have link spam that could make you irrelevant with Google. This topic is just too extensive a topic to cover here, besides this is about an epiphany concerning my transition from offline marketer to full time online marketing. But I will add this tidbit that I have learned about being perceived as relevant with Google. Avoid duplicate content. websites with identical pages, scraped content, heavily distributed articles and boiler plate pages/sites can hurt your relevancy perception with Google.

Even when you are the original author putting the exact same article into e-zines and blogs can hurt your relevancy if the same articles are posted on the websites you want to come up in the Google search engines. On the sites you want to rank in Google keep all content as original and unique as possible.

Source: http://ezinearticles.com/?SEO-Epiphany-of-an-Offline-Marketer-Turned-Online-Marketer&id=6713183

Kerry Finch Writing Announces Streamlining of Content Writing Services

Content writing business, Kerry Finch Writing, has announced the restructuring of the services it provides to businesses around the world. In revealing the changes, owner, Kerry Finch, gave some insights into the reasons behind the decision.

“The digital business world is dynamic in the true sense of the word: it is constantly changing. Rather than simply reacting to those changes, it makes better business sense to anticipate them, and to be nimble in the implementation of newer and more effective strategies,” said Ms Finch. “Because I am active in several vibrant online business communities, and lead a tightly knit, responsive, team, we are in a great position identify new opportunities and to apply them quickly and seamlessly to the content we are writing.”

“Very often my clients don’t even recognize the often subtle changes in the written content we deliver to them, for instance the increased use of niche-relevant semantics, the more engaging ‘calls to action’, the up-to-the-minute topic selections, and, where appropriate, moving towards more geo-targeted content,” she continued. "Geo-targeting is becoming very important for connecting with people using mobile devices for search queries."

“All of the modifications are implemented with a focus on current best-practices.”

In redefining the types of blogs offered by Kerry Finch Writing, Ms Finch has her sights on keeping her business, and those of her clients, both competitive and relevant.

Blog Posts

The changes include removing the term ‘articles’ from her product offering, and replacing these with ‘blog posts’.

“Seven years ago, when I established Kerry Finch Writing, a key service I provided was article writing. These were often published on article directories with the objective of acquiring valuable links back to websites, but things have changed dramatically since then,” Ms Finch explained.

“These days I help each of my business clients, regardless of their size, to understand the need to grow their own websites with original, relevant content. I help them to understand that their website is a business asset with a dollar value - but unlike other assets, if it is well-maintained, it will appreciate over time - not depreciate.”

“Once a website has been built and populated with well written content, the best way for it to grow is by having a News or Blog section, which we can feed with new posts on a regular basis. Those clients who have a recurring monthly blog post order with us, are the ones who are reaping the rewards - their websites are being found by the search engines, and they are enjoying better engagement with their ideal customers.”

Kerry Finch Writing now offers three types of blog posts: Evergreen (general niche-relevant posts), News (relating a current news story to the clients’ businesses), and Authority (which are longer, more deeply researched industry-focused posts).

“News-focused blog posts are now one of the most effective and types of content that we offer. They demonstrate to readers that the blog is current, and that the business owner is aware of what’s happening in the real world. These posts also have the added benefit of riding off the popularity of trending news stories,” said Ms Finch. “The practice of not dating blog posts is old-school, people need to see that what they are reading is accurate right now, not 2, 3 or 4 years ago - and this is especially important for my many law firm clients.”

Removal of Services

Some of the services that Kerry Finch Writing has provided in the past are no longer being offered, but the decision to remove them are also seen as a positive step.

“I am not in the business of offering some writing services just because we can. If a particular type of content is not delivering results, then I’d rather not include it in our suite of services,” said Ms Finch.

“The more streamlined content options we are now providing have been carefully considered, and as new opportunities arise, I will no doubt be changing them again. For instance I see more downloadable White Papers in the near future. Being adaptable is vital to succeeding in the online world.”

Source: http://www.virtual-strategy.com/2014/06/26/kerry-finch-writing-announces-streamlining-content-writing-services