Comments on: A very rough proxy for anger at the Pope http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/ Mon, 12 Dec 2011 17:53:49 +0000 hourly 1 http://wordpress.org/?v=3.0.4 By: Chris http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/comment-page-1/#comment-6841 Chris Thu, 05 Feb 2009 02:22:17 +0000 http://www.explananda.com/?p=2875#comment-6841 Ah, thanks. I figured there had to be a way to do that. Ah, thanks. I figured there had to be a way to do that.

]]> By: Steve Laniel http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/comment-page-1/#comment-6838 Steve Laniel Wed, 04 Feb 2009 22:48:45 +0000 http://www.explananda.com/?p=2875#comment-6838 Oh, and yes: Google Trends compares search volumes, not page counts. Quite so. So I *did* misunderstand. Oh, and yes: Google Trends compares search volumes, not page counts. Quite so. So I *did* misunderstand.

]]> By: Steve Laniel http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/comment-page-1/#comment-6837 Steve Laniel Wed, 04 Feb 2009 22:48:08 +0000 http://www.explananda.com/?p=2875#comment-6837 If you want to fake Google out, and have them not block urllib, I think you just need to fake the user-agent string. Try setting it to something like this: "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2008122010 Iceweasel/3.0.5 (Debian-3.0.5-1)" I think they just automatically block anything that looks like a spider. But on the Internet, no one needs to know you're a spider, you know? If you want to fake Google out, and have them not block urllib, I think you just need to fake the user-agent string. Try setting it to something like this:

“Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2008122010 Iceweasel/3.0.5 (Debian-3.0.5-1)”

I think they just automatically block anything that looks like a spider. But on the Internet, no one needs to know you’re a spider, you know?

]]> By: Chris http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/comment-page-1/#comment-6836 Chris Wed, 04 Feb 2009 21:13:24 +0000 http://www.explananda.com/?p=2875#comment-6836 That's kind of an amazing spike in 2005, though, isn't it? And of course it coincides with the month he was named Pope. That’s kind of an amazing spike in 2005, though, isn’t it? And of course it coincides with the month he was named Pope.

]]> By: Chris http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/comment-page-1/#comment-6835 Chris Wed, 04 Feb 2009 21:11:07 +0000 http://www.explananda.com/?p=2875#comment-6835 Doesn't the Google Trends figure out how many times the term has been searched for? I was looking at how many results the search returned in a given period. Doesn’t the Google Trends figure out how many times the term has been searched for? I was looking at how many results the search returned in a given period.

]]> By: Chris http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/comment-page-1/#comment-6834 Chris Wed, 04 Feb 2009 21:09:51 +0000 http://www.explananda.com/?p=2875#comment-6834 Oh cool. Oh cool.

]]> By: Steve Laniel http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/comment-page-1/#comment-6833 Steve Laniel Wed, 04 Feb 2009 21:09:44 +0000 http://www.explananda.com/?p=2875#comment-6833 Wouldn't Google Trends do the same thing? http://www.google.com/trends?q=nazi+pope Or maybe I misunderstand. Wouldn’t Google Trends do the same thing?
http://www.google.com/trends?q=nazi+pope

Or maybe I misunderstand.

]]> By: ben wolfson http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/comment-page-1/#comment-6832 ben wolfson Wed, 04 Feb 2009 21:00:45 +0000 http://www.explananda.com/?p=2875#comment-6832 There are officially-approved ways to do what you want, but I can't remember them. It is possible. You need an API key or whatever. There are officially-approved ways to do what you want, but I can’t remember them. It is possible. You need an API key or whatever.

]]> By: Chris http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/comment-page-1/#comment-6831 Chris Wed, 04 Feb 2009 20:23:11 +0000 http://www.explananda.com/?p=2875#comment-6831 Oh, should have known it wouldn't be so easy to write a script to grab anything from google. This, as an experiment: import urllib html_data = urllib.urlopen('http://www.google.com/search?hl=en&%3Aen-US%3Aofficial&hs=wOT&q=%22Nazi+Pope%22++daterange%3A2454466-2454496&btnG=Search') for line in html_data: print line Gets me a long message including this: Your client does not have permission to get URL <code>/search?hl=en&%3Aen-US%3Aofficial&hs=wOT&q=%22Nazi+Pope%22++daterange%3A2454466-2454496&btnG=Search</code> from this server. You can hardly blame them. I imagine people are constantly trying to harvest info from them. Oh, should have known it wouldn’t be so easy to write a script to grab anything from google.

This, as an experiment:

import urllib
html_data = urllib.urlopen(‘http://www.google.com/search?hl=en&%3Aen-US%3Aofficial&hs=wOT&q=%22Nazi+Pope%22++daterange%3A2454466-2454496&btnG=Search’)
for line in html_data:
print line

Gets me a long message including this:

Your client does not have permission to get URL /search?hl=en&%3Aen-US%3Aofficial&hs=wOT&q=%22Nazi+Pope%22++daterange%3A2454466-2454496&btnG=Search from this server.

You can hardly blame them. I imagine people are constantly trying to harvest info from them.

]]> By: Chris http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/comment-page-1/#comment-6830 Chris Wed, 04 Feb 2009 19:42:15 +0000 http://www.explananda.com/?p=2875#comment-6830 Thanks! Skimming the new results and comparing them with the old results suggests that they produce roughly the same Julian dates. But your way is much nicer. Thanks! Skimming the new results and comparing them with the old results suggests that they produce roughly the same Julian dates. But your way is much nicer.

]]> By: ben wolfson http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/comment-page-1/#comment-6829 ben wolfson Wed, 04 Feb 2009 19:29:10 +0000 http://www.explananda.com/?p=2875#comment-6829 One of the pages you link says that a Julian date is "the number of days that have passed since January 1, 4713 B.C.". Trial and error suggests that this is equal to: 1721425 + (datetime.date(year,mon,day) - datetime.date(1,1,1)).days One of the pages you link says that a Julian date is “the number of days that have passed since January 1, 4713 B.C.”. Trial and error suggests that this is equal to:

1721425 + (datetime.date(year,mon,day) – datetime.date(1,1,1)).days

]]> By: Chris http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/comment-page-1/#comment-6828 Chris Wed, 04 Feb 2009 19:08:11 +0000 http://www.explananda.com/?p=2875#comment-6828 Or am I just being silly? Or am I just being silly?

]]> By: Chris http://www.explananda.com/2009/02/04/a-very-rough-proxy-for-anger-at-the-pope/comment-page-1/#comment-6827 Chris Wed, 04 Feb 2009 19:07:43 +0000 http://www.explananda.com/?p=2875#comment-6827 It guess it wouldn't be all that hard to write a little tool that took a search string and a date range, constructed a list of appropriate urls, extracted the number of hits from the resulting pages, and then plotted hits over time. I'd want to run it from the public library though, just to be on the safe side. It guess it wouldn’t be all that hard to write a little tool that took a search string and a date range, constructed a list of appropriate urls, extracted the number of hits from the resulting pages, and then plotted hits over time. I’d want to run it from the public library though, just to be on the safe side.

]]>