bpgergo: 2012

Religion cybernetics and getting things done

Two days ago we had this conversation about astrology and a friend mentioned it would be interesting, from a "religion cybernetics" aspect, to see the list physicists who has the zodiac Taurus. Religion cybernetics: I'd heard this term many times, and I was still puzzled what the heck this would be, but this time I had a second thought: I was wondering how long would it take to actually get this done and produce such a list from the Wikipedia.

Well, here you go, eventually I did this during my daily telecommuting and in my spare minutes yesterday and today. All in all, it took cc. 2-3 hours which is not bad.

The export format

I decided in the first place, that I will start from the List of physicists and visit each individual page and extract the date of birth. I knew the Wikipedia can be downloaded but I was too lazy to download the whole thing. On the other hand, I also knew that it is not nice to crawl the human readable HTML version of the site, and I found the middle ground: the export format, which is basically the wiki content wrapped up in some minimal XML metadata.

The date of birth

After a short investigation, it turned out the date of birth can be found usually at three different places.

1) Somewhere in the article text.

E.g. Johann Jakob Balmer's page starts with this: "Johann Jakob Balmer (May 1, 1825 – March 12, 1898)"

2) In the so called infobox,

which is on the right side of the page. It can contain the dates in different formats, see the birth_date and death_date for example.

"{{Infobox scientist

|name = Johann Jakob Balmer

|image = Balmer.jpeg

|image_size = 220px

|caption =

|birth_date = May 1, 1825

|birth_place = [[Lausen]], [[Switzerland]]

|death_date = {{dda|1898|3|12|1825|5|1}}

|death_place = [[Basel]], [[Switzerland]]

|residence =

|citizenship =

|nationality = [[Switzerland]]

|ethnicity =

|field = [[Mathematics]]

|work_institutions =

|alma_mater = [[University of Basel]]

|doctoral_advisor =

|doctoral_students =

|known_for =

|author_abbrev_bot =

|author_abbrev_zoo =

|influences =

|influenced =

|prizes =

|religion =

|footnotes =

|signature = }}"

3) Persondata.

I have learnt, that every biographical article have some metadata, that is not visible on the human readable page. It's purpose is exactly to help my work, that is help the automatic extraction of information. What I did not expect, this can contain the dates in different formats as well.

"{{Persondata
| NAME = Balmer, Johann

| ALTERNATIVE NAMES =

| SHORT DESCRIPTION =

| DATE OF BIRTH = May 1, 1825

| PLACE OF BIRTH = [[Lausen]], [[Switzerland]]

| DATE OF DEATH = March 12, 1898

| PLACE OF DEATH = [[Basel]]

}}"

The cache

Since I knew I will run the whole process several times until I get everything right, I decided to save every webpage I download, so that I do not have to download them unnecessary several times. I also saved the dictionaries I have built into files, so that later I can load them and rerun a partial step of the process without running every other preceding step.

Epilog

Just after I have finished writing up this, I've made one last Google search. It's a shame I only read this Stackoverflow thread after I have finished. I should have started with that. There are several tools that I could have used:
- Pywikipediabot: collection of python scripts automating work on wikipedia articles
- How to extract Persondata from the SQL dump files
- Wikidata
- this is the worst: forget export format. there is an API which lets you retrieve data in JSON format. Next time start here: http://www.mediawiki.org/wiki/API/Tutorial
- there is a Wiki format parser module that will return the date of birth
I could rewrite this in a few lines of code. But I won't do this now, because it is DONE. Next time I grab info from Wikipedia, I will be smarter.

The code:

Problem:
Suppose you flip coins for a certain times, say n. You want to know your chance of getting certain number of heads in a row, say k heads in a row.

Let's start with the function that'll do a certain number of coin flips:
Let's do 10 flips right away:

do_n_flips(10)

'0100111001'

Let's suppose you get 1 euro if you get the given number of heads in a row while doing a certain number of flips. Here's the function that'll calculate your payoff:
Let's try our payoff function with doing 10 flips and getting payed for 3 heads in a row. sometimes it is 0, sometimes it is 1:

payoff(10, 3)

0.0

payoff(10, 3)

1.0

Now, what is your chance to get your 1 euro for 3 heads in a row from 10 flips? This brings us to the Monte Carlo simulation, which I'wont describe here at all, just give you a function which answers our question regarding the chance:
Running with a million iteration gives us a good approximation for the chance:

monte_carlo_solve(10, 3, 1000000)

0.507753

This post was inspired by a similar post by Remis.

bpgergo

Thursday, October 4, 2012

Use Python to extract date of birth from Wikipedia pages

Physicists grouped by zodiac

Tuesday, January 31, 2012

Monte Carlo Simulation in Python

Wednesday, January 4, 2012

How to use wildcard in Google search