human
I am working on the file "Luke_the_Downloader/EconstorDownloader.ipynb".
The first task is:
RePec crawler
The third way to obtain RePec handles is based on RePec's search engine called IDEAS. It's obviously build to interface humans, but it can also be used by robots, as we do it.
This methods produces a RePec handle given the title of a document.
It mimics a human user that queries the search engine with the title of a document. It "clicks" the first match (if there is a match) and extract the desired information from the detail page.
It should be noted that this method is fragile and error-prone. In case the layout of the website changes, the corresponding xPath's need to be adapted appropriately. Moreover, this method can produce false-positive values. There is no guarantee that the RePec handle is the one you where looking for. But random sampling showed, that the results are quite reasonable (supposedly, because the titles from EconBiz match those on RePec very well, although not perfectly).
Can you write Python program for it?
assistant
#Here is the Python program you requested.

import urllib
import urllib2
import json
from lxml import etree
import unicodedata
import re

def determineRepecHandle_ideasCrawler(query):
    numResults = '1'
    xpathFirstResult = '//*[@id="content-block"]/dl/dt/a'
    
    def genXpathRepecDetailPage(row=1, col=1, bold=False):
        xpath = u'//*[@id="biblio-body"]/table/tr[' + str(row) + ']/td[' + str(col) + ']'
        if bold == True:
            xpath += '/b'
        return xpath
    
    # normalize text 
    #    replaces e.g. ä with a
    unicodedata.normalize("NFKD", query).encode("ascii", "ignore").decode("utf8")
    #    remove everthing that's not alphanumeric
    query = re.sub(r'[^A-Za-z0-9 ]*', '', query)
    
    # percentage encoding
    queryPercentageEncoded = urllib.quote_plus(query)
    htmlParser = etree.HTMLParser()
    # Request result list
    ideasHdl = urllib2.urlopen('http://ideas.repec.org/cgi-bin/htsearch?ul=&q=' + queryPercentageEncoded + '&cmd=Search%21&wf=4BFF&s=R&dt=range&db=&de=&m=all&fmt=long&sy=1&ps=' + numResults)
    # parse received page
    tree = etree.parse(ideasHdl, htmlParser)
    # find first match in result list
    match = tree.xpath(xpathFirstResult)
    
    # is there a match?
    if len(match) > 0:
        urlsDetailPages = match[0].values()
    else:
        return None
    
    for url in urlsDetailPages:
        if validateURL(url):
            detailPageHdl = urllib2.urlopen(url)
            detailsPageTree = etree.parse(detailPageHdl, htmlParser)
            
            for i in xrange(1, 10):
                '''
                Go through the table util you find 'Handle:'. In this
                case return value from the same row and the next 
                column, which is hopefully the RePEc handle
                '''
                xpath = genXpathRepecDetailPage(row=i, col=1, bold=True)
                matchRow = detailsPageTree.xpath(xpath)
                if len(matchRow) > 0 and matchRow[0].text.strip() == 'Handle:':
                    xpath = genXpathRepecDetailPage(row=i, col=2)
 ...