#!/usr/bin/python
# -*- coding: UTF-8 -*-
u"""
===============================================================================
sitesearch.py - redirect a site-specific search query to a search provider.
Author/Copyright: 2016-2020, M. Lutz (learning-python.com)
License: provided freely, but with no warranties of any kind.
Versions:
Sep 18, 2020 - Fix Py3.X+Linux redirect-page print excs on non-ASCII terms.
Try but skip alt format for non-ASCII Location search URLs.
Fix log-file write excs in Py2.X for non-ASCII term encodes.
Jun 28, 2019 - Document and improve the privacy and permissions of the
search-terms file saved on the server.
Jun 28, 2018 - Improve and describe handling of Unicode in search terms,
remove Ixquick (it's now StartPage), add Baidu and Yandex,
unescape URL displayed to user in UTF-8 fallback page.
Jan 25, 2016 - Log search terms to a server file for metrics, with date
and time. For privacy, set this file's Unix permissions
to 0200 (or 0600) per "ABOUT PERMISSIONS..." ahead.
A Python CGI script: runs on the server, reads form (or URL query) input,
prints HTTP headers + HTML text to the client. This script builds a query
URL and submits and delegates it to a search provider via an HTTP redirect.
Example - search entire site:
user inputs = [site=Entire site, term=spam, host=Google]
search string = "spam site:learning-python.com"
URL = https://www.google.com/search?q=spam+site%3Alearning-python.com
Example - search individual parts (if any):
user inputs = [site=Books only, term=decorator, host=DuckDuckGo]
search string = "decorator site:learning-python.com/books"
URL = https://duckduckgo.com/?q=decorator+site%3Alearning-python.com%2Fbooks
Normally invoked by the "action" tag of the form in the companion HTML page at:
http://learning-python.com/sitesearch.html
Use your browser's "View Source" to see the form in this HTML page's code.
To use for other sites, edit the HTML select list and the "sites" dict below.
As usual, this script can also be invoked from a browser or script using a
GET-style URL with query parameters at the end like this (all on one line):
http://learning-python.com/cgi/sitesearch.py?
searchhost=DuckDuckGo&searchsite=Entire+site&searchterm=fortran
This script's scheme exits the hosting site for search results, in exchange
for a relatively simple way to add multiple-provider search that requires
neither JavaScript nor proprietary widgets.
Usage note: although search providers can be selected for comparison,
DuckDuckGo or StartPage are strongly recommended. Other providers may
insert ads and unrelated photos in results and track searchers, and Google
occasionally disables the Back button. See the HTML page for more details.
Coding note: uses \n instead of \r\n for line breaks, because all known
clients accept it; print adds a \n by default; and Windows may expand \n
to \r\n automatically, which could change an explicit \r\n to \r\r\n.
See https://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.3.
Coding note: this script is portable to both Python 2.X and 3.X, but its code
is constrained by the need to run on Python 2.4 -- the most recent Python at
the ISP hosting this site (Godaddy). OTOH, 2.4 works fine, which begs the
question: were all the Python changes since 2004 really that important?...
Update, June 2017: learning-python.com is now a single site/part; the former
"Books only" in both code here and older examples is defunct but harmless
(it's disabled altogether in the input form's HTML - no changes made here).
Updates, June 2018: Ixquick has been merged into StartPage (and is disabled
in the form's HTML); this site's host now runs Python 2.6 by default (though
this has no impact here); Baidu and Yandex have been added to the list of
supported providers; and Unicode handling is modestly improved (per below).
-------------------------------------------------------------------------------
ABOUT PERMISSIONS AND PRIVACY OF THE SEARCH-TERMS FILE
This script logs search terms to a server file for metrics, adding date
and time but no user information. For best privacy, set this file's Unix
permissions to 0200 (and possibly omit it from its directory's index).
Permission 0200 allows this script to update the file, but prevents the
file from being accessed on the web at large -- by either direct URL,
or general file-viewer scripts (e.g., this site's cgi/showcode.py).
Permission 0600 suffices for sites with no file-viewer script (it prevents
direct-URL access only), and may be required to download the terms file;
change as needed. Use permission 0755 (executable) for this script itself.
-------------------------------------------------------------------------------
ABOUT SEARCH TERMS WITH ARBITRARY UNICODE CHARACTERS
TBD: this script may or may not support non-ASCII Unicode in search terms.
While it could encode the redirect URL to something like UTF-8 bytes, it's
not clear that servers would use this in "Location:" lines, even given a
content line of "Content-type: text/html; charset=UTF8". Resolve me.
UPDATE, Jun 2018: for non-ASCII characters (e.g., BMP symbols and emojis),
the HTML form that triggers this script now uses an [accept-charset="UTF-8"]
attribute to force inputs to be encoded to UTF-8 bytes and URL-escaped ("%XX")
for transmission (a charset meta tag for the entire page is now set as well).
This is an improvement, not a complete fix, though the reasons are subtle.
Before this, browsers would send characters outside the default charset
oddly: as URL-escaped HTML entities. For example, a user's searchterm
input "aaa☞bbb" would be sent from the browser as "aaa%26%239758%3Bbbb",
and become "aaa☞bbb" after URL unescaping. URL-escaping this again
here for the redirect would recreate the browser's HTML-entity string,
which only Google seemed to recognize well (StartPage returned a few hits,
but all other search sites listed here returned all or no pages in response).
With the new accept-charset attribute, a searchterm input "aaa☞bbb" is now
instead sent from browsers as "aaa%E2%98%9Ebbb" (the URL-escaped version
of Python bytes-string b'aaa\xe2\x98\x9ebbb'), which URL-unescaping turns
into the original input string, and URL-reescaping here restores.
This avoids having to use an HTML parser to munge entities, if required.
Unfortunately, though, its only real benefit today seems to be making the
term clearer in the generated URL. Google and StartPage are still the only
search sites that support such search terms reliably (HTML-entity or not),
and Google seems to be the better of the two; Baidu and Yandex support
varies per character, and others ignore the search term altogether.
Hence, barring a future revelation (and time to stumble onto one), search
users should try Google first for most searches using non-ASCII characters.
Alas, Unicode on the web is still riddled with intellectual black holes.
This very note required the charset declaration in line 2 above for the
host's Python 2.X (though 3.X's UTF-8 default happily groks any and all ☞).
For background details on such dark matters, try:
https://www.w3schools.com/tags/att_form_accept_charset.asp
https://tools.ietf.org/html/rfc2616#section-14.2
https://www.w3schools.com/html/html_forms.asp
Update: the next section's item #2 explores a later failed try on this front.
-------------------------------------------------------------------------------
SEP-2020: THREE NON-ASCII IMPLEMENTATION UPDATES (search for "SEP-20" for all)
1) Fix silently failing "print(manualredirect)" in 3.X for non-ASCII terms
When using Python 3.X on Linux, this fallback-page print near the end of
the script raised an exception for non-ASCII search terms. This was never
noticed because this text followed the redirection headers (it was meant
only for redirect failures), and clients had already redirected before the
exception text was seen. The later save of the search term to the local
log file, however, had been failing for non-ASCII terms ever since the host
website moved to a server with 3.X (earlier non-ASCII results are unknown).
TO FIX, the fallback-page text is now encoded to UTF-8 bytes in 3.X
before printing, and printed as bytes; this prevents automatic ASCII
encoding for str in print(). A "charset" header parameter was also added
for the UTF-8 encoding. With the fix, the log file now records non-ASCII
terms (even for emojis), and the redirect page both properly displays
decoded non-ASCII text and redirects to the search engine (if ever needed).
2) Try alternative non-ASCII term formatting added to showcode - and punt
This script experimented with using the HTTP header parameter formatting
applied to fix non-ASCII filenames in the raw-text replies of showcode.py.
For the full story on this scheme, search for "Sep-2020" in:
https://learning-python.com/cgi/showcode.py
In showcode.py, this yields properly encoded-and-then-escaped filenames,
which prevent print() exceptions in Py 3.X on Linux, and is better for
clients in general:
Content-Disposition: inline; filename*=UTF-8''%c2%a3%20spam%20%e2%82%ac%20eggs
Here, applying this same formatting yields Location headers like this
(the first of these searches for a smiley emoji, the second for '真Л⇨'):
Location: UTF-8''https%3A%2F%2Fwww.google.com%2Fsearch%3Fq%3D%F0%9F%98%8A%2Bsite%3Alearning-python.com
Location: UTF-8''https%3A%2F%2Fwww.google.com%2Fsearch%3Fq%3D%E7%9C%9F%D0%9B%E2%87%A8%2Bsite%3Alearning-python.com
This differs from the original's encoded-and-then-escaped parts:
Location: https://www.google.com/search?q=%F0%9F%98%8A+site%3Alearning-python.com
Location: https://www.google.com/search?q=%E7%9C%9F%D0%9B%E2%87%A8+site%3Alearning-python.com
The new alternative formatting DIDN'T WORK here for the value of the Location
header (as coded and tested, at least). In all cases, the alternative
formatting produced "Not Found" reply pages in the client. This is perhaps
because the standard seems for HTTP header _parameter_ values, not the entire
header value; the net effect appears to invalidate the redirect altogether.
Hence, this script is sticking with its original formatting scheme of
URL-encoding individual parts of the redirect URL. The good news is that
this seems to work on most search engines listed here for non-emoji non-ASCII
terms such as '真Л⇨' (though emoji support outside Google is still spotty).
Yandex also now calls out 'automated' searches like those here; sigh...
3) Fix exceptions in 2.X (only) on log-file-line encode of non-ASCII terms
Finally, the line log-file write() at the end could fail for non-ASCII
terms, because it was an already-encoded-bytes str in 2.X, which bombed in
the manual encode. It's a decoded-codepoints str in 3.X, which does require
the encode. Alas, 2.X/3.X Unicode portability may not be worth it today.
===============================================================================
"""
####################
# CODE STARTS HERE #
####################
from __future__ import print_function # enable py3.X print() in py2.X
import cgitb; cgitb.enable() # route python exceptions to browser/client
import cgi, sys, os, time
UsingPython3 = sys.version[0] == '3'
UsingPython2 = sys.version[0] == '2'
# fetch url reply, url-encode/decode text
if UsingPython3: # py 3.X/2.X compatible
from urllib.request import urlopen # *urlopen not yet used*
from urllib.parse import quote_plus, unquote_plus
else:
from urllib import urlopen, quote_plus, unquote_plus
# Jan 2016: save search terms to this server file for metrics
SAVETERMSFILE = 'sitesearch-savedterms.txt'
# testing
MOCK = False # True=simulate form/url-query inputs
DEBUG = False # True=display url without submitting it
#==============================================================================
# Fetch and map inputs to url parts
#==============================================================================
searchsites = {
'Entire site': 'learning-python.com', # Jun 2017: now Entire only
'Books only': 'learning-python.com/books', # [url-encodes '/' ahead]
#'Training only': 'learning-python.com/training' # [not yet (ever) supported]
}
searchhosts = {
'DuckDuckGo': 'duckduckgo.com', # no tracking, ads, or images
'Google': 'www.google.com', # all of the above + Back breaks...
'Bing': 'www.bing.com', # ads + images + less fruitful?
'StartPage': 'startpage.com', # google results, no tracking
'Yahoo': 'search.yahoo.com', # you be the judge
'Baidu': 'www.baidu.com', # Jun 2018: popular in China
'Yandex': 'www.yandex.com', # Jun 2018: popular in Russia
'Privado': 'www.privado.com', # Sep-2020: privacy-focused newcomer
'OneSearch': 'www.onesearch.com', # Sep-2020: privacy-focused newcomer
#'Ixquick': 'ixquick.com', # [Jun 2018: now redirects to StartPage]
}
#------------------------------------------------------------------------
# Fetch inputs.
# From html form fields, url query parameters, or mock-up.
#------------------------------------------------------------------------
if not MOCK:
form = cgi.FieldStorage() # parse live form/url input data
else:
class Mock: # simulate form input to test
def __init__(self, value):
self.value = value
form = {'searchsite': Mock('Entire site'),
'searchterm': Mock('"class decorator" ☞'), # or 真Л⇨'),
'searchhost': Mock('Google')}
# => Location: https://www.google.com/
# search?q=%22class+decorator%22+%E2%98%9E+site%3Alearning-python.com
#------------------------------------------------------------------------
# Map inputs.
# Missing and invalid keys in url queries trigger cgitb KeyError displays.
#------------------------------------------------------------------------
site = searchsites[form['searchsite'].value]
host = searchhosts[form['searchhost'].value]
if 'searchterm' in form: # but handle missing term: starts empty
term = form['searchterm'].value # py 2.4 has no 'a if b else c' expr
else:
term = ''
#-----------------------------------------------------------------------
# Jun 2018: Drop 'atag' prefix in favor of per-provider patterns,
# because Baidu and Yandex URLs are too custom for prior scheme.
# The former "/" before an empty atag is optional (ddg adds one).
# urltemplate = 'https://%(host)s/%(atag)s?q=%(term)s+%(pref)s%(site)s'
#
# Sep-2020: Yandex now may detect and call out automated search URLs
# (requiring a user verify), and Bing may require multiple Backs to
# exit. Newly added: Privado and OneSearch are both privacy-focused
# newcomers (there may be more like them). Regrettably, most here
# may show sponsored ads first - even those pushing privacy.
#-----------------------------------------------------------------------
urlgeneric = 'https://%(host)s?q=%(term)s+%(pref)s%(site)s'
# keys should match searchhosts above
urltemplates = {
'DuckDuckGo': urlgeneric,
'Google': 'https://%(host)s/search?q=%(term)s+%(pref)s%(site)s',
'Bing': urlgeneric,
'StartPage': 'https://%(host)s/do/search?q=%(term)s+%(pref)s%(site)s',
'Yahoo': 'https://%(host)s/search?q=%(term)s+%(pref)s%(site)s',
'Baidu': 'https://%(host)s/s?ie=utf-8&wd=%(term)s+%(pref)s%(site)s',
'Yandex': 'https://%(host)s/search/?text=%(term)s+%(pref)s%(site)s',
'Privado': urlgeneric, # Bing-powered
'OneSearch': urlgeneric # caveat: requires an extra click
}
urltemplate = urltemplates[form['searchhost'].value]
#-----------------------------------------------------------------------
# Encode and escape parts, expand URL template.
# URL-escapes the search-term text input, site '/', and label ':'.
# This escapes individual parts of the redirect URL, not all of it.
# Also explicitly encodes term's str codepoints to utf8 bytes in py3.X;
# quote_plus() does this auto in 3.X only, but that's far too implicit.
# quote_plus() also always returns an all-ASCII _str_, for any input.
#-----------------------------------------------------------------------
# Sep-2020: formerly relied on quote_plus() utf8 default in py3.X (only)
if UsingPython3:
term = term.encode('utf8') # it's already encoded bytes in 2.X
term = quote_plus(term) # pass encoded bytes in both 3.X and 2.X
site = quote_plus(site) # '"A/B C:D E"' -> '%22A%2FB+C%3AD+E%22'
pref = quote_plus('site:') # unquote_plus() reverses the %-encoding
# expand template
searchurl = urltemplate % vars() # Jun 2018: now table-driven
#==============================================================================
# Pass 'term site:xxx' query to providers
#==============================================================================
#------------------------------------------------------
# Fallback-option page (or via HTTP Refresh header).
# Caution: the body's HTML may contain non-ASCII text.
# Jun 2018: unescape link displayed to user (only).
#------------------------------------------------------
manualredirect = """
Redirecting to search host...
If this fails, please click this link instead:
%s
""" % (searchurl, searchurl, unquote_plus(searchurl))
if not term:
# error check, custom reply
print('Content-type: text/plain\n') # start reply: hdr + blankline=\n
print('Please provide a search term in field "Search for this:".')
elif DEBUG:
# disply built link only
print('Content-type: text/plain\n') # start reply: hdr + blankline=\n
print(searchurl) # with encoded-and-escaped term
else:
#--------------------------------------------------------------------
# Valid and live: redirect client to a search-provider's site/page,
# by sending an HTTP reply = [headers + blank line + HTML page]
# having a 'Location:' header with a formatted search-query URL.
# Hosting site auto generates a 3xx status header if 'Location:'.
# Print HTML reply too, in case client doesn't redirect as planned.
#--------------------------------------------------------------------
#print('HTTP/1.1 302 Found') # <= added by host if 'Location:'
print('Location: %s' % searchurl) # cgi http redirect header
#print('Connection: close') # <= this seems optional or auto
print('Content-type: text/html; charset=UTF-8') # Sep-2020: add charset for body
print('') # '' else 2.X tuple sans __future__
#--------------------------------------------------------------------
# HTML page shown if 'Location:' header redirect fails (unlikely).
#
# Sep-2020: The original print() below produced a to-ASCII encoding
# exception in Python 3.X on Linux for all non-ASCII search terms;
# redirects silenced the exceptions, but later log writes didn't
# happen. To fix, 3.X encodes manually and writes as bytes instead.
#--------------------------------------------------------------------
if UsingPython2: # py2.X: already encoded bytes
print(manualredirect) # py2 stdout allows str of bytes
else: # py3.X: manually encode str
manualredirect = manualredirect.encode('utf8') # match charset header encoding
try:
sys.stdout.flush() # mar20: else headers last on 3.X
sys.stdout.buffer.write(manualredirect) # py3 stdout is str: use io layer
except:
print('Page error', repr(sys.exc_info()[1]), file=sys.stderr)
#--------------------------------------------------------------------
# Jan 2016: Save search term (pre-encode) to flat file on server.
# Open with exclusive lock for possibly concurrent web access.
# Encode str to bytes for Py 3.X, no-op on Py 2.X unless non-ASCII.
# Use permission 0200 for updates but inaccessibility on the web.
# Sep-2020: don't encode line in 2.X - encoding fails for non-ASCII.
#--------------------------------------------------------------------
try:
filename = SAVETERMSFILE
message = '[%s] [%s]' % (time.asctime(), unquote_plus(term)) # jan25
if not os.path.exists(filename):
open(filename, 'w').close() # make file first time
fd = os.open(filename, os.O_EXCL | os.O_APPEND | os.O_WRONLY) # binary on Unix
line = message + os.linesep
if UsingPython3:
line = line.encode('utf8') # Sep-2020: it's already encoded bytes in 2.X
os.write(fd, line)
os.close(fd)
except:
print('Log error', repr(sys.exc_info()[1]), file=sys.stderr) # to error_log
pass # neither server nor client care if this fails
# And the client reroutes to the search provider (which hopefully plays nice)