#!/usr/bin/python3 """ =============================================================================== ip-anon.py, a tactical script coded in Python3 Why this script? - https://learning-python.com/privacy-policy.html Admin: © 2019-2020, by M. Lutz (learning-python.com), no warranties provided. For all HTML pages in a website's folder tree, add a line to Google Analytics JavaScript code to anonymize IP addresses, iff this line has not already been added by static template expansion (e.g., by genhtml) or manual edits. See __main__ logic at the end of this file for command-line usage or the console screenshot at https://learning-python.com/ip-anon.png. This can be used as a one-time update for all the web pages in a site, and rerun by build scripts to catch new files. In the host site's case, it was used to temporarily convert code in oddballs that are edited manually outside the website tree and copied in (e.g., program docs and Apache auto-index text) or are otherwise outside the scope of template expansion, until these outliers can be manually updated and propagated to the site folder. It: - Handles mixed Unicode encodings by using bytes-mode files and ASCII-only text - Handles mixed eoln (end-of-line) sequences by inspecting file content - Converts code in both HTML files and .htaccess files' IndexHeadInsert text - Assumes a specific code/line structure - adapt for your files as needed CONFIGURATION Edit the code settings below to tailor this script for your site as needed. You may also need to expand the "isHtml" test below to check/fix additional file types. At this site, just one other file was harboring the analytics code and required a manual edit - a ".txt" HTML template used by the showcode.py file-viewer script. Your site's mileage may naturally vary. UNICODE CAUTION This script's use of bytes-mode files and ASCII-text matching assumes that all the HTML-related files in your site's tree use Unicode encodings that store ASCII text with one ASCII byte per character. This scheme works for mixes that include UTF-8, Latin-1, CP-1252, and others, but will fail to find matches in any UTF-16 or UTF-32 files lurking in your tree (their ASCII text is not simple bytes). If the latter applies, you may need to instead use text-mode files and try multiple Unicode encodings in turn until one succeeds. See this site's genhtml and showcode programs for examples of this approach. Luckily, UTF-16 and its multibyte ilk are rarely used for web content today. =============================================================================== """ import os nhtml = nsans = nfixed = 0 # global counters # Edit the following uppercase settings for your site's files Sitedir = '/Users/me/MY-STUFF/Websites/UNION' # Catalina path [Apr-2020] Copymodtimes = True # keep modtimes? [Apr-2020] Sendcmd = b"ga('send', 'pageview');" # insert set before this Anoncmd = b"ga('set', 'anonymizeIp', true);" Htmlfiledoc = b' // AUTO INSERTED Jun-2019' Htaccessdoc = b' /* AUTO INSERTED Jun-2019 */' # where eolns stripped Addline = b' ' + Anoncmd + Htmlfiledoc # fixed for htaccess later # Edit the following if your site's filenames or needs differ isHtml = lambda filename: filename.lower().endswith(('.html', '.htm')) isHtaccess = lambda filename: filename.lower() == '.htaccess' def convert(path, isHtaccess, nofileupdates=True): """ ------------------------------------------------------------------------------ Convert a single file, adding a JavaScript "set" line before "send". Auto-inserts work, but some files are copied from originals in Code/ and still need to be manually patched there too. Per ahead, this is bytes-based (to avoid encoding guesses), and checks the file to see which eoln to use on the added line (to retain the file's eoln formatting). Special case: adds a trailing '\' in Apache .htaccess IndexHeadInsert text. For .htaccess, also replaces '//' comment in added line with '/* */' form: a '//' line comment won't work, because Apache strips eolns in the inserted text - all lines after the '//' would be ignored by JavaScript as comment. New [Apr-2020]: Copymodtimes=True retains a changed file's prior modtime, so it won't register as a difference in incremental backups; use this for site publishing scripts that rerun files though this script on each build, else this script's change can mask the status of files' actual content. ------------------------------------------------------------------------------ """ global nfixed lines = open(path, 'rb').readlines() # use the same elon as file beoln = b'\r\n' if any(b'\r\n' in x for x in lines) else b'\n' # add trailing slash in .htaccess addline = (Addline + b' \\' if isHtaccess else Addline) + beoln # change comment forms for code in .htaccess insert if isHtaccess: addline = addline.replace(Htmlfiledoc, Htaccessdoc) for ix in range(len(lines)): if lines[ix].lstrip().startswith(Sendcmd): lines.insert(ix, addline) # add set before send print('\t' + '-'*40, end='\n\t') print(b''.join(lines[ix-1:ix+2])) # preview the update print('\t' + '-'*40) if nofileupdates: pass else: nfixed += 1 if Copymodtimes: prevtime = os.path.getmtime(path) file = open(path, 'wb') file.writelines(lines) # update the file file.close() if Copymodtimes: os.utime(path, (prevtime, prevtime)) break else: print('****NOT AUTO CONVERTED:', path) # matching differed def findAndConvert(listfilesonly=False, nofileupdates=True): """ ------------------------------------------------------------------------------ Find HTML files without a JavaScript "set" line, and pass on to convert(). This now uses bytes-mode files to avoid Unicode encodings altogether. This scheme works here because text matching is ASCII only. By contrast, genhtml tags matched may be any text (including non-ASCII), and showcode's text reply requires an encoding type for proper display in web browsers. Bytes mode here sidesteps the issue of guessing encodings in a mixed-encoding tree. Caveat: bytes mode also assumes all HTML files use encodings that store ASCII text as simple bytes; they do at this site, but see UNICODE CAUTION above. About eolns: naively adding a '\n' on Unix would result in mixed '\r\n' and '\n' eolns in files created in Windows, because bytes files do not map eolns to/from '\n'. Addressed by a file precheck to determine which to use (and assume the file's eolns are uniform); this seems better than text mode, which would map all eolns read and written to Unix's '\n', thereby dropping any Windows eolns in the (few) files that have them. Prior (abandoned) text-mode notes: the latin1 encoding works as a utf8 fallback (instead of cp1252) because of "passthrough": if text is both loaded and saved as latin1, any cp1252 bytes are retained intact because they are loaded and saved as their raw byte values by the latin1 decoder (see genhtml, showcode). ------------------------------------------------------------------------------ """ global nhtml, nsans for (dir, subs, files) in os.walk(Sitedir): for file in files: if isHtml(file) or isHtaccess(file): nhtml += 1 path = os.path.join(dir, file) byts = open(path, 'rb').read() if (Sendcmd in byts) and (Anoncmd not in byts): nsans += 1 print(path) if not listfilesonly: convert(path, isHtaccess(file), nofileupdates) if __name__ == '__main__': # #----------------------------------------------------------------------------- # When run as a script: # python3 ip-anon.py 1 1 => list unconverted files only # python3 ip-anon.py 0 1 => list unconverted files and preview changes only # python3 ip-anon.py 0 0 => list both and apply updates to unconverted files #----------------------------------------------------------------------------- # import sys if len(sys.argv) == 3: listfilesonly, nofileupdates = (int(x) for x in sys.argv[1:]) # 2 optional args else: listfilesonly, nofileupdates = False, True # show filenames + preview changes findAndConvert(listfilesonly, nofileupdates) print('DONE: %d html-related files read, %d files unconverted, %d files converted' % (nhtml, nsans, nfixed))