{"id":232,"date":"2017-06-23T17:19:34","date_gmt":"2017-06-23T22:19:34","guid":{"rendered":"https:\/\/cdrhdev.unl.edu\/log\/?p=232"},"modified":"2017-06-26T15:45:42","modified_gmt":"2017-06-26T20:45:42","slug":"repairing-nebraska-newspapers-space-time-continuum","status":"publish","type":"post","link":"https:\/\/cdrhdev.unl.edu\/log\/2017\/repairing-nebraska-newspapers-space-time-continuum\/","title":{"rendered":"Repairing the Nebraska Newspapers Space-Time Continuum"},"content":{"rendered":"

\"Nebraska<\/a>Nebraska Newspapers<\/h2>\n

Since 2007, the CDRH has been cultivating our web-based time machine, Nebraska Newspapers<\/a>, in partnership with the Library of Congress’s National Digital Newspapers Program (NDNP)<\/a> and funded by grants from the National Endowment for the Humanities (NEH)<\/a>. At time of writing, we currently have published 45 newspapers with full text and high resolution scan images. And there are more on the way after receiving our third NDNP grant in 2016.<\/p>\n

\"National<\/a><\/p>\n

Plattsmouth Papers<\/h3>\n

\"Newspaper<\/p>\n

Upon ingest of the Plattsmouth, Nebraska newspapers, CDRH staff noted signs of a rift in the space-time continuum: some problems with the dates assigned to some of the newspaper issues. Papers from the early 1900s were miskeyed as being from the early 1800s — a time when few white settlers were living in Nebraska and certainly when no newspapers were being published here. These errors were not caught in the validation process, because the validator only checks to see if the date value has a valid date format, not whether the date is actually the correct one. Typos like this sound simple to fix, but changing the date across hundreds of filenames and within the XML of said files would take hours and hours to complete manually.<\/p>\n

\"\"
Laura’s timeline of Plattsmouth papers’ title shifts, publication dates, and LCCNs<\/figcaption><\/figure>\n

The Plattsmouth Public Libraries<\/a> soon also noted that title shifts had not been identified. Plattsmouth has had two major newspapers, The Plattsmouth Herald<\/em> and The Plattsmouth Journal<\/em>, but both of these newspapers have undergone title changes throughout their history. According to standard cataloging practice, each newspaper title shift requires a unique Library of Congress Control Number (LCCN). Additionally, the NDNP guidelines require the use of print LCCNs, not those identifying the microfilm versions. Although the Plattsmouth papers had not been selected for inclusion in the NDNP, CDRH staff decided they should comply with the Library of Congress standards in hopes of being able to include the papers in Chronicling America<\/a> in the future.<\/p>\n

Given the information from the Plattsmouth Public Libraries, and armed with the US Newspaper Directory<\/a>, our Metadata Encoding Specialist Laura Weakly<\/a> investigated and sorted out what had been mixed up with the aid of her trusty whiteboard. Laura documented a few date ranges where LCCNs had been misapplied when the paper’s name had changed and identified where the microfilm LCCN had been used. Like correcting the dates, repairing these LCCN problems would involve similarly monumental amounts of manual work. Rather than undertake this time-consuming process, Laura approached the dev team about writing scripts to save time and minimize the possibility of new typos.<\/p>\n

\"Flux<\/h2>\n

Python Scripts<\/h2>\n

To write the needed scripts, I decided to use our flux capacitor Python<\/a>, because it powers the Library of Congress’s Chronicling America and other NDNP websites’ Chronam software<\/a>, including Nebraska Newspapers. We usually use Bash or Ruby scripts for our projects, but I wanted this to be written in the language familiar to others working on NDNP projects. I had only written a small handful of Python scripts in the past, but I looked forward to refreshing my time-saving skills with it.<\/p>\n

Fix Dates by LCCN<\/h3>\n

Source: fix_dates_by_lccn.py<\/a><\/p>\n

I chose to start by focusing on the incorrect dates as it seemed a little less complex than updating LCCNs.<\/p>\n

\"Time<\/p>\n

The script begins by using argparse<\/a> to implement common command line options and the necessary arguments to control the script, similar to how I had begun using getopts<\/a> for my Bash scripts and OptionParser<\/a> for my Ruby scripts.<\/p>\n

# Arguments\r\n# ---------\r\nparser = argparse.ArgumentParser()\r\n\r\n# Optional args\r\nparser.add_argument(\"-d\", \"--dry_run\", action=\"store_true\",\r\n help=\"don't make any changes to preview outcome\")\r\nparser.add_argument(\"-q\", \"--quiet\", action=\"store_true\",\r\n help=\"suppress output\")\r\nparser.add_argument(\"-s\", \"--search_dir\",\r\n help=\"directory to search (default: \/batches)\")\r\nparser.add_argument(\"-v\", \"--verbose\", action=\"store_true\",\r\n help=\"extra processing information\")\r\n\r\n# Positional args\r\nparser.add_argument(\"lccn\", help=\"LCCN to be fixed\")\r\nparser.add_argument(\"bad_date\", help=\"incorrect date\")\r\nparser.add_argument(\"new_date\", help=\"corrected date\")\r\n\r\nargs = parser.parse_args()<\/pre>\n
This allows us to execute the script and do a dry run<\/a> to verify what files it finds from our arguments with more verbose output without modifying any files before we run it again in quiet mode and let it silently work its magic for us:<\/p>\n
.\/fix_dates_by_lccn.py -dv sn95069723 1821 1921<\/pre>\n
\n
Searching \/batches\/\r\n\r\nSearch for bad dates in batch_pm@delivery2_ver01\/data\/sn95069723\r\n\r\n  Search for bad dates in \/00000000036\/1821111001\r\n    Fix date in file 0045.xml\r\n    Fix date in file 0042.xml\r\n    Fix date in file 0047.xml\r\n    Fix date in file 1821111001.xml\r\n\u00a0     Replace 1821 in file name with 1921\r\n\u00a0   Fix date in file 0044.xml\r\n\u00a0   Fix date in file 0043.xml\r\n \u00a0  Fix date in file 0046.xml\r\n \u00a0  Replace 1821 in dir name with 1921\r\n\u00a0   Update dates in batch XML covering sn95069723\r\n...<\/pre>\n<\/blockquote>\n
.\/fix_dates_by_lccn.py -q sn95069723 1821 1921<\/pre>\n
The script modifies the date arguments and stores them in each of the formats necessary to use in matching filenames and XML strings. Then the script calls functions to find and collect the directory paths corresponding to the LCCN argument and the directories within which match the incorrect date argument. It accomplishes this by calling os.walk<\/a>.<\/p>\n
\"Moonwalking\"
Python’s os.walk doing its thing<\/figcaption><\/figure>\n
After gathering the desired directory paths, the script identifies which files were ALTO XML or METS XML and uses the ElementTree XML module<\/a> to read and modify them. Special care is needed here to preserve some parts of the XML documents.<\/p>\n
XML Namespaces<\/h4>\n
In our original XML file, we see the list of namespaces at the top of the file:<\/p>\n
<mets TYPE=\"urn:library-of-congress:ndnp:mets:newspaper:issue\" PROFILE=\"urn:library-of-congress:mets:profiles:ndnp:issue:v1.5\" LABEL=\"The Plattsmouth Journal, 1821-10-20\" \r\n xmlns:mix=\"http:\/\/www.loc.gov\/mix\/\" xmlns:ndnp=\"http:\/\/www.loc.gov\/ndnp\" xmlns:premis=\"http:\/\/www.oclc.org\/premis\" xmlns:mods=\"http:\/\/www.loc.gov\/mods\/v3\" xmlns:xsi=\"http:\/\/www.w3.org\/2001\/XMLSchema-instance\" xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xmlns=\"http:\/\/www.loc.gov\/METS\/\"\r\n xsi:schemaLocation=\"\r\n http:\/\/www.loc.gov\/METS\/ http:\/\/www.loc.gov\/standards\/mets\/version17\/mets.v1-7.xsd \r\n http:\/\/www.loc.gov\/mods\/v3 http:\/\/www.loc.gov\/standards\/mods\/v3\/mods-3-3.xsd\" ><\/pre>\n
For the script to understand and retain namespaces, one must register all of the namespaces used within a file with ElementTree before it parses the XML:<\/p>\n
# Set namespaces before parsing\r\nET.register_namespace(\"\", \"http:\/\/www.loc.gov\/METS\/\")\r\nET.register_namespace(\"mix\", \"http:\/\/www.loc.gov\/mix\/\")\r\nET.register_namespace(\"ndnp\", \"http:\/\/www.loc.gov\/ndnp\")\r\nET.register_namespace(\"premis\", \"http:\/\/www.oclc.org\/premis\")\r\nET.register_namespace(\"mods\", \"http:\/\/www.loc.gov\/mods\/v3\")\r\nET.register_namespace(\"xsi\", \"http:\/\/www.w3.org\/2001\/XMLSchema-instance\")\r\nET.register_namespace(\"xlink\", \"http:\/\/www.w3.org\/1999\/xlink\")\r\nET.register_namespace(\"np\", \"urn:library-of-congress:ndnp:mets:newspaper\")<\/pre>\n
The namespace for the <structMap> element further down in the file is not preserved by ElementTree though, so we manually re-add it:<\/p>\n
# Restore structmap namespace that ET doesn't write\r\nfile = fileinput.FileInput(file_path, inplace=1)\r\nfor line in file:\r\n    print line.replace('<structMap>', '<structMap xmlns:np=\"urn:library-of-congress:ndnp:mets:newspaper\">'),<\/pre>\n
XML Comments<\/h4>\n
There are a few comments scattered throughout each XML document. I learned that retaining XML comments with ElementTree requires defining a custom XML parser<\/a>:<\/p>\n
# XML parser to retain comments\r\nclass CommentRetainer(ET.XMLTreeBuilder):\r\n\r\n    def __init__(self):\r\n        ET.XMLTreeBuilder.__init__(self)\r\n        # assumes ElementTree 1.2.X\r\n        self._parser.CommentHandler = self.handle_comment\r\n\r\n    def handle_comment(self, data):\r\n        self._target.start(ET.Comment, {})\r\n        self._target.data(data)\r\n        self._target.end(ET.Comment)\r\n\r\n# Parse XML using custom parser\r\ntree = ET.parse(file_path, parser=CommentRetainer())<\/pre>\n
XML Rewriting and Name Updates<\/h4>\n
With ElementTree, XML element attributes are usually rewritten in a different order than in the original file, but thankfully this is one area where XML is flexible and remains valid. Once the XML is rewritten, the script updates file and directory names.<\/p>\n
Date and LCCN String Management<\/h4>\n
I didn’t realize until I had spent a fair amount of time testing the functionality above that it was also necessary to update the issue records within the batch XML files. The tricky part of that was extracting the tail of the incorrect date path including the LCCN to match against the corresponding strings in the XML. There are a lot of date and LCCN strings to manage throughout this process.<\/p>\n
# Determine batch file and bad date paths\r\nbatch_path_re = re.compile('^(.+)\\\/sn[0-9]{8}\\\/')\r\nbatch_path = batch_path_re.match(bad_date_path).group(1)\r\nbatch_files = [os.path.join(batch_path, \"batch.xml\"), os.path.join(batch_path, \"batch_1.xml\")]\r\n\r\nbad_date_path_tail_re = re.compile(\".+\\\/({0}\\\/.+)$\".format(lccn))\r\nbad_date_path_tail = bad_date_path_tail_re.match(bad_date_path).group(1)<\/pre>\n
Fix LCCN by Date<\/h3>\n
Source: fix_lccn_by_date.py<\/a><\/p>\n
\"Doc
Wearing Doc Brown’s mind reading helmet is necessary for keeping track of all this information<\/figcaption><\/figure>\n
Very similarly, this script begins by handling command line options, necessary arguments, and finding the paths to the directories and files out of order. Updating the METS XML files doesn’t differ much either.<\/p>\n
Things get interesting and complicated though when the script needs to track which related reel files are copied and deleted, and whether the changes empty the directory of the incorrect LCCN. Removed reel files and emptied directories have to be removed when updating the batch XML file as well. Moved reels also have to be inserted in the correct order in the batch XML file.<\/p>\n
# Add copied reel to batch reels\r\nif not copied_reel_added:\r\n    if not args.quiet and not batch_file[-6:] == \"_1.xml\":\r\n        print \" Adding copied reel {0} to batch XML\".format(reel_copied)\r\n\r\n    reel_number = reel_copied.split('\/')[1]\r\n    reel_element = ET.Element(\"reel\", {\"reelNumber\": reel_number})\r\n    reel_element.text = reel_copied +'\/'+ reel_number +'.xml' \r\n    reel_element.tail = '\\n\\t'\r\n\r\n    if copied_reel_index:\r\n        root.insert(copied_reel_index, reel_element)\r\n    else:\r\n        root.append(reel_element)<\/pre>\n
Testing the Scripts<\/h2>\n
While writing the scripts, I manually identified and downloaded a small handful of XML files from the affected newspapers from the server without their accompanying high resolution scan images. I kept an unmodified set of the files and repeatedly copied them to another location, ran the scripts on them, evaluated the output, and deleted them. In hindsight, I could have just made a temporary Git repository and reset or checked out the unmodified files after each time I ran the scripts on them. But this slip of the mind didn’t cost me much time.<\/p>\n
Downloading Bulk Newspaper Files<\/h3>\n
When I felt the scripts were ready to test on the entire corpus of Plattsmouth papers, I researched how to exclude the image files so I could save time by downloading approximately 4GB of files instead of 300GB. I found an rsync option that provided exactly what I was looking for:<\/p>\n
rsync -ahu --info=progress2 --exclude '*.jp2'<\/strong> server_name:\/batches\/pm_delivery* .\r\n<\/tt><\/pre>\n
Logical Volume Manager (LVM) Snapshots<\/h3>\n
\"DocEven after excluding the image files, re-copying the 4GB of XML files would be too time consuming to do after each time I ran the scripts on them. If I had thought to use Git, this may<\/em> have been simpler to accomplish. I write that hesitantly because with this much data, even Git may have been significantly slower. I had been looking for a good reason to learn to use LVM snapshots since I had read about them a few years ago and have partitioned my disks for the possibility since. More specifically, I have been partitioning using LVM thin-provisioning, also referred to as LVM2.<\/p>\n
If you’re unfamiliar, I recommend reading the Red Hat documentation on Snapshot Volumes<\/a> and Creating Snapshot Volumes<\/a>. Here is a brief description and their particular use in this situation:<\/p>\n
The LVM snapshot feature provides the ability to create virtual images of a device at a particular instant without causing a service interruption.<\/p>\n
…<\/p>\n
Because the snapshot is read\/write, you can test applications against production data by taking a snapshot and running tests against the snapshot, leaving the real data untouched.<\/p><\/blockquote>\n
This feature requires a relatively modern Linux system and the necessary underlying LVM2 partitioning scheme. I created my thin-provisioned snapshot and mounted it with the following commands:<\/p>\n
lvcreate -s -kn --name newspapers_snapshot data\/newspapers\r\nmount \/dev\/data\/newspapers_snapshot \/data\/newspapers_snapshot<\/tt><\/pre>\n
I then ran my scripts on the files in \/data\/newspapers_snapshot. After the scripts would finish, I could nearly instantly both delete the snapshot which had been modified and create another snapshot, repeating as needed. To be even more thorough, I later downloaded the full 300GB data set. Applying the LVM snapshot process was still lightning fast with all the full-size page images.<\/p>\n
A cursory search shows that Windows provides a similar snapshot feature via Shadow Copy<\/a> and Mac OS users may gain the capability with the third-party software Paragon Snapshot<\/a>. I haven’t read much into or tried them though, so I can’t say whether they work as well or not.<\/p>\n
Updating Production Files<\/h2>\n
\"Doc
My face upon seeing errors on the server command line<\/figcaption><\/figure>\n
After I was convinced that everything was working correctly, I copied the scripts to the production server and tried to run them. But I had provided the wrong fuels for the flux capacitor!<\/p>\n
Python 2.6<\/h3>\n
I developed the scripts on my Fedora Workstation desktop, but the production server runs the latest CentOS 6 release which uses Python 2.6. Thankfully the only change I had to make was to add positional argument specifiers to the replacement fields in the formatted strings<\/a> I was passing to the print function.<\/p>\n
print \"\u00a0 Could not find batches with bad date {}\".format(bad_date)   # Python 2.7\r\nprint \"\u00a0 Could not find batches with bad date {0}\".format(bad_date)  # Python 2.6<\/pre>\n
ElementTree Module<\/h3>\n
The version of the ElementTree module on CentOS 6 differed from the version on my desktop as well, so some of the functions the script calls don’t exist there. I learned that Python will first search in the directory of the script being executed for modules imported<\/a>, so I tried copying the module files from my workstation to the scripts’ directory on the server. Great Scott! It worked!<\/p>\n
\"DeLorean<\/p>\n
And that was how we prevented a paradox that could have caused a chain reaction that would unravel the very fabric of the space-time continuum and destroy the entire universe… or at least could have left some inaccuracies in the Plattsmouth newspapers’ history.<\/p>\n
Many thanks to Laura Weakly, Karin Dalziel, and Jessica Dussault for their proofreading and feedback on this post.<\/p>\n","protected":false},"excerpt":{"rendered":"
Nebraska Newspapers Since 2007, the CDRH has been cultivating our web-based time machine, Nebraska Newspapers, in partnership with the Library of Congress’s National Digital Newspapers Program (NDNP) and funded by grants from the National Endowment for the Humanities (NEH). At time of writing, we currently have published 45 newspapers with full text and high resolution…<\/p>\n Read More<\/span> Repairing the Nebraska Newspapers Space-Time Continuum<\/span><\/a>","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"Layout":"","footnotes":""},"categories":[40],"tags":[44,41,42,43],"class_list":["entry","author-techgique","post-232","post","type-post","status-publish","format-standard","category-utilities","tag-lvm","tag-newspapers","tag-python","tag-xml"],"_links":{"self":[{"href":"https:\/\/cdrhdev.unl.edu\/log\/wp-json\/wp\/v2\/posts\/232","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cdrhdev.unl.edu\/log\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cdrhdev.unl.edu\/log\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cdrhdev.unl.edu\/log\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/cdrhdev.unl.edu\/log\/wp-json\/wp\/v2\/comments?post=232"}],"version-history":[{"count":41,"href":"https:\/\/cdrhdev.unl.edu\/log\/wp-json\/wp\/v2\/posts\/232\/revisions"}],"predecessor-version":[{"id":289,"href":"https:\/\/cdrhdev.unl.edu\/log\/wp-json\/wp\/v2\/posts\/232\/revisions\/289"}],"wp:attachment":[{"href":"https:\/\/cdrhdev.unl.edu\/log\/wp-json\/wp\/v2\/media?parent=232"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cdrhdev.unl.edu\/log\/wp-json\/wp\/v2\/categories?post=232"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cdrhdev.unl.edu\/log\/wp-json\/wp\/v2\/tags?post=232"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}