It is with great excitement I announce that after an almost four-year pause, the  DCHAS-L archives have resumed adding posts and is now fully up to date and VASTLY improved. If you've forgotten about this resource, the home page is https://www.ilpi.com/dchas/index.html 

You may recall that preserving the wisdom of DCHAS-L was a project I launched years ago with help from Ralph Stuart: https://acsdchas.wordpress.com/wp-content/uploads/2015/08/m3-toreki-12000-thoughts.pdf  At the time we presented that ACS award address, I had archived 12,132 posts. Today, the total is almost 29,000!

Background: In mid 2022, Apple changed its mail program and the custom scripts I had written to archive the posts broke. This project was always a challenge because there is - still to this very day  - actually no one standard for email body content like there is for say, HTML. Posts can arrive in text, rich text (RTF), HTML, binary encoded (base64), and others, and all of those can have different text encodings. Which meant there were A LOT of glitches in the previously archived posts with incorrectly displaying characters and other artifacts. And when Apple revamped their mail behavior, I realized the time had come to reimagine the project using modern standards. But was a heavy lift to plan that and execute it as  I lacked the skill set to make it happen. So, until this week, just over 3,000 posts had accumulated but not been processed.

A Modern Architecture: I spent a good week contemplating the architecture behind this before writing the new code in a weekend blitz. While I had been identifying important data (author, date, subject, message ID, In-Response) in the posted files, it really needed to have the metadata tags that will allow me (and others, eventually) to exploit the data for some really neat uses (more on that later). To do that requires a formal database. And do that requires laying out a complete ontology where we formally define, identify, and map data contained in the post such as author, date etc. 

As a result, new posts are processed into an SQLite database (open source!) on the back end: https://sqlite.org/index.html using custom Python scripts (https://www.python.org/) This is a quantum leap in the project as the data can now be extracted one time and injected into anything. So to generate the latest set of posts, I have a template file where that metadata is injected into the header, body and footer to create each post. The same template standardizes the navigation elements etc. and another template builds the yearly indices. And if I ever reimagine the presentation of data or want to do something with that data it can simply be done across all the files in the database with a new template file and one command instead of having to scan all the existing posts. Powerful stuff. If you're a data scientist type person (looking at you, Leah) and you peek into the HTML files, you'll find a bunch of metadata tags in each one:

The backend database also stores the original complete content. So while I only present munged email addresses on the site: ralph**At_Symbol_Here**rstuartcih.org, the backend database has the normal version. Previously, I had been discarding posts once processed, but now if it becomes necessary to go back and reimagine things again, the original pure unchanged data is there. As well as the original backup mailbox files since mid-2022.

Public-facing changes: Here's how it impacts the user experience:

1. My new solution uses Python's (open source!) mail encoding packages to extract the post content and coerce the data into a standard encoding. What this means for the reader is that the glitchy character issues have basically disappeared and in many cases the formatting will closely match that you saw in your mail program. It's not 100% perfect - so in plain text posts a quoted section might have ">" at the beginning of each lin instead of a vertical bar and indent, but that's livable and not worth fixing.

2. The Previous by Date and Next by Date tables have been replaced with a nice CSS implementation that looks like little data cards/buttons. Unlike the old clunky HTML tables (take a look at a post prior to 2022) which were unforgiving on resizing the page to small widths, the new implementation handles small screen sizes gracefully, moving them on top of each other at a mobile size. See, for example, https://www.ilpi.com/dchas/2025/20251108.html 

3. The annual indices such as https://www.ilpi.com/dchas/2023/index.html which one could describe as a "wall of text" now highlight in yellow when you hover over an article link. This makes it clear exactly what you are clicking on and is handy on small screens. When scrolling with a mouse it’s like having a finger on your page to keep your place.

4. The main index page used to be just a bullet list of 2022, 2023, 2024 with the same long list issue. I replaced that with a tiled array that gracefully resizes with the window, shows the # of posts per year, and follows the same button/card visual motif used in the Previous/Next navigation used in individual posts: https://www.ilpi.com/dchas/index.html 

5. The list started accepting attached files in 2023. Which is yet another coding nightmare. Attachments are now archived with their posts - the only issue you may note is that they aren't inline. So if someone included a picture in the middle of the post, it now appears after the post body as a clickable link that opens in a new window. See, for example, https://www.ilpi.com/dchas/2023/20231024.html 

6. Dead link cleanup. Before starting any of this I wrote scripts to crawl the existing files and find URL's that were not responding or 404. As this is an archive, I wanted to preserve the actual URL's and post format, so the link stays there. It looks like a link but because it's dead it has a little frowny face emoticon after it. If you mouse over the dead link a little box pops up that says "dead link". But it also shows you the original URL in case you want to see if there's an archived copy on the Internet Wayback Machine (https://web.archive.org/), search the original site (if it exists) or so some web searching. And as it's not clickable, you don't experience the frustration of opening a 404 page and searchbots are not dinging the site for so many dead links. Example: https://www.ilpi.com/dchas/2022/20221223a.html 

And when I say cleanup - wow. As the Archive has so many news articles, links to personal pages, links to one-time events etc.the half-life of links probably two years. I ended up deactivating approximately 8,000 dead links!

7. Differing timezones were a nightmare. Posts would come in with all sorts of different timezones and formats and determining the chronological order was a pain. The new architecture converts all posting times to UTC (Coordinated Universal Time).

LabLocks™ - The first device that can lock out standard laboratory ball valves: https://www.safetyemporium.com/07400

US-Made NIOSH and FDA-approved N95 respirators: https://www.safetyemporium.com/safety-items/respirators/

Safety Emporium - Laboratory and Safety Supplies 
https://www.SafetyEmporium.com
esales**At_Symbol_Here**safetyemporium.com  or toll-free: (866) 326-5412
Fax: (856) 553-6154, PO Box 1003, Blackwood, NJ 08012