DCHAS-L Archives · Post dated October 12, 2026
From: Rob Toreki <info**At_Symbol_Here**ILPI.COM>
Subject: Re: [DCHAS-L] Even more Archive improvements
Date: Oct 12, 2026 02:44 UTC
Reply-To: ACS Division of Chemical Health and Safety <DCHAS-L**At_Symbol_Here**PRINCETON.EDU>
In-Reply-To: [DCHAS-L] Even more Archive improvements
We reconciled the archive’s web pages with its underlying records and found 757 existing posts that were not recorded in the database used to rebuild the website. Those posts are now included, with their existing web addresses and corrected content preserved.We also restored missing or truncated message text from older backups and original email files. In some cases, some of the text did not display on the page because the importing software mistook part of the message for an attachment. That problem has been corrected, and future imports will flag messages that appear to have no readable content.We removed 41 duplicate posts (which can happen for multiple reasons, humans and technical) while retaining repeated announcements and reminders that represented separate messages. Three genuinely empty posts were also removed. Automatic redirect rules on the server will invisibly send readers using older duplicate addresses to the retained copies.The entire archive was then rebuilt, including its yearly indexes, navigation, statistics, and the lists that help search engines discover its pages.
Thousands of small formatting and conversion errors were repaired across hundreds of posts. These included broken quotation marks, damaged formatting instructions, and leftover email encoding that could interfere with how a page displayed.Some web addresses, email references, and passages of text had become invisible because browsers mistook them for formatting instructions. Those were brought back into view. We also corrected page information that was confusing search-indexing softwareThese were targeted repairs to presentation and readability, not changes to the text. The original wording and historical context remain.There is still a lot of HTML spinach/junk in the old posts, most of which you wouldn’t see on the page anyway, but this cleaned out quite a bit of it.
We checked nearly 34,000 distinct external web links remaining since my last dead link cleanup pass. 2,881 occurrences of confirmed dead links were marked using the archive’s dead-link convention, preserving their visible text and original addresses while deactivating them as hyperlinks.The checks were deliberately conservative. A timeout, access restriction, or temporary server error was not enough to declare a link dead. Those uncertain links were left alone. Where verified, older links were also updated to working secure addresses.Missing pictures were an unaddressed area and now get a consistent treatment. Approximately 1,000 broken remote image URL's were replaced with compact notices saying “Original image not available,” with small icons used for very small images. When a missing image linked to another page, the replacement includes a clearly visible "Open external link" option.TIP: as the list now accepts attachments it is always better to attach images rather than link to them on a remote server given how pervasive link rot is. The archive will keep an attachment image forever; a picture of a seminar announcement stored on a random web site will usually stop working after a couple months.We also removed 82 defunct tracking and spacing images that served no useful purpose for archive readers.
One of the most important improvements is behind the scenes. The original database design holds all the post metadata (author, date, reply-to thread etc.) in an SQL database, but the actual post content was stored in the original emails. However, if I needed to rebuild the archive to accommodate new found files like we did this weekend, any edits to the previously generated post files would be overwritten.Therefore, the post content is now also stored in the database. When I go in and deactivate a dead link in my local HTML post file, that edit will be written back to the database. We still have the original source material stored elsewhere so it is possible to retrieve that independently if ever needed.One of the key aspects of this was automatically adding a digital “fingerprint” (SHA-256 content hash) to each pages’ message content when it is generated. This hidden string helps the archive system recognize whether a file has changed since it was last saved. If even one character is changed, the fingerprint will no long be valid.For routine upkeep, fingerprints let the updating process quickly skip unchanged pages and identify corrections that need to be saved into the archive’s underlying records. They also distinguish changes to a message from changes to the surrounding page layout. This makes maintenance more efficient and helps ensure that future rebuilding preserves our corrections.
Finally, the fingerprints lay the foundation if someday we want to have multiple editors maintaining the archive. Suppose one editor repairs a message and another later submits changes based on an older copy. The system can recognize that the second editor’s starting version is out of date and stop for review, rather than silently overwriting the newer correction. No plans for it, but implementation will be easy.
