End of Term Web Archive
| End of Term Web Archive (EOT Archive) | |
|---|---|
 | |
| Mission statement | The End of Term Web Archive captures and saves U.S. Government websites at the end of presidential administrations. |
| Commercial? | No |
| Type of project | Collaborative government web archive |
| Country | United States |
| Established | 2008 |
| Funding | $0 |
| Website | eotarchive |
The End of Term Web Archive is a collaborative project to archive the web presence of U.S. federal agencies during administration changes.
The data produced from the project is publicly available for bulk download through the Internet Archive or Amazon Web Services Open Data Registry. Limited keyword search and replay capabilities are available through the Wayback Machine.
Background
[edit]The End of Term Web Archive was set up following a 2008 announcement from National Archives and Records Administration (NARA) that they would not be archiving government websites during transition, after carrying out such crawls in 2000 and 2004.[1]
From a legal perspective, the Federal Records Act (FRA) of 1950 mandates federal agencies to preserve public copies of all the materials they created. The Federal Depository Library Program administered by the US Government Printing Office (GPO) is responsible for distributing these copies to a network of libraries nationwide with the goal of long-term preservation and access. However, FRA rulings do not apply to electronic records, leaving websites, online articles, datasets, and other forms of information particularly vulnerable to permanent loss.
Following the 2025 inauguration, the US government conducted a mass removal and modification of online resources, highlighting the importance of preservation efforts such as the End of Term Web Archive.[2]
Project history
[edit]
The project was initially sketched out after a General Assembly of the International Internet Preservation Consortium in 2008.[3] The founding members included Library of Congress, Internet Archive, University of North Texas, and California Digital Library.
The first capture started in 2008 and concluded in 2009. Each members conducted their own harvest which led to issues of duplicated data. After consolidation, a full copy was stored at the Library of Congress and partial copies were distributed among members.[3] The University of North Texas received funding from the Institute of Museum and Library Services to conduct research on classification strategy, collection development protocol, and value assessment method.[4]
In 2012, Harvard University joined as a partner, followed by Stanford University and George Washington University in 2016. The seed list of URLs for crawling grew each year (43,674 in 2016 compared to 4,622 in 2008) as the project came more aware of the scope of US government web presence. Individual citizens and interested organizations are encouraged to nominate URLs through the crowdsourcing nomination tool. The 2016 harvest was also the first time the project included social media accounts on platforms such as Twitter and Tumblr in the collection scope.[5]
In 2020, the Environmental Data & Governance Initiative (EDGI) and infoDOCKET joined as new partners. After the 2020 harvest concluded, all previous crawls from 2008 were uploaded to Amazon Web Services through the AWS Open Data Program.[6] In 2024, WebRecorder joined as a new technical partner and enabled high-fidelity captures of selected domains, which resulted in mirrored replays of these websites, including complex interactive experiences. The Common Crawl Foundation also joined to provide technical solutions and consultation. The total uploaded size of the 2024 EOT Web Archive (WARC compressed) was 2.29 PB compared to 266.04 TB in 2020[7].
In 2025, the project started collecting URL nominations for the US Government Web & Data Archive, marking the first harvest outside of an end-of-term year[8].
Dataset
[edit]In 2022, all previous crawls of End of Term Web Archive was replicated in AWS S3 Storage for public access, using tools and knowledge developed by Common Crawls. In 2024, more remedial work took place to produce the EOT Web Archive Dataset[9] to improve accessibility, discoverability, and re-usability. This included creating derivative datasets in WAT (text and links) and WET (text-only) formats, augmenting metadata (language identification, charset dection, 404 detection), and producing compressed indexes in ZipNum CDXJ and cloud-optimized Parquet format. These indexes can publicly accessed and queried to request partial download of the web archive data.
See also
[edit]References
[edit]- ↑ Webster, Peter (2017). Brügger, Niels (ed.). "Users, technologies, organisations: Towards a cultural history of world web archiving". Web 25. Histories from 25 Years of the World Wide Web: 179–190. doi:10.3726/b11492. hdl:2318/1770557. ISBN 9781433140655. Archived from the original on 2020-10-21.
- ↑ "Here's how you can help save government data". Freedom of the Press. 2025-02-09. Retrieved 2026-08-02.
- 1 2 Seneca, Tracy; Grotke, Abbie; Hartman, Cathy Nelson; Carpenter, Kris (2012). "It Takes a Village to Save the Web: The End of Term Web Archive" (PDF). DTTP: Documents to the People. 40: 16. ISSN 0091-2085. Archived from the original (PDF) on 2015-09-08.
- ↑ R., Murray, Kathleen; Lauren, Ko,; Edward, Phillips, Mark (2011). "Curation of the End-of-Term Web Archive". UNT Digital Library. Archived from the original on 2025-07-20.
{{cite journal}}: CS1 maint: extra punctuation (link) CS1 maint: multiple names: authors list (link) - ↑ Phillips, Mark E.; Phillips, Kristy K. (2017-12-01). "End of Term 2016 Presidential Web Archive". Against the Grain. 29 (6). doi:10.7771/2380-176X.7874. ISSN 2380-176X.
- ↑ Edward, Phillips, Mark; Sawood, Alam, (2022-06-24). "Moving the End of Term Web Archive to the Cloud to Encourage Research Use and Reuse". UNT Digital Library. Archived from the original on 2025-07-03.
{{cite journal}}: CS1 maint: extra punctuation (link) CS1 maint: multiple names: authors list (link) - ↑ Archive, End of Term Web. "Datasets". End of Term Web Archive. Retrieved 2026-08-02.
- ↑ "Nomination Tool: About Project". digital2.library.unt.edu. Retrieved 2026-08-02.
- ↑ Phillips, Mark E.; Phillips, Kristy K.; Alam, Sawood (2023-06). "End of Term Web Archive Dataset: Longitudinal Web Archive of .GOV and .MIL Domains". 2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL): 98–101. doi:10.1109/JCDL57899.2023.00024.
{{cite journal}}: Check date values in:|date=(help)
