Webrecorder Blog

Execution Time Addons, Robots.txt, Profile Refreshes, Custom Schedules, and More

Tessa Walsh — Thu, 18 Dec 2025 00:00:00 GMT

We are excited to announce a new release of Browsertrix, 1.21. This blog post covers the changes in this new release, as well as the previous two releases, 1.19 and 1.20.

Each of these bring exciting new features, as well as bug fixes and performance improvements for Browsertrix. This blog post highlights some of the key features from each release, starting with the newest.

Browsertrix 1.21

Purchase additional execution time

A common pain point that we’ve heard from customers is that it can be frustrating when your org reaches its monthly execution time limit. Previously, any running crawls were automatically stopped at this point and could not be later resumed, and there wasn’t much users could do (short of upgrading their plan) other than wait for the limits to reset at the beginning of the next month.

In Browsertrix 1.21, it is now possible to purchase additional execution minutes at any time from right within Browsertrix itself. Org admins can go to Settings → Billing & Usage, and then click the button to purchase additional execution time. This will lead you directly to Stripe to complete the transaction. We have set some preset amounts of minutes that we expect users might want to use, but the amount you purchase is also fully configurable in Stripe. Once the purchase is complete, you will be returned to your org in Browsertrix and are able to begin archiving again immediately.

Purchase additional execution minutes through the Billing & Usage section of your organization settings.

Additional minutes purchased this way are not tied to the month in which they are purchased and do not expire, so you are welcome to add and use these additional minutes however you would like.

Pausing crawls when org limits are reached

We’ve also changed what happens to crawls when limits are reached. Running crawls are now paused rather than stopped when the org’s storage or execution time limits are reached. This gives users up to a week to resume paused crawls from where they left off. At any point during that week, you can free up some storage space, purchase additional execution time, or wait until the monthly limits reset, and then continue archiving without needing to restart your crawls from the beginning.

Robots.txt support

Another long-requested feature from some of our user base has been support for the Robots Exclusion Protocol, more commonly known as robots.txt. This is a convention that allows website administrators to specify which content on their sites should not be captured by crawlers and other bots. This new option, disabled by default, is available in the Scope section of the crawl workflow editor.

At this time, Browsertrix’s support for robots.txt will skip any web pages disallowed by that host’s robots.txt policy, if one exists for that host. Our support for robots.txt does not yet check each resource on a page, as that could quickly break Browsertrix’s promise of high-fidelity web archiving. If there is sufficient demand from our users, we may revisit other options to expand the scope of robots.txt support in future releases.

Browsertrix 1.20

Browser profile refresh

One of the most significant changes in Browsertrix 1.20 is a reworked user interface for browser profiles. The new browser profile interface includes a number of improvements that make it easier for users to manage their browser profiles and the crawl workflows that use them.

Sites that have been visited with a browser profile are now prominently displayed in the Saved Sites list. Clicking on the saved site will open the site with the browser profile, making it easier than before to verify that a site is still logged in or otherwise configured the way you want.

Configure profiles with a more powerful and intuitive interface and manage related workflows.

To add additional sites to a browser profile, you can now click the Load New Url button to open the browser profile to a new URL of your choosing.

The Load Profile dialog opened by clicking Load New Url or Load Profile from the Actions menu now also has an option Reset previous configuration on save, which can be used to reconfigure a profile from scratch. This provides a much smoother mechanism than before to update a browser profile and all of its related crawl workflows. This setting can be used to reconfigure a browser profile to use a different logged-in user account or different cookie settings without any interference from previously saved data, without needing to create a new browser profile and then manually update the browser profile used in all of the related crawl workflows. This has saved us a lot of time in managing browser profiles for social media and other sites, and we expect our users to enjoy the same benefits.

The interface for the interactive browser used to configure browser profiles has also been updated to improve the user experience.

Auto-updating browser profiles

Starting in Browsertrix 1.20, browser profiles are now also automatically updated following each crawl, with the data from the browser profile used during crawling. This ensures that crawls do not run with outdated browser profile data, and more closely matches what a real user browsing the site would look like to the site host. In our testing, this reduced the frequency of logouts on some social media sites significantly when crawl workflows are configured to use the browser on a regular (e.g. daily or weekly) schedule.

Browser profiles and crawling proxies

Previously, it was possible to create a browser profile with one crawling proxy, and then specify a different crawling proxy in the associated workflow. This could result in sub-optimal crawls or even getting blocked by websites.

Starting in Browsertrix 1.20, if a crawling proxy is set on a browser profile, all crawl workflows that use that browser profile will automatically use the same crawling proxy. If you want to change the crawling proxy that a browser profile and its related crawl workflows use, you need only reset the browser profile and select a new crawling proxy. This change ensures that users have the best experience of using browser profiles in their crawl workflows.

Crawling tab refresh

Browsertrix 1.20 also brings a revamped user interface for the Crawling page. We made a few changes here.

First, while the Crawling page still defaults to showing all of your crawl workflows, we’ve now added a Crawl Runs tab that lists all of the crawls run in your organization, including any failed and canceled crawls that have previously been difficult to find information about. Both the Workflows and Crawl Runs tabs also have expanded filtering and sorting options so it’s easier than ever to locate the exact crawl workflow or crawl run you are looking for.

Easily access all of your crawl runs in one place, making it simpler to find failed and canceled crawls.

Host-specific global proxies

Another change in Browsertrix 1.20 that is invisible to most users of our hosted service is that Browsertrix now supports setting global proxies that are applied only to specific hosts. In our hosted service, we are using this to always crawl Youtube through a specific proxy, which enables our users to crawl Youtube links and videos without the need for a logged-in browser profile.

If a crawling proxy is set in a crawl workflow, that proxy will override the global host-specific proxies.

Browsertrix 1.19

Custom cron schedule for crawl workflows

Browsertrix has long supported scheduling crawl workflows to run at daily, weekly, or monthly intervals. In Browsertrix 1.19, we added the option to specify custom schedules. These schedules can be input using the Unix Cron syntax, providing our users the greatest possible flexibility in setting schedules. Some helpful macros are also supported, including @yearly and @hourly. More details and resources for working with the Cron syntax are available in the User Guide.

Set custom cron schedules for your crawl workflows for granular control over when your crawls run.

Filter archived items by tag

We’ve added tag filters to the Archived Items page, as part of a general strategy of improving our filtering and sorting options across Browsertrix in recent releases. This makes tags more useful in Browsertrix and should help you better organize your archived items and make finding crawls and uploads easier than ever.

Crawl download improvements

Previously, downloading crawls from the action menu’s Download Item button or the Download All button in a crawl’s WACZ Files tab would always download a multi-WACZ, even if the crawl only contained a single WACZ file. This added an unnecessary layer of nesting to some downloads. Starting in Browsertrix 1.19, the Download Item and Download All options will only generate a multi-WACZ if the crawl has multiple WACZ files, otherwise the crawl’s WACZ file will be downloaded as-is rather than repackaged as a multi-WACZ.

Our statement on Conifer sunset announcement

Ilya Kreymer — Thu, 18 Dec 2025 00:00:00 GMT

On December 15th, Rhizome announced the sunset, or “twilight” of the Conifer service. The service will be taken down some time around June 2026. As some of you know, the Conifer service was a previous-generation archiving service which was developed during my collaboration with Rhizome between 2016 and 2020.

Having led the development of Conifer for many years, I am a bit bittersweet about this announcement, but I understand this is the right move as technology evolves and Conifer’s approach to archiving is no longer sustainable. Dragan Espenschied, the Preservation Director at Rhizome, has coordinated closely with us around this process and I believe the result is a carefully planned platform wind-down.

As explained in the post, Rhizome is offering many options for Conifer users, including self-hosting using ReplayWeb.page, and continued hosting through by Rhizome (also via ReplayWeb.page). As part of this process, Rhizome will migrate all Conifer collections to WACZ files in the future, and we at Webrecorder will provide guidance throughout the process as needed.

I’m especially excited to partner with Rhizome in inviting Conifer users to sign up for Browsertrix. Once the migration to WACZ files is complete, Conifer users will be able to upload their collections into Browsertrix. Conifer users will also receive a discount code for Browsertrix for one year. Of course, this is only one of the options available to Conifer users, and no data will be transferred between Conifer and Browsertrix without users’ consent and opt-in.

For Conifer users who’d like to continue to manually web archive in their own browsers, similar to how Conifer operated, we recommend the ArchiveWeb.page extension or desktop app. The extension can also be integrated with Browsertrix for hosting locally created WACZ files as uploads in Browsertrix.

I believe Rhizome is offering a model example of how to wind down a long-running service with flexible options and plenty of advance notice to meet the varied needs of Conifer users. I want to especially thank Dragan Espenschied as well as Mark Beasley, Rhizome’s former Senior Developer, who has kept Conifer running for the past 5 years, in preparing a careful transition process.

As the final sunset/twilight date gets closer, we will continue to coordinate with Rhizome and hope to welcome interested Conifer users to Browsertrix in the future!

Browsertrix 1.18: Large URL Lists and Beautiful Emails

Emma Segal-Grossman — Mon, 11 Aug 2025 00:00:00 GMT

In the new version of Browsertrix we’re adding support for large URL lists, and making some steps towards better and more consistent communication.

Large URL Lists

A long-requested feature in Browsertrix has been support for crawling a very long list of URLs. While we’ve supported large numbers of pages in crawls for a long time via crawl scopes that allow the crawler to discover pages, in some archiving workflows that doesn’t make sense. We’d limited the number of individual URLs you could enter on the workflow configuration page to 100, but in some cases users were splitting crawls in multiple workflows just to be able to get all of their pages crawled, which then made other operations in Browsertrix unwieldy and in some cases a little broken.

Now, you can upload or paste your huge URL list into Browsertrix and everything will just work! You can still enter URLs manually if you want, but if you have a large number of URLs you can upload a plain text file with one URL per line, and we’ll save it for you and use it in your workflow.

Upload a seed file to crawl a large number of URLs

For more details, check out our documentation.

Pretty Emails

As of now, emails for various account-related interactions with Browsertrix (such as new sign-up invitations, password resets, etc.) now more clearly come from us, and are now much more easy to read. Welcome emails have some more clear instructions for getting started, and updates related to your subscription will include more details.

One of the new email templates now in use

For a while now, these types of emails had been… functional, but very plain. We’d heard they were sometimes mistaken for spam, so we’ve built a new look for all of our transactional emails, and updates to our marketing and newsletter emails should follow soon. Implementing this posed some interesting technical challenges, detailed below if you’re interested.

How We Built Our Own Emails

Our email templates were originally implemented as plain-text Jinja templates. While this got the job done early on in Browsertrix’s development, it wasn’t ideal for a number of reasons: editing these templates was a tedious process that required a lot of trial and error, and styling emails is notoriously messy and difficult because of how few HTML and CSS features are consistently supported across email clients.

When looking into other alternatives we initially considered other Python libraries that could more easily drop in as a replacement for the existing templates in the backend, but ultimately found that React Email was the best maintained and easiest to use library available, despite being a Javascript library. I started putting together a template for the welcome email, and within a few hours had templates for almost all of the other email types done as well.

One of the nice things React Email does, especially compared to other libraries, is let us use style tokens from our design system Hickory as part of a a Tailwind config. It’s got a bunch of built-in components that provide abstractions for some of the more frustrating aspects of email development, i.e. all the style and layout constraints imposed by email clients, and a very simple API that can output both HTML and plain text from the same template. I originally intended to use it to just generate HTML that I’d then convert back to Jinja2 templates, but realized that with some of the more advanced templating I ended up using (e.g. date and time formatting; Python’s date and time formatting utilities don’t support the same types of formatting as the ones in Javascript’s Intl.DateTimeFormat) it would be easier to just use React Email directly. I put together a simple API server for generating email templates in both HTML and plain text along with subject lines, and got it all set up to deploy alongside the main Browsertrix application in the helm chart.

If you’d like to check out the changes, mess around with templates yourself, or contribute to the project, feel free to check out the codebase on GitHub. The email templating service lives in the emails folder, and has instructions for running the very handy dev server that React Email provides.

What’s Next

There’s a few more updates to email communication in the works, including some improvements to the onboarding and trial experience as well as improvements to styling and layout for marketing and newsletter emails. We’re also working hard on a few exciting crawler updates,

Other Updates

You can now use some of our Quality Assurance tools without running QA Analysis on crawls! While we initially wanted users to try out our QA tools with QA Analysis, we’ve found the tooling we built to be useful even without it, so we’re making them available and more easily accessible from workflow detail pages. We’ve updated the documentation to reflect this as well, it’s worth another read even if you’ve already been using QA.
We’ve added a crawler setting that will fail a crawl if the site you’re crawling logs you out. This will work on specific sites we’ve built detection for, which at the moment includes Facebook, Instagram, TikTok, and X. For more details, check out the docs.

As always, you can view the full list of changes on GitHub.

Browsertrix 1.17: Crawl Pause/Resume and Lower Numbers of Browser Windows

Tessa Walsh — Wed, 23 Jul 2025 00:00:00 GMT

We are excited to announce the release of Browsertrix 1.17. This release includes some commonly-requested improvements to crawling, including the ability to pause and resume crawls, and the ability to crawl with only 1 or 2 browser windows.

Pausing and Resuming Crawls

A common piece of feedback we’ve received since launching Browsertrix is that it would be nice to be able to pause crawls and then later resume them from where they left off. We listened, and this feature is now available via a new Pause button in the crawl workflow.

After clicking the Pause button, a running workflow will tidy up any remaining pages and upload WACZ files containing all of the data crawled so far. Once the workflow is successfully paused, you can replay all of the pages that have been crawled so far, download your WACZ files, and inspect the logs. You are then free to inspect the crawl thus far at your convenience for up to a week. If you forget to resume or stop the paused crawl within that week, Browsertrix will stop it for you, preserving all of your already-crawled data.

Based on previous conversations with many of you, we anticipate pausing will be especially useful for conducting test crawls. Not sure how well a website will be captured with a given workflow’s settings? Start the crawl with its full scope, pause it after however many pages you want to use as your sample have been crawled, and then inspect the replay and logs to see if you’re happy with the results. If all looks good, you can simply resume your crawl and it will pick up right where it left off. If not, you can modify the workflow settings before resuming the crawl, or cancel the crawl without having used many of your execution minutes in the process.

This functionality is made possible by the new Latest Crawl crawl workflow tab, which consolidates several pre-existing tabs such as Watch Crawl and Logs into a simpler interface. Latest Crawl displays information about the currently active crawl, or if the workflow is not currently running, the last crawl that was run from that workflow. This also means you can now replay a workflow’s latest crawl without needing to navigate away from the workflow!

Browser Windows

Another commonly requested feature is being able to crawl with a lower number of browser windows to avoid issues with sites that aggressively rate limit users. Now you can do just that. In the workflow editor, it’s now possible to configure crawls to run with 1, 2, or 3 browser windows, in addition to the multiples of 4 that were previously offered. In combination with other politeness settings such as Delay Before Next Page, this should help significantly with avoiding rate limiting and IP bans.

For our users that primarily interact with Browsertrix via the REST API, you’ll note in the API documentation that the scale field in the /crawlconfigs/ endpoints has been deprecated in favor of a new, simpler browserWindows field, which overrides scale when used and can be used to configure workflows to use an amount of browser windows lower than the increments available via scale. But don’t worry, in order to avoid breaking any existing user tooling and integrations, we have made sure to continue to support scale when it is used for creating and updating crawl workflows.

Create, Use, and Automate Actions With Custom Behaviors in Browsertrix

Tessa Walsh — Wed, 28 May 2025 00:00:00 GMT

We are thrilled to introduce a new feature available starting in Browsertrix 1.15 that will be very exciting for our Browsertrix power users: support for custom behaviors that let you automate in-page actions while crawling specific websites. You can now easily specify which custom behaviors to use directly in the crawl workflow editor. We’ve also updated our documentation to guide developers and advanced users in creating their own custom behaviors. Plus, we’ve added support for a new type of custom behavior that can be set up right in the Chrome DevTools, with no coding required.

The Story of Behaviors in Browsertrix

A big part of Browsertrix’s promise of high-fidelity web archiving is its ability to automate actions inside real browsers during crawling. This is made possible through Behaviors, code, or JSON documents (more on that later) that specify what actions the browser should take when visiting a web page.

Behaviors themselves aren’t new to Browsertrix. In fact, Browsertrix and Browsertrix Crawler have supported built-in behaviors for several years. A number of these are background behaviors, which quietly run on every web page, constantly checking for changes and taking action when needed. This includes some behaviors that always run on every web page no matter what, like autoplay, which plays video and audio on the page to ensure it is captured. It also includes some behaviors that can be enabled or disabled in Browsertrix, like autoscroll, which scrolls down the page until it hits the end or its timeout is reached.

Another type of built-in behavior that has long been supported in Browsertrix is site-specific behaviors. These only run on particular websites and are designed to perform actions tailored to those sites. This includes our built-in behaviors for social media sites like Instagram, Twitter/X, Facebook, and TikTok. You can find more detailed information about built-in behaviors in the Browsertrix Crawler documentation.

But here’s the exciting part: with this release, creating and using your own custom behaviors is easier than ever. And if you encounter a website that is tricky to crawl because it requires interactivity, you can now create and use your own behaviors immediately!

Browsertrix Support for Behaviors

You’ll now find everything related to behaviors in the new Page Behavior section of the crawl workflow editor. This update combines our new autoclick behavior and support for custom behaviors with existing settings like autoscroll, page timeouts, and delays (which were previously scattered across the workflow editor).

Introducing Autoclick Behavior

One exciting addition in the Page Behavior section is the autoclick behavior. This built-in feature automatically clicks on elements in the page without navigating away. By default, this will click on anchor (<a>) tags, which can be useful for websites that use these anchor links in non-standard ways. For example, some sites use JavaScript in place of the standard href attribute to create a hyperlink, while others use <a> tags in place of <button>s to reveal in-page content.

Need it to click on something other than links, like all the <button> elements on a page? No problem! Just specify a different CSS selector for autoclick directly in the workflow editor.

Custom Behaviors

Want to use new and existing custom behaviors in your crawls? Starting in Browsertrix 1.15, you can now specify custom behaviors to use in crawl workflows by pointing to behavior files that are hosted at any public URL or in a public Git repository. This means you can not only create and use your own custom behaviors in your crawls, but also tap into the Browsertrix community’s shared behaviors.

How To Create Custom Behaviors

Adding support for custom behaviors in Browsertrix is just one part of the solution. We also want to make it easier for you to create them. That’s why we’ve created new documentation that walks you through two ways to build custom behaviors: JavaScript behaviors and JSON Flow behaviors.

Traditionally, custom behaviors in Browsertrix have been written in JavaScript. This approach is still the most flexible and powerful, but it does require coding skills. For developers, our updated documentation covers how to make a JavaScript behavior, including an overview of the expected format, as well as important references and helpful suggestions.

We’ve also added a much more accessible option that doesn’t require writing a single line of code: JSON Flow behaviors. Thanks to Chrome’s built-in DevTools Recorder, you can simply record your actions on a webpage: click around and interact with the content, and when you’re done, export the recording as a JSON file. Upload that file somewhere with a public URL, like a GitHub Gist, Pastebin, or a public Git repository, and you’re ready to go! Just point your crawl workflows to that JSON file and Browsertrix will replay your recorded actions on that page automatically while crawling.

For visual learners, we recommend checking out the following Youtube video, which demonstrates how to use the DevTools Recorder and download your recording as a JSON file:

Browsertrix will even extend some of the actions in your JSON Flow behavior. For example, if it detects that you repeated an action (like clicking “Next” in a paginated list) more than three times, it will keep repeating that step until it can no longer do so successfully.

Of course, for more complex behaviors that involve loops, such as scrolling through and loading comment threads on a social media site, or other complicated actions, JavaScript behaviors will still be the go-to solution. But we are happy to offer a simpler and more accessible way that lowers the barrier to entry for anyone wanting to create custom behaviors.

Behaviors: one more way Browsertrix makes it easy to capture the web exactly the way you want.

Our Commitment to Provide Accessible Tools

Camille Lawrence — Wed, 30 Apr 2025 00:00:00 GMT

Funding for cultural institutions has been a constant uphill battle for decades. No matter the organization’s size, from small community archives to local public libraries and even larger national libraries and universities, we have all encountered similar funding challenges. Thankfully, budget cuts and limited staff support have not stopped archivists, librarians, and web archivists from creating new pathways forward. Archivists across the globe continue to take unique approaches to maximize budgets to cover costs for various archives initiatives, especially web archiving.

As communities continue to create and publish content online, the scope of web archiving initiatives has grown. But even with funding challenges, budget cuts, and staffing uncertainty, our wide range of users in the Webrecorder community are tapping into our robust resources and tools.

Our Full Suite of Tools

Since 2014, Webrecorder has been committed to developing a full suite of tools to empower users across the globe to archive, curate, and replay content captured on the web. Our team continues to create, update, and prioritize providing multiple access points for users to capture complex, and interactive content on the web – including dynamic websites, social media platforms, and content behind paywalls.

We encourage you to take advantage of our various tools to accomplish your web archiving goals; no matter your budget, we have solutions accessible to you:

Browsertrix Hosted Service is our browser-based, high-fidelity crawling tool, with innovative quality assurance and collaborative organization that empowers users to preserve, curate, and share archived web content confidently. Users can curate and publish unlimited collections, capture content behind paywalls/logins, and work collaboratively on the same multi-user platform. Whether you invite your organization and colleagues as Admin, Crawlers, or viewers, our tool is designed to encourage collaborative workflows bringing departments out of silos. You can access a free 7-day trial here.
- People can also access the open source code to deploy Browsertrix locally and connect directly with users worldwide in the Webrecorder forum online.
ArchiveWeb.page is a free Google Chrome browser extension that allows users to create high-fidelity web archives directly in their browser. When enabled, ArchiveWeb.page will record the network traffic on a given tab and store the data in the browser for later viewing. Archives created with ArchiveWeb.page can be viewed from right within the app or using Webrecorder’s free ReplayWeb.page viewer. Files can be exported in standard WARC and WACZ formats.
ReplayWeb.page is a free, browser-based tool that allows users to view archived items. Features also include full text search, Flash support, and Google Drive support.
pywb, the oldest of Webrecorder’s tools, is a free and open source web archive replay system, or “wayback machine”. At its core, pywb provides a calendar-based interface for exploring and replaying web archive collections. It also includes some related functionality, such as a recording mode for creating web archives, as well as a proxy mode for serving web archives. More information on pywb’s features can be found in the documentation.

If you are an organization or individual archivist that needs support, feel free to send an email to info@webrecorder.org. If you’d like access to our pricing plans for larger crawling projects, schedule a meeting with us here.

You can also explore our Resources page for more information on our tools, forums, presentations, and publications about Webrecorder!

Our New Resources Page for Web Archivists

Webrecorder Team — Wed, 16 Apr 2025 00:00:00 GMT

We have launched a brand new Resources page where you can learn more about web archiving, how our tools work, and how people are using them across different fields.

Our goal with this evolving hub is to support and inspire everyone in the web archiving community — whether you’re just getting started or already deep in the work. We’ve gathered helpful guides, reference materials, presentations, and research, all in one spot: webrecorder.net/resources

What You’ll Find

Introduction to Web Archiving: An overview of what it is, why it matters, and how our tools fit in.
Glossary of Terms: A go-to reference for common web archiving terms and concepts.
Formats and Specs: Technical documentation for formats like WACZ and CDXJ.
Talks and Webinars: Recordings of past presentations, demos, webinars, and more.
Academic & Press Mentions: A collection of research papers and media pieces featuring our tools in action.

Send Us Your Feedback

Whether you’re here to learn, teach, build, or contribute, this section is designed to grow with you and the wider community.

If you want to see something else included, email us at info@webrecorder.org. We’d love to hear from you!

Introducing GovArchive.us & Mirroring Entire Sites with Web Archives

Ilya Kreymer — Tue, 25 Mar 2025 00:00:00 GMT

We’re excited to announce the launch of GovArchive.us, a dedicated site for exploring our US Government Web Archive on Browsertrix. The project also introduces a brand new approach for viewing web archives: the ability to host a full-site “mirror” from any web archive, keeping original links intact while hosting them on a new domain.

One example of this is our archived version of the previous usaid.gov website, which is now accessible at usaid.govarchive.us. Unlike traditional web archive replay, this “mirror” archive preserves the original URL structure, making the site as easy to navigate and reference as the original site. For instance, the archived version of a page originally hosted at https://usaid.gov/about-us/mission-vision-values can be viewed at https://usaid.govarchive.us/about-us/mission-vision-values, by simply replacing the domain usaid.gov with usaid.govarchive.us.

We’ve reserved the *.govarchive.us domain and subdomains to be able to dynamically add more archives of US Government sites from our collections to this system.

What is Available Now?

Here’s a selection of a few ‘mirror’ sites that we have available from govarchive.us. Each mirror is a static site that loads an archived version from our collection, hosted on a dedicated domain:

usaid.govarchive.us as a mirror of usaid.gov
cdc.govarchive.us as a mirror of cdc.gov
fema.govarchive.us as a mirror of fema.gov
epa.govarchive.us as a mirror of epa.gov
climate.govarchive.us as a mirror of climate.gov.

Check GovArchive.us for an up-to-date list as we add more mirrors from our archives!

Mirroring Sites with Web Archives — Getting Started

This approach can be used by anyone to mirror a dynamic website hosted as a static site powered by web archives!

If you run a particular domain, you can set up a web archive as a static site, and point the domain to the static version of the site instead!

Or, you can host a mirror elsewhere, as we have done. This can be used to migrate off costly or obsolete infrastructure, while still preserving a site at the highest fidelity!

We provide the following template to get started with a single site mirror created from a web archive:

Web Archive Site Mirror Template

Using the above template, you can host your own web archive mirror entirely on GitHub Pages!

How it works: GovArchive and Wildcard Subdomains

GovArchive.us demonstrates a more complex setup with wildcard subdomains.

We’ve set up a wildcard DNS to point to a static site for any *.govarchive.us. (For this, we use Bunny CDN as GitHub pages does not support wildcard subdomains pointing to the same repo.)

Then, we dynamically choose the correct site to mirror in the browser based on the subdomain. A specific Browsertrix collection is chosen based on the current subdomain, allowing for maximum flexibility to add more collections.

Nested subdomains are flattened by replacing the . with the - so that more.subdomains.example.gov would become one-level of subdomain with more-subdomain-example.govarchive.us so that we can use a wildcard SSL cert easily. For example, nca2023-globalchange.govarchive.us mirrors nca2023.globalchange.gov.

With GovArchive.us, we also provide a custom banner and loading screen. If the archive is already initialized, it will load right away, otherwise the bootstrap script runs and a loading screen is shown while the service work is being initialized. Finally, the top-level site just provides a landing page index, hosted in a different repo.

As always, whole thing is open source, and further details are available on our GitHub repos:

The replay itself is provided with our low-level browser-based replay engine, wabac.js, which is also used in ReplayWeb.page. (In the future, the mirror capability may be added to ReplayWeb.page itself).

We hope GovArchive.us provides a much needed resource, as well as an example of how web archive-powered site mirrors can be done at scale.

If you need help setting up your own web archive mirror, reach out and we’d be happy to support your efforts!

Introducing Public Collections in Browsertrix

Webrecorder Team — Wed, 05 Mar 2025 00:00:00 GMT

At Webrecorder, we’re always working to improve Browsertrix, our high-fidelity web archiving service. With our latest release (1.14), we’re excited to introduce major improvements to our collections feature, introducing the Public Collections Gallery, a new way to personalize, curate, and share your very own web archives with the world!

Collections provide a way to dynamically combine and group multiple individual crawls and uploads into a contextual, unified archive replay experience.

With this release, you can now curate and showcase collections on a public gallery page for your organization, customizing thumbnails, titles, descriptions, and more. You can also choose to make collections fully downloadable, and allow embedding of collections in other websites using ReplayWeb.page.

The public collections gallery is available to all users of Browsertrix, and we’re excited to get your feedback on this feature!

US Government Web Archive

We previously shared our work on archiving federal US government websites in collaboration with others (end-of-term crawling).

We are now excited to share our first batch of public collections, available in our US Gov Web Archive!

As we add more US government collections to this archive, they will appear at this URL — check back for more updates!

Of course, we also continue to support unlisted collections, only viewable to those with a specific collection URL, as well as fully private collections, available only to logged-in users (the default option).

Browsertrix now also includes a private collections gallery, available to all members of your archiving organization.

Start Using Public Collections Today

Using Browsertrix already? Check out our new walkthrough Enabling Public Collections Gallery in the Browsertrix docs for detailed steps and additional videos on how to enable public collections for your organization.

Not yet using Browsertrix? You can sign-up for a free trial today and test out this feature for yourself!

If you have feedback, questions, or more ideas on how to make Browsertrix even better for you, drop us a line at info@webrecorder.net or join the conversation on Bluesky, Mastodon, and X.com.

Preserving Government Websites with Browsertrix

Ilya Kreymer — Thu, 06 Feb 2025 00:00:00 GMT

Update 2025-03-07: This work is now publicly available on Browsertrix as the Webrecorder US Gov Archive

At Webrecorder, we’re dedicated to making web archiving easy and accessible for everyone. We believe that preserving digital history is essential — especially when it comes to vital records of public information, such as government websites. That’s why we’re proud to be one of the partners in the End of Term Web Archive (EOT) effort to capture these sites and keep them accessible, using our own high-fidelity web archiving service, Browsertrix.

Why the End of Term Web Archive Matters

Every four years, as the U.S. transitions into a new presidential term, government websites change — some are updated, some move, and others disappear entirely. The EOT partners work to safeguard this history, ensuring these sites are archived before they are lost. This effort has special urgency this time, given the extensive changes and deletion of federal government websites.

We have selected sites that were nominated by EOT crawl participants, submitted by users through their URL nomination tool, and contributed by other partners like The Common Crawl Foundation (CCF). Our focus has been on identifying and crawling high-risk federal content, such as environmental justice, healthcare and climate change, as well as other content vulnerable to removal like LGBTQIA+ sites.

How Webrecorder Ensures Accurate Archiving

Our tools at Webrecorder go beyond simple static archives — they are designed to preserve modern, interactive websites exactly as they appear, handling everything from dynamic content, maps, and dashboards, to complex graphics. With Browsertrix, we can capture everything in high-fidelity and ensure that archived government websites can be presented as accurately as possible, and be navigated as they originally existed. To achieve this, we set up a dedicated space to run crawls of the selected federal government websites.

Additionally, we are using our browser extension ArchiveWeb.page to augment high-fidelity crawls with manual archiving of difficult-to-archive and highly interactive content. Both manual and automated captures are then merged into collections hosted with our Browsertrix service.

What’s Next?

Webrecorder is honored to support the EOT effort in this large-scale preservation task, and we hope to share the results of our EOT crawling in the near future. All of the data will be publicly accessible as part of the EOT initiative on eotarchive.org, and if you have other pages to contribute, you can submit them here.

Browsertrix 1.13: The Translations and Internationalization Release

Ilya Kreymer, Emma Segal-Grossman, Sua Yoo, and Clara Itzel — Wed, 18 Dec 2024 00:00:00 GMT

Web archiving tools are overwhelmingly only available in English, and while Webrecorder tools are no exception to this, we want to change that. To further our mission of making web archiving more accessible for all, we have started the process of translating Browsertrix, our premier crawling service, into other languages, including Spanish, French, Portuguese, and German.

To achieve this, we’ve integrated support for the popular translation tool Weblate, which provides a friendly UI for entering and editing translations.

We’d love your help getting Browsertrix translated into more languages! Join our Weblate project to help out.

Setting Your Preferred Language and Formatting Options

In your Account Settings, you’ll see a Language section where you can set your preferred language, as well as how you’d like values such as dates, durations, and numbers to be formatted. The Use browser language settings for formatting numbers and dates setting will let you override your selected language with your browser settings, which might be more familiar to you.

Contributions and Development

Thanks to community contributions, the Spanish translation of Browsertrix UI is over 50% complete — immense thanks to Weblate user Kamborio, who made 628 contributions in one day!

For users deploying Browsertrix on their own infrastructure, we have documentation on how to add new languages locally, if you’d like to test out a local instance with a new language.

We also welcome requests for more languages! If you’d like to see Browsertrix in your language, please submit a request on GitHub.

To see all of the changes in this update, check out the release on GitHub.

Not a Browsertrix user yet? Archive what matters to you with a 7-day free trial

Browsertrix 1.12: Proxies, Crawling Defaults, and Simplified Workflow Creation

Tessa Walsh and Emma Segal-Grossman — Thu, 07 Nov 2024 00:00:00 GMT

Proxies

We are very excited to announce the release of Browsertrix 1.12, including a long-anticipated new feature: crawling through dedicated proxies! Browsertrix can now be configured to direct crawling traffic through dedicated proxy servers, allowing websites to be crawled from a specific geographic location regardless of where Browsertrix itself is deployed. Want to crawl geographically-restricted content? Ensure your archived items reflect a local user experience? Crawl from a static local IP without the maintenance burden of self-hosting? All of this is now possible!

In our hosted Browsertrix service, proxies will be an optional paid feature available to users of Pro plans. We’ve started testing proxies with a few existing Pro users and will continue to refine our proxy offerings based on what we learn in practice.

If you are a current Browsertrix service customer, and want to try a local proxy in your region, let us know.

We’ve also added detailed documentation for folks who self-deploy Browsertrix on how to configure proxies to use with Browsertrix.

Once proxy servers are configured and made available to an organization, they can be selected per-crawl workflow or set as an organizational default so that all new workflows use a specific proxy by default (see more about crawling defaults below!).

Simplified Workflow Creation

We’ve simplified the process for creating a new crawl workflow. Previously, users had to select between URL List and Seeded Crawl workflow types before any other configuration. We heard loud and clear that this was confusing.

Now all crawl scopes types are available in the same interface without needing to go through a dialog, and we’ve quick-linked the different scope types from the New Workflow button in the Crawling tab. Your browser will even remember the last crawl scope type you used and set it again the next time you create a crawl workflow.

We’ve also added a Single Page scope type to make it as clear as possible what to do when you only need to archive a single page.

In Case You Missed It

We added a few features in 1.11 point releases since the last blog post, so here’s a few words on those!

Crawling Defaults

Browsertrix now includes new per-org crawling defaults! We heard from users that it would be helpful to be able to set certain crawling defaults for things like crawl limits, browser profiles, user agents, and the browser’s language setting. You can now find these in the Crawling Defaults section of Org Settings. Set it once and forget it, with the confidence that your default settings will be used in every new crawl workflow unless you manually override them.

We’ve added breadcrumb navigation to many pages in the app to help you navigate and situate yourself. When you’re viewing crawl workflows, archived items, collections, and browser profiles, take a look up towards the top of the page for the new breadcrumbs.

Last but certainly not least, we’ve integrated documentation right into Browsertrix in specific places where referencing the docs might help you most! Specifically, check out the “Setup Guide” button in the upper-right corner of the workflow editor. Getting help with understanding the purpose of workflow options has never been easier.

What’s next?

We’re already hard at work on features that will go into Browsertrix 1.13, including custom org storage. Soon you’ll be able to bring your own S3 bucket to use for crawling outputs, browser profiles, and other data generated by Browsertrix, giving you more flexibility and control than ever.

Browsertrix 1.11: Self Sign-Up, QA Improvements, Easier Downloading and new APIs

Emma Segal-Grossman, Henry Wilkinson, Tessa Walsh, and Ilya Kreymer — Tue, 06 Aug 2024 00:00:00 GMT

A lot of our work in this release cycle has been focused on our internal tooling to allow you to sign up for our Browsertrix hosted service in a fully automated way, with billing integration via Stripe. You can now sign up to use Browsertrix on your own, choosing from one of our newly offered plans. Once you sign-up, you can always update your subscription from the new “Billing” pane in Org settings, including automatically switching to a different plan as your crawling needs change!

Better Quality Assurance (QA) Stats

In our last release, we introduced our new QA system. In 1.11, we’ve made a few improvements to make the system even more useful.

The QA analysis meters are now updated in real-time as the analysis is running, allowing you to see immediate results on how the analysis going without having to wait for it to check all the pages. This should provide more immediate feedback about the quality of a larger crawl!

We’ve also added a few extra stats that help our image and text comparison meters make a little more sense. If you haven’t noticed anything funky up until now, that’s great, continue on as you were! For the inquisitive folks, explaining this change fully is a little more involved…

When archiving a website, sometimes the crawler encounters media such as PDFs, images, or video files that are surfaced on pages as links. These are treated as “pages” in the archive because they were linked to like any other HTML page would be (as opposed to being embedded as part of a page) but unlike actual webpages, these “pages” are just static files. Based on this fact, we can say with 100% certainty that, if these files are present within the archive, they’re going to be accurate copies of what was downloaded, and we don’t have to bother assessing them in a QA run, saving you time (and money!)

We’ve never actually run analysis on these files for the reason above, but our bar graph breakdowns didn’t account for this properly and grouped these non-HTML files captured as pages — along with objective failures — in with un-assessed pages, meaning that a 100% complete analysis run might look like it has some un-assessed pages when really they’re just not relevant! In Browsertrix 1.11, we list these above the meters as a separate stat, meaning that the HTML page match analysis graphs should always be fully filled when an analysis run is 100% complete, as it will only display HTML pages that can be analyzed!

Who knew bar graphs could be so involved?!

Easier Archived Item Downloads

Archived items often contain multiple WACZ files; typically one will be generated for each crawler instance, and they are all split in ~10GB increments. You’ve always been able to download these from the “Files” tab within an archived item’s details page, but we’ve never been satisfied with the amount of clicks required to accomplish this task. Today, that changes! Much like collections, any archived item can now be downloaded with a single click from the actions menu. This packs the associated WACZ files into a single “multi-WACZ” file, which can be replayed in ReplayWeb.page as you’d expect.

Fixes & Small Things

As always, a full list of fixes (and additions) can be found on our GitHub releases page. Here are the highlights:

If you’re a part of multiple orgs, the list of orgs in the quick switcher is now alphabetically sorted.
We’ve turned off behaviors for crawl analysis. This greatly reduces the time it takes to run!

Changes for Self-Hosting Users and Developers

While the bulk of the work in this release has been focused on our hosted service, users who are self-hosting Browsertrix can also benefit from a number of improvements to our API and webhooks.

New Webhooks

We now have additional webhooks which can notify when a crawl has been reviewed in the QA process, and when assistive QA analysis has started or finished.

Org Import & Export

Superadmins on self-hosted instances can now export an organization’s data from the database to a JSON file, and import an organization from an exported JSON file into the database. This API-only feature can be used to move organizations from one instance of Browsertrix to another, or to export all information from an organization for backup purposes.

Documentation for org import and export has been added to the Browsertrix deployment documentation.

Org Deletion & Crawling Restrictions

Superadmins on self-hosted instances can now delete orgs. To make sure you know what you’re deleting before you remove it from existence forever, we’ve added a nice verification screen that makes you type in the org name.

Superadmins can also turn off all crawling abilities for an org.

What’s next?

Support for crawling through proxies in different geographic locations and custom storage options for crawling to your own S3 bucket are currently being worked and will be available in a coming release.

To sign up and start crawling with Browsertrix, check out the details at: Browsertrix.com

Browsertrix 1.10: Now with Assistive QA!

Henry Wilkinson — Mon, 10 Jun 2024 00:00:00 GMT

After some rest and a few solid weeks of polish after our demo at IIPC’s 2024 Web Archiving Conference, we’re proud to release Browsertrix 1.10: Now with Assistive QA!

Assistive Quality Assurance

Quality assurance for web archives has long been a challenging and time consuming task. The best methods of ensuring a page was captured properly typically fall to a discerning archivist manually scoring various aspects of the page replay to get an overall picture of crawl quality. While we wanted to retain the human element of curation given the vast diversity of the web, our goal in developing these features is to dramatically speed up the review process by providing meaningful heuristics that help direct attention towards pages that need it most.

The crawl analysis and review process is the culmination of these efforts! Browsertrix can now analyze archived webpages by crawling pages from the captured WACZ files and comparing their replay with what the browser saw on the page during the initial crawl — a feature uniquely made possible due to the tight integration of Browsertrix, Browsertrix Crawler, and ReplayWeb.page.

Crawl Analysis

After crawling has completed, the first step in the review process is conducting an analysis run for the archived item in question. On the crawl’s details page in the Quality Assurance tab, press the Review Crawl button to begin the analysis process.

Like crawling, running crawl analysis will also use execution time. While we would generally recommend sticking it out and letting analysis runs complete to get a full picture of the success of your crawl, there may be some instances (a website with many similar pages that you believe to be captured successfully but still want some evidence that is the case) where a stopped analysis run may be enough to get the data you’re looking for.

Once analysis is complete, Browsertrix graphs the matching scores of two analysis dimensions — screenshot and text match comparison.

Details on what to expect when re-running crawl analysis can be found in the documentation!

Reviewing Crawls

Now that crawl analysis has finished, let’s take a look at its results in context and decide if our archive is any good! Generally, pages with high scores are less problematic, and you’ll want to direct more attention to pages with lower scores. For an archived item with at least one finished analysis run, pressing the Review Crawl button will open the Crawl Review page.

In just a few seconds, we can sort the list of pages by the heuristic we’re interested in (here text comparison) and jump straight to where there might be problems, and indeed there are! Some older ReplayWeb.page embed example pages aren’t loading everything that was found when crawling — in this case it’s because we’re trying to load ReplayWeb.page inside ReplayWeb.page and it doesn’t seem to handle recursive instances of itself very well, a known limitation! This example is pretty specific to our tools and website, some more common text matching discrepancies we’ve encountered are the result of video players’ UI text, cookie & consent popups, embedded content that loaded while archiving but 404s due to a replay issue — any time you see ReplayWeb.page’s “Archived Page Not Found” error show up here there’s probably something worth investigating further!

You can vote the page down and leave a comment if it’s a serious issue you’d like to flag for others, or vote the page up and do the same if the issues aren’t ones you need to worry about. Generally, we wouldn’t recommend assessing and voting on every page; instead, try to assemble a few key examples that exhibit common problems or consistent successes to give other curators concise information about the page quality they can expect from the archived item.

Once you’re satisfied with your assessment of pages, press the Finish Review button to score the success of the entire crawl and update the description with any additional details! This assessment score is reflected in the Archived Items list and will be used elsewhere in the app to assist with organization and discovery in the future.

That’s the general process, but it’s not quite everything! The other tabs not covered here allow you to compare screenshots taken while crawling and on replay, a standard replay tab to check the heuristics against the real thing, and a resource comparison table displaying counts of loaded vs unsuccessful page resource fetches. For more information on exactly what you can expect from each heuristic, check out the documentation for crawl analysis!

We’ve been working on this feature since November of 2023 and we’re all very excited to finally get it into your hands! As you might have noticed in the screenshots above, we’re releasing our crawl analysis and review tools with a “beta” tag attached and we’d really like your feedback! Your thoughts are always appreciated on the forum or on GitHub.

Fixes & Small Things

As always, a full list of fixes (and additions) can be found on our GitHub releases page, here are the highlights:

Browsertrix joins ReplayWeb.page with updated branding! You can see it in the screenshots above, and on this very website!
Emails are now displayed for both pending and current users in the org settings.
You can now offset the crawl queue to view URLs from any part of the upcoming pages list.

Changes for Developers

As a key component of Browsertrix, Browsertrix Crawler 1.1+ CLI also supports analysis runs! Running QA with Browsertrix Crawler can also output screenshot diffs to a separate directory for local debugging when using the --qaDebugImageDiff option. Check out our crawler QA documentation and full list of CLI options for more info about this feature.

We have created a Helm repo for Browsertrix! You can add our repository with helm repo add browsertrix https://docs.browsertrix.com/helm-repo/. See our deployment documentation for details.

What’s next?

We’re already hard at work on Browsertrix 1.11 with the beginnings of proxy support and improvements to the heuristic meters above currently in progress. Look for them in the next major release!

If you’re interested in signing up to crawl (and assess the quality of your captures) with Browsertrix, check out the details at: Browsertrix.com

ReplayWeb.page 2.0

Henry Wilkinson and Ilya Kreymer — Tue, 23 Apr 2024 00:00:00 GMT

We’re thrilled to announce the release of ReplayWeb.page, our embeddable browser-based web archive viewer. This release features updated branding, reorganized documentation, various UI improvements, experimental ad-blocking support, and a more robust codebase with TypeScript.

Read below for additional details.

New Branding

ReplayWeb.page has a new logo and new app icons for macOS, Windows, and Linux! Behind the scenes we’ve been working on updating our branding since October of 2023, and ReplayWeb.page is the first of our primary tools to launch with the new logos! More of these are on their way with Browsertrix and ArchiveWeb.page to follow.

Multiple captures of the same URL can now be accessed directly in the navigation bar! While all page snapshots are still listed in the Pages list, this should make finding snapshots of the same page a little faster.

Page Thumbnail Support

A thumbnail picture might not say a thousand words, but it’s still helpful when combined with page titles and URLs!

All new WACZ files created with Browsertrix include thumbnail screenshots captured while crawling. For those using the command line Browsertrix Crawler application, see the various screenshot options available in the Browsertrix Crawler documentation… But also the one you want is --screenshot thumbnail.

Adblock Embed Option

Most web advertisements will dynamically load content from ad servers which will send a relevant advertisement to the user. This setup poses a few challenges when replaying archived content, namely the client-side Javascript may not request the same advertisement files that it did originally. This makes ads a very challenging element and one of the least replicable parts of web archives to display. While some archivists work diligently to preserve advertising — a traditionally under-preserved aspect of culture and cherished element of the web loved by all — an alternate solution to these issues is to simply hide them and pretend they don’t exist!

The useAdblock="" embed attribute will use the Easylist filter rules by default to hide ads on archived webpages loaded in embedded ReplayWeb.page. adblockRulesUrl="https://urlhere.com/file.txt" can also be used to set a custom filter list.

Just like rendering advertisements, blocking them also brings new and unique challenges! We’re releasing ad blocking as a beta feature only available through the embed option above and expect it to evolve over time, please let us know what works for you and what might need further attention!

Update Favicons Embed Option

Adding the updateFavicons="" attribute will update the favicon of the page ReplayWeb.page is embedded within to the favicon of the embedded website. Currently this is only supported on Chrome.

Details regarding the above embed options (and all the rest) can be found in the “embedding” documentation section.

Documentation Overhaul

We have completely overhauled the ReplayWeb.page documentation and converted to MkDocs. We’re also using the homepage to highlight some of the great organizations that have been building their tools around or integrating ReplayWeb.page.

Go check them out and if you think your project belongs in this list, get in touch!

All pages have received some level of attention with hierarchy improvements and content corrections across the board. If something seems amiss, unclear, or incorrect, please let us know at: docs-feedback [at] webrecorder.net

Fixes & Small Things

As always, a full list of fixes (and additions) can be found on our GitHub releases page, here are the highlights:

The nav bar controls have been reordered. Navigation controls are now grouped on the left side, full screen has been moved to the right.
- The main navigation controls are now visible at smaller screen sizes.
Archive info has been moved to a dialog available under the More Replay Controls (three dots) menu.

Changes for Developers

The ReplayWeb.page codebase has been converted to TypeScript which should make the code more accurate and efficient. We have also added support for the Shoelace component library (also used in Browsertrix) to further improve UI consistency between our tools in future releases, and streamlined the build process to no longer commit prebuilt artifacts to simplify merging. We hope these changes — along with the documentation updates — will improve the developer experience, especially for new open source contributors!

Although a major release, this version should generally be compatible with previous ReplayWeb.page releases, but if you run into any issues, please let us know!

Browsertrix 1.9

Henry Wilkinson — Wed, 31 Jan 2024 00:00:00 GMT

A Quick Look Back at 2023

It has been almost two years since we initially announced Browsertrix! Since then, we’ve been pretty much solely focused on developing our next generation cloud-based archiving platform. One of the downsides of this sole focus is that sometimes you forget to update the company blog and actually tell people about what you made! I’m not going to go back and write update posts for every major release we’ve done (there have been eight!), but here are some of the more recent highlights if you missed them:

Collections

In 1.6 we added collections, the ability to add archived items to multiple different groups for sharing and export! Collections serve as the base for future curation features, but right now they allow for both crawls and uploads to be replayed, downloaded, or shared together as one package.

Because both uploads and crawls share their data within a collection, they also allow you to manually patch automated crawls created through our ArchiveWeb.page browser extension. If elements on a site you’ve tried to capture in Browsertrix aren’t available when replaying the crawl but are available when you capture the page with ArchiveWeb.page, try uploading a WACZ from ArchiveWeb.page and adding both to a collection!

Dashboard & Execution Time

In 1.7 we added the Overview page which displays key org metrics for storage and crawling. In 1.9 we’ve updated the usage history table to give you more granular stats on your execution time, separately listing the execution minutes used per-month based on how you’re charged for them.

Documentation

We may not have updated the blog much, but we definitely upgraded our docs! Browsertrix now has a full user guide using the excellent Material for MKDocs theme. One page I’m personally proud of is our extensive list of every crawl workflow setting, a handy reference if the in-app help text isn’t quite enough.

1.9 Release!

Crawler Version Selection

We frequently release beta versions of Browsertrix Crawler, the core component of our software actually responsible for capturing websites. Up until now, the version the app uses has been set by us, soon we’ll be providing you with some options! Release channels can now be set on a per-crawl basis so if we’ve implemented a fix in the latest beta for a site you’re trying to crawl, you can use it — while keeping in mind that there may be other unresolved issues that aren’t quite ready for prime time yet and that’s why it’s a beta version. But you knew that already, right?

More information can be found in the Crawler Release Channel section of the workflow setup page.

Custom User Agents

Release channels aren’t the only new crawling feature though, we’ve also added the ability to set a custom user agent that the crawler will use to identify itself to websites. In addition to bypassing sites that try to restrict which browsers can access them — something they generally shouldn’t be doing anyway — this feature is also useful for some of our larger clients for coordinating with publications to ensure their crawls don’t get blocked.

Updated Collection Selection UI

While we updated some of this to remove the clunky multi-stage setup and editing process for collections in 1.8 (if you know, you know), the release of 1.9 completes our overhaul to the collection content editing process. “Auto add” can be toggled for workflows right in the collections editor, and archived items in and out of the collection are now displayed in the same window giving us more room to display information about each item.

Fixes & Small Things

As always, a full list of fixes (and additions) can be found on our GitHub releases page, but here’s some of the big small stuff:

The “workflow settings” tab that displays the current workflow settings and crawl settings tab that displays the workflow settings used for that crawl now display the same data without any discrepancies. #1473
- Useful for nailing down what might have changed when crawling the same site multiple times with different settings!
We’ve increased the max width of the app. More data on the screen at once! #1484
Fixed a memory leak, now the server doesn’t have to restart every day! #1468
Run on Save is now only toggled on by default when creating a new workflow. #1458
- No more workflows running by accident!

What’s next?

We’re busy developing the initial version of our assistive quality assurance tools, currently focused on screenshot analysis of captured content. While there’s still a little ways to go before it’s ready for testing, everyone is pretty excited to get that into your hands. Look for it in the next major release! 🙂

If you’re interested in signing up to crawl with Browsertrix for your institution, check out the details at Browsertrix.com.

An update on the WACZ format

Ilya Kreymer — Wed, 03 May 2023 00:00:00 GMT

New WACZ Tools and Integrations

It has been over two years since we’ve first introduced the WACZ format and I wanted to provide a brief update on exciting new tools and integrations of WACZ, and also provide a glimpse of what’s next in the evolution of the format.

WACZ support in Perma Tools

We are thrilled to share that our colleagues at Harvard LIL, who run the perma.cc have released several new tools that leverage the WACZ format, including:

js-wacz — Javascript library for WACZ, designed to be compatible with our original Python py-wacz
Scoop — a standalone archiving tool for generating signed WACZ files for single pages. The tool adheres to the WACZ Signing spec and also uses our Browsertrix Behaviors for improved high-fidelity capture.
wacz-exhibitor — a tool for bootstrapping ReplayWeb.page viewer embed system embed with a bundle Nginx servicer, custom cache layer and additional wrapping via an iframe.

You can read more about Scoop on this blog post from Matteo Cargnelutti, the lead developer of the tooland the rest of the Perma Tools suite at https://tools.perma.cc/.

It is exciting to see a growing open source ecosystem around the format and high-fidelity archiving!

Save WACZ Now!

We are also excited to share that Internet Archive’s Save Page Now system now provides support for emailing users a copy a WACZ files created from an on-demand capture using this service.

Our colleague ed-summers describes testing out this feature in a recent blog post

We appreciate IA’s support in helping make web archives more portable via the WACZ format!

WACZ in AP News

Our collaboration with Starling Lab has led to an experimental use of a signed WACZ for an embedded Tweet in an AP News article, which you can read here.

The tweet uses our archival ‘receipts’ provenance view to indicate that this archived tweet was created on the specified date and time, by Starling Lab server! Even if the original tweet is removed or edited, we can digital proof that this web archive was created by Starling Lab (signed by certificate issued to authsign.starlinglab.org) This is all possible by using a WACZ signed according to the WACZ Signing and Verification Specs created using an instance of Browsertrix operated by Starling.

What’s next For WACZ Spec

These are just some of the examples of the growing adoption for the WACZ format. (If you have more examples, please share with us).

We also wanted to provide an update on new specification work happening around the WACZ format.

WACZ on IPFS Custom Storage Spec

One of the new things we are working on is how to put WACZ files on IPFS, in a way to maximize deduplication by splitting the files in a certain way along file and WARC record, and WARC payload boundaries.

The spec covers how to put general ZIP files, how to put WARC files (compressed or uncompressed) onto IPFS and the various trade-offs involved.

By following the spec, it will be possible to leverage IPFS’s content addressing to automatically deduplicate the same archived content, even if stored in different WARC files inside different WACZ files!

You can read the current draft of the spec here

The ArchiveWeb.page extension and our Save Tweet to IPFS tool are already using this spec.

For example, if two different users save the save tweet, the actual content will be automatically deduplicated, while the WARC headers will be new, resulting in storage savings overall.

Look forward additional blog posts describing this spec in more detail!

Multi-WACZ or WACZ Collections Spec

We are also working on a spec for how to combine multiple WACZ files to create collections. A single WACZ file can only be so big (though we’re exceeded 1TB with the format last year), and we need a way to group WACZ files, either from the same crawl, or multiple crawls, into a user-defined collection.

The ‘Multi WACZ’ spec will be all about creating collections of persistent web archives, and will be a Frictionless Data package that specifies URLs to WACZs files logically grouped together.

If you are interested in this spec, please see the pull request and GitHub Issue and feel free to provide feedback!

We will provide additional information as this spec develops!

Improvements and Suggestions Wanted!

We are always looking for further improve the spec as web archiving continues to evolve. Are there other data you’d like to see in the WACZ format, or other feedback in general? If so, feel free to leave an issue directly on our specs repo!

EDIT 2024-05-22: “Browsertrix” was previously referred to here as “Browsertrix Cloud”. This post has been updated to reflect the new name.

Announcing pywb 2.7.0 release

Tessa Walsh and Ilya Kreymer — Wed, 23 Nov 2022 00:00:00 GMT

After several betas and months of development, we are excited to announce the release of pywb 2.7!

This release includes a new banner and calendar user interface for pywb written in Vue. The new banner has the same localization/multi-language support as pywb 2.6 with a number of new additions and improvements, including an interactive timeline for navigation between captures and easier local theming of the banner via the config.yaml configuration file.

We hope that this new UI will be more flexible and easier to modify to meet user needs.

The new timeline and calendar can be independently toggled on and off as needed:

The pywb documentation now has a section on the how to use and custom the new Vue UI, and a complete list of changes is also available in the pywb Changelist on GitHub. This release also adds a new contributing guide to the pywb GitHub repository with information about how to submit issues, propose new features, and contribute code to pywb.

This release builds on previous rounds of work which were supported by the IIPC. Webrecorder wishes to thank the IIPC membership for their beta testing and feedback for the 2.7.0 release.

Next Steps

The next release of pywb is planned to include support for the WACZ format created and used by other Webrecorder open source tools, including browsertrix-crawler, ArchiveWeb.Page, and ReplayWeb.Page.

Community consultation and roadmapping for a future pywb 3.0 release is also currently underway - be sure to stay tuned in the coming months for updates!

Showing Provenance on ReplayWeb.page Embeds

Ilya Kreymer and Henry Wilkinson — Thu, 10 Nov 2022 00:00:00 GMT

The ReplayWeb.page viewer allows web archives to be embedded and displayed almost anywhere. It can be used as a single-page embed, or integrated into existing services and digital repository systems.

Previously, ReplayWeb.page embeds either showed the location and nav bar UI, or without any additional UI or context. With the interface hidden, there is limited information to signal to users that the content is being served from a web archive, and a high fidelity web archive should look and feel the same as the original. At the same time, we want people viewing web archives to understand that the content within an archive may not be a perfect record of a live website, and that the content is frozen won’t be updated or deleted like other content embedded through traditional means.

To help with this, we’ve added a new embed mode for ReplayWeb.page embeds to be seamlessly added without extra UI, but with a dropdown ‘archive receipt’, which shows the provenance of the archive, as seen below:

This mode can be enabled by setting the embed="replay-with-info" attribute in the <replay-web-page> tag.

Archive Provenance

The embed (above) now includes a dropdown, which expands to show the ‘archive receipt’ with info about the web archive:

In the technical info section, the archival receipt can show basic provenance Original URL and Archived On metadata, as well as what tools were used to create the archive. Additional provenance info can be added here as needed. A download link at the bottom provides a link to download the full archive locally as well, and the size of the full archive file is included.

Validating Signed Web Archives

A key use case for the receipt is to also show cryptographic signature metadata about the web archive. If the web archive is a signed WACZ files, created as per the WACZ Signing Spec, the real-time validation status, cryptographic keys, as well as hash of the full data package in the WACZ file, are also shown.

When loading a signed WACZ file, all data (WARC records, indexes, page lists, etc…) is validated as it is loaded on the fly (via wabac.js). The receipt includes a Validation field, which shows the number of hashes validated thus far. If the WACZ file has at all been tampered with, hashes would not match, and this will also be displayed to the user.

The cryptographic metadata also relates to provenance, and includes either a public key, or a link to a trusted third party observer certificate, to establish the creator of the web archive. (We’ll discuss how this works in a future blog post!)

Trusting Web Archives

We hope the display of this information will be a first step in making distributed web archives more trusted. While some of the data may not be applicable to most users and is fairly technical, we believe this can encourage independent verification of archived content in the future.

In a follow up blog post, we will discuss the different WACZ signing approaches and their implications for authenticity!

Perma.cc Upgrades to ReplayWeb.page

Ilya Kreymer — Wed, 17 Aug 2022 00:00:00 GMT

I am thrilled to share that our colleagues at Harvard Law Library Innovation Lab (LIL) who run the Perma.cc service have recently switched to using ReplayWeb.page as the default replay system for all Perma.cc archives! Read the announcement on their blog.

ReplayWeb.page provides a fully browser-based viewer / web archive replay system with a variety of embedding and customization options.

ReplayWeb.page includes a number of replay fidelity improvements, and Perma.cc users should see improved replay across the board. The new system also simplifies the replay architecture by allowing web archives to be loaded directly from Perma’s cloud-based storage. For added security, Perma.cc replays WARC files in a separate iframe that is only accessible from a Perma.cc URL.

Perma.cc has a large archive of single-page WARC files and ReplayWeb.page will download and index the WARC file on first load. Since Perma.cc captures a single page at a time, this is generally very performant with most of their archives. ReplayWeb.page supports both WARC and WACZ replay, allowing Perma.cc to easily experiment with using WACZ files, should they wish do so in the future.

Perma.cc continues to be a pioneer and early adopter of the latest Webrecorder tools and technologies, and we have a long tradition of working together! In 2014, Perma.cc was one of the first adopters of pywb system from when it was barely in alpha!

In 2019, Perma.cc switched to a customized version of the then-current Webrecorder stack (which still powers the Conifer service). Over the years, Perma.cc developers have also contributed to various Webrecorder open source tools, supported collaborative development efforts, and contributed to important research around web archives and security, including the WACZ signing specification.

I look forward to continuing our long-standing collaborations!

Webrecorder receives $1.3M open source development grant from the Filecoin Foundation

Ilya Kreymer — Tue, 21 Jun 2022 00:00:00 GMT

I’m really thrilled to announce that Webrecorder has received a two-year, $1.3M open source development grant from the Filecoin Foundation!

The grant will support our mission of developing quality open source web archiving tools for all!

This funding will help us to grow the Webrecorder team and make improvements across the broad Webrecorder ecosystem of tools.

(Check out the jobs page for more info on current and future job postings!)

From the beginning, Webrecorder’s mission has been to support decentralized web archiving that can be performed directly in the browser, where web archives can live anywhere that data can be stored. A key part of enabling decentralized web archiving, is a system of decentralized or distributed storage. The IPFS protocol provides a powerful foundation and the Filecoin storage network can go a long way in making decentralized web archiving a reality.

Dietrich Ayala, IPFS Ecosystem Lead at Protocol Labs, agrees:

Our collaboration with Webrecorder is key to the IPFS and Filecoin mission: making a web that works for the most impacted users in critical situations, and ensuring the safety of the digital human experience for future generations. Webrecorder provides the specs, libraries, tools and services to build bridges between the HTTP web and any of these new technologies, and bring the last 30+ years of the web along too.

I am very grateful for Filecoin Foundation’s continued support of the Webrecorder project and our web archiving mission, and thankful to everyone who has supported our efforts thus far!

You can also read our project spotlight on the Filecoin Foundation blog.

Highlights from Recent Work

This grant supersedes and expands on our previous open source development grant from Filecoin Foundation, which focused on design and research.

Here are a few highlights from the progress we’ve made in design, research and browser-based tool integration over the last few months:

UX Research with New Design Congress

As part of our previous grant, New Design Congress has been working on extensive UX research around browser-based web archiving. They’ve shared their initial findings in our last community call and a more detailed report is forthcoming! We will continue to collaborate around research, and examine use cases and risks associated with browser-based web archiving.

WACZ Spec + Use Cases Development

We have formalized the WACZ spec, and added additional web archiving related specs, including a spec for the CDXJ format, and a spec for signing WACZ files. The specs are available at: specs.webrecorder.net, thanks in large part to the work of ed-summers, our technical writer and editor for this effort. The work on the WACZ spec continues, focusing on full-text search and additional metadata, and additional recommendations for WACZ storage on IPFS.

ArchiveWeb.page Browser Integration

Part of our work in making web archives more accessible is to attempt to integrate web archiving directly into browsers. Mauve Software has released an update to their Agregore Browser which includes a proof-of-concept integration of web archiving via ArchiveWeb.page extension.

The Agregore Browser supports IPFS natively as well as many other p2p protocols, and support for browser-based archiving will soon allow users to share web archives they created directly through the browser itself.

Project Goals

This new funding will help us continue these existing efforts, as well as support our software development goals for the Webrecorder ecosystem in several key areas.

We will share a more detailed timeline in the later, but a few high level goals for the next two years include:

Browsertrix - Continued development of our open-source, federated cloud-native SaaS service, with support for archival storage of data on IPFS/Filecoin as one option. The service will allow institutions as well as independent communities to be able to create archives on their own.
Scalable web archive data model/specification and necessary tooling - Building on the WACZ file format, and implementing a robust data model for storing larger web archive collections, from a single WACZ file to multi-TB or even multi-PB collections. The data model would support encryption, signing and storage of all data on IPFS/Filecoin, and an optional IPFS-based search index.
Web archive signing and validation framework - Building tools and specifications for signed and verifiable web archives, to prove identity and authenticity. To support a variety of use cases, this will include multiple approaches, such as PKI-based, DID, and possibly blockchain-based solutions for verifying authenticity. Standalone validator tools that are deployable by independent institutions will also be created.
Replay Viewer and Embedding - Continued development of ReplayWeb.page viewer to keep up with complexity of web, including integration with additional CMS/digital preservation systems. Improvements for self-hosting the viewer for loading web archives from IPFS, and implementation of validation features in the viewer.
Browser-Based Web Archiving Tooling: Continued development and research around several browser-based archiving approaches, including our ArchiveWeb.page extension and the extension-less archiving via ArchiveWeb.page Express.

Next Steps

The next two years will be an exciting time for Webrecorder, as we expand and continue to build on this previous work, and expand our efforts on Browsertrix, which has gained a lot of use over the last few months. (More details in an upcoming blog post!)

In the short term, we will also be looking for additional help! If you would like to work with Webrecorder on achieving our mission of web archiving for all, please do not hesitate to reach out!

EDIT 2024-05-22: “Browsertrix” was previously referred to here as “Browsertrix Cloud”. This post has been updated to reflect the new name.

Introducing: Browsertrix

Ilya Kreymer — Wed, 23 Feb 2022 00:00:00 GMT

I’m excited to announce that Webrecorder is working on embarking on perhaps our most ambitious development effort to date: the collaborative development of Browsertrix!

You can read more on our official site browsertrix.com, including a list of key features.

Browsertrix is a fully integrated open source browser-based crawling platform that will allow users to create their own high-fidelity web archives in an automated way at scale.

Current Status

The full source code for Browsertrix is available on GitHub.

Development is currently in the early stages, with a focus on implementing core features and a user-friendly UI.

Our Senior Frontend Developer Sua Yoo has been tackling the creation of a brand new user interface to manage crawls and crawl configurations. To keep up with the development progress, please follow the project on GitHub.

Planned Service and Collaborative Development with IIPC Community

Webrecorder plans to eventually offer Browsertrix as a service, and we will be rolling out testing gradually over the next few months.

If you’re interested in participating in early testing, please sign-up for the Browsertrix info list

But, Browsertrix won’t just be another siloed online service!

Browsertrix is still in early stages of development, but we believe it is important to share this work, and more importantly, develop it in collaboration with our partners in the web archiving community.

Towards this goal, we are also very excited to announce our collaboration with the IIPC community!

The International Internet Preservation Consortium (IIPC) has agreed to contribute funding towards the development of Browsertrix over the next one or two years.

In addition to this funding and as part of this collaboration, several IIPC members, including The Royal Danish Library, British Library, the National Library of New Zealand, and the University of North Texas will also be deploying Browsertrix within their institutions.

The goal of Browsertrix is to provide a kind of federated web archiving system, which can be deployed not just by Webrecorder, but by other institutions. The close collaboration with IIPC members from the start will ensure that this system can meet the broader goals of the web archiving community, from smaller institutions to large national libraries.

Open Source and Open Web Archive Data

Webrecorder believes strongly that web archiving tools should be fully open-source to ensure long-term viability of the digital record. Web archiving is too critical to be relegated to proprietary processes and the whims of individual vendors.

While the WARC format provides a standard way to store raw HTTP data, there are no standards formats for everything else, from crawl specifications and crawl logs, page lists and indexes, full text search data, and curatorial metadata. With the WACZ format, Webrecorder is beginning to standardize some of these remaining components of the web archiving workflow.

One of our goals with Browsertrix is to allow crawl outputs (WACZ or WARC) to be stored in any storage of the users’ choosing. Browsertrix will allow output to any S3-bucket, so that even if the service were to disappear or stop working, all of the data will still be accessible using existing tools like ReplayWeb.page. This federated approach to storage will allow crawled data to be stored almost anywhere, from custom institutional repositories like Archipelago, to existing WARC data centers at national libraries, to any cloud S3 provider (like Amazon or Digital Ocean), to local file system, as well as decentralized storage systems like IPFS.

With Browsertrix, we hope to enable users to truly own all of their web archive data, and to be able to access and make use of it without relying on infrastructure from any single vendor (including Webrecorder!)

Collaborations Welcome

We know this will be an ambitious project, and we are just getting started! Web archiving is becoming more critical and more difficult.

If you would like to contribute to the development, testing, or are interested in a custom deployment of Browsertrix, please feel free to reach out directly via e-mail, GitHub or our forums.

If you would like to support Webrecorder financially, please consider supporting Webrecorder via our Open Collective or GitHub Sponsors accounts and don’t hesitate to reach out to us with any questions.

EDIT 2024-05-22: “Browsertrix” was previously referred to here as “Browsertrix Cloud”. This post has been updated to reflect the new name.

Launch of Open Collective and First Institutional Sponsor

Ilya Kreymer and Lorena Ramírez-López — Tue, 15 Feb 2022 00:00:00 GMT

We are excited to announce the launch of Webrecorder’s Open Collective page. Open Collective is a crowd-funding platform that allows many kinds of groups, including open source projects, to fundraise and keep track of funding and expenses in an open and deliberate way.

And, we’re thrilled to announce Rhizome as Webrecorder’s first institutional sponsor through the Open Collective platform!

Our team is extremely appreciative of Rhizome’s continued support. Rhizome was an early benefactor of the Webrecorder project, and Webrecorder found its home at Rhizome from 2016-2020. Thanks to the support from Rhizome and generous grant funding from the Andrew W. Mellon Foundation, we developed the original webrecorder.io hosting service for high-fidelity browser-based interactive web archiving.

In 2020, the webrecorder.io service has been renamed to Conifer and has become an integral part of the Rhizome digital preservation program.

Through this support, Rhizome remains committed to continued success of open source web archiving tools, as well as web archiving as a cultural practice, and continuous development of infrastructure and research.

We hope to further collaborate with Rhizome around improvements to shared tools and implementing new features, such as adding support for our WACZ format to the Conifer service.

In addition to Open Collective, we are continuing to accept support via GitHub Sponsors. We also want to thank all of our sponsors thus far who have supported Webrecorder via GitHub Sponsors!

In particular, we would like to also thank Kiwix, who have supported the initial development of Browsertrix Crawler, and have continued to sponsor Webrecorder via GitHub sponsors.

With starting an Open Collective page, we hope to provide our community with more ways to support the project, and we hope to further explore ways to use the Open Collective structure to share updates and progress on our work.

We would like to again thank Rhizome, Kiwix and all of our other supporters - your continued support helps makes our work building source web archiving tools possible!

Improving browser-based web archiving with standards and design research

Cade Diehm, Ed Summers, and Lorena Ramírez-López — Tue, 18 Jan 2022 00:00:00 GMT

After 30 years, we see how much the web and its users have grown and evolved and web archiving — “the process of collecting portions of the World Wide Web” must adapt technology, tools and workflows to evolve with it to ensure the access and use of these preserved collections in an archival format.

The Webrecorder project has been working to both broaden and deepen web archiving practice, by allowing every day users of the web to create and share high fidelity archives of web content using their web browser with our suite of open-source tools: ArchiveWeb.page, ReplayWeb.page, pywb, Browsertrix Crawler and many other tools that can be found on our Tools section.

And thanks to the support from the Filecoin Foundation’s Open Source Development Grant, our team is developing new efforts to improve browser support for web archive by initiating three new streams of work: standardization of our new format, Web Archive Collection Zipped (WACZ), design research, and integration beginning of integration of this work into existing tools.

The main value of this project is to create a formally standardized approach to creating, storing and accessing web archives on p2p/decentralized systems, which is why as part of these efforts, Webrecorder has partnered with the New Design Congress, a research organization that recognizes all infrastructure as expressions of power, and sees interfaces and technologies as social, economic, political and ecological accelerants, on design research and Agregore Integration Developer, Mauve Software, for browser integration.

And while we hope that our tools will further user’s ability to safely and easily create browser-based high-fidelity web archives, store them on decentralized systems, and efficiently access them, it’s important to remember that tools and even file formats can change in unanticipated ways.

Web archiving remains a niche discipline but is a profoundly important one. The preservation and curation of digital material for cultural, legal or historical reasons, is just as crucial as its physical equivalents. Both within the US and internationally, the landscape for tooling is small. Almost all archive systems rely on just a handful of base tools to capture and maintain their collections. As the new decade begins, we have new challenges to consider. Gone are the days of technological optimism; as tool builders, we must acknowledge the challenges of what we make, and how those challenges evolve and change over regions, time and from unexpected outside influences.

In response. the attitudes of tool-builders, policy makers, and infrastructure designers have not kept pace. Far too often in digital spaces, hasty and purportedly ethical answers are offered at scale in response to structural harms before the actual underlying problem is fully identified or its complexities accounted for. Perhaps unsurprisingly, there is little publicly available research or critical evaluation of the existing beliefs and practices of web archiving and how they manifest consequences for those who are involved with, subjected to or interact with the web archiving process.

Through the WACZ specification, we aim to produce a collection of deeper understandings of these threats, alongside proactive recommendations that will — alongside making WACZ a more resilient format — provide real, tangible examples of responding to the challenges inherent in designing and building digital tools.

As part of our milestones for this project, we’ve already begun gathering use cases as well as threats, and anti-use cases on our GitHub. We’ve taking the conversation publicly on our last two community calls in November and December of 2021 and over the coming weeks, we’ll be conducting a series of interviews with archivists, journalists and researchers with the specific goals of both getting a better sense of individual and institutional archival practices, and uncovering deeper concerns specific to archival tools. Each interview will take approximately 1-1 ¼ hour to complete. We will use these contributions to help inform a set of design recommendations for Webrecorder that are more resilient to the effects of weaponised design and other threats.

Do you web archive, and want to help? Get in touch or fill out our Google Form.

Web Archives on, of, and off the Web

Ed Summers — Fri, 26 Nov 2021 00:00:00 GMT

This is a guest post by Ed Summers, who is working with Webrecorder as a technical writer and designer on the WACZ spec. The post is cross-posted from his blog at: https://inkdroid.org/2021/11/24/wacz/

Last month Webrecorder announced a new effort to improve browser support for web archives by initiating three new streams of work: standardization, design research and browser integration. They are soliciting use cases for the Web Archive Collection Zipped (WACZ) format, which could be of interest if you use, create or publish web archives…or develop tools to support those activities.

Webrecorder’s next community call will include a discussion of these use cases as well as upcoming design research that is being run by New Design Congress. NDC specialize in thinking critically about design, especially with regards to how technical systems encode power, and how designs can be weaponized. I think this conversation could potentially be of interest to people who are working adjacently to the web archiving field, who want to better understand strategies for designing technology for the web we have, but don’t necessarily always want.

The next Webrecorder Community Call will be on:

Nov 30th, 9am PT / 12pm ET / 17:00 GMT

We’ll be discussing use cases for WACZ format and plans for UX research around browser-based web archiving

More details and sign-up: https://t.co/hXWFwjnzkS
#WebArchiveWednesday
— Webrecorder (@webrecorder_io)
November 24, 2021

I’m helping out by doing a bit of technical writing to support this work and thought I would jot down some notes about why I’m excited to be part of the project, and why I think WACZ is an important development for web archives.

So what is WACZ and why do we need another standard for web archives? Before answering that let’s take a quick look at the web archiving standards that we already have.

Since 2009 WARC (ISO 28500) has become the canonical file format for saving content that has been collected from the web. In addition to persisting the payload content, WARC allows essential metadata to be recorded, such as the HTTP requests and response headers that document when and how the web resources were retrieved, as well as information about how the content was crawled. ISO 28500 kicked off a decade of innovation that has resulted in the emergence of non-profit and commercial web archiving services, as well as a host of crawling, indexing and playback tools.

In 2013, after three years of development, the Memento protocol was released as RFC 7089 at the IETF. Memento provides a uniform way to discover and retrieve previous versions of web resources using the web’s own protocol, HTTP. Memento is now supported in major web archive replay tools such as OpenWayback and PyWB as well as services such as the Internet Archive, Archive-It, archive.today, PermaCC, and cultural heritage organizations around the world. Memento adoption has made it possible to develop services like Memgator that search across many archives to see which one might have a snapshot of a specific page, as well as software extensions that bring a versioned web to content management systems like Mediawiki.

More recently, the Web Archiving Systems API (WASAPI) specification was developed to allow customers of web archiving services like Archive-It and LOCKSS to itemize and download the WARC data that makes up their collections. This allows customers to automate the replication of their remote web archives data, for backup and access outside of the given services.

So, if we have standards for writing, accessing and replicating web archives what more do we need?

One constant that is running through these various standards is the infrastructure needed to implement them. Creating, storing, serving and maintaining WARC data with Memento and WASAPI usually requires the management of complex software and server infrastructure. In many ways web archives are similar to the brick and mortar institutions that preceded them, of which only “the most powerful, the richest elements in society have the greatest capacity to find documents, preserve them, and decide what is or is not available to the public” [@Zinn:1977]. This was meant as a critique in 1977, and it remains valid today. But really it’s a simple observation of the resources that are often needed to create authoritative and persistent repositories of any kind.

The Webrecorder project is working to both broaden and deepen web archiving practice, by allowing every day users of the web to create and share high fidelity archives of web content using their web browser. Initial work on WACZ v1.0 began during the development of ArchiveWeb.page and ReplayWeb.page, which are client-side JavaScript applications for creating and sharing archived web content. That’s right, they run directly on your computer, using your browser, and don’t require servers or services running in the cloud (apart from the websites you are archiving).

You can think of a WACZ as a predictable package for collected WARC data that includes an index to that content, as well as metadata that describes what can be found in that particular collection. Using the well understood and widely deployed ZIP format means that WACZ files can be placed directly on the web as a single file, and archived web pages can be read from the archive on-demand without needing to retrieve the entire file, or by implementing a server side API like Memento.

WACZ, and WACZ enabled tools, will be a game changer for sharing web archives because it makes web archive data into a media-type for the web, where a WACZ file can be moved from place to place as a simple file, without requiring complex server side cloud services to view and interact with it—just your browser.

It’s important to remember that games can change in unanticipated ways, and that this is an important moment to think critically about the use cases a technology like WACZ will be enabling. You can see some of these threats starting to get documented in the WACZ spec repository alongside the standard use cases. These threats are just as important to document as the desired use cases, perhaps they are even more consequential. Recognizing threats helps to delineate the positionality of a project like Webrecorder, and recognizes that specifications and their implementations are not neutral, just like the archives that they make possible.

However, it’s important to open up the conversation around WACZ because there are potentially other benefits to having a standard for packaging up web archive data that are not necessarily exclusive to the ArchiveWeb.page and ReplayWeb.page applications. For example:

Traditional web archives (perhaps even non-public ones) might want to make data exports available to their users.
It might be useful to be able to package up archived web content so that it can be displayed in content management systems like Wordpress, Drupal or Omeka.
A WACZ could be cryptographically signed to allow data to be delivered and made accessible for evidentiary purposes.
Community archivists and other memory workers could collaborate on collections of web content from social media platforms that are made available on their collective’s website.
Using a standard like Frictionless Data could allow WACZ metadata be simple to create, use and reuse in different contexts as data.

Webrecorder are convening an initial conversation about this work at their November community call. I hope to see you there! If you’d rather jump right in and submit a use case you can use the GitHub issue tracker, which has a template to help you. Or, if you prefer, you can also send your idea to info [at] webrecorder.net.

Webrecorder receives a grant for Design and Standardization of Browser-Based Web Archives

Ilya Kreymer — Wed, 13 Oct 2021 15:00:00 GMT

I’m excited to announce that Webrecorder has received a $100,000 Open Source Development Grant from the Filecoin Foundation to work on standardization and design around creation of browser-based web archives.

The creation of web archives through the browser has been a key goal for the Webrecorder project, and this work will help make this goal closer to reality. The grant will be focused on three strands of work, explained in more detail.

I especially wish to thank the Dietrich Ayala of Protocol Labs for collaborating and supporting this work!

WACZ Standardization

The first area of work will be a more formal definition of the WACZ Format, a format designed to package standard WARC files along with other requisite components, such as CDXJ indexes, page lists and other metadata. (See our previous post about WACZ). We hope this will help more formally define the other formats that make web archives useful such as CDXJ, but which are currently underspecified. The WACZ format allows random access to web archives of any size, making it possible to efficiently retrieve a single page out of a larger collection, thus making it possible to efficiently load web archives from IPFS, Filecoin, as well as any storage that supports random access. In this work, we hope to focus on browser-based web archives first as well as plan for how to store and access much larger crawl-based collections.

On this effort, I will be collaborating with a long-time friend of Webrecorder and colleague Dr. ed-summers, who will work as a technical writer and designer on the WACZ specification.

UX Research on Privacy-preserving Web Archiving

Suppose browsers could natively create web archives of anything you browse? How do we ensure users privacy is protected, and users are able to make intelligent choice about what to archive and what not to, where to store the data, and with whom to share it. The second strand of this work will focus on critical UX research around privacy-preserving web archiving, threat modeling and UX design taking into account different scenarios for user-based web archiving. We hope to focus on use cases and users outside the traditional web archiving communities, including users with different and high threat risks, due to the nature of their work, such as journalists, human rights researchers, etc…

The UX research will be led by cade-diehm, along with New Design Congress(NDC), an independent research organization he founded which “recognises all infrastructure as expressions of power, and sees interfaces and technologies as social, economic, political and ecological accelerants.”. I am super excited to be collaborating with Cade and NDC on this effort and supporting much-needed privacy research around new form of web archiving.

Implementation and Browser Integration

Finally, the last strand of work will focus on beginning to integrate the design and research from the other strands into our existing tools. We will likely update tools such as py-wacz and ArchiveWeb.page to support the latest WACZ format specification and UX recommendations.

In this phase, we will also be joined by Mauve, developer specializing in open source decentralized tools and the creator of Agregore Browser, a “minimal web browser for the distributed web” which already natively supports IPFS, hyper:// and other decentralized protocols. Mauve will work to integrate web archiving support into Agregore via our ArchiveWeb.page extension, combining a web browser, built-in web archiving support and native decentralized storage via IPFS.

Taken all together, I hope that this work will make a significant impact on the web archiving field, helping advance not only the technology for web archiving in a more decentralized ways, but also our understanding of how more personalized archiving can empower users (and the risks involved) involved. This grant helps support the core of Webrecorder’s mission of bringing ‘Web Archiving for All’!

I look forward to sharing more updates on this work in the upcoming months!

Webrecorder Website Update

Ilya Kreymer — Wed, 13 Oct 2021 14:50:00 GMT

The Webrecorder site got a slight overhaul today, thanks to the work of UX Designer Thomas Walskaar and our generalist developer, Emma Dickson!

As one of the main changes, we’ve add a new page for our community, which includes information about upcoming community calls and links to previous calls.

We’ve also updated the tools page to more accurately reflect our current tools.

Here’s a screenshot of the on-going design draft from Figma, created by Thomas:

We are still tweaking a few things, but we hope this update will make it easier to learn about our tools, upcoming events and ways to reach us all in once place!

Announcing pywb 2.6.0 release

Ilya Kreymer — Wed, 11 Aug 2021 00:00:00 GMT

After several betas and months of development, I’m excited to announce the release of pywb 2.6!

This release, supported in large part by the IIPC (International Internet Preservation Consortium), includes several new features and documentation as well as many replay fidelity improvements and optimizations.

The main new features of the release include improvements to the access control system and localization/multi-language support. The access control system has been expanded with a flexible date-range based embargo, allowing for automated exclusions of newer or old content. The release also includes the ability to configure pywb for different user access levels, when running pywb behind an Nginx or Apache server. For more details on these features, see the Access Control Guide and Deployment Guide

With this release, pywb also includes support for running in different languages and configuring the main UI to support switching between different languages. All text used is automatically populated into CSV files and imported back. For more details, see the Localization / Multi-Language Guide section of the documentation.

A complete list of changes is also available in the pywb Changelist on GitHub.

This work is a follow-up to the first package of work supported by the IIPC, which resulted in the creation of a transition guide for users of OpenWayback. Webrecorder wishes to thank the IIPC for their support of pywb development.

The next release of pywb, corresponding to the final batch of work sponsored in this collaboration with IIPC, will include several improvements to the pywb user-interface and navigation.

For more discussion on this release and upcoming work, please join the upcoming IIPC-hosted webinar on pywb on Tuesday, August 31, 2021 or stay tuned for the restart of our Webrecorder Community Calls starting this fall!

Autopilot: Testable Automated Behaviors for ArchiveWeb.page and Browsertrix

Ilya Kreymer — Wed, 21 Apr 2021 00:00:00 GMT

Autopilot in ArchiveWeb.page

Web archiving can be complex and often tedious work, especially when trying to archive dynamic, infinitely complex content such as social media. A key goal of Webrecorder tools is to make web archiving simpler, and we’ve taken an important step with latest update to our tools. Over the last week, the Webrecorder team has been quietly testing our new automated, in-page behavior system, sometimes also known as Autopilot!

The system is available in the latest release of ArchiveWeb.page extension and desktop app.

The ArchiveWeb.page Guide has been updated with a new page on how to use Autopilot.

The automated behavior system allows the browser to be automated to perform custom interactions with a page to automate repetitive tasks, such as clicking and scrolling. The default, Autoscroll behavior, is designed to support any site with infinite scroll. (It works well on Yahoo Answers, helping archive those pages before they disappear!)

The system includes site-specific behaviors for the most commonly requested sites: Twitter, Instagram, and even Facebook!

The behavior for Facebook pages is the newest and most experimental, but we hope will it will make the job of those trying to archive social media slightly easier.

These behaviors perform complex interactions designed to capture the highly interactive elements of these sites, including infinite feeds, videos, photos and comments. The guide also includes a detailed overview of each behaviors functionality and limitations.

Autopilot in Browsertrix Crawler

The behavior system that forms the basis for Autopilot is actually part of the Browsertrix suite of tools, and is known as Browsertrix Behaviors.

The behaviors are also enabled by default when using Browsertrix Crawler, and can be further customized with command-line options for Browsertrix Crawler.

Browsertrix Crawler provides additional options for choosing which behaviors are enabled and provides options to view the behavior status log as the behavior is running.

The Hard Part — Automated Automated Testing

The first iteration of Autopilot was initially launched for Webrecorder hosted service (now Conifer), and Webrecorder Desktop App.

Over time, we’ve learned that as hard as it is to make the automated behaviors, maintaining them is even harder! Social media sites are not only complex, but also change frequently, and the web archiving community must inevitably play catch-up.

There is no doubt that the site-specific behaviors will break and need to be maintained, and require consistent upkeep.

To make this a bit easier, all Autopilot/Browsertrix Behaviors are automatically tested daily, using GitHub actions.

The tests run a small crawl using Browsertrix Crawler on a fixed social media account, created specifically for testing, to ensure the basic functionality of a behavior (clicking on photos, playing videos, going through feed, etc…) remains unchanged. Each branch or pull request for the behavior system is also tested with a basic crawl. Of course, these tests are a bare minimum to the potentially infinite complexity of archiving dynamic social media sites, but we hope this is a start to making behaviors more maintainable.

We’ve also learned that it is important to help uses manage expectations. With these tests, we can quickly find out when particular behaviors break, and users of Webrecorder tools can also see which behaviors are currently working and which ones from the behaviors overview page or from GitHub.

With this testing in place, we hope to be able to address broken behaviors more quickly, and let users know when they are broken.

Browsertrix Behaviors — Just add Browser

The behavior system is intentionally designed to run entirely in the browser and can work on any modern browser. While we test it with Browsertrix Crawler, the behaviors can be injected directly into a browser in any way (including just copy and paste! and is not tied to a particular crawler.

The goal was to make the behavior system usable in any kind of browser-based crawler, and encourage community contributions of new behaviors!

Are there certain site-specific behaviors you’d like to see, and can you help create? If so, feel free to open an issue on GitHub, or discuss on the forum.

We hope to create more guidelines and documentation on how to contribute behaviors in the future. Stay tuned!

Announcing New ArchiveWeb.page App, Deprecating Older Tools

Ilya Kreymer — Mon, 22 Feb 2021 00:00:00 GMT

Over the years, the Webrecorder project has developed a lot of tools to make web archiving easier and accessible for all. To continue pushing the boundaries of high-fidelity web archiving and make tools that are easy to use and easy to maintain, it is sometimes necessary to discontinue older tools and focus on new ones.

If you are currently using the following tools, we recommend transitioning to the newer tools mentioned below.

If you’re using Webrecorder Desktop, you should switch to the ArchiveWeb.page Extension or Desktop App. See below for more details on ArchiveWeb.page. Webrecorder Desktop development has been discontinued.
If you’re using Browsertrix, you should switch to Browsertrix Crawler, a more modular, self-contained crawler. See below for more details on Browsertrix Crawler
If you’re using Webrecorder Player, you should switch to ReplayWeb.page App or use the https://replayweb.page web site. ReplayWeb.page was released last year, and Webrecorder Player development has been discontinued last year.

ArchiveWeb.page Desktop App Now Available

Last month, we announced the release of the ArchiveWeb.page Chrome Extension

During our last community call, we also announced initial beta release of the ArchiveWeb.page App which complements the extension.

The desktop app uses the same code base as the extension and updates will be released to both at around the same time.

Extension vs App

The extension is preferable to many use cases, as it integrates directly with the browser and may be easier to start recording. When using the extension in their existing Chromium-based browser, users can archive exactly what they see, including all sites they’re already logged into.

The app may be useful in cases where the extension has difficulty, particularly due to certain restrictions in the browser. For example, in Chrome, many Google sites have native apps, and security settings may prevent archiving Google Docs, etc.. Archiving these sites should work in the standalone app.

The extension does require a Chromium-based browser (Chrome, Brave, Edge), so the app may be an alternative for those who do not wish to install one of these browsers.

Users familiar or have existing workflow with Webrecorder Desktop should find the ArchiveWeb.page App easy to use.

Webrecorder is committed to making it as easy as possible to archive any site, and will continue to offer ArchiveWeb.page as both an app and an extension.

Deprecation of Webrecorder Desktop

With the release of ArchiveWeb.page App and extension, the existing Webrecorder Desktop app is now deprecated.

The Webrecorder Desktop was developed by migrating a system designed to be a cloud-based service into an app, and resulted in an overly complex architecture that made it difficult to maintain. While the app was based on Electron, it also bundled two separate native executables, a Python App and an external Redis binary, which made it very hard to keep up-to-date for latest MacOs and Windows releases.

The ArchiveWeb.page app and extension are designed from the ground up to run as local archiving systems on your machine.

If you are starting a new archive, please use ArchiveWeb.page

If you have existing collections in Webrecorder Desktop, you can export them as WARC files and view via ReplayWeb.page.

ArchiveWeb.page app and extension will both have a way to import WARC files from Webrecorder Desktop in and upcoming update.

We plan to release more instructions for how to migrate in the near future!

Crawling Tools Update: Refactoring Browsertrix into the new Browsertrix Crawler

With the release of the modular Browsertrix Crawler crawling system, the older, all-in-one Browsertrix is no longer being developed in favor of Browsertrix Crawler. The original system had too many ‘moving parts’: a crawler, a remote browser system, behavior system, a scheduler, a UI and a CLI tool, all split across many Docker containers and repos.

All of those are important, but it became difficult to maintain all of the components as designed. The idea of Browsertrix lives on in a more modular setup with Browsertrix Crawler, which focuses on the core use case of being able to run an automated high-fidelity crawl of small or medium-size site.

Additional features, such as a scheduler or a UI may be added in the future, but will be separate from the Browsertrix Crawler. Above all, we want the core Browsertrix Crawler to be easy to use and focus on providing high-fidelity crawling via a single command.

See the Browsertrix Crawler repository issues for more details on current development of the crawler.

Introducing Browsertrix Crawler

Ilya Kreymer — Mon, 22 Feb 2021 00:00:00 GMT

I wanted to more publicly announce Webrecorder’s new automated browser-based crawling system: Browsertrix Crawler.

The premise of the crawler is simple: to run a single command that produces a high-fidelity crawl (based on the specified params and config options).

The Browsertrix Crawler is a self-contained, single Docker image that can run a full browser-based crawl, using Puppeteer.

The Docker image contains pywb, a recent version of Chrome, Puppeteer and a customizable JavaScript ‘driver’.

The crawler is currently designed to run a single-site crawl using one more Chrome browser in parallel, and capturing data via a pywb proxy.

The default driver simply loads a page, waits for it to load and extracts links using Puppeteer and provided crawler interfaces. A more complex driver could perform other custom operations target at a specific site fully customizing the crawling process.

The output of the crawler is WARC files and optionally, a WACZ file.

The goal is to make it as easy as possible to run a browser-based crawl on the command line, for example (using Docker Compose for simplicity):

docker-compose run crawler crawl --url https://netpreserve.org/ --collection my-crawl --workers 3 --generateWACZ

After running the crawler:

a WACZ file will be available for use with ReplayWeb.page at ./crawls/collections/my-crawl.wacz
the WARC files will also be available (in standard pywb directory layout) in: ./crawls/collections/my-crawl/archive/
the CDX index files will be available (in standard pywb directory layout) in: ./crawls/collections/my-crawl/indexes/

Currently, Browsertrix Crawler supports a number of command line options and a more extensive crawl config is coming.

Browsertrix Crawler represents a refactoring of the all-in-one Browsertrix system into a modular, easy-to-use crawler.

Use Case: Zimit Project

The development of Browsertrix Crawler was initially created in collaboration with Kiwix to support their brand new crawling system, Zimit.

The system is publicly available at: https://youzim.it.

Kiwix is a non-profit producing customized archives for use primarily in offline environments, and open source tools to view these custom archives on a variety of mobile and desktop platforms.

Kiwix’s core focus includes producing archived copies of Wikipedia. Kiwix maintains an existing Docker-based crawling system called “ZIM Farm” that runs each crawl in a single Docker container. To support this existing infrastructure, Browsertrix Crawler was architected to run a full crawl in a single Docker container. This versatile design makes Browsertrix Crawler easy to use as a standalone tool and adaptable to other environments.

The Zimit system produces web archives in the ZIM format, the core format developed by Kiwix for their offline viewers and the basis of their downloadable archives. ZIM files created by Zimit include a custom version of wabac.js service worker, which also powers ReplayWeb.page. Support for loading ZIM files created via Zimit is being further developed by Kiwix for all of their offline players.

To stay up-to-date with the Zimit project, you can follow it on GitHub at: https://github.com/openzim/zimit.

The development of the Zimit system was supported in part by a grant from the Mozilla Foundation. Webrecorder wishes to thank Kiwix, and indirectly, Mozilla, for their support in creating Browsertrix Crawler.

Announcing WACZ Format 1.0

Ilya Kreymer and Emma Dickson — Mon, 18 Jan 2021 00:00:00 GMT

The Webrecorder team has just finished a new release for WACZ and we’re delighted to share it with you!

WACZ stands for Web Archive Collection Zipped, and is a new file format designed to make creating and hosting web archives quicker and easier. The format has been in development for a few months, and we’re excited to announce the release of WACZ Format 1.0. The spec for the format can be found on github.

ReplayWeb.page and the newly announced ArchiveWeb.page extension both support the WACZ format 1.0. (ReplayWeb.page continues to support earlier iterations of the format as well)

ZIP-Based packaging for WARCs, Indices and Metadata

WACZ serves as a zipped package format for WARCs. Normally WARC files contain mostly the raw network data. WACZ files take the raw WARC files and zip them up, along with a CDX or compressed CDX index, and a full text index.

This gives WACZ files a few distinct advantages over plain WARC files. Because WACZ files are essentially Zip files, they benefit from a property of Zip files allowing them to be read on-demand over network without downloading the entire file. WACZ files they come packaged with everything you need to create and host a web archive collection: A random-access index of all raw data, a list of entry-point pages into the archive, and a user-defined, editable metadata about the web archive collection. As an added bonus, the full-text data extracted from web pages is also included, ready to be ingested into search engines like Solr or loaded on-the-fly along with the replay.

When using WACZ, ReplayWeb.page can quickly load large web archives without downloading the entire file. Using the ReplayWeb.page Google Drive extension, WACZ files can be loaded directly from Google Drive without downloading the entire file.

Frictionless Data Package

In an effort to base WACZ on established formats, starting from 1.0, WACZ also conforms to the Frictionless Data Package standard. The Data Package manifest adds integrity checks (via SHA-256 or MD5) for each file contained in the WACZ. We hope to expand this specification as well as collaborate with the Frictionless Data community to better standardize formats that are used in web archives.

Tools for creating and verifying WACZ

We have released wacz 0.2.0 Python package, the official reference implementation for creating and validating WACZ files. The library supports packaging up WARC files, simple full-text extraction and a variety of other options. The library can also validate existing WACZ files to spec. (For extracting WACZ files, any unzip tool can be used since WACZ files are also ZIP files).

See the py-wacz page for more details on options or run wacz -h after installing the python package.

The ArchiveWeb.page extension has built-in Javascript-based support for creating WACZ files for web archives stored in the extension. We hope to release additional JS-based tools for working with WACZ files in the future.

Community Feedback

While Webrecorder is leading the development, we would like the WACZ format to be responsive to the needs of web archiving communities. If you have any suggestions or comments, feel free to open an issue on the WACZ format GitHub or attend one of our upcoming community calls.

Introducing ArchiveWeb.page - Local High-Fidelity Web Archiving directly in your browser

Ilya Kreymer — Mon, 18 Jan 2021 00:00:00 GMT

Introducing ArchiveWeb.page Chrome Extension

I am excited to announce the launch of ArchiveWeb.page, a brand-new high-fidelity web archiving system available as a Chrome extension from the Chrome Web Store

The extension has been tested in latest versions of Chrome, as well as with the Edge and Brave browsers.

In classic Webrecorder style, the extension allows users to ‘record’ highly interactive websites, including social media, video, customized content, and even local intranet content.

When the original webrecorder.io was launched nearly six years ago, the goal was to allow users to record/capture exactly what is loaded in their browser. At the time, it was not possible to do entirely with a browser extension and an outside proxy server (running on webrecorder.io) was necessary. Now, thanks to evolution of the browser technologies, this original vision of archiving entirely in your browser can finally be realized!

The ArchiveWeb.page extension turns the browser into a full web archiving system, allow users to turn ‘recording’ mode on any tab, which will then capture/record all the elements of a page exactly as they are loaded. The archived data is then stored in the browser itself, and can be replayed/accessed even when offline. ArchiveWeb.page builds on and complements the ReplayWeb.page system, announced last year.

User Guide

To get users started with the extension right away, we’ve also launched a detailed User Guide, created by our Community Manager, Lorena Ramírez-López.

Read on below for an overview of some key features in ArchiveWeb.page.

Archiving Flash with Ruffle Emulator

ArchiveWeb.page embeds the Ruffle emulator, allowing users to archive and replay Flash-based works. Ruffle is automatically enabled on pages that have Flash.

Not all Flash pages will work with Ruffle, but many will. See our on-going efforts to ensure Flash remains accessible.

Page-oriented Archiving and Deduplication

In ArchiveWeb.page, the smallest unit is the page. The extension archive keeps track of which resources are loaded from which page. This allows for individual pages to be downloaded, and deleted, as necessary, and will help ensure archived pages are accurately replayed. Resources shared across multiple pages are automatically deduplicated to save storage.

This is a bit different than in Webrecorder Desktop, where the smallest unit was a session and individual pages could not be deleted or separated. Support for removing individual pages was an oft-requested feature, and this is now available in ArchiveWeb.page.

Full-Text for Web Pages and PDFs

ArchiveWeb.page includes built-in full-text search support. When recording a page, the text of the page is automatically extracted and indexed (when the page is first loaded and again when leaving the page). Text for any PDFs recorded is also extracted.

When replaying pages, enter text queries in the location bar to search pages by text.

Download Archives in WACZ or WARC

The extension fully supports exporting entire web archive collections or individual pages in the new WACZ Format 1.0. This format, which contains WARCs, indices and other data, makes it easy to share web archives and load them quickly using ReplayWeb.page.

See our blog post on the WACZ format.

Of course, the extension also supports downloading as plain WARC files as well.

The ArchiveWeb.page extension includes experimental peer-to-peer sharing of web archives, using IPFS. This feature allows users to share a web archive collection from directly from their browser!

ReplayWeb.page has been updated to support loading web archives directly from IPFS, allowing shared archives from the extension to be quickly shared with others, without having to download and send full WACZ files.

This feature is still experimental, see the guide page on sharing for some caveats.

Video

Here’s a brief video of the ArchiveWeb.page extension being used to archive a Twitter feed, including video, archive a MOMA exhibition page with Flash, replay each page, search by text, and then download selected pages in WACZ format:

Further Work / Coming Soon

This is only the initial release of ArchiveWeb.page. Here’s some additional work that is in the pipeline for future improvements.

ArchiveWeb.page Desktop App

For those that may prefer a standalone desktop app instead of an extension, we’re also working on an ArchiveWeb.page desktop app.

This app will shares the same system as the extension, but will run as a standalone desktop app. The ArchiveWeb.page App will replace the existing Webrecorder Desktop app, and we hope to offer a migration path to the new app once its available. Stay tuned for more details.

A development version can be built locally using the ArchiveWeb.page GitHub repository

Autopilot System

The autopilot system from Webrecorder Desktop, which runs automated behaviors on certain sites is not yet in this version of the extension, but rest assured that we plan to add this system to ArchiveWeb.page, both extension and app, in an upcoming release. We’ll be sure to make an announcement once it is ready!

Feedback

Try out archiveweb.page, read the guide and let us know if you have any feedback on this new tool! We want to hear from you!

You can reach out via the forum or attend our upcoming community call.

Flash Ain't Dead Yet! Even More Ways to Run Flash using OldWeb.today

Ilya Kreymer — Mon, 04 Jan 2021 00:00:00 GMT

Faster Emulation, More Browsers

A new version of OldWeb.today was released two weeks ago, switching to in-browser Javascript and WebAssembly emulation. Today, one of the emulators used in OldWeb.today, v86, received a major upgrade with WebAssembly.

OldWeb.today has now also been updated to support this new version, allow for Windows and Linux based browsers to run even faster. This should also offer a noticeable upgrade to Flash emulation in these browsers.

OldWeb.today now supports five versions of Flash (including a version of Shockwave with Director support) in nine different browsers. Three versions of Java are supported as well.

Old Linux Browsers with Latest Flash

Thanks to the updated v86, two new browsers: Opera 12 and Firefox 10 ESR are added, using a recent version of Tiny Core Linux.

These browsers are pre-installed with the latest Flash plugin for Linux, Flash Player 32, ensuring that even the latest Flash player is covered.

And of course, OWT also includes Ruffle, which is a Flash-specific emulator that runs Flash directly in your current browser.

You can try out these browsers and more at: https://oldweb.today.

Announcing the New OldWeb.today

Ilya Kreymer — Wed, 23 Dec 2020 00:00:00 GMT

Just over five years ago, at the beginning of December 2015, I released the initial version of OldWeb.today, which demonstrated running emulated browsers connected to web archives. This system used Docker to run emulated versions of browsers in the cloud, and required significant resources to maintain and could only support a fixed number of users at a time. (The old version of OWT is still available as classic.oldweb.today)

I imagined that this would be a temporary solution, and that eventually emulators will run fully in the browser.

Today, thanks to the work of numerous emulator developers and advances in Javascript and related technologies, I am excited to announce a fully Javascript/WebAssembly, browser-based OldWeb.today. This version supports three different emulators, all running entirely in the browser, limited only by your own CPU! Sound should fully work in this version as well.

The Basilisk II emulator is used for running MacOS up to System 7 and is used to run several early browsers, including early versions of MacLynx, Mosaic, Netscape, IE.

The v86 emulator is used to run Windows 98, and is presented with a version of Netscape and IE 5 and IE 6. (That’s right, IE6 is back!)

These emulators were modified to support a custom in-browser network stack (ethernet, tcp/ip) implementation, developed by the bwFla Emulation as a Service team. The stack allows connections from thee emulators to be handled in your current browser, and directed to either a web archive or to the live web.

The final emulator included is Ruffle, an open-source Flash emulator that is used to replay any web page with Flash enabled.

Differences from classic oldweb.today

Unlike the original, the entire system can be deployed as a static site, and easily integrated with existing web archives, if desired.

More details on the architecture and deployment can be found on GitHub.

Compared to classic oldweb.today, this version includes only browsers that can be run in a JS-based emulator.

This version includes IE 5, IE 6 and additional 68K Mac based browsers. All emulated browsers also support sound output.

Currently, this version focus on Mac, Windows browsers and Flash, but Linux based browsers could certainly be added as well.

Support for multiple web archive sources, similar to the original, is planned for a future update.

(Not) Gone in a Flash?

Much has been said about Flash ‘no longer being available’, but in reality, the end of Flash is not really the end. Thanks to easily accessible emulation, Flash can continue to be made accessible in a variety of emulation environments. OldWeb.today features old version of Macromedia Shockwave in MacOS browsers, Flash player 9 in IE5 and IE6, and the new Ruffle emulator running in your own browser. Does this cover all Flash works? Not yet, but advancements in emulation technology will continue to ensure that Flash remains accessible.

Here are just a couple of examples of Flash-based works, loaded with Ruffle:

For example, here are two ways to view another MOMA exhibition piece that requires Flash:

The Ruffle version is the fastest, but may not work as accurately. Until then, you can try the IE 6 version, which is more sluggish — an entire Windows 98 + IE6 is loaded in your browser after all! However, the result should generally be more accurate.

For example, an animation (from the excellent list of Physics Flash Animations), can be loaded in multiple browsers using OldWeb.today:

As the underlying emulators, Basilisk II, v86 and Ruffle will continue to be improved, I expect the number of viable options for viewing Flash to only continue to grow!

Remember Java Applets?

Before Flash, there was another technology that reached the end of life: Java Applets. Yet, many applets do remain in web archives and online, especially within academic department websites.

Today these applets can continue to be accessed directly in your browser using OldWeb.today.

Old Web and Web Archives

With the relaunch of OldWeb.today, Webrecorder is committed to making access to archived web as accessible as possible.

These examples above include both live web site and sites loaded from public archives. Currently, OldWeb.today can load from the Internet Archive Wayback Machine, and future improvements will support loading from multiple archives, similar to the previous version of OldWeb.today.

It is easy to deploy OldWeb.today as part of your web archive, pointing to a different wayback machine endpoint. See the README for more details or get in touch with any questions.

The system can support additional browsers, and the goal is to add more browsers in the future when time permits.

The biggest risk to Flash, like any web content, is that the data is no longer available.

Web archiving with Webrecorder tools, such as Webrecorder Desktop can help you archive any Flash content to ensure that it does not disappear.

Stay tuned for additional tools from Webrecorder to make archiving Flash even simpler!

An open source thank you!

It is important to acknowledge that this project was only possible by building on many open source tools and projects. In particular, it was possible thanks to the tireless work of emulator developers, Fabian who created v86, James Friend who ported to Javascript and Christian Bauer who developed Basilisk II, Rafael Gieschke of Emulation as a Service who built the Javascript network stack, and of course the work of all developers working on the Ruffle project. Continued access to digital web heritage is only possible through continued commitment and support of the open source software ecosystem.

Have feedback?

Have any thoughts/feedback/suggestions on OldWeb.today? Feel free to discuss and share on our forum!

OpenWayback to pywb Transition Guide and pywb update

Ilya Kreymer — Tue, 15 Dec 2020 00:00:00 GMT

Earlier this year, members of the IIPC (International Internet Preservation Consortium), after an internal survey, recommended the adoption of Webrecorder pywb as the primary replay system for their members’ web archives. Webrecorder and IIPC established a multi-part collaboration to help with this transition and advance the development of pywb.

To meet these goals, I’m excited to announce the launch of an official guide for migrating from OpenWayback to Webrecorder pywb, available at:

https://pywb.readthedocs.io/en/latest/manual/owb-transition.html

This guide was created with input from IIPC members and marks the completion of the first package of the IIPC project on pywb. This guide is now part of the standard pywb documentation and provides examples of various OpenWayback configurations and how they can be adapted to analogous options in pywb. The guide covers updating the index, WARC storage and exclusion systems to run in pywb with minimal changes.

For best results, deployment of OutbackCDX, an open-source standalone web archive indexing system developed by the National Library of Australia, alongside pywb is the recommended setup for managing web archive indexes. See the guide for more details and additional options.

Sample Deployment Configurations

With the guide, pywb now also includes a few working deployments (via Docker Compose) of running pywb with Nginx, Apache and OutbackCDX.

These deployments will be part of the upcoming pywb release and will be updated as pywb and configuration options evolve.

Next Steps

Next on the immediate roadmap for pywb is an upcoming release, which will feature numerous fixes in addition to the guide. (See the pywb CHANGELIST for more details on upcoming and new features)

The next iteration of pywb, which will be released in the first half of 2021, will include improved support for access controls, including a time-based access ‘embargo’, location-based access controls, and improved support for localization, in line with the work outlined in pywb project Package B.

Feedback Wanted!

We hope the guide will be useful for those updating from OpenWayback to pywb. We are also looking for input from IIPC members about any use cases for improved access control and localization for the next iteration.

If you have any questions, run into issues, or find anything missing, please send feed feedback via the IIPC mailing lists or directly to Webrecorder, via email or via the forum

Web Object Encapsulation Complexity (Part I)

Ilya Kreymer — Mon, 09 Nov 2020 00:00:00 GMT

What does it take to archive a web page/project?

As the web transitioned from static documents to interactive web applications, the challenge of archiving and preserving the web have only increased.

But some web pages/projects/publications - lets refer to them as ‘web objects’ - are easier to archive than others. Some require no effort at all, while others require significant effort and still can not be correctly archived. Sure, the number of pages there are, or how ‘interactive’ a project is, plays a role, but it’s not the fully story.

This is a key question for Webrecorder, and something I have wondered about for a while in determining what is possible and how much effort may be required.

To fully express this difficulty, a new methodology is needed, something I’ve decided to call ‘encapsulation complexity’.

Introducing Encapsulation Complexity

At its core, web archiving is really a reproducibility problem, the problem of capturing web objects, and replaying, or reproducing them later, as accurately as possible in an isolated environment from archival storage. To reproduce a captured web object, it must first be encapsulated, meaning all dependencies must be determined and also captured.

How difficult it is to encapsulate any web object and reproduce it later can be called the ‘encapsulation complexity’ of the object, which depends on a number of factors, such as external dependencies, explained further below.

Different levels of encapsulation complexity require different digital preservation approaches and tools and lead to different expectations. Sometimes a web object can be saved as a single page, or a web archive stored in WARC or WACZ files is sufficient. But sometimes it is necessary to also encapsulate a fully running web server running in an emulation system, and for other cases, even that is not feasible.

This complexity can be categorized with the following levels, which each level being an order of magnitude ‘harder’ than the previous one.

Level 0 (Single-Page Encapsulatable)

A single-page web object, with zero external dependencies. Images, CSS, Javascript, if any, are fully inlined in the page. The page can be directly loaded in any browser. The page need not be fully static, but it should not have any external resources, and does not require any web server.

A Level 0 object does not require much effort for encapsulation, as it can simply be saved using the browser’s ‘Save Page’ functionality and replayed again, as it is already self-encapsulating in a sense.

A screenshot, or a PDF can also fit into this category, as they are single-page web objects that can be loaded in a web browser.

Possible Tools: Built-in Save Page As, Single-Page Extension, Taking a Screenshot, Saving as PDF.

Level 1 (Web Archive Encapsulatable)

Level 1 web objects that consist of a finite number of URL resources that can be exhaustively captured or crawled. The resources can be loaded from any number of different web servers, including embeds. The object can be arbitrarily complex on the client, running complex Javascript, and requiring arbitrary user input. The web server interaction of this object must be limited to a fixed amount of data from a fixed number of URLs. Other dynamic network data, such as websocket connections, can not be included. The Level 1 object can be fully encapsulated as a web archive in a single WARC or WACZ format, and can load directly in a browser.

Most web objects that work well within web archives fit into this level of complexity, from small or single-page projects to large sites requiring crawling.

Possible Tools: Browser-based capture (Webrecorder), web crawling (Browsertrix, Heritrix, etc…)

Level 2 (Web Archive + Server Emulation Encapsulatable)

Level 2 web objects require a fixed, known web server to also be encapsulated to be fully functional, along with a WARC/WACZ based web archive. The web server must be encapsulated and reproducible in a self-contained computing environment. The server can have other dependencies, such as a database, as long as the database is deployed alongside the service and not externally. The client web object can make any number of dynamic URL requests to the fixed web server(s), including with websockets, that are running within the encapsulated environments.

Possible tools: Orchestrated web server containers (Docker Compose, Kubernetes) combined with web archives, web server emulation

Level 3+ (Not Fully Encapsulatable)

Any web objects that have an unknown number of external dependencies, or dependencies that simply can not be enumerated in a encapsulation/preservation system are Level 3 objects or higher. Web objects that make dynamic requests to external servers that are outside the control of the user, such as doing a search on Google. Web objects that rely on dynamic external data, such as specific camera, microphone, geolocation inputs or network speed. Of course, there is no limit to how complex such objects can get, and examining them further is not useful, as they are not ‘encapsulatable’ at full fidelity.

Possible Tools None currently, requires migration to a Level 2, 1, or 0 object.

Determining Encapsulatability: So what does it take to archive a particular web object?

Coming back to the original question, when looking at a particular web object, determining its encapsulation complexity can greatly inform on how difficult it will be to encapsulate/preserve and what the options may be.

Given the above methodology, this determination can still be tricky without examining ‘how’ a web object ‘works’, looking at the network traffic, interacting with it, and even examining the code.

For one, web objects consisting of multiple pages, the levels of complexity of each individual page may vary. For example, a mostly static blog (Level 1) may contain a page with a YouTube embed (a Level 3 object). Therefore, the whole blog would be a Level 3 object, because it involves a YouTube video which brings with it an infinitely interactive external dependency (including recommendations, related videos, etc..).

There are a few telltale signs that can help, though, such as embeds of external services like YouTube. As another example, a web object that has server-side search will be at least a Level 2 object, as it must make dynamic requests to the server for search.

If the search is implemented entirely client-side using a JSON search index and a client-side framework like FlexSearch, then the same object could become Level 1, all other things being equal. However, without looking at the network traffic, it may not be evident if server or client-side search is used.

For the author of a web object that will need to be encapsulated/preserved, it may be worth it to choose client-side search over server-side search to make encapsulation as a web archive easier down the road.

Alternatively, it may be reasonable trade-off to ‘migrate’ a Level 2 object with server-side search to a Level 1 encapsulated web archive, which could have a built-in web archive search, but lose its original search features. Indeed, web archives can fully encapsulate web objects up to Level 1, and anything higher is necessarily migrated to Level 1 to be encapsulated as a web archive.

The encapsulation complexity level provides an upper bound on how hard it may be to encapsulate a particular web object, as well as what the maintenance costs can be.

In a future blog post, I’ll provide additional approaches to determine this complexity and discuss migration of web objects to a lower complexity level and the trade-offs involved.

Lorena Ramírez-López joins Webrecorder as Community Manager

Ilya Kreymer — Mon, 09 Nov 2020 00:00:00 GMT

I’m excited to announce that Lorena Ramírez-López (DaleLore.com) has joined Webrecorder team as a part-time community manager!

Lorena is a trained moving image specialist for film, video and digital collections. Her main interests focus on the preservation and conservation of time-based media art as well as research and access to Net Art and web archives. A native New Yorker from Queens, Lorena believes in access and sharing resources, which is why she participates and collaborates in open-source projects, hackathons, and international communities with the Audiovisual Preservation Exchange from NYU.

Lorena will help develop and improve documentation, help answer questions on the forums and generally make all Webrecorder tools more accessible to all!

Stay tuned for more updates soon!

Web Archives as Digital Publications / Digital Publications as Web Archives

Ilya Kreymer and Matthew Conlen — Thu, 22 Oct 2020 00:00:00 GMT

Web archiving is often done after the fact — a digital publication is designed, built, published and only then, archived for preservation.

But what if the archiving process became part of the publication pipeline, complementary to online publishing, or an alternative distribution medium free from hosting requirements and available for offline use?

Parametric Press, a digital magazine focused on interactive, data-driven content has been experimenting with this approach from the beginning. With the release of each issue, they’ve also simultaneously released a downloadable high-fidelity web archive, which can accessed and viewed online or offline using ReplayWeb.page and the ReplayWeb.page app.

The magazine highlights interactive, browser-based articles created using the Idyll, an open-source authoring tool that helps writers produce digital work that combines text with interactive graphics and data visualizations. To use it, authors write in a markdown dialect that’s been imbued with a reactive variable system and syntax to embed dynamic JavaScript components. Idyll comes with a set of useful components but users can also bring their own as well, using libraries like React, D3, or P5.

Parametric Press utilizes and builds upon Idyll and is designed to serve as a platform for digital writers who want to incorporate more of the interactive potential of the web into their work. For example, in the newly released second issue, authors utilize simulations and data visualizations to highlight different aspects of the climate crisis. All of the project’s code is open-source and available as a technical blueprint for others wishing build their own interactive publishing platform.

As the issue was being prepared, I worked with Matthew Conlen, senior editor of Parametric Press and creator of the Idyll project, using Webrecorder tools, including Webrecorder Desktop, to capture each of the articles. Webrecorder’s high fidelity web archiving ensures that all of the interactive elements, visualizations and even external maps created in Idyll can be captured and replayed.

The complete web archive of this issue is packaged in the WACZ format and available for either online or offline use. The WACZ file of course also be hosted elsewhere, such as in a digital preservation system or any static hosting service.

You can thus access the interactive content in Parametric Press Issue 02 in the following ways:

“Live” Version - https://parametric.press/issue-02
Archived Version - View in ReplayWeb.page
Archived Version - Download for Offline Viewing with ReplayWeb.page App

Web archiving can be seen as a process of encapsulating online content and making it available as a standalone digital object that can be reproduced in the browser and eventually stored in digital preservation systems.

For traditional print-based or static publications, this can also be accomplished by creating a PDF or EPUB. But for dynamic publications, we hope that the approach taken by Parametric Press can serve as an example for interactive publishing. By releasing publications as web archives, publishers can blur the line between an archived version and the published version, creating self-contained, offline-accessible and preservable digital objects of their publications along from the beginning.

Emma Dickson joins Webrecorder as Generalist Developer

Ilya Kreymer — Wed, 07 Oct 2020 00:00:00 GMT

Emma Dickson Joins Webrecorder as a Generalist Developer

I’m excited to announce that the Webrecorder team has expanded, and that Emma Dickson has joined Webrecorder as a part-time Generalist Developer!

Emma is fascinated by outdated technology and the process of translation and obsolescence in technical languages. They love creating archives and archival tools. In addition to their work as a conservation technician on time-based media projects, Emma also produces net art and new media sculptures. Their art, including the 2018 piece “Mixed Connections”, explores identity, community, and longing through tech.

Emma’s expertise in software development and digital preservation will be of great help in improving all aspects of Webrecorder’s open source toolset.

Stay tuned for more updates!

Next Generation Web Archiving: Loading Complex Web Archives On-Demand in the Browser

Ilya Kreymer — Wed, 12 Aug 2020 00:00:00 GMT

I’m excited to present an exciting new milestone for Webrecorder, the release of six high-fidelity web archives of complex digital publications, accessible directly in any modern browser. These projects represent the entire catalog of Stanford University Press’s Mellon-funded digital publications, and are the culmination of a multi-year collaboration between Webrecorder and Stanford University Press (SUP).

You can read more about this collaboration, and additional details on each of the publications on the corresponding blog post from SUP.

It has been really exciting to collaborate with SUP on these boundary-defining digital publications, which allow Webrecorder to also push the boundaries of what is possible with web archiving.

The web archives cover a variety of digital publication platforms and complexities: Scalar sites, embedded videos, non-linear navigation, 3D models, interactive maps. Four of the projects are also fully searchable with an accompanying full-text search index.

All six projects are preserved at or near full fidelity (or will be soon: two are in final stages of completion). But digital preservation and web archiving require long-term maintenance, and my goal, in addition to creating web archives, is to lower the barrier to web archive maintenance.

For example, maintaining the live versions of these publications requires hosting Scalar, or Ruby on Rails, or other infrastructure. Maintaining these web archives simply means hosting six large files, ranging from ~200MB to 17GB, and a static web site online.

To present fully accessible, searchable versions of these projects, only static web hosting is necessary. There is no ‘wayback machine’, no Solr, and no additional infrastructure. The web site page for this project is currently hosted as a static site via GitHub while the web archive files, ranging from ~200MB to 17GB, are currently hosted via Digital Ocean’s S3-like bucket storage.

In the near future, we plan to transfer hosting from Webrecorder to SUP, which will simply involve deploying the static site from GitHub in another location and transferring the static files from one cloud host to SUP’s digital repository, and that’s it! Once transferred, SUP will not require any additional expertise, beyond simple website hosting, to keep these web archives accessible.

My hope is that this will demonstrate a new model for more sustainable web archiving, allowing complex web archives to be stored and hosted alongside other digital objects.

Below, I’ll explain the technologies used in making this possible, and steps taken to create these archives.

ReplayWeb.page and WACZ

This is all possible due two new technologies from Webrecorder: The ReplayWeb.page system and a new collection format currently in development, the Web Archive Collection Zip (WACZ) format, explained in more detail below.

ReplayWeb.page, initially announced in June allows loading of web archives directly in the browser, and is a full web archive replay system implemented in Javascript.

To better support this project, the replayweb.page embedding functionality has been further improved, and now supports search features, including page title and text search.

For example, it is now possible to link directly to certain pages or search queries in an embedded archive:

ReplayWeb.page embeds can be added to any web page. Here are some further examples of ReplayWeb.page embeds presented on this site, including the Filming Revolution project:

Web Archive Collection Zip (WACZ) - More Compact, Portable Web Collections

An astute reader might wonder: loading web archives in the browser works great for small archives, but does this work for many GBs of data? Surely loading large WARCs in the browser will be slow and unreliable? And what about full text search and other metadata?

Indeed, loading very large WARC files, the standard format in web archiving, directly in the browser is not ideal, although ReplayWeb.page natively supports WARC loading as well. An individual WARC file is intended to be part of a larger collection, and lacks its own index. Given only a WARC file, the entire file must be read by the system to determine what data it contains.

Further, the WARC is not designed to store metadata: titles, description, list of pages, or any other information about the data to make it useful and accessible for users (though Webrecorder has managed to squeeze this data into the WARC in the past).

The Web Archive Collection Zip (WACZ) format attempts to address all of these issues.

The format provides a single file bundle, which can contain other files, including WARCs, indexes (CDX), page lists, and any other metadata. All of this data can be packaged into a single bundle, a standard ZIP file, which can then be read and created by existing ZIP tools, creating a portable format for web archive collection data.

The ZIP format has another essential property: It is possible to read parts of a file in a ZIP without reading the entire file! Unlike a WARC, a ZIP file has a built-in index of its content. Thus, it is possible to read a portion of the CDX index, then lookup the portion of a WARC file specified in the index, and get only what is needed to render a single page. The WACZ spec relies on this behavior and ReplayWeb.page takes full advantage of this functionality.

For example, the Filming Revolution project is loaded from a 17GB WACZ file, which contains 400+ Vimeo videos. Using only a regular WARC, the entire 17GB+ file would need to be downloaded, and this would take far too long for most users and would not be a good user experience.

Using WACZ, the system loads only the initial HTML page when loading the project. When viewing additional videos, they are each streamed on-demand. The system would only load the entire 17GB if they watched every single video in the archive, but a more casual user can get a glimpse of the project by browsing a few videos quickly.

Even though the web archive is presented in the browser, the browser need not download the full archive all at once!

(This requires the static hosting to support HTTP range requests, a standard HTTP feature supported since the mid-90s)

Creating WACZ Bundles

WACZ files are standard ZIP files and initially were created existing command-line tools for ZIP files.

To simplify the creation of WACZ files, a new command-line tool for converting WARCs to WACZ is being developed. The tool is not yet ready for production use, but you can follow the development on the repo.

The format is still be standardized, and if you have any suggestions or thoughts for what should be in the WACZ format, please open an issue on the main wacz-format repository on github or leave a comment on the discussion thread on the Webrecorder Forum.

Creating the SUP Web Archives: From Web Archives to Containers and Back

I wanted to share a bit more about how these six archives were created.

All of the archives were created with a combination of the Webrecorder Desktop App and Browsertrix and warcit with the exception of Enchanting the Desert, which was captured by SUP’s Jasmine Mulliken using the Webrecorder.io a few years ago and converted to WACZ and transferred to the new system for completeness.

At the beginning of our collaboration, Jasmine at SUP provided full backups of each project and my initial preservation approach was to attempt to run each project in a Docker container, and then overlay them with web archives. An early prototype of the system employed this approach, with ReplayWeb.page seamlessly routing to the WACZ or to a remote, containerized server.

Ultimately, one by one, I realized that, at least for these six projects, running the full server did not provide any additional fidelity. Most of the complexity was in the web archive anyway, and little was gained by the additional server component.

To run the server + web archive version, a Docker setup and Kubernetes cluster operation were necessary, significantly complicating the operations, while a web archive only version could be run with no extra dependencies or maintenance requirements.

The Filming Revolution project is a PHP-based single-page application, but contains 400+ Vimeo Videos. The PHP backend was not at all needed to run the single page application, but the Vimeo videos needed to be captured via a web archive. After obtaining the list of video ids from the data, it was just a matter of running a Browsertrix crawl to archive all of them at the right URL.

The three Scalar based projects, Black Quotidian, When Melodies Gather and Constructing the Sacred at first also seemed like they would require a running Scalar instance. However, using the Scalar APIs, along with warcit to get any resources in the local Scalar directory, it was possible to obtain a full list of URLs that needed to be crawled. Using Browsertrix to archive ~1000-1200 URLs from each Scalar project, and using Webrecorder Desktop App to archive certain navigation elements, resulted in a fairly complete archive.

Finally, The Chinese Deathscape is a Ruby on Rails application, with a database and dependent data sets. It would have take some time to migrate it to run in Docker and on Kubernetes — but that would not matter. The entire set is just a few pages, easily captureable with Webrecorder Desktop, while most of the complexity lies in the numerous embedded maps. The project contains four or five different maps of China, at different zoom level, loaded dynamically from OpenStreetMap and ArcGIS. To fully archive this project, I first used Webrecorder Desktop to chart out the ‘bounding box’ of the map, and attempt to automate capturing the remaining tiles via a script.

The Chinese Deathscape requires another QA pass to check on the map tiles, and the Constructing the Sacred requires another pass to capture 3D tiles of 3D models used in that project. Webrecorder tools are capable of capturing both, though it is currently a manual process.

Overall, it turned out that running the server in a container did not help with the ‘hard parts’ of preservation, with perhaps one exception: full text search.

Embedded Full-Text Search

The Scalar projects all include a built-in search, and I wanted to see if a search could be implemented entirely client-side, as part of the ReplayWeb.page system.

By using Browsertrix to generate a full-text search index over the same pages that were crawled, it is possible to create a compact index that can be loaded in the browser, using the FlexSearch javascript search engine. Over the last week, I was able to add an experimental full-text search index to the three Scalar projects and even Chinese Deathscape. This results in a nearly-100% fidelity web archive. Since the live Chinese Deathscape does not come with a search engine, the web archive is arguably more complete than the original site.

Search can be initiated by entering text into the archive’s location bar, as shown in the video below.

The following video demonstrates searching for “Shirley” in Black Quotidian web archive to a page containing the video of Shirley Chisholm:

Future Work and Improvements

One of the remaining challenges is how to create better automation around capturing complex projects. I believe a combined automated + manual capture by a user familiar with a project will be necessary to fully archive such complex digital publications.

For example, archiving Scalar and list of Vimeo videos can be easily automated, thanks to the existing of concise APIs to discover the page list, but some pages may require a manual ‘patching’ approach using interactive browser-based capture. Archiving map tiles and 3D models may continue to be a bit more challenging, as the discovery of tiles may prove to be complex or always require a human driver.

Other projects, with more dynamic requirements beyond text search, may yet require a functioning server for full preservation. For these projects, the ReplayWeb.page system can provide an extensible mechanism for routing web replay between a web archive and a remote running web server.

A major goal for Webrecorder is to create tools to allow anyone to archive digital publications into their own portable, self-hostable WACZ files. The plan is to release better tools to further automate capture of difficult sites, including server preservation, combined with the user-driven web archiving approach that remains the cornerstone of Webrecorder’s high fidelity archiving.

If you have any questions/comments/suggestions about this work, please feel free to reach out, or better yet, start a discussion on the Webrecorder Community Forum

Supporting IIPC community in transitioning to Webrecorder pywb

Ilya Kreymer — Wed, 17 Jun 2020 00:00:00 GMT

I’m excited to announce an exciting new collaboration between Webrecorder and International Internet Preservation Consortium (IIPC), a group of national, university and regional libraries and archives involved in web archiving all over the world. The IIPC will recommend the adoption of Webrecorder pywb, the core Python web archiving toolset developed by Webrecorder as the ‘go to’ web archive replay system.

To support IIPC members in switching to pywb, I will be developing a migration guide, additional documentation and features to ensure a smooth transition for users of OpenWayback.

The pywb project was started in 2014 and has grown to be a fully-fledged replay and capture system for web archives, and has already been adopted by several IIPC members, including UK Web Archive, arquivo.pt, and the Australian Web Archive. The latest, pywb 2.4.x release includes a new new access control system and was supported in part by the UK Web Archive. I am especially thankful to these early adopters for paving the way for broader use, and look forward to working with IIPC Tools Development group and broader community on this project!

A New Phase for Webrecorder Project, Conifer and ReplayWeb.page

Ilya Kreymer — Thu, 11 Jun 2020 00:00:00 GMT

Today, I’m excited to announce a new phase for the Webrecorder Project, and several major releases/updates.

First, welcome to https://webrecorder.net/ - the new official site of the Webrecorder Project. Feel free to look around, and pardon the dust.

This site will contain all news and updates from Webrecorder, and the tools page is being updated to maintain a current index of all Webrecorder software.

Long Live Webrecorder, Long Live Conifer

In 2014, I created Webrecorder with the goal of building high quality tooling to support “web archives for all”, allowing anyone to create and share exactly what they experience in their browser, capturing interactive web sites as accurately as possible. Webrecorder started with a hosted service at webrecorder.io, but has since grown into an ecosystem of open source tools and free products to support web archiving capture and replay.

Thanks to generous support from the Mellon Foundation, Webrecorder was able to join Rhizome and together this hosted service was developed with a robust set of features, including remote browser and Autopilot automation, providing high-fidelity web archiving with a trusted cultural and digital arts institution. The hosted service known as webrecorder.io has now been renamed to Conifer and Rhizome will continue to run this unique service and build new features around this service. See Rhizome’s blog post for more details about Conifer and a brief post on the Conifer blog.

With the launch of Conifer, Rhizome will focus on running a brand new web archiving service, while Webrecorder will focus on software development.

As of 2020, Webrecorder is once again an independent project. Organizationally, I have formed ‘Webrecorder Software LLC’ to represent my work but the goals of the project remain the same as ever.

The Webrecorder Project will continue to work with Rhizome, and with many other partners to develop and maintain the best free and open source web archiving tools, and to further push the web archiving field forward with accessible, free and easy to use tools for all.

With that, I’d like to introduce the newest addition to the Webrecorder tool suite.

Introducing ReplayWeb.page

In an uncertain world, web archiving is becoming critical more than ever. A key, if not defining reason, for creating web archives, is to be able to access (“replay”) the archived web sites at a later time. The Webrecorder project has always focused on the accuracy and fidelity of capture and replay — recording and reproducing interactive websites as accurately as possible compared to the original.

Six years ago, when Webrecorder.io was started, high-fidelity web archive replay was possible only through a hosted service/centralized server. But with advances in browser and web technology, along with growth in decentralized web and storage, this is certainly no longer the case! Today’s browsers can natively run all sorts of applications, including a fully-featured web archive replay system.

Previously, I’ve introduced wabac.js, as a fully Javascript, browser-based experiment for rendering web archive replay. Today, I want to announce ReplayWeb.page, a new, fully featured browser-based replay system (or ‘wayback machine’) that further develops this idea.

The entire system is implemented as a static web page running from GitHub (https://github.com/webrecorder/replayweb.page), and is bundled as just two Javascript files: one for UI and one for the backend/service worker. ReplayWeb.page can load web archives located anywhere on the web, or from your local machine. No data is uploaded anywhere, and the browser stores the web archive (or loads it directly from the file system). ReplayWeb.page provides a brand new interface and a new replay engine, but should remain fully compatible with the existing Webrecorder (now Conifer) system, including supporting familiar curatorial features such as lists.

Webrecorder’s original goal of ‘web archives for all’ can only be realized when users have the tools to create and view web archives, on their own devices, or having a choice as to where to store their data. ReplayWeb.page takes a step further in this direction, by allowing a wide array of options for web archive storage. Have web archives on an institutional repository, or S3, or any cloud storage? No problem! ReplayWeb.page can load web archives directly from there!

Have web archives on Google Drive that should only be shared with select collaborators? The ReplayWeb.page Google Drive integration should allow that! Interested in storing data on decentralized web? ReplayWeb.page is designed to be able to support IPFS and Dat/Hypercore protocols in the future as well.

For larger local archives, or for archives requiring Flash, there is also the ReplayWeb.page App, which provides the same interface, but can support Flash even if your current browser can not. The ReplayWeb.page App fully replaces Webrecorder Player for offline use.

See the full User Docs for more details or just try out https://replayweb.page/ for yourself!

Not sure where to start? Here’s some example of web archive links to get started with replayweb.page:

More Updates / Summary of Changes

That was a lot of info! Here is a few more recent releases and overall summary of changes.

The release of pywb 2.4.0 is now out, and includes a whole host of features and fixes developed in partnership with the UK Web Archive, including a brand new Access Control system.
A new release of Webrecorder Desktop 2.0.2 is now out. This release features a few minor improvements, including a new Twitter Autopilot behavior, capture and fidelity improvements with pywb 2.4.0
ReplayWeb.page App along with https://replayweb.page supersede the Webrecorder Player, which will no longer be maintained. But don’t worry, ReplayWeb.page should support all the same features and work better.
Webrecorder.io is now https://conifer.rhizome.org and fully rebranded! All existing features of Webrecorder.io are maintained by Rhizome. More info in blog post from Rhizome
The webrecorder/webrecorder repository will be rebranded for Conifer. It will be maintained for Rhizome’s hosted service, but will not be developed separately as ‘webrecorder’.
The previous Webrecorder Blog from 2016-2019 is now the new Conifer Blog
Other GitHub repositories associated with Conifer (such as the user guide) are also being renamed.

Webrecorder or Conifer?

Perhaps you are confused about what tools will be provided by whom. Don’t worry! Here is a more clear delineation:

If you are interested in running web archiving tools on your own, running a desktop app, pywb, ReplayWeb.page, etc… Webrecorder will continue maintaining these tools! If you would like to integrate web archiving into other software or service, Webrecorder is here to help!
If you are looking for a well-established arts non-profit that has been committed to digital preservation to provide a dedicated web archiving service, then Rhizome’s Conifer is for you!

Of course, you can continue to use both and we will continue to work together in expanding the web archiving ecosystem.

Thank Yous

I wanted to thank Ashley Blewer for help in making this site and the new Webrecorder logo.

I would like to thank the folks at Rhizome for their continued support for Webrecorder from 2016-2020. The Webrecorder team that was at Rhizome over the years: Mark, Pat, Anna, John and especially Dragan, and Zachary Kaplan for supporting Webrecorder all the way as Executive Director of Rhizome. I look forward to further collaboration as we continue to work on Webrecorder and Conifer.

Stay in touch

Follow @webrecorder_io or @IlyaKreymer or check back on this blog for latest updates on Webrecorder.

You can also reach me via e-mail.

Stay tuned for a lot more updates in the coming weeks!

Ilya

Webrecorder Blog

Execution Time Addons, Robots.txt, Profile Refreshes, Custom Schedules, and More

Browsertrix 1.21

Purchase additional execution time

Pausing crawls when org limits are reached

Robots.txt support

Browsertrix 1.20

Browser profile refresh

Auto-updating browser profiles

Browser profiles and crawling proxies

Crawling tab refresh

Host-specific global proxies

Browsertrix 1.19

Custom cron schedule for crawl workflows

Filter archived items by tag

Crawl download improvements

Our statement on Conifer sunset announcement

Browsertrix 1.18: Large URL Lists and Beautiful Emails

Large URL Lists

Pretty Emails

How We Built Our Own Emails

What’s Next

Other Updates

Browsertrix 1.17: Crawl Pause/Resume and Lower Numbers of Browser Windows

Pausing and Resuming Crawls

Browser Windows

Create, Use, and Automate Actions With Custom Behaviors in Browsertrix

The Story of Behaviors in Browsertrix

Browsertrix Support for Behaviors

Introducing Autoclick Behavior

Custom Behaviors

How To Create Custom Behaviors

Our Commitment to Provide Accessible Tools

Our Full Suite of Tools

Our New Resources Page for Web Archivists

What You’ll Find

Send Us Your Feedback

Introducing GovArchive.us & Mirroring Entire Sites with Web Archives

What is Available Now?

Mirroring Sites with Web Archives — Getting Started

How it works: GovArchive and Wildcard Subdomains

Introducing Public Collections in Browsertrix

US Government Web Archive

Start Using Public Collections Today

Preserving Government Websites with Browsertrix

Why the End of Term Web Archive Matters

How Webrecorder Ensures Accurate Archiving

What’s Next?

Browsertrix 1.13: The Translations and Internationalization Release

Setting Your Preferred Language and Formatting Options

Contributions and Development

Browsertrix 1.12: Proxies, Crawling Defaults, and Simplified Workflow Creation

Proxies

Simplified Workflow Creation

In Case You Missed It

Crawling Defaults

Breadcrumb Navigation

New Documentation Sidebar

What’s next?

Browsertrix 1.11: Self Sign-Up, QA Improvements, Easier Downloading and new APIs

Self Sign-up

Better Quality Assurance (QA) Stats

Easier Archived Item Downloads

Fixes & Small Things

Changes for Self-Hosting Users and Developers

New Webhooks

Org Import & Export

Org Deletion & Crawling Restrictions

What’s next?

Browsertrix 1.10: Now with Assistive QA!

Assistive Quality Assurance

Crawl Analysis

Reviewing Crawls

Fixes & Small Things

Changes for Developers

What’s next?

ReplayWeb.page 2.0

New Branding

Page Snapshots Dropdown

Page Thumbnail Support