Blader door bijdragen

When slowing down means moving fast: A blog about iPRES 2022 from the IISG

Commercial Social Media Archiving Tools: Summary of First Survey by NDE/IISG

Social media archiving is one of the latest hot topics in digital preservation and information and records management. It is now becoming widely recognized that the contemporary record, whether it is meant for evidence, research, memory or any combination of the above, can and will very often contain material originating from various social media platforms. Background of the project In the Netherlands too there has been recent mounting interes...

Examples of social media

Social networking sites e.g. Facebook, LinkedIn, WhatsApp, Telegram, Viber, Signal, I-message, Facebook Messenger, WeChat; dating apps such as Tinder, Bumble, Inner Circle, Grindr– sites that connect friends, groups based on interests or professions. Blogs e.g. Medium, Wordpress, Blogger, and Tumblr; Twitter can be considered as micro-blogging Image and video sharing platforms e.g YouTube, Vimeo, Tumblr, Instagram, Snapchat, Imgur, TikTok, a...

Structured data formats (JSON, CSV, XLSX)

The turn to more data-intensive access methods to web and social media archives, as indicated by the use of big data and digital humanities methods to analyze social media content calls for capturing social media in formats appropriate for these activities. Usually in formats like JSON, CSV, and XLSX, collections made up of structured data are more amenable to computational methods such as network analysis, topic modelling, and many other visu...

WARC

The WARC format is widely accepted enough to be considered one of the default formats for storing captured content from the web. It followed its predecessor, the ARC, as the main file format in use by the Internet Archive, and is maintained by the International Internet Preservation Consortium (IIPC). The rationale behind the WARC format is that one file format for web archiving should preferably be able to hold not only the archived resources...

File formats for social media collections

The two general approaches to social media archiving presented here ("look and feel" and "structured data") also have implications for file format selection, which by extension has implications for preservation and collection quality. The choices made when capturing and preserving, part of which is selecting appropriate formats according to one's purpose, will affect the possible uses the collection can be put into, and by extension, the types...

"Structured data" tools

This is a list of tools that captures social media content in the form of structured data, focusing on the information included e.g. text, URLs, number of posts, etc., and not on the visual features of the content.

"Look and feel" tools

This is a list of tools that capture the experience of browsing social media, i.e. visual features, media, etc.

References

TAGS

It is safe to say that most of the tools that output structured data are not the easiest or most intuitive to use. One notable exception then is TAGS (Twitter Archiving Google Sheet). TAGS is in essence an app built on Google Sheets, that uses the Twitter API to fetch structured data based on queries the user inputs in the spreadsheet. TAGS makes use of an already authenticated Twitter API app for its operation, but you are able to use your ow...

Munin-Indexer (Munin)

Munin (Munin-Indexer) uses Docker to wrap different scraping and archiving tools together and offer a scraping solution for Facebook, Instagram, and VKontakte. It indexes and scrapes posts, then crawls and captures them, and finally uses pywb to display them. Suitable for public social media content The important thing to note about Munin is that it is only able to archive public posts, i.e., only posts that do not sit behind a log-in. Conseq...

Crocoite

Note: According to its GitHub page, this tool is not in active development anymore at the time of this writing (January 2022). However, it is still available for download and it still functions as expected. In a way, crocoite is a good example of a tool arising from the open-source community that could prove problematic to use in a professional setting because of lack of ongoing support. As browser-based crawling seems to become central in th...

Browsertrix Crawler

Initially known as Browsertrix, the Browsertrix Crawler is the latest and revamped version of what used to be a system of multiple browser-based crawlers that worked together to capture complex web content, such as social media. Browsertrix Crawler is built by the team behind the online web recording service Conifer and the desktop app ArchiveWeb.page (formerly known as Webrecorder and Webrecorder Player respectively) and uses the Chrome and C...

Brozzler

For those looking for large-scale harvesting solutions, Brozzler, like Browsertrix, is an interesting choice. Brozzler was developed and is still being maintained by the Internet Archive, and it is already used by organizations such as the Portuguese Web Archive. It is a browser-based crawler which uses Chrome or Chromium to access web content and harvest it in a WARC file. Brozzler is one of the newer-generation capturing tools which leverag...

Survey of free and open source tools

The tools are divided based on the type of output they produce, and each page includes presentation of features as well as a more detailed review based on personal experience with implementing and using them.

Requirements and features for review and selection of tools

Part of the research into social media archiving tools that was performed by NDE/IISG had to do with finding out what the requirements would be to consider these tools suitable for different kids of usage and users. In these pages, more general quality attributes for social media archiving software are presented, that could be relevant for any type of tool, free and open-source or not. Additionally, the functional requirements that were first ...

Tool surveys by the NDE

This wiki page includes the tool surveys that NDE/IISG performed between 2020 and 2021. The aim is to keep these pages up-to-date as much as possible, and encourage other organizations involved with web archiving in the Netherlands to contribute to them with content they consider useful.

The "structured data" approach

Instead of focusing on the "look and feel" of the material, its visual form and its multimedia affordances, the "structured data" approach focuses on informational qualities and the raw data that derive from the captured social media content. The output of this is structured textual data, usually in tabular form. Social media platforms make structured data derived from their websites available via API services, i.e., specific interfaces create...

The "look and feel" approach

This method is based on common web archiving practices, which makes sense as social media archiving can be seen as an offshoot of web archiving. The most common method of web archiving, namely web crawling or web harvesting, attempts to preserve the so-called “look and feel” of online content, meaning the layout, structure, and style of a website, as well as its navigational features, like buttons and menus. However, this has proven to be rela...

Approaches to archiving social media

Social media appears in many shapes and forms, aimed at many different audiences and serves many different purposes. This variety also affects the ways we can consider what a social media archival collection is, and how it is created. In this section, two broad approaches to social media archiving will be presented, i.e. "look and feel" and "structured data". These are not mutually exclusive, and in many cases it could be preferable or advisab...