Digital Methods: Tools and Utilities

The various scrapers, crawlers, etc. listed below may be useful for different digital methods. For convenience, the list is loosely divided into tools and utilities, with the former denoting more complex operations that are either method- or device-specific, and the latter referring to relatively simple actions with multiple applications. Please note that of the prescribed uses for these tools and utilities, none are required.

Tools

Issue Crawler and Allied Tool Set

• Issue Crawler - Enter URLs and the Issue Crawler performs co-link analysis in 1-3 iterations, and outputs a cluster graph
• Extract URLs - Extracts URLs from an Issuecrawler .xml file
• Actor Profiler - The script will calculate the top 10 nodes out of the selected issuecrawler network map by indegree, query googlenews for the issue you specify, get the pagerank for those top 10 nodes and returns them and the actor profile graphic for those results. The colors for the actor profile are taken from the svg. >howto<
• Issue Geographer - Geo-locates organizations from an issue network and visualizes the results on a map.
• Compare networks over time - Compare Issue Crawler networks over time. Displays ranked actor lists from a scheduled set of Issue Crawler results.
• Google Network Cloud - enter an Issue Crawler xml id. The script gets the urls of the map and queries these urls in google for certain keywords and makes a tag cloud of it. Good for issue analysis in a network
• Triangulation tool - Enter two or more lists of URLs or other items, and the triangulation tool finds commonalities among them.

PageRank Tools

• Pagerank - Discover a website's Google Pagerank per issue/query.
• Issue Dramaturg - Scheduled list of websites' Pagerank per issue.

Google Scrapers

• Google Scraper - Query Google for a particular keyword/issue, possibly in particular sites.
• Google Scraper Frequency Tool - Use a results file from the Google Scraper and discover the frequency with which the original query appears in each individual host. (Tip: visualize these results in a tag cloud.)
• Google Teaser Text Ripper - Use a results file from the Google Scraper to get unique phrases from the teaser (or lead) text for each google search return.
• YouTube video discovery - Use a results file from the Google Scraper to discover, count, and rank YouTube? , Ikbis, and Google Video links in the descriptions. Obtain such a result file by querying a set of sites for e.g. 'youtube.com/watch', 'ikbis.com', 'videoplay?'.
• Split Results - Split the results from Google Scraper into two lists (e.g. blocked and unblocked sites).
• Google News Scraper - Query news.google.* with one or more keywords. It's only possible to scrape articles of the last 30 days.
• Google Images Scraper - Query images.google.com with one or more keywords, and/or use images.google.com to query specific sites for images.
• Scrape Google Blogsearch (Open Kapow) - Scrapes titles and URLs for a google.nl/blogsearch query.

Technorati Scrapers

• Scrape Technorati Posts (Open Kapow)
• Scrape Technorati Inlinks (Open Kapow)
• Actor Resonance Tool - Queries Technorati to measure the resonance of an actor for an issue in the Blogosphere.
• Actor Resonance Chart Builder

Other Tools and Scrapers

• Open Calais - Discovers the most relevant words and phrases among a set of websites, within a text, or within an issue network (i.e. an Issue Crawler .xml file). This tool is based on Reuters Open Calais.
• Issue Discovery - Discovers the most relevant words and phrases among a set of websites, within a text, or within an issue network (i.e. an Issue Crawler .xml file). >howto<
• Significance Measure - Enter a set of keywords. They will be scraped in various search engines and their specificity to eachother will be returned.
• RSS discovery - Discovers RSS/ATOM/RDF feeds in websites.
• Yahoo inlink Scraper - Gets all the inlinks to a site from Yahoo.>howto
• YouTube Response Retworks - enter a YouTube? movie and get all the responses to that video.
• Surfer Issue Pathways - Building upon Alexa's related sites feature, this tool determines which sites are likely to be in the actual surfer paths of other sites related to the same issue.
• Wikipedia Network Analysis - Find a hyperlink network around a Wikipedia topic (see here for more information).
• De.licio.us Related Tags Cloud Generator - Create a tag cloud showing URLs and tags related to a specific issue or keyword.
• Delicious tags for url - Get delicious tags for a specfic url (tagcloudable).
• Webpage History Generator - Uses the Internet Archive's Wayback Machine to make screenshots of all different versions of a site and output a webpage history scroll.
• WeScrape (Meta-Tool) - A howto guide for building your own scraper.

Utilities

• Language Detection - Detects the language from websites.
• Link Ripper - Capture all internal and/or external links from a page.
• Compare Lists - Compare two lists of urls for commonalities and differences.>howto<
• Image Ripper - Scrape images from a single page.
• Robots.txt Ripper - Display a site's robot exclusion policy.
• Censorship Explorer - Check accessiblity of a URL through proxies located around the world.
• Text Ripper - Rip all non-html (i.e. text) from a specified page.
• Rip Sentences - Rip text from a specified page and force line breaks between sentences.
• Timestamp - Rips and displays a web page's last modification date (using the page's HTML header).
• Whois - Use a site such as http://www.whois.net/ to check who has registered a particular domain name.
• Counting Tag Cloud Generator - Takes and counts raw text or a Google result and returns an ordered, unordered or alphabetically ordered tagcloud. >howto<
• Tag Cloud Generator - Input tags and values to produce a tag cloud. Output is in html, svg, and pdf.
• Counting Tree Map Generator - Input a text and visualize the word counts in a Tree Map. Output is in svg.
• Tree Map Generator - Input tags and values to produce a Tree Map. Output is in svg.
• Dorling Map Generator, aka the Bubbles Generator - Input tags and values to produce a Dorling Map. Output is in svg.

External Tools and Wishlist

• External Tools - Useful tools from around the Web.
• DMI Tool Wishlist - An internal list of tools desired by members of the Digital Methods team.

Using the tools

• On this page you can find some pre-DMI methods which use the tools. See also MethodsByTheme and the DMI Course Outline

Crystal Ball Tool (reveals future 'look' of all DMI tools):