Tag Archives: @sbreal

Tandem Week 13 Update


We are happy to announce that the initial version of our near-polished UI is up and functioning on http://dhtandem.com/. This development means that you can now go to the site and walk through uploading files as well as review some early versions of our documentation.

Immediate next steps for our team include updating the text on the documentation pages to the more robust things we have patiently waiting in the wings while we finalize the connection of the front and back components of the app. We have been powering away at creating thorough documentation and user information to be present on the final site. This also includes our exploration of the Mother Goose corpus which is beginning to take shape (in part thanks to some TANDEM supporters and volunteers from the praxis class). Basically, we’re pushing our data set through various tools for discovery and analysis. These results will become incorporated in the Sample Data section on the TANDEM website, which is intended as an example of the apps potential, and as a learning tool for new users.

As we continue to work on bugs and high priority action items, such as fixing an error with zipping files that originated from a change in processing in this iteration, we are realizing areas that could use strengthening post-dhpraxis. Our function May 19th MVP is so close we can taste it.

The zipping problem mentioned above may be related to another problem, which only happens on the server and cannot be replicated in a development environment. What appears to happen follows: when a user starts a new project, TANDEM builds three folders on the server, one for the uploaded files, one for the final output which is subsequently zipped for download. The third folder is a staging or intermediate directory that can contains files after any pre-processing that is required. For example, PDF files must be converted to JPG for our image analysis software to work. Another example is that the text must be extracted into TXT files via an OCR step for NLTK to be able to consume the content.

These new folders appear to be created successfully, and their locations are saved to global variables in the program. However, when it comes time to write files to the newly created folders, it seems that the file are being written to a previously used set of folders. The problem is intermittent. To make diagnosis more difficult, the zip step sometimes zips the older folder which delivers content from multiple projects to the user. However, other times the zip step zips the new folder which is empty delivering an empty file to the user. At still other times, the files are all read and written properly.

Zipping issues aside, we are moving along. Given all the amazing progress we have made, it is not surprising that buzz for the launch is growing. (Also Jojo invites anyone and everyone she speaks to). With new details regarding presentations, we are ready to get this party started. The DH community at CUNY and in New York has been a part of these projects whether actively or abstractly, and it seems a grand opportunity to celebrate.


TANDEM Project Update 4.11.15

TANDEM Week 9 Presentation

TANDEM: A Brief Agenda

I. Review our project goals

  • Discuss new interested users (advertising, biodiversity cataloging)
  • Discuss output applications in “Mother Goose Counts”

II. Describe our development drive

  • Branches of Dev underway
    • UI/UX dynamic pages
    • Django framework
    • TANDEM tool python script

III. Explain our development steps

  • Two parallel paths were followed building Python “backend” code to run the analytics on the users’ input files
  • The paths were merged and tested on a laptop
  • The Python environment was then built on the server
  • A command line versionTANDEM will now run on the server using local server-based files.
  • @sreal19 will Demo TANDEM! (Fasten your seatbelts, folks!)

IV. Discuss next steps

  • What still needs doing hooking up front and back ends.
  • Getting polished examples of our output up along with clear links to available datavis resources.
  • Getting Kelly’s best practices documentation live.
  • Outreach (not just to beta testers, but to users who might not have considered these tools before — looking for education applications/journalism
  • Now is also the time to start considering the life beyond Praxis:
  • Grants for continuing work?
  • How much labor/manpower/development would be needed to move beyond MVP?
  • What does 1.0 look like?

Thanks for following and stay tuned for updates!

@dhTANDEM #picturebookshare

tufte retweet




Publishing Case:

Chris is a Data Analyst for the Advertising department of XYZ Publishing. He has the banner ads from this year’s holiday campaign. He is interested in analyzing what generated the highest click-through rates for the company. Chris has previously downloaded and installed TANDEM to his desktop tool. Chris drag-and-drops his folder of ads onto the TANDEM interface. A progress bar appears. A .csv file is generated in the backend to store the output. The completion page gives Chris a downloadable CSV. Chris is directed to brief guides on how the data could possibly be used/visualized. Chris goes the basic route and enters excel to explore his data. He compares the data to the clickthrough rates in the ad server and notices a trend in the relationship between brightness and saturation, along with the number of words on the advertisement, and how many users clicked the ad. The brightest ads with 10 words or less had the highest click through rates. Chris is able to make an data-driven argument with the design team for brighter ads with minimal text in future campaigns.

Scholar Case:

Professor Plum is studying how advertising strategies have been affected by a significant historical event such as World War I. He has collected a corpus of print advertising materials spanning multiple product categories both before and after the event which is being studied. Plum wants to know what has changed and has developed theories regarding a number of features among which are the following questions:

  • Has the proportion of text to image changed? How?
  • Has the word usage changed? How?
  • Has the iconography changed? How?
  • How has the visual style changed? Are the different colors being used? Are the images more contrasty?

Using a tool outside of TANDEM, Professor Plum scans the materials into a digital format such JPG, TIFF, PDF or GIF. After the image files have been built, he downloads a copy of TANDEM from the Internet and installs it on his desktop computer. Plum launches TANDEM and starts the analysis process by inputting the name of the folder that contains  the electronic documents being studies. TANDEM outputs OCR, NLTK and FeatureExtractor data into a database, which can be saved.

Professor Plum can now use TANDEM (or some other visualization tool) to produce visualizations or tables on the parameters that are of particular interest to the scholar. Based on the results of these visualizations, Plum may make some adjustments to the settings in TANDEM to produce a more useful result. He may choose to export the results database to another application for further work or study.

Educator Case:

An early childhood educator, Yasya Berezovskiy, wants to study the effects of children’s literature on neurological development, exploring factors such as narrative, image representations, and lexiles (or word complexity/reading level) together. To date, Berezovskiy has worked with empirical evidence and collected fieldwork data.

Berezovskiy will be analyzing a number of children’s books with varying factors, ranging from author collections, time published, and theme.

Using TANDEM Berezovskiy can upload page images or entire works to process the work’s text in comparison to the visual information. Once complete, Berezovskiy can visualize the processed files in split screen, with the original image beside the visualized data. From there, Berezovskiy can choose to isolate individual elements to analyze, such as opacity, density, text to image ratio, text to color ratio, shape to text ratio, and more. Alternately, Berezovskiy can download the raw processed data to analyze using a separate visualization program.

The processed data will be complementary to other observational research being done by Berezovskiy’s colleagues. Without TANDEM, the evidence from the children’s books would have been only descriptive. Further, without TANDEM it would have taken Berezovskiy multiple programs and more effort.

Fairy Tale Nerd Case:

The user, a woman interested in creating a datavisualization for a pop lit site like Toast.net — let’s say Ella, wants to look at Victorian illustrated fairy tale collections. Ella wants to analyze captions for art plates in all available published works. She wants a computer to process all available picture books to give her more information on the content of a work based on its visual properties as well as its textual content. She wants to get a computer to pull all the words included in the illustrations, as well as the ratio of those words in relation to what is written in the story (Are they direct quotes? Are they distinct?). She goes to the TANDEM interface. There, she sees a simple description of what files the application will yield. It’s so understandable! All the fields are so well explained! She clicks the upload button, finds the files on her computer, uploads the picture book scans, and runs the application. Once the TANDEM program has run, another window appears offering a number of file types. Each file type has a scroll over description of its applications and recommended datavis links. Once she has selected, she can download the data file (CSV or …. …..).

Ella takes it to her favorite datavis site and goes wild with joy at the new capabilities and bases for comparison. All her dreams have been answered. Thanks, TANDEM!