Overview

MILC studies (Multi-dimensional In-depth Long-term Case studies) is a research method which is well adapted to the study of the creative activities that users of information visualization and visual analytics systems engage in.   The method was designed to assess the value of research prototypes by documenting 1) usage (observations, interviews, surveys, logging etc.) and 2) expert users’ success in achieving their professional goals.  

A MILC study has unusual characteristics which makes it practical for evaluating visual analytics research prototypes:  

• Participants are professionals trying to solve real problems with real data. They often have already analyzed their data with off the shelf tools, knew about our research, and expressed interest in trying our prototypes.
• We first negotiate a custom interview schedule and ask them to define what a measure of success might be for their use of the tool.
• After some training (an hour to a week) they work on their own, at their own pace (for 1-6 weeks), Sometime they do the analysis with a researcher driving the interface (we call that “chauffeur mode”).  This avoids having to fix all usability issues before testing, but of course it needs to be reported. 
• MILC studies also document how the software changed over time, possibly in response to the analyst’s feedback or requests. 
• Individual MILC studies do not always succeed, often for good reasons (e.g., mismatch between the tool and the data or task; need to collect new data; change of work priority) so researchers should report cases that showed benefits as well as those that did not – and what roadblocks were encountered.
• The flexible MILC format allows researchers to work with analysts who use confidential data and report usage and benefits in general terms. Consent forms and questionnaires identify potential issues early on.

The measure of success is the success of the analysts conducting their own work

MILC studies have become important parts of our PhD students’ dissertations. They are used as evaluation methods in scientific papers, and have led to scientific papers co-authored with MILC partners in the scientific journals of our partners’ professional fields.

Original Publication

Shneiderman, B., Plaisant, C. Strategies for Evaluating Information Visualization Tools: Multi-dimensional In-depth Long-term Case Studies. Proc. BELIV’06 – Beyond Time and Errors: Novel Evaluation Methods for Information Visualization 1-7

In 2016, for the 10 year anniversary of BELIV the MILC paper received the Test of Time Impact Award asBELIV’s most viewed, cited, and downloaded BELIV paper [see acceptance speech slides]. As of Nov. 2020 the paper has 560 citations according to Google scholar.

Sample IRB application

To help you use MILCs we provide this Sample IRB Application (as a WORD DOCUMENT)

• Sample IRB Application (p. 2)
• Sample recruiting email (p. 5)
• Consent Form (p. 6)
• Entry Questionnaire  (p. 9)
• Exit Questionnaire and consent form (for publication permissions) (p. 11)

Warning: This sample application is from 2010, in Maryland.  Rules varies over time and between countries so please adapt the sample application and consent form as needed.

Examples of the use of MILC studies in our research

•     Taowei Wang’s 2010 PhD thesis (Interactive Visualization Techniques for Searching Temporal Categorical Data) included several six MILC studies (see p. 134). 
  After his defense Taowei summarized the medical case studies in a paper which appeared in the Journal of Medical Systems  [Published Version] [PDF]

• John Guerra Gomez had the highest number (13) MILC studies, reaching out to a lot of people who used John’s prototypes at different stages of development – often with John driving, which we called “chauffeur mode”). See John’s Phd Thesis: Exploring Differences in Multivariate Datasets Using Hierarchies: An interactive information visualization approach Published version ~ TR version
  The table below is on page 6.

• Sana Malik’s Ph.D Dissertation (A Visual Analytics Approach to Comparing Cohorts of Event Sequences ) describes 10 MILC studies (see p. 87.)

• Fan Du’s 2018 PhD thesis reports on 3 MILC studies that were conducted by partners working with confidential data (e.g. student records or log data used by Adobe analysts during Fan’s internship there) so the case study summaries had to remain general. No names and few or no screenshots were included.   The MILC partners reviewed the materials before release. See: Fan Du, Explainable Recommendation for Event Sequences: A Visual Analytics Approach (the table below is on p. 146)

Contact

• Catherine Plaisant, Senior Research Scientist, UMIACS
• Ben Shneiderman, Professor, Computer Science

Related materials 

Beliv’ Workshops

Valiati, E.R., Freitas, C.M. and Pimenta, M.S., 2008, April. Using multi-dimensional in-depth long-term case studies for information visualization evaluation. In Proceedings of the 2008 Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization (pp. 1-7)