Project Overview

In this course, you will complete a quarter-long research project. This project will be completed in groups of three.

At a high level, successful projects will raise an important research question, and plan and execute a methodology for answering that question. Often, this methodology will include building and evaluating a prototype system, but hacking is not strictly necessary. All projects require a study — obviously a much more thorough study will be expected of projects that do not involve system building. The goal of the project abstract draft is to help you scope your work appropriately.

To get a sense of what a good scope for a project is, here are some examples of final papers. Because several of these projects went on to become a work-in-progress publications, their page length may differ from your final deliverable:

Groups who complete excellent projects will be encouraged to submit their research to the late-breaking work track for a top-tier human-computer interaction conference.

For information on how the project will be evaluated, see the individual rubrics under each assignment and the grading page.

Articulating Contributions in HCI

Choosing an ambitious and yet feasible project is the subtle art of research. There are two important keystones to making this work: 1) read a lot of papers in an area of interest, and 2) generate a lot of ideas. This is an individual assignment; you will form teams next week.

What makes something "research"? The high-order bit is that you are introducing an idea into the world. The terms to avoid are "advanced development" (vs. systems research) and "usability study" (vs. social science research), both of which indicate iterating on someone else's idea. If you are thinking of a systems-oriented contribution, then your idea needs to be the first one to propose an approach to the design of interactive systems. If you are thinking of a social science-oriented contribution, then your idea should be teaching us something general about people or their interactions with technology.

The goal of this assignment is to learn to distill contributions to HCI research. Refer to our slides about what constitutes a research contribution (e.g., new problem+old solution, old problem+new solution). A good scaffold for doing this initially is to build on prior research. So, for this assignment, pick papers that we've read so far and brainstorm potential follow-up research projects. Come up with three possible projects and write an abstract explaining each idea. Make sure to name which paper the idea is a follow-up on. Search for the original paper in Google Scholar and use the "Cited By" link to make sure that later work citing it hasn't already done your idea—research needs to be novel!

Here is an example abstract riffing on the Labeling Images with a Computer Game paper:
"Games with a Purpose suffer from collecting common knowledge rather than expert information. In this project, we propose a mechanism to collect expert knowledge through a betting framework. Players can "double down" on a tag they suggest. Both players then try to agree on a web URL that argues the term is right, or that the idea was incorrect. The main contribution of this research is a mechanism to acquire more uncommon and high-expertise terms."

Refer to the example projects at the top of this page to scope appropriately for a one-quarter project.

Grading rubric

Collectively, the abstracts will be graded out of 10 points.

Category Insufficiency Adequacy Proficiency Mastery
Research contribution
7 points
1: Ideas are present, but they do not clearly articulate why they represent significant new knowledge to HCI or why they have wide applicability. 3: The ideas demonstrate incremental new knowledge in HCI and have only minor generalizability. 5: The ideas introduce moderate new knowledge to HCI and are typically generalizable. 7: The ideas introduce creative new knowledge to HCI and are strongly generalizable.
Feasibility
3 points
1: The ideas have not been scoped to be completable in ten weeks. 3: The ideas have been scoped to be completable in ten weeks.

Project Ideas

Forming Groups

All project deliverables from here forward will be completed in groups of three. You can choose your own team members. When discussing a potential partnership with someone, you should discuss your background (e.g., programming proficiency or other skills you bring), availability (e.g., do you plan to primarily work evenings or mornings? weekdays or weekends?), and motivation level (ambition for a Turing award? Or to just barely graduate?). It's important to be honest with your partners up front, and to follow through on commitments you make.

Assignment

If you want to have good ideas, you must have many ideas.
Linus Pauling, Nobel Prize winning chemist

The purpose of this assignment is to help you generate many ideas. Along the way, you will end up reading many papers. For this assignment, work as a group to generate fifteen proposed research projects that would fit within the scope of this ten-week course. Do not split up and each contribute a few ideas — work together on this. The ideas may cover any topic and any method covered (e.g., building systems, studying people): check with the staff in office hours if you are unsure of fit. If you have particular interest in any of the ideas you generated in the last assignment, you may repurpose one of them (per person) here as well (but are not expected to).

This brainstorming assignment will be graded less leniently than the previous. As a reminder here are some suggestions when generating ideas:

  • Just proposing using a technology in a new place is not enough. "New problem old technique" only works if the old technique is NOT KNOWN to HCI. Example: machine learning on electrical noise to detect electrical events in the home. You can't take an HCI technique and apply it to another HCI problem — that's no longer research.
  • Feasibility. Remember this is a 10-week class. Make sure you scope your project so it's accomplishable in that timeframe. This includes ensuring that the population needed for your study is available.
  • Beware of super sensors. Building new technology takes time.
  • Using Google Scholar's "cited by" feature to discover related work.
  • If you're going to base the idea on another paper, you need to read the other paper.

For each project idea, give it a title, describe the core contribution in 2-3 sentences (i.e., what the HCI community will learn), and explain what exactly you're planning to build or what the method will be for the study you are planning. In addition, for each idea, cite one published paper or article that is a central piece of published, academic related work, and describe its relevance to your idea.

From those fifteen ideas, select 1) your most logical idea, 2) your most unexpected idea, and 3) your idea most likely to inspire.

Use Google Scholar as well as the references section in papers listed in this course to start poking through related work. If your idea is related to an area taught in the course, you can also take a look at past years' lectures on those topics by going to the syllabus, changing 347 to 376, and changing the year in the URL to last year. Look ahead to topics and readings later in the class for inspiration. Use past projects listed in the overview to help ground what may be possible in one quarter. We encourage you to join existing HCI research opportunities for your class project. Doing so will connect you with a mentor who can help guide the project. Use the HCI Group's research webpage to connect you with relevant folks. You are welcome to use riffs on these projects in your list of ideas for this assignment.

The grading rubric is identical to that of the previous assignment. Instead of grading all 15 ideas, the TA's will select the top 4 and grade based on those. This will produce the highest scores and will also allow us to nudge you towards projects that we feel have the greatest potential. If you want to pursue an idea outside of the 4 you were given feedback on, please meet with one of the TA's to discuss that idea prior to submitting your project abstract.

Project Abstract Draft

You must submit a draft of your project abstract. The abstract should aim to be around one full typewritten page (~500 words) with citations in APA format. Course staff will provide feedback on the draft to assist in the preparation of a final version.

Option 1: If you are designing or developing a new technology, your abstract should cover the following topics:

  • Problem: What is the problem that you are solving, and why is it important? For strong examples, read the first couple of paragraphs in the Introduction section of systems/applications papers in this course or on the HCI Group web site.
  • Insight and theoretical contribution: What is the unique perspective that you bring to the research? In other words, what is your "aha!" insight that others haven't established? How does it solve the problem? Keep in mind that even impressive technology prototypes are, at their core, answering a research question or establishing a thesis.
  • System: What is the exact system that you are building to instantiate this insight? What will it do? Include a low- or medium-fidelity sketch of the envisioned system.
  • Related work: Write a paragraph describing 4-5 pieces of published research, how they inform your project, and where your work trancends this knowledge.
  • Evaluation plan: how will you know if your system solved the original problem? (This should include the design of your study.) Grounding this in methodologies that other researchers have used (e.g. by drawing from the class readings) is a good idea. There are three major points you should hit here.
    • Study design: What are you going to do? Be detailed and precise. This can involve a large range of example outputs from your system, a field deployment, or a controlled study.
    • Measures: How will you know you succeeded? What will you measure? How will you measure it?
    • Ecological Validity: Why does your study answer your research question? Why does your evaluation address your hypothesis? Make sure your study, and the variables you're measuring, properly address the question you are asking.
  • Recruitment Plan: most evaluations involve participants. How will you get them? For pilot studies, we suggest you recruit from within the class -- "trading" participation with other groups is a great way to learn about what others are doing. For larger studies (e.g. for those not building a system), you need a clear recruitment plan.

Option 2: If you are developing a study, a new method, or data analysis, your abstract should cover the following topics. Keep in mind that the staff's grading bar for study design and execution here is much higher than if you are building a system. We recommend only going this route if one of your team members has prior background in social science research:

  • Research Question: What are you trying to answer? What are the theoretical contributions of your work to existing research in your area?
  • Hypothesis: what do you think the answer to your question is, and why? State your hypothesis in terms that you will actually be able to deliver on within the space of a quarter. For example, having a new technology increase someone's income might be your ultimate goal, but you may not be able to measure a change in income in 10 weeks. In this case, increasing income could be part of your motivation, but not your hypothesis. Your hypothesis needs a more proximal measure.
  • Related Work: Write a paragraph describing 4-5 pieces of published research, how they inform your project, and where your work trancends this knowledge.
  • Method: how will you explore your hypothesis, and why is that the right approach? (This should include the design of your study.) Grounding this in methodologies that other researchers have used (e.g. by drawing from the class readings) is a good idea. There are three major points you should hit here.
    • Study design: What are you going to do? Be very detailed and precise.
    • Evaluation: How will you know you succeeded? What will you measure? How will you measure it?
    • Ecological Validity: Why does your study answer your research question? Why does your evaluation address your hypothesis? Make sure your study, and the variables you're measuring, properly address the question you are asking.
  • Study Recruitment Plan: how will you get participants for your study? For pilot studies, we suggest you recruit from within the class -- "trading" participation with other groups is a great way to learn about what others are doing. For larger studies (e.g. for those not building a system), you need a clear recruitment plan.

For a guide to the APA format, go to APA Style. Note that the information on the site is possibly too detailed for the abstract. If you want a good example of the detail expected for the final paper, look at Dynamic Speedometer: Dashboard Redesign to Discourage Drivers from Speeding, Manu Kumar and Taemie Kim.

We encourage you to iterate multiple times on this abstract. While there is only one formally defined point for receiving feedback from course staff, you should seek out more informal feedback as you work on this. E-mail us at any point if you'd like us to take a look at your current submission, or come to office hours if you'd like to discuss in person. You are free to change directions after submitting your draft, but the sooner you nail down a direction, the better your project is likely to be.

Systems abstract grading rubric

Category Insufficiency Adequacy Proficiency Mastery
Problem
7 points
1: The problem is unclear, unimportant or not well-motivated. 3: The problem is clear but only motivated at a mediocre level. The solution feels like a hammer trying to find a nail. 5: The problem is clearly articulated and well-motivated. 7: The problem poses a novel perspective or a major opportunity for innovation.
Insight
7 points
1: The insight is not well described or has been covered previously in prior work. 3: The insight is clear, but makes only an incremental contribution to the field. 5: The insight delivers a reasonably novel idea relative to the literature. 7: The insight makes a significant contribution to the literature.
System
3 points
1: The system description is unclear or incomplete. 3: The main idea of the system is made clear through writeup and sketch.
Related work
3 points
1: Major related work is missing. 2: Much of the relevant related work is present, but either some is missing or its relationship to this project is weak. 3: Related work is present and well-represented. Its relationship to the project is clear.
Evaluation plan
7 points
1: The method is unlikely to lead to an evaluation of the insight or system. 3: The evaluation plan might work, but is vague or has methodological flaws, or there are obvious better methods. 5: The evaluation is likely to test the proposed insight or system, but has some minor methodological flaws or is overly complicated. 7: The evaluation is methodologically valid, not overly complicated and well thought through.
Feasibility
3 points
1: Project may need to be substantially scaled back to finish in ten weeks. 3: Project can likely be completed in ten weeks if the team works hard.

Study abstract grading rubric

Category Insufficiency Adequacy Proficiency Mastery
Research Question
7 points
1: The research question is absent or trivial (the answer is obvious). 3: There is a promising question but it is not clearly stated. 5: The question is clearly stated but has only minor impact on the field. 7: The question is clearly stated and its answer has major impact on the field.
Hypothesis
7 points
1: The hypothesis is the same as the research question or is trivial. 3: There is a hypothesis, but it is unclear, too ambitious, or not clearly motivated. (What makes you hypothesize so?) 5: The hypothesis is clearly stated, but is only vaguely motivated. (What makes you hypothesize so? Can you ground it in prior work?) 7: The hypothesis is clearly stated and well motivated.
Related work
3 points
1: Major related work is missing. 2: Much of the relevant related work is present, but either some is missing or its relationship to this project is weak. 3: Related work is present and well-represented. Its relationship to the project is clear.
Study design
7 points
1: The method is unlikely to lead to an evaluation of the hypothesis. 3: The evaluation plan might work, but is vague or has methodological flaws, or there are obvious better methods. 5: The evaluation is likely to test the proposed hypothesis, but has some minor methodological flaws or is overly complicated. 7: The evaluation is methodologically valid, not overly complicated and well thought through.
Measures
3 points
1: The measures being used are not reflective of the concepts being studied, are difficult to measure, or are likely to exhibit bias. 3: The measures being used have ecological validity and construct validity, and can be measured.
Feasibility
3 points
1: Project may need to be substantially scaled back to finish in ten weeks. 3: Project can likely be completed in ten weeks if the team works hard.

Project Abstract Revision

This is the final revision of your project abstract. It is weighted much more heavily than the draft.The formal requirement is that you revise your project abstract draft to address the staff's comments on your draft.

The rubric is the same as with the abstract draft.

Project Fair (Round One)

This is your first checkpoint for the project. The goal is to focus on prototyping or piloting the core element of your research idea. Reflect on your abstract and its core insight or question. What is the pilot study or prototype that would most clearly examine it? Do not spend much time working on aspects of the project that are orthogonal to your research question. For example, if you are testing a new debugging interface, focus on implementing one or two scenarios with special case data, not the entire underlying infrastructure or the overall IDE plug-in architecture. As another example, if you are studying whether people think that online comments that contain self disclosure (e.g., "I'm scared of Michael Bernstein's eyebrows") are more likely to occur on ephemeral or permanently archived social media, focus on gathering the status updates for the study and analyzing the results from a pilot run: don't worry about the details of your labeling interface.

If you are creating a system, the deliverable is a basic working prototype of your research idea. It does not need to be polished, but you should have the basic idea at a functional prototype level. If you are running a study, the deliverable is pilot results from at least five people who ran through an early version of your study, including a first round of data analysis.

Submit any relevant materials, including a one-page document describing your project status and goals between now and the Round Two project fair, via the class website. We'll also be using the last 30 minutes of class to show off our projects to each other, so bring any demo or study materials you've got.

Finally, each team member must complete this survey to give us a sense of how your team dynamics have been going thus far. Your responses at this point will not affect your grades or your teammates' — the intent is for us to check-in with your group mid-way through the quarter, and to calibrate you on what we're eventually looking for in terms of equal participation. We will not release your grades for this assignment until all team members have completed the survey.

Category Insufficiency Adequacy Proficiency Mastery
Depth
7 points
1: The system or study is too thin to evaluate the research question. Critical components are missing. 3: The system or study has a basic pilot/prototype, but it is only barely at a depth to point to next steps for the research. 5: The system or study has a pilot/prototype that exercises the main research idea, provides evidence of its final feasability, and points to next steps for the research. 7: The system or study has struck at the core of the idea and demonstrates clear directions for the research.
Focus
3 points
1: Too diffuse: attempting to prototype/pilot too much of the idea at once meant that the core question was largely left unexplored. 2: The system or study explores the main question or insight, but also spent significant time on less critical aspects of the research. 3: The system or study focuses directly on the most important parts of the research question.
Plan
2 points
1: Plan is underspecified, not ambitious enough, or too ambitious. 2: Plan clearly details the goals for Round Two and a realistic timeline to achieve them.

Project Fair (Round Two)

We are nearing the end of the quarter, so it's time to start showing off your best stuff. This is your final checkpoint before the final project presentations. The goal is to have the core element of your research project complete. If it's a system, that means that the core research functionality should be implemented, and (perhaps with an additional coat of paint to come soon) ready for an evaluation. If it's a study, you should have your final materials complete, the study should have at least ten participants so far, and you should have a first statistical/qualitative analysis of your data complete. If it's a study, it is OK for this version to have null results as long as you also have an idea of how to proceed.

By the end of the project fair, submit relevant materials and a one page document via the class website. In this document, describe your project status as well as your goals to accomplish between now and the final project presentations. We'll also be using the last 30 minutes of class to show off our projects to each other, so bring any demo, study materials, or data that you've got.

The rubric is the same as for Round One.

Final Presentation

Note: Due to classes being held remotely during the COVID-19 outbreak, there will be no Final Presentation assignment this year. The Final Paper portion of the project will weighted more highly to compensate.

At the end of the quarter, you will present your research results to the class and outside guests. We have invited a couple HCI luminaries. Feel free to invite interested friends and colleagues! Example slides from previous years are here and here.

Rubric for Final Presentation

  • Each group will have 5 minutes to present and 1 minute for setup. This time limit will be strictly enforced – groups should set up during the question session of the group before them. To enable this, unplug the video cable from your laptop before answering questions.
  • Test (and debug) your laptop video projection before presentations begin. Time spent fiddling with display settings will count against your presentation time.
  • Structure your presentation like a pyramid — begin with a one-sentence statement of your research result. This will get everyone on the same page. Then, explain things in detail.
  • This presentation is short enough that you can write out everything you want to say long-hand. Do this! This will allow you to convey information efficiently and effectively. Read through it enough times so that you have it basically memorized, but not so memorized that you get flustered if you skip a word or someone asks a question.
  • Know your audience! You can expect that everyone in the class knows everything you learned in class. So, you don't need to re-introduce the whole field of HCI. A sentence or two to situate your work in the field is good, but spend the rest of the time telling us what you did.

Category Insufficiency Adequacy Proficiency Mastery
Clarity
7 points
1: The research question, method and results are unclear. 3: The basic thesis of the work was communicated, but major components were buried or confusing. 5: The main ideas were clear, but the details were confusing, incomplete, or signaled a misunderstanding. 7: The presentation clearly sets out the project's goals, results, and implications, and all details were clear.
Effectiveness
7 points
1: The presentation was unconvincing in its communication of the core problem, question, and solution. 3: The presentation was unconvincing in its communication of at least one of the core problem, question, or solution. 5: The presentation made a reasonable case for the importance, novelty and validity of the research. There was at least one noticeable weak point in the argument. 7: The presentation made a strong case for all aspects of the project, and did so with forcefulness or flair.
Timing
3 points
1: The presentation ran out of time or misappropriated time between components of the research. 3: Balanced coverage of introduction, idea, system or study design, and results.

Final Paper

In addition to the presentation, you will present your findings in a final paper. This paper should be written in the same style as the papers we have been reading all quarter. However, it will be shorter, so pay attention to space limits. See examples here, noting that your length and format may differ as detailed below.

Page limit: Final papers should be 4-5 pages long in the CHI Proceedings format. Use the SIGCHI proceedings template to write your paper. While this may sound short, it is much harder to write an effective, complete short paper than it is to ramble. A good approach to writing a great short paper is to write a long one first, and then trim it down to the most vital parts. Appendices are acceptable and optional (they don't count towards the page limit), but won't be graded. Add one for materials you want an interested reader to see (for example, when we post your project on the website for next year), but don't need to be graded. The page limit includes references.

Each team member should fill out this Team Dynamics Survey prior to the assignment deadline.

Much of the advice from above for preparing your presentation applies to the paper as well. Here are a few more suggestions for preparing your paper:

  • Find a paper that you particularly like because of how it's written, and use it as a template. This paper needn't be on the same topic, but a close mapping in terms of type of contribution (e.g. a tool paper vs. a theory paper) will give you more guidance as to how to structure your paper.
  • Use collaborative editing tools such as Overleaf if using the LaTeX template or Office 365 if using the Microsoft Word template.
  • The title and abstract are the most important parts of a paper, and should clearly convey what you did. Motivate your specific problem (not the field as a whole), and focus on what you did. After reading the abstract, the reader should know what your contribution is – don't speak in generalities. For example, instead of saying "We analyze different methods for preparing cookies with interesting ingredients by running a user study.", say "We present three new recipes for chocolate chip cookies each employing a unique ingredient: jellybeans, tofu, and corn nibblets. Cookies were compared using a blind, within-subjects taste test with 30 individuals. The cookie with tofu was found to have superior mouth feel when compared with the other two, but subjects preferred the taste of the corn cookie by a 2:1 margin."
  • Review the Project Abstract assignment. Make sure you clearly address each of the important bullets from the abstract in your final paper.
  • Please use the APA heading structure to describe your study & results. Clearly tie your analysis to your hypotheses.
  • Use pictures to show your interface and graphs to present your data. Graphs should generally aggregate across participants, and show standard error bars. (Only show individual data points if the reader learns something more by doing so.) Figures should be captioned with what you believe the reader should infer from the figure (e.g. Participants rated tofu cookies to have 25% better mouth feel. Differences between jellybeans and corn nibblets were not significant). However, figures should be understandable without reading their captions.
  • Category Insufficiency Adequacy Proficiency Mastery
    Research idea
    7 points
    1: Unclear or indistinguishable from standard application development. 3: Vague, only of tangential relevance to the HCI research community, or incremental in nature. 5: A clear research question or goal, but not cleanly articulated as a novel contribution beyond prior work or is not a major contribution. 7: A clear research question or goal that makes a significant addition to HCI research.
    Execution
    7 points
    1: The project has some complete components, but critical aspects are incomplete. 3: The basic elements of the research are complete, but either they are flawed or important aspects are still missing. 5: All relevant aspects of the project have been completed, but they have flaws. 7: The research question has been explored and answered thoroughly through the project and writeup.
    Evidence
    5 points
    1: The evaluation or study is incomplete or has major analysis flaws. 3: The evaluation or the study is complete, but has significant analysis flaws. More leniency is applied to systems projects on this rubric item. 5: The evaluation or the study is complete and convincingly argues for a result.
    Related work
    3 points
    1: The description of related work is incomplete or surface-level. 3: The writeup covers major points of related work, and explains how this research extends them.

cs347@cs.stanford.edu