LIGN 6 Final Project Writeup Rubrics

THIS RUBRIC WILL CHANGE SLIGHTLY BEFORE WEEK 6. PLEASE CHECK BACK FOR THE FINAL VERSION

For a successful final project, you will be graded on the following dimensions, each contributing in varying degrees to your final grade. They are presented roughly in order of importance, and the four ‘Levels’ of mastery can be thought of as corresponding roughly to ‘95%’, ‘80%’, ‘60%’, and ‘0%’. Your overall grade for the project will be based roughly on your overall level across the categories, adjusting for other factors (unclarity, missing sections, exceptional or low effort, etc).

Three Big Things

1: This is about language

The biggest thing to remember is that this is a class about language and computers. Some students write very compelling papers about human computer interaction, or interaction design, or troubleshooting Python installations for NLP, or the difficulties of designing tools to interface with a certain task, but completely fail to demonstrate that they’ve paid any attention in LIGN 6, by neglecting the important, language-focused elements of the virtual assistant or NLP tools. A paper which is of the proper length, mentions all of the steps of the interaction process (where applicable), and describes an interesting tool can still be an “F” quality paper if it never engages with natural language processing and linguistics. So, in each step of your paper writing process, ask yourself ’How am I showing the linguistic knowledge about natural language processing that I’ve gained from LIGN 6?”

2: Effort Matters

Every year, I get one or two students who attempt to submit exceptionally short, shoddy, or last-minute work, figuring ‘oh well, I’ll probably get a C and that’ll be OK’. And every year, those students are shocked when they realize that they’ve just earned a 15% on something worth 30% of their grade. Papers which do not even come close to meeting the length guidelines, just engage with the rubric in a few, token measures, or show little understanding of course material are really frustrating as an instructor, and are a sign that you simply didn’t try very hard.

So, please, start early, put in effort, and make sure that you’re meeting the standards in the rubric. And if you choose not to do these things, please act with integrity and understanding when you fail the assignment.

3: Self Grading

We’re experimenting with self-grading this quarter. So, in addition to writing and completing your project, you’re going to grade it too, according to the rubric below, giving yourself a score for each rubric item as well as your honest appraisal of what you’ve earned on the project.

This will be treated as the ‘default’ grade for your project, and as we look at your project, the main question we’ll be asking is ‘Has this student accurately assessed their own work?’. If we feel you were fair to yourself and your classmates, your grade will stand. But if we feel that your grade does not reflect your work and efforts, we’ll adjust the grade down (or up!) accordingly.

To see what this looks like, see the self-grading Guidelines with sample cover sheet

Project Proposal

Your project proposal should be around one single spaced page, plus or minus, providing and discussing the following information:

• Your name
• Who you’re working with
• What you’re planning to do in your final project
  • If you’re planning to do Option 1:
    • What’s your system designed to do?
    • What are some sample queries/commands? What answers should the system give?
    • What are a few of the complexities that make this different from what existing assistants do?
    • What do you think will be ‘the hard part(s)’ for your system?
  • If you’re planning to do Option 2:
    • What task do you want to work on?
    • What tool are you interested in using?
    • What kind of experience do you have with similar tools or languages? (e.g. explain why this is possible)
    • What kind of data do you want to use to test your tool?
  • If you’re planning to do Option 3:
    • What do you have planned?
    • What is the final product?
• Why is this topic interesting to you and your group members?

I’ve uploaded a sample project proposal which is basically perfect. It’s describing an ‘option 3’, off the beaten path project, and it’s a bit extra, so you can get full credit doing a bit less work than this, but this is exactly the kind of work I’d love to see in your proposal.

Grading Rubric

A Masterful proposal will…

• Discuss all of the elements above, providing concrete examples and deliverables
• Provide clear discussion, particularly in option 2 projects, of how you plan to implement this
• Lay out the steps required to complete this project in a clear outline
• Show evidence of careful consideration of the problems and difficulties involved.
• Show clear understanding of the challenges, particularly for option 2, and discuss the how the student(s) can meet them

An Acceptable proposal will…

• Discuss all of the elements above, but with fewer examples or direct deliverable items
• Provide discussion of how you plan to implement the project, but with some areas of vagueness or large missing steps
• Focus more on deliverables than the process of getting there, with little evidence of planning
• Feel a bit more rushed, or like a rough gesture in the direction, rather than a clear set of plans
• Show understanding of the challenges involved, but doesn’t show how the student(s) are equipped to meet them

A Novice proposal will…

• Discuss most or all of the elements above, but perhaps omitting some examples.
• Provide only vague information about the planned direction
• Promise an outcome with no discussion of the process
• Feel last-minute, under-planned, or show little effort.
• Be substantially shorter than the one page target length

A Way Off proposal will…

• Fail to discuss many or all of the elements above.
• Describe only a vague plan, with little differentiating information
• Give no details about the planned process
• Feel like little effort was spent in creating the proposal
• Be substantially shorter than the one page target length

Rough Draft

You are encouraged (but not required) to submit a rough draft to me at any point prior to the start of finals week. This should be a largely structurally complete paper, although leaving some schematic areas, pending a question for me (e.g. “Should I go into [subtopic] here?”) is very much acceptable. This is a great opportunity to make sure you’re on the right track, and to check in to make sure you’re executing your topic well. Rough drafts will get general and specific feedback, but of course, there may be issues which I don’t ‘catch’ during that initial read or new issues slipping in, so you’re not guaranteed to be 100% problem free even if you address all my feedback. Most importantly, this is a great way to avoid being ‘surprised’ by a poor final grade on the paper, if you’ve missed a major component of the rubric or of the complexity of your system.

Given that you have the opportunity to submit a draft, as well as to discuss your paper with me at any point, your final paper grades are final and non-negotiable. Even if you were ‘way off’ and misunderstood the prompt, rubric, or goals of the assignment, you had ample opportunity to get feedback or submit a draft, so you will not be offered opportunities for a ‘do over’.

Required Cover Page and Self-Assessment

Project Option 1: Design a System

Cover Page and Self-Grading

All projects are required to include a cover page and self-grading. To see what this looks like, see the self-grading Guidelines with sample cover sheet

Projects which do not include a cover page with self assessment will have 30% deducted

• Masterful: Completes and includes a cover page including at least:
  • Project Title
  • Name of the Project Host (both the person and the organization)
  • Name of the Faculty Advisor (if different)
  • Name of all participants
  • List of any supplementary files or urls (e.g. source code links, website link, etc)
  • Whether you want detailed feedback, or just a final grade
  • Your self-assessment, including a rubric leve
  • Your suggested project grade
• Way Off: Any element of the cover page is missing.

Scope of Writeup

• Masterful: Discusses, indirectly or directly, the ‘Things to consider’ for all elements in the Virtual Assistant Interaction process guide in sufficient depth to demonstrate understanding of the difficulties (or simplicities) of their particular system, or, when a given step doesn’t apply, explains why. If additional steps are required, they are described in detail, and the ethical concerns in the process guide are addressed as well.
• Acceptable: Discusses all of the elements in the process, but in one or two areas shows lacks nuance or fails to capture or discuss important complexities faced by their particular system.
• Novice: Fails to thoroughly discuss some of the elements of the interaction process (without justification of why that step doesn’t apply), or shows repeated failures of understanding of the process and complexities.
• Way Off: Writeup focuses exclusively on one or two elements of the process without prior approval, does not address the NLP complexities of multiple steps, or does not demonstrate understanding of the many intermediate steps in the NLP chain.

Demonstration of Knowledge

• Masterful: Student paper shows considerable and nuanced knowledge and attention to the material from the course, making regular reference to concepts from class and homeworks, fully engaging with the material taught in class, and potentially, goes beyond it into independent research. Additionally, the student shows some realistic understanding of the ease or difficulty of designing the system as described.
• Acceptable: The paper shows attention and engagement with the class materials, but occasionally neglects some important facts or discussions from the course, or occasionally shows confusion with the nature, functioning or limitations of some of the NLP processes discussed.
• Novice: The paper makes occasional reference to the concepts discussed in class, but repeatedly shows failures of understanding of the concepts discussed or their applicability to the system in design.
• Way Off: Writeup largely fails to demonstrate attention to or knowledge of the material from the course, instead focusing on non-NLP related elements or giving explanations so superficial as to be available even to those who haven’t taken LIGN 6. Note that papers which make little reference to concepts learned in class will be penalized disproportionately, as you had one job!

Richness

• Masterful: Project describes a system with considerable interactional depth, discusses 5-10 different commands, involving back-and-forth interaction with the human, creation of a discourse context, and complexity of commands at or beyond current commercial offerings (e.g. Siri or Alexa).
• Acceptable: Project describes a system with some richness, but which primarily processes single complex commands and gives output, without considerable interaction or discourse. Or, the student focuses exclusively on one or two commands, without prior approval.
• Novice: Student describes a simple call-and-response system with low NLP complexity, where pre-specified commands are responded to with canned phrases or actions. (e.g. “Alexa, what is your name?” “My name is Alexa, thanks for asking.” or “What is the best menu choice for a vegetarian?” “Pasta Primavera”)
• Way Off: The system described makes only limited use of natural language processing, or the system’s principal described complexities are outside of the NLP world (e.g. for a pizza ordering system, the bulk of the paper is spent discussing the robotic creation of the physical pizzas).

Formatting and Length

To do this assignment well, expect to write 4000-5000 words. Note that students with strong command of the material might be able to excel in a bit less, and students who are struggling could easily provide 10,000 words without showing their knowledge. Note that papers submitted by groups will be expected to have a greater scope and detail than individual papers, so a 5000 word paper from five people would be considered ‘suspiciously short’, but it’d be fine from a single person. Additionally, feel free to use APA or MLA formatting as a baseline, particularly for citations, and you should use hierarchical formatting (e.g. labeled sections, subsections, subsubsections), but I will not be grading on deviations from an arbitrary set of formatting laws, because we both have better things to do with our lives.

• Masterful: The paper is sufficiently long to demonstrate knowledge, richness of the system, and show proper scope, but without dragging or feeling like words are being added for the sake of talking. Formatting is reasonable, readable, and enhances the text. Citations, where given, are reasonably formatted according to an accepted standard, and contain the information needed to be followable.
• Acceptable: Paper length is sufficient, although perhaps a bit too brief or too wordy. Formatting is acceptable, if occasionally distracting or adding difficulties.
• Novice: Paper is ‘a bit too short’ to accomplish the needed demonstration of knowledge, or buries lack of knowledge in a mountain of text. This could also apply if the formatting is distracting, problematic, or hurtful to the argumentation.
• Way Off: The paper is so short as to be unable to address the appear ‘last minute’ or ‘low effort’, and not given enough space to demonstrate knowledge. Or is typeset using ‘Comic Sans MS’ or ‘Papyrus’ fonts, is presented without formatting or sectioning, citations are missing crucial information

Structure and Organization

Please structure your paper with sections, subsections, and subsubsections (where needed), to make things easier both in reference and readability. Don’t worry about ‘transitions’, just as long as the organization is OK.

• Masterful: The paper is well organized, with the introduction discussing the structure of the paper to guide the reader. Sections are numbered and cross-referenced throughout the paper where needed. Transitions between sections are clear, and the sections make sense. (Hint: The Interaction Process guide is a pretty reasonable outline!)
• Acceptable: Structure of the paper is acceptable, if somewhat difficult to understand or poorly explained. The reader has a rough understanding what’s going on, even if it’s never explained.
• Novice: The paper’s structure is counterintuitive and not understandable, with the structure (or lack thereof) hurting the argumentation considerably.
• Way Off: The paper has no discernable structure nor organization.

Language and Argumentation

• Masterful: Student language use is consistently clear and understandable, with a reasonable level of formality for academic writing. Additionally, the paper was clearly proofread, with typoes and English language grammar issues relatively rare and not considerably affecting the argumentation.
• Acceptable: Student language use is largely understandable, but in places unclear, or with a sufficient number of typoes or English language grammar issues as to start to hinder my understanding of the argumentation.
• Novice: The argumentation is regularly unclear, and my ability to understand the content of the paper is considerably hindered by typoes, difficult to understand argumentation, or English language grammar issues.
• Way Off: The paper is not written in English, is largely incomprehensible, or shows evidence of automatic machine translation.

Academic Integrity and Citation

Note that plagiarism or other academic integrity issues will result in an automatic ‘0’ on the paper.

• Masterful: All direct quotes or external references are cited, with the source given clearly in a ‘Works Cited’ section, in the format of your choice (APA is a fine default) including the necessary information to track down the resource. All other words are your own.
• Acceptable: All direct quotes or external references are cited, but the citations do not include enough information for the reader to locate the original resource.
• Way Off: Sources are not cited, or there is evidence of academic dishonesty. Be careful to cite all your sources and quotes (outside of the class material).

Project Option 2: Implement some NLP

Cover Page and Self-Grading

The rubric here is identical to that for Option 1.

Scope of Writeup

• Masterful: Discusses, indirectly or directly, all ten elements of a successful project option 2 writeup described in the course syllabus guide in sufficient depth to demonstrate their engagement with that particular system, or, when a given step doesn’t apply, explain why. If additional steps are required, they are described in detail.
• Acceptable: Discusses all of the elements, but in some areas shows lack of depth, missing elements, failure to capture or discuss important complexities faced by their particular system.
• Novice: Fails to discuss some of the elements of the interaction process (without justification of why that step doesn’t apply), or shows repeated failures of understanding of the process and complexities.
• Way Off: Writeup focuses exclusively on one or two elements of the process without prior approval, or does not demonstrate understanding of the many intermediate steps in the NLP chain.

Demonstration of knowledge

• Masterful: The student has successfully installed the tool locally, and has demonstrated the use of the tool on real natural language data.
• Acceptable: The student shows that they’ve managed to make the tool work, and demonstrates learning in that process.
• Novice: The student has not managed to successfully implement the tool, but presents deep discussion as to the difficulties of this above and beyond ‘computers are hard’, engaging with the NLP aspects of the process.
• Way Off: The student fails to install the tool without discussion, or demonstrates the ability to install or implement the tool, but does not engage with the NLP elements of it, going little further than ‘Hello World’. Or, student has used an online implementation of this tool, e.g. a web interface.

Richness

• Masterful: Student paper shows considerable and nuanced engagement with natural language processing and the tool(s) of their choice, showing that they’ve not just ‘managed to install it’, but have begun to demonstrate skill and facility with it, and have applied it in ways that are revealing and demonstrative for NLP. Student also provides nuanced discussions of the failings and stengths of the system, making concrete reference to their work.
• Acceptable: The student has implemented the tool, but without discussion of the implications for NLP and for the implementation of this tool in production settings. Discussions of the failings and stengths of the system are general, or fail to engage in consistent depth.
• Novice: Results from the tool’s run are presented with major problems, or without discussion and analysis.
• Way Off: The tool is not implemented, or results are only on a test file.

Formatting and Length

The rubric here is the same as for Option 1, but the expected length will be shorter (5-7 single spaced pages) because more of the work is happening ‘outside the paper’.

Structure and Organization

Option 2 projects, in particular, lend themselves well to a section-by-section write-up, with large chunks of code/output interspersed. The rubric for this portion is the same as for option 1.

Language and Argumentation

The rubric here is identical to that for Option 1.

Academic Integrity and Citation

The rubric here is identical to that for Option 1.

Option 3: Build-Your-Own

With a build-your-own project, it’s difficult to describe the exact expectations, as they vary from paper to paper. However, you can always email me to get clarifications if you’re unsure what I’m looking for. These do tend to be a bit longer, closer to 10-15 pages single spaced, particularly if they’re more research-based papers (e.g. “The history of TTS”).

However, the even for ‘Build your own’ papers, you’ll still grade based on the ‘Formatting and Length’, ‘Structure and Organization’, ‘Language and Argumentation’, and ‘Academic Integrity and Citation’ rubrics for option 1.