Capstone Rubric

Overview

This rubric applies to all three capstone projects (Public Health Data Investigation, Business Analytics Report, and Social Justice Data Audit). Each criterion is assessed on a four-level scale: Excellent, Good, Developing, and Needs Improvement. The weighting of each criterion varies slightly by project — see the Assessment section of each project specification for the weights — but the performance level descriptions below are consistent across all three.

Use this rubric before you start (to understand expectations), while you work (to check your progress), and before you submit (to self-assess).

Criterion 1: Question Formulation

Does the project begin with a clear, focused, answerable question that drives the entire analysis?

Excellent (90-100%)

The research/business/investigation question is specific, clearly stated, and well-motivated. The student explains why the question matters in the relevant context (public health, business, or policy). The question is appropriately scoped for the data available — neither too broad to answer nor too narrow to be interesting. The student states an initial hypothesis and explains their reasoning. The connection between the question and the chosen analytical methods is explicit and logical throughout the project.

Good (75-89%)

The question is clearly stated and relevant, though the motivation could be more developed. The question is answerable with the data, and the student demonstrates awareness of scope. An initial hypothesis is stated. The connection between question and methods is present but not always explicit — some analytical choices appear without clear justification of why they serve the research question.

Developing (60-74%)

The question is too broad, too vague, or not well-connected to the data available. The student may have a general topic but hasn't sharpened it into a specific, testable question. The motivation is superficial ("this is an important topic"). Some analytical methods are used without a clear connection to the question, suggesting a "try everything" approach rather than a question-driven approach.

Needs Improvement (Below 60%)

No clear question is stated, or the question is unanswerable with the data provided. The project reads as a collection of disconnected analyses rather than a coherent investigation. No hypothesis is stated. The student appears to have started with the techniques rather than the question.

Criterion 2: Data Handling

Does the student demonstrate competence in loading, cleaning, preparing, and documenting data for analysis?

Excellent (90-100%)

The data is loaded, inspected, and profiled systematically. A complete data dictionary is provided with variable names, types, descriptions, and notes. Missing values are identified, quantified, and handled with clearly justified strategies (with discussion of alternatives considered). Outliers are detected and addressed with reasoned criteria. All cleaning decisions are documented in the notebook with explanations. Derived variables are created where appropriate and clearly defined. The analysis-ready dataset is well-organized, and the student can articulate how cleaning decisions might affect results.

Good (75-89%)

The data is loaded and inspected. A data dictionary is provided but may lack some detail. Missing values and outliers are addressed, though documentation of decisions is inconsistent — some cleaning steps are explained, others are performed without justification. The data is usable for analysis. Derived variables are created where needed. Minor issues (e.g., unexplained filtering, undocumented recoding) are present but don't substantially affect the analysis.

Developing (60-74%)

The data is loaded but not thoroughly inspected. The data dictionary is incomplete or missing. Some missing values or quality issues are addressed, but others are ignored without explanation. Cleaning decisions are made but not documented — the reader cannot tell why certain rows were dropped or how missing values were handled. The data preparation may introduce subtle biases that the student doesn't address.

Needs Improvement (Below 60%)

The data is loaded with minimal or no inspection. No data dictionary is provided. Missing values and outliers are either ignored entirely or handled in ways that clearly bias the results (e.g., dropping all rows with any missing value without considering the impact). Data cleaning steps are not documented. The dataset used for analysis may contain errors, duplicates, or inconsistencies that undermine the validity of the results.

Criterion 3: Statistical Analysis

Does the student apply appropriate statistical methods correctly and demonstrate genuine understanding of what the methods do?

Excellent (90-100%)

The student applies at least three distinct statistical methods (e.g., hypothesis test, confidence interval, regression), each chosen because it serves the research question. For each method: - Hypotheses (where applicable) are correctly stated in words and symbols - Conditions and assumptions are verified with evidence (not just asserted) - The test is executed correctly (correct test statistic, correct degrees of freedom, correct p-value) - Effect sizes are calculated and interpreted - Results are interpreted in context, distinguishing between statistical significance and practical significance

The student demonstrates understanding of why each method works, not just how to run it. When assumptions are violated, the student either uses an appropriate alternative or discusses the robustness of the method.

Good (75-89%)

The student applies at least three methods appropriately. Hypotheses are correctly stated. Conditions are checked, though verification may be perfunctory in places. Calculations are correct. Effect sizes are reported for most (but perhaps not all) analyses. Interpretations are generally correct and in context, though they may occasionally slip into generic language ("we reject the null hypothesis") without enough problem-specific interpretation. The distinction between statistical and practical significance is addressed but not deeply explored.

Developing (60-74%)

The student applies methods but with notable gaps. Some of the following issues are present: - Wrong test chosen for the data type or study design - Conditions checked superficially or not at all - Computational errors in test statistics or p-values - Effect sizes missing or calculated incorrectly - Interpretation limited to "significant" or "not significant" without context - No discussion of practical significance - Over-reliance on a single method when the question calls for multiple approaches

The student appears to understand the mechanics of the methods but not the deeper logic.

Needs Improvement (Below 60%)

Statistical methods are applied incorrectly or inappropriately. Multiple issues are present: - Hypotheses are missing or incorrectly stated - The wrong test is used (e.g., a t-test on categorical data) - Conditions and assumptions are completely ignored - Calculations contain errors that lead to wrong conclusions - No effect sizes - Results are misinterpreted (e.g., "the p-value is the probability the null hypothesis is true") - The analysis does not address the stated research question

Criterion 4: Visualization

Are the visualizations well-designed, correctly implemented, and effective at communicating patterns?

Excellent (90-100%)

Visualizations are thoughtfully chosen to match the data type and the message. Each graph: - Has a clear, descriptive title - Has labeled axes with units - Uses appropriate scales (no truncated axes unless justified and noted) - Includes legends where needed - Uses color effectively and accessibly (colorblind-friendly palettes) - Tells a clear story — the main pattern is immediately visible

The student uses a variety of graph types appropriate to the data (histograms, box plots, scatterplots, bar charts, etc.). Visualizations are integrated with the narrative — they appear where they're discussed, not in a separate "figures" section. At least one visualization goes beyond the basics (e.g., annotated plots, small multiples, overlaid distributions, confidence interval plots).

Good (75-89%)

Visualizations are appropriate and correctly implemented. Titles, labels, and legends are present. The graphs communicate their intended message. Some variety in graph types is demonstrated. Minor issues may include: inconsistent formatting, overly busy charts, suboptimal color choices, or graphs that are correct but not as polished or informative as they could be. Visualizations are generally integrated with the narrative.

Developing (60-74%)

Visualizations are present but have notable weaknesses. Some of the following issues: - Missing titles or axis labels - Inappropriate graph type for the data (e.g., a pie chart for 15 categories) - Graphs that are technically correct but hard to read - Default styling with no customization - Graphs that don't clearly connect to the analysis or narrative - Missing or misleading legends - Limited variety (e.g., only histograms and scatterplots)

Needs Improvement (Below 60%)

Visualizations are missing, incorrect, or misleading. Multiple issues: - Few or no visualizations included - Graphs with misleading scales or presentations - Charts that do not match the data (e.g., a line chart for unordered categorical data) - No titles, labels, or legends - Visualizations appear to be unmodified default output with no thought to communication

Criterion 5: Interpretation

Does the student interpret statistical results correctly, in context, and with appropriate nuance?

Excellent (90-100%)

Every statistical result is interpreted in the context of the original question — not just "we reject the null" but what this means for the health/business/policy question. The student: - Correctly interprets confidence intervals (including what they do NOT mean) - Correctly interprets p-values (not as the probability that the null is true) - Distinguishes between statistical significance and practical significance with specific discussion - Discusses limitations of each analysis (confounders, assumptions, sample size) - Draws appropriately cautious conclusions — avoids causal language with observational data - Synthesizes multiple results into a coherent overall interpretation - Connects findings back to the initial hypothesis and discusses whether it was supported

Good (75-89%)

Interpretations are generally correct and in context. The student avoids the most common misinterpretations (e.g., doesn't call p-values the probability of the null being true). Limitations are discussed, though perhaps not for every analysis. Conclusions are reasonable and connected to the research question. Minor issues may include: occasional generic language, insufficient discussion of confounders, or conclusions that slightly overreach what the data supports.

Developing (60-74%)

Interpretations are partially correct but show gaps in understanding. Some of the following issues: - Generic interpretations that could apply to any analysis ("the p-value is less than 0.05, so we reject the null hypothesis") without context-specific meaning - Confidence interval misinterpretation (e.g., "there is a 95% chance the true mean is in this interval") - Confusing statistical significance with importance - Causal language used with observational data without caveats - Limitations acknowledged but superficially ("our sample might not be representative") - Individual results interpreted but no synthesis across analyses

Needs Improvement (Below 60%)

Interpretations are incorrect or missing. Multiple issues: - p-values, confidence intervals, or effect sizes are misinterpreted - Conclusions do not follow from the analysis - No limitations discussed - Results are stated without any interpretation at all (just numbers, no meaning) - Causal claims made without any acknowledgment of study design limitations - Contradictory statements within the project

Criterion 6: Ethics

Does the student thoughtfully address the ethical dimensions of their analysis?

Excellent (90-100%)

The student demonstrates genuine ethical reasoning, not just a checklist of concerns. The ethical section: - Addresses data provenance, consent, and privacy with specifics (not just "data should be collected ethically") - Identifies who might be harmed by the analysis and how - Discusses representation — who is in the data, who is missing, and why it matters - Considers how findings could be misinterpreted or misused, with concrete examples - Reflects on the student's own assumptions and potential biases - Proposes specific mitigation strategies, not just abstract principles - Is integrated throughout the project, not just an afterthought at the end

For the Social Justice Data Audit specifically: the student addresses the ecological fallacy, the difference between documenting disparity and proving discrimination, the risk of stereotyping, and the responsibilities that come with publishing group comparisons.

Good (75-89%)

The student addresses ethical issues with some depth. Several relevant concerns are identified and discussed. The student goes beyond generic statements to engage with ethics specific to their analysis. However, the ethical discussion may be limited to one section rather than woven throughout the project, and some important concerns may be underdeveloped. The student shows awareness of the "so what" — why these ethical considerations matter for real people.

Developing (60-74%)

Ethical issues are addressed but superficially. The student lists concerns (privacy, bias, consent) without applying them specifically to their analysis. Statements are generic: "data should be collected ethically" or "researchers should be careful about bias." The ethical section reads like an obligation rather than a genuine engagement. Important concerns specific to the dataset (e.g., differential missingness by race, the impact of algorithmic bias on marginalized communities) are not addressed.

Needs Improvement (Below 60%)

Ethical considerations are missing entirely, or are addressed with only a sentence or two. The student shows no awareness of how their analysis might affect real people. No reflection on limitations, potential misuse, or researcher responsibility. The absence of ethical discussion suggests the student views statistics as a purely technical exercise divorced from human consequences.

Criterion 7: Communication

Is the project clearly written, well-organized, and effectively communicated to the intended audience?

Excellent (90-100%)

The project is a pleasure to read. The writing is clear, well-organized, and appropriate for the audience. The technical notebook tells a coherent story from question through analysis to conclusion. The executive summary/policy brief is genuinely written for a non-technical audience — no jargon, no formulas, no p-values in the main text. Statistical results are translated into plain language that a manager, legislator, or health official could understand and act on.

The project structure is logical and easy to navigate. Section headers guide the reader. Transitions between sections are smooth. The narrative explains not just what was done but why — the reader understands the reasoning behind each analytical choice. Figures are placed where they're discussed, not orphaned in an appendix.

Good (75-89%)

The project is clearly written and well-organized. The narrative is coherent, and the reader can follow the analysis from start to finish. The executive summary/policy brief is mostly non-technical, though some jargon may slip through. Writing is competent but may occasionally be dense, overly formal, or lacking in clear transitions. The project structure is logical. Minor issues (typos, inconsistent formatting, a few unclear explanations) are present but don't impede understanding.

Developing (60-74%)

The project is understandable but has organizational or clarity issues. Some of the following problems: - Writing is disorganized — the reader has to work to follow the logic - The executive summary/policy brief reads more like a shortened version of the technical notebook than a document written for a non-technical audience - Statistical jargon is used without definition or translation - Some sections are much more developed than others - The narrative explains what was done but not why - Typos, grammatical errors, or formatting issues are frequent enough to distract

Needs Improvement (Below 60%)

The project is difficult to follow. Multiple issues: - No clear organizational structure - Writing is unclear, disjointed, or missing - No executive summary/policy brief, or it's essentially a copy of notebook output - Technical terms used without explanation - The reader cannot determine what question was investigated, what was found, or what it means - The project appears hastily assembled

Criterion 8: Reproducibility

Could another analyst replicate the analysis from the submitted materials?

Excellent (90-100%)

The Jupyter notebook runs from top to bottom without errors when the kernel is restarted. All data files are included (or clear download instructions are provided with specific URLs and access dates). All library imports are at the top of the notebook. Every data cleaning decision is documented with justification. Random seeds are set for all stochastic procedures. Variable names are descriptive. Code is well-commented — not every line, but every logical block has a comment explaining its purpose. A methods section summarizes the analytical approach. The project could be handed to another analyst and they could reproduce every result without asking any questions.

Good (75-89%)

The notebook runs without errors. Data files are available. Most cleaning decisions are documented. Code is reasonably commented and organized. Some minor reproducibility issues: a missing random seed, an undocumented filtering step, a few variable names that aren't self-explanatory. An analyst could reproduce the results with minor effort.

Developing (60-74%)

The notebook runs with minor modifications needed (e.g., a file path that needs updating, a missing package installation). Some of the following issues: - Data cleaning steps are performed but not all are documented - Code comments are sparse - Variable names are unclear (e.g., df2, temp, final_final) - Random seeds are not set for bootstrap or simulation - Some code cells depend on variables defined elsewhere in a non-linear order

An analyst could probably reproduce the results but would have to do detective work.

Needs Improvement (Below 60%)

The notebook does not run without errors. Multiple issues: - Missing data files with no instructions for obtaining them - Errors that prevent execution (undefined variables, missing imports, broken code) - No documentation of data cleaning - No code comments - Code is disorganized with cells that must be run in a specific non-linear order - Another analyst could not reproduce the results without significant effort and guesswork

Scoring Summary

For each criterion, assign a percentage score within the appropriate level:

Multiply each criterion score by its weight (see the Assessment section of each project specification) and sum for the final project grade.

Example calculation (Public Health project):

Self-Assessment Checklist

Before you submit, score yourself honestly on each criterion. This isn't graded — it's for your benefit.

Question Formulation - [ ] My question is clearly stated in one sentence - [ ] I've explained why the question matters - [ ] My question is answerable with my data - [ ] I've stated an initial hypothesis

Data Handling - [ ] I've inspected the data and reported its basic properties - [ ] I've created a data dictionary - [ ] I've addressed missing values with documented reasoning - [ ] I've handled outliers with documented reasoning - [ ] I've created any needed derived variables

Statistical Analysis - [ ] I've used at least three distinct statistical methods - [ ] Each method is appropriate for the data type and question - [ ] I've verified conditions/assumptions for each method - [ ] I've calculated and interpreted effect sizes - [ ] I've distinguished between statistical and practical significance

Visualization - [ ] Every graph has a title, axis labels, and legend (if needed) - [ ] I've used appropriate graph types for each variable type - [ ] My graphs clearly communicate their intended message - [ ] I've included a variety of visualization types - [ ] My visualizations are integrated with my narrative

Interpretation - [ ] I've interpreted every result in context (not just "significant" or "not significant") - [ ] I've correctly interpreted confidence intervals and p-values - [ ] I've discussed limitations and confounders - [ ] I've been careful with causal language - [ ] I've synthesized results into an overall conclusion

Ethics - [ ] I've addressed data provenance and consent - [ ] I've considered who might be harmed - [ ] I've discussed representation and missing voices - [ ] I've considered potential misuse of findings - [ ] My ethical discussion is specific to my project, not generic

Communication - [ ] My notebook tells a coherent story from start to finish - [ ] My executive summary/policy brief is written for a non-technical audience - [ ] I've translated statistical results into plain language - [ ] My project is well-organized with clear section headers - [ ] I've proofread for clarity, grammar, and formatting

Reproducibility - [ ] My notebook runs from top to bottom without errors (Restart and Run All) - [ ] All data files are included or download instructions are provided - [ ] All imports are at the top - [ ] Code is commented - [ ] Random seeds are set