About the Star Quality Rating

The star quality rating indicates the quality or confidence in the results of the study producing the CMF. While the reviewers applied an objective as possible set of criteria, the star quality rating still results from an exercise in judgment and a degree of subjectivity. The star rating is based on a scale (1 to 5), where a 5 indicates the highest or most reliable rating. The review process to determine the star rating judges the accuracy and precision as well as the general applicability of the study results. Reviewers considered various factors for each study (depending on the study type used to develop the CMFs) — study design, sample size, statistical methodology, statistical significance, etc. — and judged each CMF according to its performance in the various factors (including multiple subcategories within each factor). The chart below provides a summary of the factors used for the various study types. The total score for a CMF is calculated by adding the points for the different factors, with a maximum possible score of 150.

BEFORE-AFTER AND CROSS-SECTIONAL STUDIES

FACTORS SUBCATEGORIES TOTAL POINTS
Data Number of miles/sites and number of crashes for reference/comparison and treatment groups 55
Study Design andStatistical Methodology Potential biases, accounting for trends, and appropriateness of statistical procedures 75
Statistical Significance Statistical significance of CMF 20

META-ANALYSIS STUDIES

FACTORS SUBCATEGORIES TOTAL POINTS
Methodology and Data Consistent methodology and crash type and severity definitions across included studies 55
Individual CMF Quality Quality of studies used and standard errors of the individual CMFs. 35
AppropriatenessofCombining Use of appropriate methods to estimate the combined CMF 40
Statistical Significance Overall statistical significance of the combined CMF 20

META-REGRESSION STUDIES

FACTORS SUBCATEGORIES TOTAL POINTS
Methodology and Data Consistent methodology and crash type and severity definitions across included studies 55
Individual CMF Quality Quality of studies used and standard errors of the individual CMFs. 35
Appropriateness of Statistical Analysis Use of appropriate statistical methods to estimate crash modification function including use of appropriate model and functional forms 60

The star rating is assigned based on the score and the ranges in the table below. It should be noted that information may be missing from a study report for specific characteristics such as sample size. In these cases, the rating is based on available information and the CMF will likely receive a lower rating due to the lack of information.

Score Star Rating
135-150 5 Star
110-134 4 Star
75-109 3 Star
35-74 2 Star
0-34 1 Star

DETAILED THRESHOLDS FOR CMF CLEARINGHOUSE STAR QUALITY RATING PROCESS

The following tables provide a description of the thresholds used by CMF Clearinghouse reviewers in determining the rating for a CMF. These thresholds are consistent with the rating system developed by NCHRP Project 17-72 “Update of Crash Modification Factors for the Highway Safety Manual”. These are general "rules of thumb" followed by the reviewers to ensure consistency. However, there may be certain aspects of a study that cause it to be rated higher or lower than these general rules would indicate (for e.g., a study may report sample size for total crashes but not for the disaggregate crashes used to develop a CMF, in such a case, a reviewer can make a judgement on the sample size and allocate partial or full points). This determination is made by the reviewer based on his or her experience in statistical analysis, research experiment design, and highway safety knowledge.

RATING CRITERIA FOR BEFORE-AFTER STUDIES

Factors Description/Subcategories Levels[1] Points Total Points
Data Number of miles/sites of reference/comparison group for SPF estimation and trend analysis (for analysis based on "segments", number of miles should be used; for analysis based on intersections or similar units, sites should be used) 50 or more miles/sites 10 55
25 to 49 miles/sites 5
Less than 25 miles/sites, or cannot be determined 0
Number of crashes in reference/comparison sites for SPF estimation and trend analysis 500 or more crashes and 100 or more crashes per year on average 10
500 or more crashes and less than 100 crashes per year on average 7
250 - 499 crashes 5
Less than 250 crashes, or cannot be determined 0
Reference/comparison group is appropriate to account for any spillover/crash migration Yes 5
No, or cannot be determined 0
Number of miles/sites for treatment group (for analysis based on "segments", number of miles should be used; for analysis based on intersections or similar units, sites should be used) 50 or more miles/sites 10
25 to 49 miles/sites 5
Less than 25 miles/sites, or cannot be determined 0
Number of crashes in the before period + crashes expected in the after period without the treatment, for treatment group 200 or more 10
100 to 200 5
Less than 100, or cannot be determined 0
At least one traffic volume count in the before period and one traffic volume count in the after period Yes 10
No, or cannot be determined 0
Confounding and Appropriateness of Statistical Analysis Possible bias due to RTM is addressed or the treatment was a systemwide implementation Yes 25 75
No, or cannot be determined 0
Accounts for changes in traffic volume during the study period Yes 10
No, or cannot be determined 0
Accounts for time trends and other changes during the study period Yes 10
No, or cannot be determined 0
Reference/comparison group is similar to treatment group in terms of AADT, i.e., the AADT range for the reference group overlaps the AADT range for the treatment group, and the mean AADT for the reference and treatment groups are similar to each other. Yes 10
No, or cannot be determined 0
The reference/comparison group and treatment groups belong to the same roadway type (e.g., rural two lane roads) and site type (e.g., horizontal curve).In addition, the reference/comparison groups are similar to treatment group in terms of other important site characteristics. Yes 10
No, or cannot be determined 0
The SPFs were estimated using appropriate statistical procedures and functional form is reasonable Yes 10
No, or cannot be determined 0
Statistical Significance CMF is statistically significant at 0.05, 0.10, or 0.15 levels. Significant at 0.05 level 20 20
Significant at 0.10 level 10
Significant at the 0.15 level 5
Not significant at 0.15 level, or cannot be determined 0
[1] Some journal/articles do not provide the necessary information for a reviewer to rate the CMF. In such cases, the original report (if available) may need to be reviewed.

RATING CRITERIA FOR CROSS-SECTIONAL STUDIES

Factors Description/Subcategories Levels Points Total Points
Data Number of miles/sites. Depending on the treatment being evaluated, this may include sites with and without the treatment.In other cases, the range of each independent variable of interest should be adequate (for analysis based on "segments", number of miles should be used; for analysis based on intersections or similar units, sites should be used) 100 or more miles/sites 20 55
50 to 99 miles/sites 10
Less than 50 miles/sites, or cannot be determined 0
Number of crashes for all sites combined 700 or more 25
500 to 699 15
250 to 499 10
Less than 250, or cannot be determined 0
At least two years with actual/estimated traffic volume counts in the study period Yes 10
No, or cannot be determined 0
Confounding and Appropriateness of Statistical Analysis Selection bias (similarity of sites with and without the treatment) Very similar (based on appropriate matching procedures) or other simultaneous equation procedures 25 75
Treated and untreated sites are similar in some of the characteristics (e.g., AADT or other critical site characteristics) 10
Treated and untreated sites are not similar, or cannot be determined 0
Appropriate model form (including error terms) Yes 10
No, or cannot be determined 0
Appropriate functional form (including possibility of non-traditional non-GLM[2]forms) Yes 10
No, or cannot be determined 0
Appropriate consideration of omitted variable bias (i.e., variables known to influence safety were considered as terms in the model or controlled through study design). Yes 10
No, or cannot be determined 0
Appropriate consideration of correlation between independent variables (with significant correlation, the coefficient of variables may have the wrong sign) Yes 10
No, or cannot be determined 0
Appropriate consideration of spatial and temporal correlation Yes 10
No, or cannot be determined 0
Statistical Significance CMF is statistically significant at 0.05, 0.10, or 0.15 levels. Significant at 0.05 level 20 20
Significant at 0.10 level 10
Significant at the 0.15 level 5
Not significant at 0.15 level, or cannot be determined 0
[2] GLM refers to generalized linear models

Rating Criteria for Meta-Analysis Studies

Factors Description/Subcategories Levels Points Total Points
Methodology and Data Included studies applied the same methodology and accounted for the same confounding factors, including RTM, traffic volume changes, time trends, and crash migration/spillover effects (if applicable) Study methodologies and confounding factors accounted for identical; All confounding factors controlled for 20 55
Majority of studies applied the same methodology and accounted for the same confounding factors; All or most of confounding factors controlled for 10
Substantial variation in the applied methodology and/or confounding factors accounted for 0
Crash type and severity definitions consistent between outcome measures of studies Yes 20
No, or cannot be determined 0
Individual estimates exhibit consistency in the direction of effect Yes 10
No, or cannot be determined 0
Publication bias was tested for and addressed if present Yes 5
No, or cannot be determined 0
Individual CMF Quality A majority of studies used are deemed acceptable by the CMF Clearinghouse rating scheme Yes 25 35
No, or cannot be determined 0
The standard error of at least one of the CMFs is less than or equal to 0.10 Yes 10
No, or cannot be determined 0
Appropriateness of Combining A test of homogeneity indicates that the CMF estimates can be combined Yes 20 40
No, or cannot be determined 0
Appropriate method used to estimate the combined CMF The recommended method from 17-63 guidance followed 20
Estimates were weighted by the inverse of the standard errors or other appropriate measure but CMF values and not the natural log of the CMFs were used 10
Another method was used to estimate the overall effect 0
Statistical Significance CMF is statistically significant at 0.05, 0.10, or 0.15 levels. Significant at 0.05 level 20 20
Significant at 0.10 level 10
Significant at the 0.15 level 5
Not significant at 0.15 level, or cannot be determined 0

Rating Criteria for Meta-Regression Studies

Factors Description/Subcategories Levels Points Total Points
Methodology and Data Included studies applied the same methodology and accounted for the same confounding factors, including RTM, traffic volume changes, time trends, and crash migration/spillover effects (if applicable) Study methodologies and confounding factors accounted for identical; All confounding factors controlled for 20 55
Majority of studies applied the same methodology and accounted for the same confounding factors; All or most of confounding factors controlled for 10
Substantial variation in the applied methodology and/or confounding factors accounted for 0
Crash type and severity definitions consistent between outcome measures of studies Yes 20
No, or cannot be determined 0
Treatment was applied similarly between locations or accounted for in the model Yes 10
No, or cannot be determined 0
Publication bias was tested for and addressed if present Yes 5
No, or cannot be determined 0
Individual CMF Quality A majority of studies used are deemed acceptable by the CMF Clearinghouse rating scheme Yes 25 35
No, or cannot be determined 0
The standard error of at least one of the CMFs is less than or equal to 0.10 Yes 10
No, or cannot be determined 0
Appropriateness of developing a CMFunction A test of homogeneity indicates that the CMF estimates cannot be combined and exhibit more variation than expected due to random variation Yes 10 10
No, or cannot be determined 0
Appropriateness of Combining Appropriate model form including error terms applied Yes 10 50
No, or cannot be determined 0
Appropriate functional form applied Yes 10
No, or cannot be determined 0
Consideration of omitted variable bias, considered through design or included in model Yes 10
No, or cannot be determined 0
Consideration of correlation between independent variables Yes 10
No, or cannot be determined 0
Considered the possible impacts of country of study origin and year Yes 10
No, or cannot be determined 0