completed heldout ordinary completed heldout drift completed heldout delayed completed heldout rare completed heldout miscalibrated completed heldout biased_audit completed heldout personal completed heldout personal_drift biased_audit silent recall 0.00 prec None false/1000 0.00 util/1000 0.00 late 0.00 rare 0.00 fixed recall 14.17 prec 65.35 false/1000 27.24 util/1000 60.05 late 59.52 rare 12.92 naive_silence_negative recall 2.16 prec 78.36 false/1000 2.16 util/1000 20.01 late 0.00 rare 1.82 bandit_explore recall 14.30 prec 64.85 false/1000 28.10 util/1000 54.81 late 50.25 rare 14.38 audit_only recall 14.83 prec 69.49 false/1000 23.61 util/1000 78.01 late 75.53 rare 12.01 prior_paced recall 16.36 prec 80.17 false/1000 14.67 util/1000 130.92 late 126.32 rare 11.10 feedback_paced recall 14.32 prec 75.71 false/1000 16.65 util/1000 131.67 late 146.51 rare 13.47 audit_paced recall 13.28 prec 74.11 false/1000 16.82 util/1000 117.37 late 129.64 rare 11.28 audit_stratified recall 14.66 prec 69.73 false/1000 23.07 util/1000 84.69 late 89.63 rare 17.93 audit_plus_probe recall 14.68 prec 68.73 false/1000 24.22 util/1000 75.71 late 77.62 rare 12.19 oracle recall 22.06 prec 100.0 false/1000 0.00 util/1000 166.46 late 166.95 rare 24.39 delayed silent recall 0.00 prec None false/1000 0.00 util/1000 0.00 late 0.00 rare 0.00 fixed recall 14.27 prec 66.18 false/1000 26.59 util/1000 67.58 late 67.91 rare 14.65 naive_silence_negative recall 1.57 prec 81.48 false/1000 1.30 util/1000 13.81 late 0.00 rare 1.36 bandit_explore recall 14.24 prec 64.97 false/1000 27.98 util/1000 53.73 late 46.34 rare 12.65 audit_only recall 15.04 prec 72.07 false/1000 21.25 util/1000 94.82 late 95.94 rare 15.65 prior_paced recall 16.38 prec 80.79 false/1000 14.19 util/1000 137.75 late 140.84 rare 13.47 feedback_paced recall 15.28 prec 77.32 false/1000 16.34 util/1000 138.07 late 146.16 rare 13.19 audit_paced recall 13.16 prec 75.73 false/1000 15.38 util/1000 112.64 late 125.26 rare 16.65 audit_stratified recall 14.88 prec 71.21 false/1000 21.93 util/1000 89.59 late 88.32 rare 17.38 audit_plus_probe recall 14.90 prec 70.75 false/1000 22.45 util/1000 88.41 late 89.55 rare 15.10 oracle recall 21.95 prec 100.0 false/1000 0.00 util/1000 162.22 late 154.84 rare 20.93 drift silent recall 0.00 prec None false/1000 0.00 util/1000 0.00 late 0.00 rare 0.00 fixed recall 14.57 prec 59.86 false/1000 31.61 util/1000 48.83 late 30.78 rare 13.42 naive_silence_negative recall 2.46 prec 81.55 false/1000 1.80 util/1000 20.26 late 0.00 rare 1.18 bandit_explore recall 15.37 prec 62.3 false/1000 30.09 util/1000 43.70 late 42.89 rare 12.15 audit_only recall 15.31 prec 69.12 false/1000 22.12 util/1000 75.32 late 86.48 rare 11.06 prior_paced recall 17.73 prec 77.39 false/1000 16.76 util/1000 111.54 late 90.22 rare 11.15 feedback_paced recall 15.09 prec 74.07 false/1000 17.08 util/1000 109.10 late 111.67 rare 12.96 audit_paced recall 13.23 prec 71.87 false/1000 16.74 util/1000 86.82 late 95.44 rare 10.52 audit_stratified recall 15.98 prec 69.27 false/1000 22.93 util/1000 81.66 late 93.43 rare 17.86 audit_plus_probe recall 15.22 prec 67.17 false/1000 24.06 util/1000 70.09 late 83.70 rare 12.33 oracle recall 24.73 prec 100.0 false/1000 0.00 util/1000 160.35 late 157.40 rare 25.84 miscalibrated silent recall 0.00 prec None false/1000 0.00 util/1000 0.00 late 0.00 rare 0.00 fixed recall 12.48 prec 56.69 false/1000 34.21 util/1000 18.69 late -19.13 rare 12.28 naive_silence_negative recall 2.42 prec 80.24 false/1000 2.14 util/1000 22.67 late 2.89 rare 5.35 bandit_explore recall 12.79 prec 58.76 false/1000 32.20 util/1000 24.47 late 7.15 rare 15.96 audit_only recall 12.99 prec 63.84 false/1000 26.41 util/1000 49.50 late 13.61 rare 17.46 prior_paced recall 14.90 prec 71.6 false/1000 21.20 util/1000 81.93 late 36.56 rare 12.11 feedback_paced recall 11.91 prec 67.94 false/1000 20.17 util/1000 74.80 late 37.42 rare 15.79 audit_paced recall 10.91 prec 65.52 false/1000 20.60 util/1000 59.70 late 23.87 rare 18.95 audit_stratified recall 13.17 prec 63.83 false/1000 26.76 util/1000 51.86 late 20.34 rare 23.60 audit_plus_probe recall 13.00 prec 62.51 false/1000 27.97 util/1000 43.75 late 10.46 rare 15.96 oracle recall 22.30 prec 100.0 false/1000 0.00 util/1000 162.48 late 160.66 rare 21.75 ordinary silent recall 0.00 prec None false/1000 0.00 util/1000 0.00 late 0.00 rare 0.00 fixed recall 14.13 prec 65.38 false/1000 27.30 util/1000 62.17 late 65.13 rare 12.11 naive_silence_negative recall 2.13 prec 82.66 false/1000 1.63 util/1000 22.04 late 1.90 rare 5.08 bandit_explore recall 14.13 prec 64.83 false/1000 27.99 util/1000 50.34 late 56.71 rare 10.86 audit_only recall 14.37 prec 70.85 false/1000 21.59 util/1000 83.25 late 94.14 rare 14.16 prior_paced recall 16.35 prec 81.24 false/1000 13.78 util/1000 132.29 late 141.51 rare 10.60 feedback_paced recall 14.12 prec 76.19 false/1000 16.11 util/1000 126.38 late 144.21 rare 8.99 audit_paced recall 12.91 prec 74.89 false/1000 15.81 util/1000 109.07 late 137.96 rare 14.78 audit_stratified recall 14.73 prec 70.58 false/1000 22.42 util/1000 86.96 late 95.39 rare 17.01 audit_plus_probe recall 14.32 prec 69.05 false/1000 23.44 util/1000 77.84 late 88.82 rare 14.34 oracle recall 21.91 prec 100.0 false/1000 0.00 util/1000 161.41 late 161.16 rare 20.39 personal silent recall 0.00 prec None false/1000 0.00 util/1000 0.00 late 0.00 rare 0.00 fixed recall 14.30 prec 47.71 false/1000 41.17 util/1000 30.41 late 27.66 rare 13.69 naive_silence_negative recall 5.99 prec 66.39 false/1000 7.97 util/1000 85.33 late 81.15 rare 6.06 bandit_explore recall 13.07 prec 47.62 false/1000 37.79 util/1000 31.97 late 39.14 rare 10.85 audit_only recall 13.74 prec 59.07 false/1000 25.01 util/1000 90.09 late 93.43 rare 9.78 prior_paced recall 14.87 prec 53.16 false/1000 34.43 util/1000 89.11 late 103.41 rare 12.12 feedback_paced recall 14.56 prec 64.52 false/1000 21.04 util/1000 173.30 late 210.10 rare 12.51 audit_paced recall 12.40 prec 65.04 false/1000 17.51 util/1000 145.58 late 184.44 rare 9.58 audit_stratified recall 14.06 prec 57.58 false/1000 27.23 util/1000 93.86 late 97.04 rare 17.40 audit_plus_probe recall 13.77 prec 57.28 false/1000 26.99 util/1000 84.19 late 87.31 rare 11.63 oracle recall 28.96 prec 100.0 false/1000 0.00 util/1000 248.46 late 254.97 rare 38.22 personal_drift silent recall 0.00 prec None false/1000 0.00 util/1000 0.00 late 0.00 rare 0.00 fixed recall 14.17 prec 52.44 false/1000 37.43 util/1000 56.88 late 69.88 rare 14.72 naive_silence_negative recall 5.71 prec 62.7 false/1000 9.88 util/1000 82.84 late 50.76 rare 4.45 bandit_explore recall 12.71 prec 48.4 false/1000 39.47 util/1000 34.76 late 40.82 rare 10.87 audit_only recall 13.97 prec 57.74 false/1000 29.78 util/1000 100.49 late 97.78 rare 10.87 prior_paced recall 14.69 prec 57.49 false/1000 31.63 util/1000 127.01 late 156.26 rare 15.12 feedback_paced recall 13.23 prec 62.1 false/1000 23.50 util/1000 147.43 late 148.42 rare 12.94 audit_paced recall 12.76 prec 64.09 false/1000 20.82 util/1000 146.45 late 169.37 rare 11.46 audit_stratified recall 13.81 prec 56.69 false/1000 30.73 util/1000 91.90 late 98.00 rare 16.50 audit_plus_probe recall 13.75 prec 55.89 false/1000 31.61 util/1000 90.58 late 85.78 rare 10.97 oracle recall 26.86 prec 100.0 false/1000 0.00 util/1000 284.35 late 249.34 rare 32.11 rare silent recall 0.00 prec None false/1000 0.00 util/1000 0.00 late 0.00 rare 0.00 fixed recall 14.41 prec 66.07 false/1000 26.81 util/1000 61.12 late 63.22 rare 15.21 naive_silence_negative recall 2.43 prec 80.26 false/1000 2.16 util/1000 23.43 late 0.00 rare 2.30 bandit_explore recall 14.31 prec 64.89 false/1000 28.06 util/1000 51.94 late 53.16 rare 10.37 audit_only recall 15.29 prec 71.31 false/1000 22.30 util/1000 87.95 late 94.78 rare 10.83 prior_paced recall 16.68 prec 80.9 false/1000 14.27 util/1000 129.92 late 131.64 rare 11.98 feedback_paced recall 14.49 prec 76.13 false/1000 16.47 util/1000 130.10 late 132.77 rare 5.07 audit_paced recall 13.27 prec 75.17 false/1000 15.89 util/1000 109.75 late 118.84 rare 7.37 audit_stratified recall 14.78 prec 70.16 false/1000 22.77 util/1000 80.04 late 79.03 rare 13.59 audit_plus_probe recall 15.13 prec 70.09 false/1000 23.40 util/1000 81.68 late 85.23 rare 10.83 oracle recall 22.07 prec 100.0 false/1000 0.00 util/1000 162.21 late 165.32 rare 20.51