Request for Qualification (RFQ)
Teacher and Principal Evaluation: Qualifications for Student Assessments to Be Used by New York State School Districts and BOCES in Teacher and Principal Evaluations
QUESTIONS AND ANSWERS (Posted March 1, 2013)
- If an assessment is currently on the Approved Assessment List, is the vendor expected to submit a proposal or any additional information with regard to this RFQ?
Answer: Assessments that are already on the Approved Assessment List were approved based on the validity and reliability of the assessment and alignment to the NYS Learning Standards. If the psychometric properties of an assessment have materially changed since the assessment was approved for placement on the list, then a complete, new application is required. If they have not, then no additional application or technical documentation is required. Further, if NYS has changed the Learning Standards since the time of approval of submission, evidence of alignment to the current NYS Learning Standards (i.e., the P-12 Common Core Learning Standards) is required to be submitted and approved by NYSED in order for the assessment to stay on the Approved Assessment List.
- Is there a section of the document that that provides instruction & guidance to vendors that already have assessments that are approved and on the list, and what they need to do to stay on the list if anything. Is there anything additional we need to provide at this point?
Answer: No, there is no section that provides instruction and guidance to vendors who currently have assessments on the Approved Assessment List. Please see A 11 above for additional information.
- On page 10 under section “New For This Release.” I recall these same items as “new requirements” in the last RFQ that was issued. Are these are in fact new requirements for winter 2013 specifically, or ones that have already been in place?
Answer: Some sections included in the RFQ have been updated in this round. On page 10, the NYSED provided additional information in the note regarding alignment techniques related to CTE assessments.
- Please describe the process by which an approved assessment could be removed from the list.
Answer: Pursuant to Section 2.4 of the posted RFQ, an assessment’s approval for inclusion in NYSED’s Approved Assessment List may be disqualified/terminated for good cause, including but not limited to, a determination by the Commissioner that:
- the assessment is no longer in compliance with one or more of the criteria for approval set forth in the Commissioner’s regulations and/or this RFQ or any subsequent RFQ;
- NYSED determines that the assessment is not identifying meaningful and/or observable differences in performance levels across teachers, schools, and/or classrooms; and/or
- high quality academic research calls into question the correlation between high performance on the assessment and positive student learning outcomes.
Disqualification/termination procedures would only begin after unsuccessful attempts at remediation with the Applicant and would follow the procedures established in the Commissioner’s regulations.
An assessment vendor may contact the NYSED at email@example.com if they request to have their assessment removed from the Approved Assessment List. However, the Department discourages a vendor from removing an assessment from the Approved Assessment List list mid school-year. If an assessment vendor seeks to have their assessment removed mid-school year, the Department recommends that the vendor notify any affected districts or BOCES immediately so that the district/BOCES can select another assessment for its APPR.
- Can you provide a bit more information on your expectation from vendors in terms of providing a crosswalk for the evaluation rubrics? It appears there are several of these rubrics; each with its own scale and expectations. Are vendors expected to provide a crosswalk for each rubric or an explanation of how we would equate our data to these rubrics?
Answer: Pursuant to section 2.3(E), this requirement asks the Applicant to provide recommended procedures to translate scores from the assessment into the scores on the teacher and principal evaluation metric. The range of scores used for teacher and principal evaluation are 0-20 and 0-15. Section 2.3(E)-i further specifies that the crosswalk must map native assessment scores onto both of the scales: 0-15 and 0-20.
- On page 13, section 2.3(E) the RFQ references two scales: “…the Applicant must provide recommended procedures to translate scores from the assessment into the scores on the teacher and principal evaluation metric (i.e., for the other comparable measures of student growth subcomponent for grades/subjects where there is no State growth score: [0-20] and the locally-selected measures subcomponent: [0-15 for grades/subjects where a value-added model exists and 0-20 where no value-added model exists])”. Must the growth measure for each scale be equivalent?
Answer: The NYSED does not require that the measures for the two scales be equivalent.
- On page 34, of the RFQ on Form F: “The Organization agrees that the New York State Education Department will post on its website Form C of this application which includes the following topics: Organization contact information, information about the assessment; an overview of the application; estimated service costs; and, information about coverage of the New York State Common Core Learning Standards (as appropriate).” Form C does not include a field to enter cost information. Where should vendors provide pricing?
Answer: The NYSED removed the requirement to include cost information on the publicly available service summary. A revised Form F has been posted on the RFQ website. Applicants are requested to use the revised Form F when submitting applications to the posted RFQ.
- Our CTE assessments cover many occupational areas and all of the assessments were developed in the same way, just with different SME’s. If we were to submit an application for these assessments, we would be submitting the same material for all of our assessments. Can we provide the assessment titles and competency for each assessment under one submittal or do we have to submit separate applications for each assessment?
Answer: Applicants seeking approval of an assessment for more than one grade within the same subject area may submit one application, provided the application clearly specifies any variance in the degree to which the assessment meets the criteria in Sections 2.2 and 2.3(B-D) of this RFQ for the grades for which the assessment is being proposed.
Applicants seeking approval of an assessment for more than one subject area may submit separate applications or a single application, but must clearly specify any variance in the degree to which the assessment meets the criteria in Sections 2.2 and 2.3(B-D) of this RFQ for the subjects for which the assessment is being proposed. Please note that if one application includes an assessment for more than one subject area and/or grade, the assessment may not be approved for use in all subject(s)/grade(s) for which it was submitted if the technical documentation for each subject/grade does not meet the criteria specified in Sections 2.2 and 2.3(B-D).
- It is noted in the RFQ that each application must include a copy of the administration materials. Because we are submitting multiple applications and the administration materials are the same, do we need to submit the same document each time, for each application, or can we include one copy of the administration materials in hard copy format that is applicable to all applications?
Answer: Applicants may submit a single administration manual if it covers multiple assessments, provided that the Applicant specifies in each application where the administration manual was submitted. Please see Q 7 above for additional information.
- If our organization manages the assessment revision and delivery for a certifying body and they have given us permission to submit their assessment on their behalf, can we complete Form F and submit the assessment? Section 2.1 (B) indicates that an LEA may submit an assessment on behalf of a copyright owner of an assessment.
Answer: In the case specified, your organization would be considered the “Assessment Representative” (see section 1.4 of the RFQ for definitions), and the organization who developed the assessment would be considered the “Copyright Owner.” Page 21 of the RFQ lists the sections required in the application; section VII states that if an Applicant is not the Copyright Owner, the Applicant must complete and submit an Approval to Submit on Behalf of Authorized Legal Representative Page (i.e., Form F).
- Per requirement 2.2(A)-i below, is the evidence of alignment to research-based learning standards only required when an entire subject/grade level is not included (as opposed to single skills)? “2.2(A)-i: Applicants must submit evidence of alignment to New York Learning Standards or, in instances where there are no such standards that apply to a subject/grade level, evidence of alignment to research-based learning standards.”
Answer: Applicants interested in qualifying an assessment for use in the educator evaluation system must provide documentation that shows evidence of alignment to the current NYS Learning Standards or, in instances where there are no such standards that apply to a subject/grade level, evidence of alignment to research-based learning standards. An assessment will be placed on the Approved Assessment List to be used for teachers in particular grades and subject areas. If an Applicant does not provide evidence of alignment to the NYS Learning Standards associated with a particular grade and subject area, the assessment will not be approved for use in that grade or subject area. Please see: http://www.p12.nysed.gov/ciai/cores.html for a listing of the NYS Learning Standards. In addition, for the purposes of the requirement of section 2.2(A)-i of the RFQ, for Career and Technical Education (CTE) industry-related assessments, where no NYS Learning Standards apply, acceptable research-based learning standards can include those created by the vendor through a professional job or task analysis that is conducted in accordance with best practices for industrial and organizational psychology.
- What level of support for districts is required if an assessment is approved?
Answer: LEA’s will have an opportunity to select assessments from the Approved Assessment List and may enter into an agreement with the Assessment Provider for services within the terms and conditions cited in State law and regulations, and the RFQ. Approval of an application for placement on the Approved Assessment List would not require than an LEA must contract for the entire range of services/support/technical assistance offered by the provider. The LEA and the provider would negotiate a mutually acceptable range of services within the approved program.
- If our company application is accepted or rejected, will feedback from the scoring process be released to the individual companies and, will there be an opportunity to discuss the merits of our submission?
Answer: After the application reviews are completed, applicants will receive a letter approving or denying the application. The approval letter will identify the grades and subjects for which the assessment was approved for usage. The denial letter will identify the items within the RFQ that were not adequately addressed in the application. An applicant may subsequently choose to request a de-briefing letter that gives additional detail about the items within the RFQ that were not adequately addressed in the application.
- When establishing a normative group, is the state looking for a comparison by district, region or state?
Answer: The NYSED does not require that an application established a normative group in a particular way. The Applicant should provide sufficient information for the reviewers to understand the process for establishing the normative group.
- How many forms of the test are expected and, will the same test be used as a Pre and Post test scenario?
Answer: The NYSED does not have an expectation for how an assessment should be administered or how an assessment should be used to measure growth. The Applicant should provide sufficient information for the reviewers to understand the process in order for the assessment administration to make valid inferences about student growth. It is possible that assessments included on the Approved Assessment List may be used by LEAs in a pre- and post-test scenario. Applicants should specify whether the same assessment form is administered at two points in time, or if multiple equated forms are available to make inferences about growth. Furthermore, if assessment is able to yield inferences about growth using one or more other assessments as the pre-test, please provide an explanation and references to empirical studies that best support these claims.
- Does the vendor need to provide copies of the assessment or can that be an expectation to districts, if pricing is adjusted accordingly?
Answer: Assessments that are placed on the Approved Assessment List will be used for the purposes of educator evaluation. Given that the assessments will be used for educator evaluation, Applicants must provide documentation related to assessment administration that delineates recommended guidelines for assessment security and shall also delineate threats to validity for the guidelines provided. Reviewers will assign up to five points for this item in the application.
- Can a district scoring protocol suffice to meet the requirement of teachers not scoring their own papers, or is the vendor expected to provide scoring?
Answer: Pursuant to section 2.3(E) of the RFQ, in no cases will an assessment that requires that a student’s own teacher grade the assessment be considered approvable for the purposes of this RFQ. Applicants must provide written guidance on the use of the submitted assessment for teacher and principal evaluation, as required in section 2.3(E) of the RFQ. The RFQ does not require that a vendor provide the scoring of a submitted assessment.
- From the application Form C, Bullet 4, is this section limited to 3 pages total or, 3 pages for bullet 4 only, see below? “Please provide an overview of the application for LEAs. Please include:"
- A description of the assessment
- A description of how the assessment is used
- A description of how scores are reported (include links to sample reports as appropriate)
- A description of how the assessment provider supports implementation of the assessment, including any technical assistance. (3 pages max)
Answer: The three page maximum refers to the entire “overview,” not just the last bullet.
- Can a third party be used in applying statistical evidence for the evaluation?
Answer: Applicants may submit any documentation that the Applicant deems is appropriate to meet the requirements of sections 2.2 or 2.3 of the RFQ in an Appendix to the application. Applicants must indicate within their response to each section of the RFQ what pages in the Appendix the reviewers should refer to for additional information.
- Does a score of less than 48 affect our chances for future consideration?
Answer: Applications that are rejected in this round of the RFQ may be revised and submitted using the RFQ requirements posted in future rounds of the RFQ.
ARCHIVED QUESTIONS AND ANSWERS
Questions and Answers from previous rounds of the assessments RFQ are posted below for reference: