OIE Institutional Research
INSTITUTIONAL RESEARCH
Institutional Research
The primary purpose of institutional research is to promote institutional effectiveness by providing information for institutional planning, policy formation, and decision-making. IR collects, analyzes, and interprets institutional data on educational programs, students, faculty, and administrative and support services so as to provide accurate information to support planning and decision making. The data gathered through a variety of methods helps the institution utilize its resources for efficient organization and management.
Institutional Research carries out studies with the sole purpose of finding solutions to challenges the institution might encounter in the future. At other times, the research may be conducted with the aim of finding a solution to an existing problem or adding a new program to the curriculum.
Key research initiatives are annually tracked to consider trends in student enrollment and retention, degrees granted by program, faculty productivity, student satisfaction and funds provided for institutional programs as faculty development, instructional materials, research, work-study students, and faculty and staff salary.
It compiles data for institutional self-study and accreditation by external agencies. Institutional self-study is an attempt that every institution makes to evaluate the effectiveness of its programs and activities. It reveals strengths and weaknesses in the institution. It is done with the aim of making sure the institution is working to accomplish its educational mission. Accreditation is also of significant importance to all colleges and universities because it is an indication that the academic programs of the institution conform to federal or state established standards. Accreditation acts as a benchmark for institutional effectiveness and gives the institution a higher academic ranking.
Graduate Student Exit Survey
The Office of Institutional Effectiveness gathered data about students’ experiences from enrollment through graduation using the Graduate Student Exit Survey. The survey asks recent graduates their perceptions about their experiences at DMGS. Topics addressed include, but are not limited to why students selected the institution and respective program; areas of academic concern, such as teaching, communication, and advising; interaction with peers; student satisfaction of non-academic and support; time-to-degree completion; and overall satisfaction.
Final analysis will permit the institution greater insight of student overall experiences so we may improve upon student satisfaction and academic achievement. In addition, this survey will reflect upon whether the institution is meeting its mission.
Faculty Course Evaluations
Beginning in the fall 2018 semester, all teaching faculty must complete a comprehensive course evaluation. This evaluation allows faculty to review the effectiveness of their teaching and evidence of students learning. Faculty are required to establish measurable outcomes of student learning and to reflect upon whether they achieved the learning outcomes. Faculty are required to explain how they plan to improve upon learning outcomes for the next time the course is offered (see DMGS Assessment Manual for template, pp. 29-31).
Student Course Evaluations
The DMGS continually offers students the opportunity to complete an end of course and faculty evaluation. The OIE recently instituted a new end of course and faculty evaluation that is more comprehensive in depth and breadth. The evaluation encourages students to express areas of satisfaction and concern, but more importantly, asks students for evidence of learning.
Student Satisfaction of Services and Academic Achievement
The Office of Institutional Effectiveness (OIE) strives to maintain student satisfaction of services received and student academic achievement. One example of how the OIE ensures the attainment of the aforementioned goals is through the yearly Academic Advising survey. Two surveys are distributed, that is, one to faculty and one to the student body.
In spring 2019, all faculty advisors completed a survey asking their perceptions and effectiveness of the academic advising process. Specifically, the OIE sought faculty perceptions as to their knowledge, skills, and abilities as academic advisors. Findings found faculty felt they needed additional training in academic advising.
In conjunction with the Office of Graduate Studies, the OIE secured institutional membership with the National Academic Advising Association (NACADA). As part of the institutional membership, the OIE was able to work with NACADA to retain a day of academic professional development (workshop) with one of NACADA’s experienced member faculty. All faculty advisors were required to attend; however, the workshop was open to all faculty and staff. Multiple non-academic units took part in the workshop.
During the spring/summer 2019 semesters, students also complete a survey about their perceptions of academic advising at DMGS. Findings were similar to the one completed by faculty in that students felt faculty needed additional training in the area of academic advising.
Faculty Portfolios
To ensure faculty continue to attain their roles in the areas of teaching, research, and service to the community, the Office of Graduate Studies and the Office of Institutional Effectiveness (OIE) instituted the DMGS Faculty Portfolio process. All full-time faculty must submit a completed portfolio no later than close of spring 2020 semester. Of key interest in the portfolio process is faculty self-evaluation of their teaching, review and analysis of student course evaluations, and changes implemented to improve the learning process. Faculty are expected to seek professional development as needed. The institution has funding for needed professional development.
Spring/Summer 2019 Academic Advising Survey Analysis
During the spring and summer 2019 semesters, all students were asked to participate in a survey seeking their perceptions about the academic advising process, their experience, and effectiveness by faculty. The results of this survey is designed to offer insight to the effectiveness of the process and areas for improvement as well as success.
Introduction
Currently, the academic advisors consist of three full-time faculty members who are program chairs. Each student is advised by the chair of their program. The response rate for the survey was 55 percent of the enrolled students.
Of the respondents, 55.56 percent were male, 40.74 percent were female, and 3.70 percent responded as other. Of the respondents, 40.74 percent were enrolled in the National Security Program, 25.93 percent in the Intelligence Program, and 33.33 percent in the Managing Disruption and Violence (MDV) Program. Academic advising is the responsibility of the program chairs and is an assigned duty.
Recommendations. It is recommended that all full-time faculty, regardless of their position, should take on the responsibility of being assigned advisees. To date, DMGS has five full-time faculty members, three whom are program chairs.
Advising Sessions
Of participants, 74.07 percent of respondents sought academic advising or were asked to attend an academic advising session. The majority of advising sessions lasted a duration of 30 minutes or less.
Student Satisfaction with Academic Advisors.
Of respondents, 75 percent or higher agreed or strongly agreed that their academic advisor was:
- easy to contact;
- easily accessible;
- responsive in a reasonable amount of time;
- able to offer the student as much time as needed when they met;
- able to take a personal interest in the student;
- encouraging students to express their thoughts; and
- a good listener.
Of respondents, less than 75 percent disagreed or strongly disagreed that their academic advisor:
- gave accurate information about their degree program requirements (74.08 percent);
- helped the student understand why required courses are important for their professional development and future plans (66.67 percent); and
- helped the student make important decisions (e.g. selecting courses; 66.67 percent).
Additional comments for specific ratings. All comments were positive for open-ended responses for students’ view on academic advising.
Recommendations. Faculty should be queried as their comprehension of program requirements, struggles with assisting students with decision-making, and of the ability to explain course requirements effectively. It may be that faculty may understand the requirements of their program; however, they may not understand the importance of different electives or requirements of other programs.
Roles and Responsibilities of the Advisee
Of respondents, 92.59 percent made appointments to see their advisor. A core finding was 66.67 percent, 62.96 percent, and 62.96 percent of respondents kept their appointments, were well prepared for their appointments with their advisor, and completed all assigned tasks for their advisor, respectively.
Additional comments for specific ratings. For open-ended responses for students’ view on academic advising, all comments were positive except for National Security in which the student expressed frustration in that he or she believed that the advisor did not have the time to follow-up with concerns and to meet as needed.
Recommendation. Students are proactive about making appointments with their advisor; however, they may need advice or encouragement about the importance of the academic advising process. Further study is needed to understand why students may have missed appointments, etc. as there may have been sufficient reasoning to justify missed appointments.
Advisor Characteristics
Respondents were asked about their perceptions of their advisor’s best characteristics. Of 27 possible respondents, 21 answered the open-ended question. A significant number mentioned the following characteristics:
- Good listener;
- Professional;
- Knowledgeable; and
- Cooperative
Quality of Advising
Respondents were asked their perceptions of how the quality of academic advising they received might be improved. This was an open-ended question. Of respondents, 16 out of 27 responded to the question. The following are the highlights of what respondents perceived:
- Lack of a clear plan for the advising session;
- Would like to have received more info about the thesis process earlier;
- Additional interest in career advising and counseling by faculty;
- Lack of time by advisor; and
- Advisors need to be good listeners.
Selection of Advisor
Respondents were asked whether they would recommend their current academic advisor to other students. Respondent answers were mixed in nature. Although many did agree across programs that they would recommend their advisors, there were some responses of concern. MDV and National Security advisees, in a few responses, would not recommend their advisor to other students. One student went as far as considering changing their program to avoid their current advisor.
Recommendations. Although full-time faculty did receive academic advising through the National Academic Advising Association, the two faculty who received negative responses did not attend the professional development supplied to them by The Office of Graduate Studies and Office of Institutional Effectiveness. Further training is recommended and should be required.
Additional Comments
Respondents were asked for additional comments about their academic advising experience that may have not been addressed in the survey. The comments offered suggest gaps in the advising process:
- DMGS needs to hire an experienced advisor;
- Academic boundaries are disrespected, in that students are asked to take on more courses than they can handle;
- Blame is placed on students when the wrong decision was made by their academic advisor; and
- Students are directly affected by incorrect advising decisions on the part of the academic advisor.
Recommendations. Faculty need additional professional development in the area of academic advising. Students should be permitted to ask for a new academic advisor if they are being ill-advised or have other concerns about the ability of their academic advisor. An individual should be assigned who students can meet with to address their academic advising concerns.
2018-2019 Faculty Advisor Survey
During the spring 2019 semester, all academic faculty were requested to complete a survey seeking their perceptions about the academic advising process and its effectiveness. The results of this survey will offer insight into the effectiveness of the process and areas for improvement as well as success.
Introduction
Currently, the academic advisors consist of three full-time faculty members. Prior to the distribution of the survey there were four full-time faculty members. One faculty member resigned; therefore, his perceptions were not captured by the survey instrument. One hundred percent of the remaining participants completed the survey. Academic advising is the responsibility of the program chairs and an assigned duty. The average advisee load is 16 students; however, one advisor has 22 advisees.
Recommendations. It is recommended that all full-time faculty, regardless of their position, take on the responsibility of the adviser role. To date, DMGS has six full-time faculty members, half whom are program chairs. If all full-time faculty are required to be academic advisors, each faculty member would be delegated eight advisees, allowing for a more manageable ratio.
Documentation of Advising Sessions
Of participants, two-thirds document their advising sessions to differing degrees. One faculty member ensures the advising loop is closed in that there is follow-up with the student after the advising process. The other faculty member summarizes the advising session; no follow-up.
Of the two faculty members that document the advising process, one does so using pen and paper, and the other faculty member uses a combination of pen and paper and the student management learning system.
Recommendation. Although there is no standardized practice in place for the documentation of advising sessions; the Office of the Graduate Studies should encourage documentation that is consistent across programs and faculty. This is especially important when a faculty member resigns, falls ill, or is terminated.
What tool they may chose for documentation is irrelevant; however, those who use Sonis for documentation should be queried about the usefulness of the program.
Advising Sessions
Overall, faculty do initiate contact with their advisees and check on student progress. Most faculty do not require students to meet with them on a scheduled basis. This last statement will be further analyzed upon student completion of their evaluation of faculty advising. This may have a negative effect linked to students in which faculty do not initiate contact, especially with those students who should be meeting with their advisor for assistance.
When meeting with their respective advisees, faculty most frequently advise students in the following areas: academic guidance, classwork assistance, and career planning. Important to note, faculty came to the conclusion that in addition to more traditional advisement roles, students are now seeking assistance with financial aid issues. This is concern as this is outside their purview and needs to be address with the Office of Financial Aid and Scholarships.
Overall, faculty perceive their advising experience as largely focused on advice about curricular and program requirements. Faculty have played other roles as an advisor, including assisting students with career choices and personal matters. Faculty do actively refer advisees to other units when they seek advice outside their role as an advisor and scope of expertise.
Recommendations. Further examination of student perceptions of the advising process will be conducted upon the completion of student evaluation of the academic advising process.
Each unit that directly services students should define their roles and responsibilities as it pertains to students. This should be accessible to students as well. For example, purpose of career services is to assist students in the following areas…or the purpose of academic advising is to…” This will avoid any confusion of roles and responsibility coupled with adhering to Federal law, that being HIPAA and FERPA.
Faculty Perceptions of Student Attitudes toward the Advising Process
Faculty did find that students either found the advising process pleasant and rewarding or unpleasant and frustrating. Faculty noted that a few students were miss advised in the past causing levels of frustration by students.
Recommendation. Review of the academic advising survey completed by students should offer further insight of student attitudes toward the advisement process.
Faculty Perceptions of Student Utilization of Academic Advising
Faculty answers spread across the spectrum from students under to over utilizing academic advising. These perceptions by faculty may be due to inexperience with academic advising. This is noted later in the study in which faculty desire professional development (PD) in advising to better serve their students.
Recommendation. It is recommended that faculty are offered the opportunity to further explain why they held a particular perception when answering this question
Analysis of the student survey of the academic advising process may offer additional insight.
Faculty Ratings
Faculty were asked to rate the effectiveness of their advising. The majority found the process moderately effective. Faculty noted the greatest reward of academic advising is student satisfaction. Regardless of ability level and experience as an advisor, faculty perceived that they do place the students’ interest first and foremost as is reflected in this survey.
Across this survey, a consistent theme expressed was a lack of training in academic advising and the need to hire a full-time academic advisor.
Recommendations. Faculty clearly desire training in academic advising. It is critical that they receive training as their advice directly impacts the student. It is recommended that professional development be offered to faculty in the area of academic faculty. This will help clarify what matters they may advise on beyond pure academics.
Faculty would like to see the hire or selection of a full-time academic advisor for students. Due to budget constraints, this may not occur. Consideration of redistribution of students from the program chairs to all full-time faculty members as program chairs play the role of thesis and academic advisor. This would give students an increased, diverse population of possible academic advisors increasing their chances of being matched with an advisor that more closely aligns to individual student needs and personalities. As it stands, if a conflict arises between a student and their academic advisor (program chair), at the moment, they have little opportunity or diversity to be placed with another as program chairs are field specific. Review of the student academic advising survey may offer greater insight to the aforementioned matter.
Further, coupled with training is the need for a process or written guidance so faculty can define the boundaries of the academic advising role. Faculty asked for an academic advising handbook. It is recommended that one be issued to the faculty as soon as possible.
In-Class Peer Evaluation
Although the Dean of Graduate Studies is instrumental in the in-class observation of all faculty, the Office of Institutional Effectiveness instituted the first peer in-class evaluation of all teaching faculty during the summer 2019 semester. This process occurs each semester in which program chairs evaluate teaching styles, evidence of student learning, student engagement, course learning outcomes, and course syllabi of their respective program faculty.
Each observed faculty member receives the completed evaluation for review and signature. It is the responsibility of the program chair to ensure areas of weakness are addressed using such methods as professional development or peer mentorship.
