Academics in Digital Overdrive:
An Inquiry into the Effectiveness of the Blackboard Learning Platform as Observed by Students
Ally Brillaud
Mackenzie Gordon
Christie Hart
Layla Li
Katie Lohec
Ellie O’Neill
Hypotheses & Design
The main purpose of this study is to examine the relationship between academic success and use of online learning tools. Participants of the study included college students from various institutions with experience using online learning tools as an aid or supplement to a college course. We investigated variables surrounding student experience with Blackboard in relation to engagement, usage, communication, performance, and overall perception.
After embarking on the research process, it was found that the hypotheses and research question originally proposed for this study had to be altered slightly to fit a set of more focused and operational variables. The following research question and hypotheses were chosen for this study:
RQ: Are online mediums, such as Blackboard, beneficial for student engagement?
H1: The use of Blackboard is positively related to college students’ academic performance.
H2: The use of Blackboard tools, such as the discussion board feature, is related to increased professor-student communication.
Main Variables
As can be seen from the research question and hypotheses, the variables of engagement, performance, and communication were chosen as the focus of the study and were each measured with the usage variable. Usage was calculated from a 7-point likert scale measuring the frequency in which a student visits the Blackboard site: 1 being very frequently and 7 being never. Similarly, communication was quantified by measuring the frequency of student interaction with professors through various communication platforms. Engagement was measured using a 5-point Likert scale asking students how much they agreed with statements such as, “I prefer courses that utilizes online technological resources as opposed to courses that do not.” Lastly, performance was quantified by measuring the cumulative GPA of students who participated in the study.
These measures were chosen because they each explore a key factor in the larger concern of our study: the affect of Blackboard on student learning. The measures chosen to operationalize each of this study’s variables were chosen because they provided data that had the potential to demonstrate decisively if Blackboard is a useful learning tool or not. Although the data was not conclusive in supporting our original hypotheses, as will be discussed later, the measures we used still provided useful data to show relationships and provide new explanations of our findings.
Design
The design of this study was centered on a survey that was distributed to college students who had experience with Blackboard or similar learning tools. The survey exposed participants to topics such as new media in academic settings and student usage of online learning tools. They were also asked questions specifically about the Blackboard site and it’s provided features. The survey, as well as the methods used for sampling, will be described further in the following section.
Methods
The latest trend in academic technology is the use of online learning platforms to supplement class discussion. With growing frequency, learning is being transported from the classroom to the Internet, and professors and students are relying on new forms of communication. Our research was done primarily to investigate this new aspect of online communication used by professors at exponentially increasing rates. The goal of this study was to gain new insight into student perception and usage of Blackboard (one of a few online learning platforms), and to see if the online platform promoted learning and its use was beneficial or detrimental. We also hoped to explore if Blackboard usage had become a replacement to classroom learning, instead of a supplement, affecting in-class engagement and communication with the professor. Previous studies have delved into professors’ perceptions of this new trend, but information from the students’ view is very limited. In particular, we wanted to see how students’ grades, experiences, and feelings toward their professor were changed, and if those changes were positive or negative, with the increase in Blackboard communication and reliance. In order to replicate this study, there are a few steps need to be taken. It is important to note that due our limitations as researchers, we were not able to test the entire demographic, resulting in a sample size that may not be representative or generalizable to the larger population.
Because we were working with limited time, resources, and funds, we chose a survey method for gathering data to test our variables and hypotheses. This method was the swiftest way to survey the demographic we were targeting, college-age students who use Blackboard or other academic online platforms. The survey we created had approximately 50 questions that were broken down into broad questions about demographics, introductory questions about academic performance, and categorized questions to assess our five variables of interest: usage, perception, communication, engagement, and performance. To test these variables, we used questions that were already produced in previous studies and tested for accuracy and clarity (Smith, Salaway, and Caruso 2009). The survey questions included both drop-down questions, as well as five and seven point likert scales. The drop-down questions included “What is your major,” “What is your cumulative grade point average” and “How many courses have you taken that utilized an online learning platform.” The scales varied for questions such as “I skip classes when materials from course lectures are available online” with responses categorized in a 5-point likert scale with 1 being “Strongly Agree” and 5 being “Strongly Disagree.” The 7-point likert scale questions included “I think that Blackboard is a valuable asset for my learning experience,” with 1 being “Strongly Agree” and 5 being “Strongly Disagree.”
Our survey was disseminated through Facebook in order to gain approximately 124 responses from college-age students with experience using Blackboard, the survey fully completed by 104. Of the 124 respondents, five were graduate students, and the remaining 99 were undergraduate students. While the majority of the respondents were from Boston University, ten students were from other universities. The gender characteristics of our survey respondents were almost entirely female. We found that only nine of the students in our sample were male.
We used this method, again, because of our limited time, funds, and resources. Our research team was comprised of students, who had access to, and connections with, other students making it relatively easy to gain the necessary number of responses in the limited amount of time. If we had had access to greater resources and time we would have liked to run analysis from a pre-test of the targeted demographic. With the information gained from that pre-test we would have targeted an even more representative sample. However, over a span of three weeks, we were able to obtain our total number of responses, analyze the data, and make important inferences about what it meant.
As stated, because our survey was disseminated through Facebook, we were only able to gain a sample of students much like ourselves based on our “Friends” lists. We must consider that this caused a sampling error because of the similarity with our research team members. For instance, the majority of respondents were Boston University students with only a handful of respondents from other colleges. Additionally, a large majority of the respondents were female. Though we tried throughout the process to increase our male participation, our data was still heavily based on female respondents. Despite these two distinguishing errors in our sample, we took steps to ensure that our participants had the necessary qualifications for their responses to be valid.
In order to take part in this research study, respondents were required to be either current students or recently graduated students of any college or university that uses Blackboard or similar online learning platforms outside of the classroom. The questions asked in the survey to gain this information were “Are you currently a full time or part time student” with the response options being “Full Time,” “Part Time,” or “Recent Graduate.” We also posed the question “Have you ever taken a class that utilized an online learning management system (i.e. Blackboard, ANGEL, etc.)” where respondents were given the choice between “Yes” and “No” and answering “No” then thanked them for their participation in the survey. We used these two qualifications to target our desired sample. Overall, we felt that the methods we used to draw information from our sample was our best possible option, and that the benefits of using a sample survey outweighed the risks and errors we encountered.
Results
In order to answer the question “Are online mediums, such as Blackboard, beneficial for students’ classroom engagement,” we tested Hypothesis 1: The use of Blackboard is positively related to college students’ academic performance, and Hypothesis 2: The use of Blackboard tools, such as the discussion board feature, is related to increased professor-student communication. We tested these hypotheses using the Multiple Regression Model. Although we did not predict a significant relationship between students’ academic success with proposed variables in the hypotheses, we conducted posthoc analysis to analyze these variables using a T test and inferential statistics.
Hypothesis 1
One of the concerns in our study is whether a combination of variables played a factor in influencing students’ academic performance. If the mean value of the response variable y in a least-square regression associated with a 1-unit change in an explanatory variable depends on a second explanatory variable, there is interaction effect. Therefore we ran a least-square regression test to determine whether there was an interaction effect between our five variables (performance, engagement, communication, perception, and usage) at α=0.05.
The results of the regression indicated there is no interaction between these five variables (R2=0.06486, F(5,78)=1.082, p=0.3769).
A multiple regression model was then built to test the connections each variable has on students’ academic performance. Academic performance was measured through survey responses about participants’ cumulative grade point average (GPA). The dependent variable is the GPA, and the independent variable is performance, engagement, perception, communication, and usage, respectively.
The results of the regression indicated the one predictor explained 4.9% of the variance (R2=0.0649, F(5,78)=1.115, p=0.506). It was found that usage at α=0.1 predicts students’ academic performance (β=-1.8984892, p=0.0613). The usage 7-point likert scale was designed with 1 being “Very Frequently” and 7 being “Never.” The GPA 7-point likert scale was designed where 1 represented “A” and 7 represented “F.” For every unit that the frequency scores go up, GPA scores go up by 0.1 (this is in relation to the coding of the variables). In layman's terms, grade point average is positively correlated with professor usage of Blackboard, meaning the student GPA decreases the less the professor uses blackboard. Given the .06 value, this finding is considered trending, however not significant.
Hypothesis 2
Hypothesis 2 states the use of Blackboard tools, such as the discussion board feature, is related to increased professor-student communication. We used the question “How often does your professor require you to use the Discussion feature of Blackboard?” as the independent variable on a 5-point likert scale with 1 being “Always” and 5 being “Never”, and the six Communication variable survey questions as the dependent variable.
Simple linear analysis was used to test if use of the discussion board significantly
predicted student-professor communications. The results of the regression indicated the use of the discussion board explained 38.9% of the variance (R2=0.15157989, p<.01).
Posthoc Testing
A two-tailed T test was also run to test whether use of Blackboard in classrooms had an impact on student’s GPA. The sample of respondents reported their average GPA in courses utilizing online learning system such as Blackboard as well as their average GPA in courses without the assistance of such learning systems. The null hypothesis is that there is no significant difference between the mean of students’ GPA who do or do not utilize Blackboard. The alternative hypotheses being there is a difference. At α=0.05 level.
H0: μ=μ0
H1: μ ≠ μ0
GPA using Blackboard (M = 1.53, SD = 0.52) is not significantly higher than GPA without using Blackboard (M = 1.59, SD = 0.60), t(1) = 1.296, p=0.195. From these results, we do not reject the null hypothesis.
Since these result do not correlate with our study’s original hypotheses, we ran additional tests to find an explanation for this phenomenon. The multiple regression model was used to determine a potential relationship between respondents’ class year on their overall usage of Blackboard and its impact on communication with professors.
The results of the regression indicated the two predictors explained 13.48% of the variance (R2=0.1869, F(11,72)=3.586, p=0.0057. It was found that Engagement at α=0.1 predicts students-professor communication (β=1.8308461, p=0.0719). The six questions used to measure the Communication variable section were on a 7-point likert scale with 1 being “Frequently” and 7 being “Never.” Students’ class year was attributed to 1 being “Senior” and 4 being “Freshmen.”
We also examined student’s attitude towards using technology as a replacement for in-class education. Multiple regression analysis was run to predict if students’ attitude toward the statement “I skip classes when materials from courses are available online” influenced their uses of Blackboard, student-professor communications, and academic performance. The results of the regression indicated the two predictors explained 31.52% of the variance (R2=0.3565, F(5,78)=8.641, p<0.001). It was found that Engagement at α=0.5 predicts students’ attitude towards skipping classes (β=-2.0431094, p=0.0444). Additionally findings showed that student-professor communications predicts students’ attitude towards skipping classes (β=-2.3583217, p=0.0209).
The survey also involved questions to determine the preferences respondents have for the way they receive information regarding course work. The answers were graphed in a histogram below. Over 30% of the participants answered they prefer to receive coursework information through “Blackboard” and another third answered they prefer “Email”.
In the question “Which of the following best describe you,” 52 out of 86 students chose “I usually use new technology when most people I know do.”
Moreover, the test results indicate the features of Blackboard with which students required assistance navigating. 21 students reported needing assistance on Blackboard. The results of this question are shown in the graph below.
Discussion
Discussion of Findings
Blackboard was chosen as the topic for this study because the trend of online learning tools is gaining momentum and we were interested in studying how the use of such tools is actually affecting students.
This study is relevant now more than ever because new media is proliferating many areas of the average person’s life and now new media is becoming an important part of academics. Millennial students are adept to using new technologies and many professors have responded by incorporating tools such as Blackboard into their classrooms. Yet despite the growing dependence of professors and students on these tools, there is a surprisingly limited amount of data explaining the student perspective and interaction with tools such as Blackboard. Our study attempts to remedy this situation and provide possible insights into how students learn from Blackboard.
As explained in the results section, this study incorporated a least-squares regression, multiple regression, and a two-tailed T-test to study the interactions between our chosen variables and Blackboard usage. First, the least-squares test found that none of the P-values for any combination of variables was less than 0.05. Unfortunately, this meant that there was no interaction between any of our five variables.
Next, a multiple regression test was used to measure each of the variables against academic performance. We chose multiple regression as a measure because, “it is useful in that it can take in a range of variables and enable us to calculate their relative ratings on a dependent variable.” (Cohen, Manion, and Morrison 2013). After running the test, we again found the results to be uncorrelated, however, there was an interaction between the usage variable and academic performance. Contrary to our original hypothesis, the data showed that the use of blackboard correlates with lower GPA. In other words, students with lower GPAs appear to be using Blackboard less frequently.
Finally, a T-test was used because, “this procedure is used to study two independent groups for differences.” (Wimmer and Dominick 2014). The test was able to find a connection between students’ class year and engagement; specifically how often students visit professors’ office hours. The test found that as class year rises, students are more engaged with Blackboard and visit professor’s office hour less often. The data also showed that when students strongly agree with the statement, “I skip classes when materials from courses are available online” they visit professor’s office hour less often.
Although our hypotheses, “The use of Blackboard is positively related to college students’ academic performance” and “The use of Blackboard tools, such as the discussion board feature, is related to increased professor-student communication” both proved to be statistically invalid, the data still provided some interesting findings to direct our research question. At the beginning of this study we posted, “Are online mediums, such as Blackboard, beneficial for student engagement?” and we had originally expected to find evidence that supported a beneficial relationship between Blackboard and student engagement. However, after conducting our research, we began to notice the opposite relationship.
It appears that students are more neutral to the use of technology in the classroom and may not actually benefit from it. In fact, as some of our data points out, students will even use online media tools such as Blackboard to avoid attending class or visiting their professors office hours. These findings lead us to draw unexpected conclusions from the relationships found in our data.
In order to better understand our unexpected conclusions, we looked further at the relationships we found between variables. In order to test each of our five variables, performance, engagement, communication, perception and usage, against each other we ran the least-square regression test and found that no combination of variables resulted in less than 0.05. With this result, we were able to conclude that there was no interaction between any of the variables. However, we were able to find significance in some variable relationships when we did the multiple regression tests. A notable relationship from the multiple regression test was between our two variables, usage and performance. With usage being at α=0.1 there was a correlation to students’ academic performance with a P-value of 0.0613. These results lead us to find that the use of Blackboard actually lowers student’s academic performance, as shown through their GPA. We also used the multiple regression test to determine whether there was a relationship between the students’ class year against their overall use of Blackboard and its impact of students’ communication with professors. With results shown at α=0.05 we found that as class year rises students are more engaged with Blackboard and visit professors office hours less often.
Overall our findings were eye opening and proved that our hypotheses were not valid; however, we did find success and answers to our skewed results through re-tests. We were able to put pieces together and get a better understanding of the potential reasons why our hypotheses were not valid. When we discovered that our first hypothesis was incorrect and that the use of Blackboard was, in fact, negatively related to college students’ academic performance, we were able to determine why this may be true through our multiple regression test. Discovering that students who engage more in Blackboard actually attend class less and visit office hours less was a clear indication that their grades may suffer. We began our study hypothesizing that Blackboard was an additional resource to assist students, but in fact students who engage with the platform more may be using it because they are disengaged or absent from the classroom. Through research we found, as Clark (1983) states, that there are no beneficial implications from the use of particular mediums in providing instruction to a group. This statement reaffirms why Blackboard usage does not correlating with the success in academic performance. In our second hypothesis we predicted that the use of certain Blackboard tools, such as the discussion board feature, would increase professor-student communications; however, this was the same case. Students who utilized the discussion board feature of Blackboard may be doing so as an excuse to avoid professor-student communication, and further themselves from the classroom and their instructors.
Limitations
There were some limitations to our study, which have been mentioned briefly but will now be discussed more fully. Our sample was mainly collected through the use of social media and our sample population consisted primarily of female Boston University students. We also considered a limitation in the questions used in our survey. It is possible that students may not have the reflective abilities to be aware of and understand their learning habits, and therefore they could not provide an accurate portrayal through survey response. Considering these limitations, we feel that we may not be able to generalize our findings from this sample to a larger population. Because our findings come mainly from one institution and the specific learning habits that are generated in said institution, the findings cannot be generalized to suggest that all students interact with online learning platforms in the same manner. Additionally, conclusions may not be drawn to a larger population regarding students’ ability to understand and reflect on their learning habits and preferences.
Future Research
Considering our limitations, future researchers may find it helpful to address them in the following ways. The likert scale questions in our survey were not consistent in terms of the point system used, therefore causing complications in our ability to run tests for significance of variables. This would be easily remedied by future researchers conducting similar studies. The grouping of questions into five variable categories was based primarily on the opinions of us as researchers, meaning that the answers and correlation between each variable may be varied depending on these groupings. Future researchers would benefit from a more concrete system of identifying the survey questions categorical positioning. Additionally, because students perception about their own learning methods and what is beneficial to them may not be as apparent in the context of a survey response, a focus group or other personal interview may provide a more accurate guide for assessing students perceptions.
We were able to engage in a larger part of the conversation taking place concerning the abilities and benefits of online learning platforms and what they mean for a change in learning behavior. The perspective of students who are utilizing these resources is new to the conversation around online learning. It is an important aspect that we believe must be considered in order to understand the bigger picture and implications of sites such as Blackboard as they become more popular, and even required, in classrooms.
Our most important finding was the inverse correlation between class attendance and engagement on Blackboard. That is to say that students who are more engaged with Blackboard are less likely to attend classes. This suggests to us that there is potential harm in the use of these online learning to assist with in-classroom learning. Students may be relying too heavily on the resources available to them online because they are an opportunity to gain materials that previously required class attendance and participation. This finding is important for researchers going forward, both to explore further the implications of Blackboard on student learning, but also to expand on research regarding teaching benefits of Blackboard. Perhaps there is a correlation as to the engagement and attendance of students and the information professors are providing on Blackboard.
Although our initial hypotheses may have been rejected, the findings between individual variables still remain an important step forward in the scholarly conversation about online learning tools. The results and findings that were significant in this study still assist in answering the question of whether online learning platforms are beneficial for student engagement. Considering student engagement in Blackboard and class attendance are inversely correlated, we can conclude that there is a potential disadvantage to a large reliance on online learning.
References
Clark, R. E. (1983). Reconsidering Research on Learning from Media. Review of
Education Research, 53(4). 445-459. Retrieved from
http://www.jstor.org/stable/1170217
Cohen, L., Manion, L., & Keith, M. (2013). Research Methods in Education (6th ed.). London: RoutledgeFalmer.
Smith, S., Salaway, G., & Caruso, J. The ECAR Study of Undergraduate Students and Information Technology. ECAR Publication. Retrieved March 17, 2014, from
