Each word in the analysis has its own rating.
The study was supported by a grant from the Russian Science Foundation.
Smirnov's study useda representative sample of data from the HSE longitudinal cohort panel study “Educational and Career Trajectories” (TrEC). The study tracks the career development of 4,400 students from 42 regions of Russia from universities participating in the PISA (Program for International Student Assessment) program. The study data also included data on students' VK accounts (3,483 student participants agreed to provide this information).
“Since such data, combined with digitaltraces are difficult to obtain, they are almost never used, ”explains Smirnov. Meanwhile, such a dataset allows a robust model to be developed that can be applied to other parameters. And the results can be extrapolated to all other students - high school and high school students.
As a training sample, we usedmessages from public VKontakte pages - a total of 130,575 messages from 2,468 people who passed the PISA test in 2012. The test allowed the researcher to assess the student's academic ability, as well as his ability to demonstrate his knowledge in practice. The study included only publicly available VKontakte messages from agreed participants.
During the study, unsupervised machine learning with vector representation of words was performed in the VKontakte post-corpus (1.9 billion words in total, with 2.5 million unique words).
“We have presented each post as a 300-dimensionalvectors by averaging the vector representations of all its constituent words, ”explains Smirnov. "These post views were used to train a linear regression model to predict the ratings of the PISA post authors."
By "prediction" the researcherdoes not mean predicting the future, but a correlation between the calculated results and the actual scores obtained by students on the PISA exam, as well as their Unified State Examination results (which are publicly available on the Internet in aggregated form—i.e., these are average scores at school). At the preliminary stage, the model learned to predict PISA data. In the final model, the calculations were compared with the Unified State Examination results of high school graduates and applicants.
The final model was supposed to be able toreliably recognize whether a strong or weak student wrote a specific post on a social network, or, in other words, differentiate subjects according to their academic performance. After a period of training, the model was able to distinguish messages written by students with high or low PISA scores (levels 5-6 and levels 0-1) with an accuracy of 93.7%. Regarding the comparability of PISA and USE, although the two tests differ, research has shown that student performance on the two tests strongly correlates with each other.
It turned out that “predicted academic performance is closelyconnected with the results of the exam, "- says Smirnov. “The correlation coefficient is between 0.49 and 0.6. And in the case of universities, when the projected academic performance and USE scores were compared (information available in the current study, Monitoring the quality of university admissions), the results also showed a strong relationship. The correlation coefficient is 0.83, which is significantly higher than for high school, because more data. "
Results
Smirnov highlighted common text featuresposts in relation to the progress of their authors. The use of headwords (-0.08), emoticons (-0.06), and exclamations (-0.04) was found to negatively correlate with academic performance. On the other hand, the use of Latin characters, the average message and word length, vocabulary size and the entropy of user texts are positively correlated with academic performance (from 0.07 to 0.16, respectively).
It has also been confirmed that students with different achievement levels have different vocabulary.
High performing students used:
- English words;
- Words related to literature (Bradbury, Fahrenheit, Orwell, Huxley, Faulkner, Nabokov, Brodsky, Camus, Mann);
- Reading concepts (read, publish, book, volume);
- Terms and names related to physics (Universe, quantum, theory, Einstein, Newton, Hawking);
- Words related to thought processes (thinking, memorization).
Students with low scores used words withmistakes, names of popular computer games, concepts related to military service (army, oath, etc.), horoscope terms (Aries, Sagittarius) and words related to driving and car accidents (collision, traffic police, wheels, tuning).
Thematic Clusters: Representing the highest and lowest scoring words from the training dataset
Smirnov calculated the coefficients for all 2.5 millionwords of the vector model and provided them for further study. Interestingly, even words that are rare in the training data set can predict academic performance. For example, even if the name "Newt" (like the Harry Potter character Newt Scamander) never appears in the training data, the model can assign a higher rating to messages that contain it. This will happen if the model learns that words from a series of novels are used by high-achieving students, and through unsupervised learning, the model "intuitively" understands that the name "Newt" falls into this category.
Read also
The annual mission in the Arctic has ended and the data are disappointing. What awaits humanity?
Watch the new Hummer overcome obstacles on the road, moving like a crab
On day 3 of illness, most COVID-19 patients lose their sense of smell and often suffer from a runny nose