This folder contains the detailed results of our models (Bloomsbury PILSA, Bloomsbury PILSA + S2Net, Bloomsbury PILSA + S2Net + Embedding) on the GRE Antonym questions test set. The dataset is originally collected by Saif Mohammad and can be found at: http://www.umiacs.umd.edu/~saif/WebDocs/LC-data/testset.txt
 
Each file contains 950 sections separated by empty lines, plus the summary in the end.  The first line in each section is the question and answer from the original dataset.  The model output scores comparing the target word with each choice word are listed after that. The choice word with the lowest score is selected as the answer.  Below is one typical example:
 
assuaged: thirsty devastated untrue unsatisfied foiled :: unsatisfied
assuaged thirsty -> -0.191555
assuaged devastated -> -0.168278
assuaged untrue -> -0.005451
assuaged unsatisfied -> -0.601790
assuaged foiled -> -0.130004
Selected answer: unsatisfied
 
Some of the target words or answer words are out of vocabulary, which are denoted as OOV. In "Bloomsbury PILSA + S2Net + Embedding", we try to resolve the issue by finding the embedding of them. When no embedding is found or used, the questions with OOV target words are skipped.  Questions with some OOV answer words are answered by selecting a word from the remaining choices.  Following are examples of two cases.
 
doltish: casuistic clever qualified disabled sharpened :: clever
doltish is OOV
 
hiatus: nexus atavist cognate vortex reflex :: nexus
hiatus nexus -> -0.357967
atavist is OOV
hiatus cognate -> -0.010980
hiatus vortex -> 0.018793
hiatus reflex -> -0.021047
Selected answer: nexus
 
--Scott Wen-tau Yih (scottyih@microsoft.com)
