Test collection is used to evaluate the information retrieval systems in laboratory-based evaluation experimentation. In a classic setting, generating relevance judgments involves human assessors and is a costly and time consuming task. Researchers and practitioners are still being challenged in performing reliable and low-cost evaluation of retrieval systems. Crowdsourcing as a novel method of data acquisition is broadly used in many research fields. It has been proven that crowdsourcing is an inexpensive and quick solution as well as a reliable alternative for creating relevance judgments. One of the crowdsourcing applications in IR is to judge relevancy of query document pair. In order to have a successful crowdsourcing experiment, the relevance judgment tasks should be designed precisely to emphasize quality control. This paper is intended to explore different factors that have an influence on the accuracy of relevance judgments accomplished by workers and how to intensify the reliability of judgments in crowdsourcing experiment.
This study was carried out to investigate the effect of economic globalization on economic growth in OIC countries. Furthermore, the study examined the effect of complementary policies on the growth effect of globalization. It also investigated whether the growth effect of globalization depends on the income level of countries. Utilizing the generalized method of moments (GMM) estimator within the framework of a dynamic panel data approach, we provide evidence which suggests that economic globalization has statistically significant impact on economic growth in OIC countries. The results indicate that this positive effect is increased in the countries with better-educated workers and well-developed financial systems. Our finding shows that the effect of economic globalization also depends on the country's level of income. High and middle-income countries benefit from globalization whereas low-income countries do not gain from it. In fact, the countries should receive the appropriate income level to be benefited from globalization. Economic globalization not only directly promotes growth but also indirectly does so via complementary reforms.