Promoting the development of the academic path through the National System of Researchers . A first attempt at evaluation

Mexico is an OECD member country that historically has allocated less than half a percent of its gross domestic product (GDP) to spending on research and experimental development (R&D). In this paper an attempt is made for the first time to evaluate the country's science and technology policy through its main program, the National System of Researchers (SNI). The results obtained make it possible to see whether the SNI has succeeded in improving research in Mexico. We also look at whether the program provides ways of improving and strengthening the science policy model adopted by Mexico and whether it could be extended and/or introduced into other nearby countries with levels of development or characteristics similar to those of Mexico.


Introduction
The late 1980s and early 1990s saw the appearance of new theories on economic growth, such as endogenous growth theory, which stressed the position of human capital and knowledge in explaining international differences in countries' economic growth rates (Lucas 1988;Romer 1986;1990a, 1990b).
In an increasingly knowledge-based economy, human capital is the key element enabling a territory's technological, economic and social development.It also helps to show whether a country or region can meet the needs of its technological and innovation processes so they can become tangible benefits for the population (CONACYT 2007, p. 21).The level of knowledge is therefore basic for enabling the generation of innovation and its adoption by third parties.A vast number of reports and projects stress this relationship.
On a European level, for example, the European Union (EU) itself, in both the Lisbon Strategy and the more recent Europe 2020 Strategy(Communication From The Commission Europe 2020, 2010; European Union Parliament Website), points to knowledge as the key element for economic development.Furthermore, according to the National Economic Council(NEC) (2011)one of the priorities of the United States for ensuring its economic growth and prosperity involves increased support for both the education and training of intellectual human resources.In the EU's IAREG project (Intangible Assets and Regional Economic Growth), an in-depth study is being made of the role of intangible assets in economic growth, and one of the assets mentioned explicitly is knowledge capital, human capital, social capital and entrepreneurship capital 1 .Also, Audretsch (2004) identifies three types of capital as drivers of economic growth: physical capital, human capital and knowledge capital.
One of the most important components of human capital is education, because it is through the creation of greater and above all better intellectual wealth that one of the most important ingredients for a country's competitiveness is encouraged, as recognized by Schultz (1961) and Krueger and Ruttan (1990).Furthermore, aid to developing countries consisting only of capital transfers is not enough if those countries do not have a suitable level of human capital in order to get the most possible benefit from the capital.The role of human capital and training in technology transfer is becoming increasingly more widely recognized.This means of transferring technology comes about in a number of ways including training which is specifically aimed at technology transfer management (Grosse, 1996), the use of consultants (Bessant and Rush, 1995), student training (with the international training of students becoming more important) (Nataraajan and Chawla, 1994), the exchange or transfer of staff (Hicks, 1993), and of course informal relationships between different scientific levels (Bozeman et al., 1995).
Mexico, like any other country that wants to progress in its development, needs to take steps to improve its scientific performance in order to maintain growth in a knowledge-based world.Measures have therefore been taken to improve its science and technology system, and the need now is to discover and evaluate the impact of the science and technology support programs.Analyzing them will enable Mexico's science and technology policy to be evaluated and it will then be possible to assess how advisable it would be to apply the same policy in other nearby countries or with similar levels of development or characteristics.
The National Council for Science and Technology (CONACYT) in Mexico has consolidated its position as the main institution encouraging, promoting and strengthening the high-level training of human resources.
Firstly, this is because it is the Mexican institution that awards the most grants for carrying out postgraduate studies whether at home or abroad.Secondly, because it is the home of the most important organization in this field in Mexico which is the National System of Researchers (SNI)which promotes and evaluates the quality of the scientific and technological research and innovation produced in the country.The SNI has become one of the pillars of Mexico's science policy aimed at boosting the quality of research in the country.
Belonging to the SNI is a distinction which symbolizes the quality and prestige of the scientific contributions made by someone with a vocation for research.The aim is that interest in becoming part of the SNI (at the highest possible level) will act to revitalize Mexican research.In this system the government recognizes the research activities of people in higher education, government institutions, the private sector and non-profitmaking organizations (NPOs).Researcher selection is carried out by a peer review committee.There are two main levels in this research system: the lower level is "Candidate", aimed at young people who are starting their research careers, and other levels (National Researcher) for researchers who are already established in their careers.
Within this framework, the questions that need to be asked and that this paper tries to answer are the following.Has the SNI been successful as an instrument for improving research in Mexico?Have the SNI assessment criteria been objective and appropriate?Are there any ways to improve the science policy model adopted in Mexico?Could the experience and results of this sub-program move to other countries with an economy similar to that of Mexico?
To answer these questions we will first analyze how the level of Mexican research has evolved over the last few decades.After that we will analyze the scientific productivity of SNI members in comparison to Mexican researchers as a whole so as to assess both whether the SNI selects the best researchers and whether these researchers carry out more research and of a higher quality.Finally, we will check whether the selection criteria of the assessment committees are internally consistent.To do this we will carry out a bibliometric analysis of Mexican scientific production (overall and specifically of SNI members) and an analysis of the groups of researchers belonging to the SNI.
Knowing the production profile as well as the productive capacity of those researchers who were accepted into this Mexican research elite at a particular moment in their lives will to a great extent make it possible to understand the capacity for innovation and science and technology transfer of what are considered highperformance researchers in Mexico.
Including this introduction we have divided the paper into six sections.In Section 2 we set out the paper's aims.In Section 3 we present the characteristics and main data regarding the research system in Mexico and the SNI.In Section 4 we evaluate whether the scientific results associated with SNI researchers are better than those for other researchers who are not part of the program.In Section 5 we evaluate the SNI selection process based on the applicants' research indicators and in Section 6 we present our conclusions.

Objectives.
As mentioned in the previous section, the aim of this paper is to discover the impact and importance of the best program in Mexico as regards research policy, known as the SNI.Analyzing it will, in short, enable us to: (a) detect whether the program has been of use in improving the investigation potential and capacity for science and technology transfer of the research elite in Mexico; (b) check whether the levels assigned to these researchers within the SNI are justified, based on research indicators (inputs and outputs); and (c) assess the pros and cons of the SNI program with an eye to reviewing it in Mexico.

Some relevant data.
As is the case in most Latin American countries, Mexico's science and technology system can hardly be described as a heavyweight.In relative terms, the indicators for Mexican research are similar to those for the Latin American countries which are strongest in research (Brazil, Argentina and Chile), but a long way from the indicators for the countries of Europe and North America.To begin with, the number of researchers working full-time or equivalent per thousand of the economically active population in 2008 was 0.9 compared to 2.6 in Argentina, 1.2 in Brazil, 0.9 in Chile, 5.7 in Spain and 8.0 in the United Kingdom (UK).
The average for Latin America for the same year was 0.7, while for the European Union it was 6.6 and for the member countries of the Organization for Economic Cooperation and Development (OCDE) it was 7.6 for full-time or equivalent researchers.The result of this, as an example indicator of quality science production, is that the total number of articles published by Mexican researchers according to the Institute for Scientific Information (ISI) was not very significant given that during the period 1997-2008 their participation did not exceed 1% in terms of the total number of articles published 3 .However, as can be seen in Figure 1, there is a noticeable growth pattern in the total number of Mexican articles published in someInstitute for Scientific Information (ISI) journals in line with increased spending on R&D in Mexico.Specifically, the annual average growth in Mexican articles in the ISI is 9.5%, in contrast to the 5.0% average growth for all articles in the ISI (1997)(1998)(1999)(2000)(2001)(2002)(2003)(2004)(2005)(2006)(2007)(2008).As far as the number of ISI articles per researcher is concerned, the ratio for Mexico in 2007 was 0.22, similar to Argentina (0.22) and Brazil (0.25), but lower than Spain (0.33), Canada (0.38) and even the average for Latin America (0.25).Finally, as regards the number of citations received per article published in the ISI, for period 2000-2007 Mexico obtained an average indicator of 4.9, lower than Argentina (6.4) and Chile (5.5), the USA (11.2), the UK (10.1) and Spain (7.4), but similar to Brazil (4.8).
With regard to the weight of the different scientific disciplines, Table 2 shows the ISI participation of the most important for period 1997-2008.The academic field of multidisciplinary physics stands out.However, there is a noticeable absence of certain social or humanities type of disciplines, although when interpreting these data it should be taken into account that traditions of article publication differ depending on the area of knowledge.But, at least the set of figures presented in this section has clearly shown the limitations of the Mexican science and technology system and its great potential for improvement.Evaluating the SNI and proposing ways to improve it will make it easier to take the policy decisions needed to achieve it.

(Insert table 2)
Source: Own data based on data from the Institute for Scientific Information (ISI), 2008.

The SNI as the main supporting sub-program for science and technology in Mexico.
The main instrument of science policy in Mexico is the National System of Researchers (SNI).In principle one would expect researchers registered with this system to be responsible for most of the quality scientific papers published, at least from within the Mexican science and technology system.This section aims to give a brief description of the system's characteristics.
The SNI is a sub-program within the Scientific Research &Development Program set up by the Federal Government of Mexico.Responsibility for leading and running the program, establishing its aims and functions, its organization and internal regulations, is in the hands of the National Council for Science and Technology (CONACYT).The purpose of the SNI in Mexico is to encourage and improve, by assessment, the quality of the scientific and technological research and innovation which the country produces.
Entry into this research system is voluntary and free for the applicant.Once applications have been analyzed by the assessment committees of each of the seven areas of knowledge defined by the SNI, they are informed whether their assessment is positive or negative and, in the case of the former, informed of their acceptance as a member of the SNI along with the level they have been assigned (Candidate, Level I, Level II or Level III).This level brings with it a variable financial remuneration.In the Mexican context, belonging to the SNI implies recognition of the researcher's quality and academic prestige resulting from a scientific output of considerable importance at national level and in some cases at international level.

Evaluation of the SNI sub-program
This section analyzes whether belonging to the SNI means added value as regards research capacity and scientific results to justify the program's existence.In this sense the aim is to find out whether the existence of the SNI has served to improve the proportion of published work (in quantity and quality) with regard to Mexican researchers as a whole.This indicator (publications) will be used, because it is commonly accepted on an international level as a proxy for a country's scientific level and also because the SNI has allowed access to a database enabling this analysis to be carried out 4 .
As far as the introduction of the SNI program is concerned, Figure 2 shows that the proportion of SNI participation for all full-time researchers 5 in Mexico for period 1997-2009 was 30.5%.In other words just under a third of Mexican researchers belong to the SNI (at one level or another).Based on the results issued by the SNI (approved and non-approved applications), in Figure 3 it can be seen that entering the SNI is relatively favourable for the researcher since the likelihood of being granted membership is around 75% of applicants, although this figure has decreased in the course of the decade (1997)(1998)(1999)(2000)(2001)(2002)(2003)(2004)(2005)(2006)(2007)(2008) (from 80% to 70%).This higher rejection rate along with the higher growth in numbers of researchers than in numbers of SNI members would explain, at least fairly clearly until 2006, the lower proportion of SNI participation for the Mexican researcher collective seen in Figure 2.With respect to the decision criteria the SNI uses to assess applications, two comments should be made: (a) on the requirements necessary for admission to the various levels or categories within the SNI, and (b) on the homogeneity of criteria between areas of knowledge.With respect to the first pointthe proportion of items needed to achieve a positive assessment by the SNI for each level (see Table 3) 6it is surprising that because of the relative weight on most items, growth will not occur in the outputs required as the researcher reaches a higher level.In contrast, the relative weight of the items needed to be evaluated positively aredecreased by SNI with the increasing level of researcher.Indeed, the reverse is true as the relative weight of articles, technological developments, distinctions, research stays, post-doctorates, conference invitations, books, theses supervised, etc. follow a downward trend as the researcher's level rises.The only important exception is the number of citations.This could indicate either that the criteria have become more rigorous over the years, that new candidates are better trained and have a higher research level than previous SNI researchers had in the past at the same level, or that the quality of research is measured above all according to the number of citations 7 and articles in internationally renowned publications.In this sense, the criterion that values quality over quantity is shown in Table 4, which presents the average numbers of articles published in the ISI by SNI researchers.
During period 1997-2002, those researchers who had uninterrupted membership of the SNI and had been promoted to a higher level within the research system had an average of 2.3 authorships and/or coauthorships in the ISI.It can be deduced that for these Mexican researchers promotion to a higher level in the SNI corresponded to the number of authorships and/or co-authorships in the ISI.In other words the higher SNI level corresponded to the higher average number of authorships and/or co-authorships in the ISI.As far as the homogeneity of assessment between areas is concerned, it is noteworthy that this does not exist to any great degree.Thus, the outputs needed for positively assessed projects are not the same in each of the SNI's seven areas of knowledge (see Figure 4) due to the different science production characteristics of each area.It is noticeable that there is great variability in the "citations" output given that they appear in around 50% of the positively assessed applications for the experimental areas and health (Areas I to III), while in humanities and social sciences (Areas IV and V) they are only required in 15% of successful applications.What is clear is that in all the areas defined by the SNI, the number of citations, conference invitations, articles and theses supervised was the output that in most cases had to be present for a positive assessment during period 1996-

2003.
As regards whether the existence of the SNI has served to improve the amount of work published (in quantity and quality), we will analyze the results for publication by researchers (from both the SNI and non-SNI collectives) in ISI journals.As Table 5 shows, the articles by Mexican researchers registered in the SNI that were published in the ISI and the citations received for them accounted for just under a third of the total for Mexican researchers as a whole 8 .Source: Own data based on past production as notified to the SNI.
In view of the information shown in Figure 2, which already placed the proportion of Mexican researchers in the ISI at around 31%, it can be concluded that SNI researchers did not make a significantly larger relative contribution than those researchers who remained outside the SNI.Therefore, no substantive improvement can be detected as a result of belonging to the SNI 9 .

Analysis of the SNI assessment criteria
In order to better understand the current assessment procedure followed by the SNI and to make the selection of research and development projects connected with the research policy program in Mexico more efficient, we analyze the results of applying four analysis and data clustering techniques.The results of applying these will also make it possible to suggest ways of improving the program and extracting conclusions that may be useful in other similar programs in other countries.

Multiple linear regression analysis (MLRA)
Firstly, the designation assigned to an application approved by the assessment committees of each SNI area was modeled, mainly according to the scientific production notified in the application 10 .The model to estimate was as follows: where, Y i = {0 (Candidate Researcher), 1 (Researcher Level I), 2 (Researcher Level II) and 3 (Researcher Level III); X ki are explanatory variables; ε i is a random variable associated with the estimation error.It can be seen from Table 6 that those applications approved by the SNI with a designation to National Researcher Level 1 were the ones best defined by the estimation, with an accurate assessment average of 80.3% for each of the seven areas defined by the system.However, the model is less sensitive when it comes to estimating the other SNI levels, regardless of the area of knowledge.An average determination coefficient of 46.9% is obtained for each of the seven areas of knowledge defined in the SNI.In other words the explanatory variables together contain only half the attributes enabling an SNI designation to be defined.
These results imply that the information requested from relatively young researchers (Candidate Researcher) and from researchers with obvious experience in carrying out research in Mexico (Level II and Level III) is not enough for estimating their designation clearly and does not justify the results of the assessment committees.This lack of fit is especially relevant with regard to Level III researchers.The results also imply that, in order to be able to issue a final report, the SNI assessors take into account additional information that is not detected in the scientific production notified.In other words it can be deduced that the scientific criteria included in the regression model serve to explain the assessments of those Mexican researchers designated as Level I, but that for the other levels there are complementary criteria used in deciding the assessment.Source: Our own data based on results obtained using EViews.
Table 7 shows the explanatory variables that proved to be statistically significant using MLRA.It can be seen that there is statistical evidence that different assessment criteria are used for approving applications in each SNI area of knowledge.Nevertheless, there are also various items in which these criteria clearly coincide, such as age, gender and the application's position in the SNI.It is important to mention that the items considered in order for an SNI area to approve an application are very balanced given that 45.5% of these criteria corresponded to characteristics inherent in the applicant, while the other 54.5% corresponded to characteristics associated with his or her productivity.
The MLRA model therefore indicates that the committees use more factors apart (path as academic researcher assessed the reputation of the researcher evaluated and, in general, information inherent to researcher evaluated) from those entered in the database (see Table I of Appendix) to make their assessments.This happens especially with Levels II and III.One way of improving this would be to incorporate these currently non-explicit factors (such as the quality of the publications and/or impact factor) to facilitate the assessment process and thereby avoid accusations of arbitrariness in the selection process.

Linear discriminant analysis (LDA)
LDA is a classification method that seeks to discriminate a subspace (Shih-Wei Lin, 2009) in which the patterns of belonging to the classes are grouped as tightly as possible, while patterns of belonging to other classes are as far apart as possible (see Fisher, 1936).In this section the grouping variable is again LEVEL (designation assigned to a researcher accepted by the SNI), while the independent variables are the same ones used in the MLRA (see Table I of Appendix).
The results obtained for the test statistics known as Wilk's lambda, Box's M and the equality of means test 11 show that discrimination is possible and makes sense.Another interesting result was obtained from the eigenvalues, which showed that, for all the SNI areas, the discrimination between designation to Candidate and Level I is carried out correctly, with an average of 89.1% of the total variance explained for each of the seven areas defined by the SNI.However, the discrimination for designations to Level I and Level II obtained an average of only 7.9% of the total variance explained for each of the seven areas of the SNI, and the discrimination between Level II and Level III was still less clear due to the fact that the third discriminant function absorbed only 2.9%.The summary of this analysis for all correct estimations is shown in Table 8.It is clear that the Candidate Researcher designation was estimated relatively accurately, whereas the other designations were less accurate at 60%.This result means that for the National Researcher category (at all three levels) there is a need for greater and above all better information (as the number of articles in ISI-JCR journals and total citations to these publications) in order for it to be possible to differentiate between them because, using the information now available, the designations detected are not justified.Table 9 shows the reassignment of possible changes in level during period 1996 -2003, by SNI knowledge area.Source: Own data based on data obtained using SPSS and historical data from the SNI.
LDA shows that 57.0% of all applications approved for the SNI's seven areas of knowledge were estimated correctly.However, 20.4% of total applications approved showed a scientific production that may have made them deserving of a higher SNI level (especially as regards movement from Level I to Level II).In contrast, 19.1% of total applications approved by the SNI during the same period showed a scientific production that did not correspond to the level assigned by the system's assessment committees, i.e. they could well have merited a lower level than the one designated (especially as regards movement from Level I to Candidate).
Finally, 3.5% of the total applications approved could not be grouped using LDA.
To summarizeLDA identifies only two groups of researchers or designations based on the information supplied to the SNI.This result comes about due to the fact that the linear discriminant function between the levels of Candidate and Level I is well defined.However, the National Researcher category at all three levels (Level I, Level II and Level III) presents no discrimination (based on the data considered in the database), in which case these three levels may as well be amalgamated into just one, the consequence being that there would be just two categories in the SNI: relatively young researchers starting out on their research careers and researchers who already have their ownsometimes well-establishedline of research within a national context.

∑| |
where A(x) is the vector of the real averages in each SNI level.
B(y) is the vector of the estimated averages in each SNI level.
defines the attributes of set A(x).
defines the attributes of set B(y).
X k is the k th attribute of set A(x).
n is total attributes.
The reason for using this distance is to detect the similarity existing between the real vectors of the production associated with each SNI level and the vectors estimated using the c-means algorithm.In other words, if there were no subjective criterion in the SNI assessment process, then the designations would definitely have a very different distribution to that observed during the period studied.The Hamming distance matrix is shown in Table 10.From this it can be deduced that the total applications approved by the SNI during period 1996-2003 showed a definite grouping into two clusters: three of the four designations defined in the SNI (Candidate, Level I and Level II) showed a clear convergence on a single clustercluster Bwhile the Level III researchers converged on cluster C. In addition, cluster B showed a great similarity (less Hamming distance) to the real vector relating to an SNI National Researcher Level I.This particular result implies that almost 90% of the Mexican researchers approved by the SNI during period 1996-2003 had the productive profile of a National Researcher Level I.

Table 10. Matrix of Hamming distances between the real averages and the averages estimated using c-means (Insert table 10)
Source: Based on historical data from the SNI.Results obtained using SPSS.
In order to validate the consistency of the results obtained in Table 10, the Hamming distance matrix was also calculated for the real averages of the criteria assessed for the Mexican researchers approved by the SNI during the period 1996-2003.This distance matrix is shown in Table 11 and it is noticeable that the productive profile for Candidate is very similar to that of Level I 14 .These results for the total applications approved by the SNI during period 1996-2003 were also obtained by carrying out an analysis for the great majority of knowledge areas defined by the system.In other words, in six out of the seven SNI areas two clusters at most were identified in which a large proportion of these approved applications were concentrated 15 .
Another result deduced from the analysis is that different internal assessment criteria were used in all the areas defined by the SNI because, as the cluster is different for each SNI level, it can be said that each area assessed different scientific criteria when approving an application, at least during period 1996-2003.Also, it is easy to see that in the great majority of these areas of knowledge, the highest SNI level -National Researcher Level IIIis clearly different to the other designations.
Finally, we should state that the scientific production and, even more so, the information requested by the SNI, at least during the period 1996-2003, did not correspond with the designation awarded, regardless of the area of knowledge, since all the researchers should be grouped together in two clusters at the most.

Fuzzy C-means (FCM) algorithm
The fuzzy C-means (FCM) (Bezdek, 1981) algorithm is an extension of the c-means algorithm, which, in very general terms, identifies relatively homogeneous groups on the basis of previously selected characteristics, such as the centroid if it is known in advance.The c-means algorithm is the basic clustering method, while the fuzzy c-means is the fuzzy clustering method which could well be seen as an improvement on the c-means from the point of view of data pooling (Shian-Chang et al., 2009).
In most of the current literature, fuzzy clustering techniques refer to the minimization of the distances between elements in the sample selected (Dae-Won et al., 2004).The FCM algorithm is more efficient when a set of numerical data is used.In practice, however, applications of this algorithm can be found using either numerical or categorical and sometimes even mixed data (Ricardo et al., 2009).The main advantage of using it in this analysis is that with this algorithm an element can belong to various classes or groups simultaneously (Bezdek, 1981).In other words, the clustering philosophy of fuzzy logic is that each point has a degree of belonging to certain clusters, expressed by fuzzy logic, instead of belonging completely to a single group or class.The points at the edge of a particular group can still be in that group, but to a lesser degree than the points at the center of it.
As the internal assessment criteria in each SNI area of knowledge do not have to be the same, and indeed are not, the FCM algorithm was applied for each of these areas of knowledge for the purposes of detecting possible approved applications that might be considered benchmarks for distinguishing between one designation and another.The FCM algorithm also supplies the maximum probability of belonging to a particular group or class.
Based on the scientific production notified by the Mexican researchers approved by the SNI during period 1996-2003, we detect the optimal group or class to which an application should belong.This group or class, for the purposes of this analysis, would correspond to the level or designation awarded by the SNI's assessment committees.In this section we again use the data shown in Table I of the Appendix and the estimations are carried out using the FuzMe (The University of Sydney) 3.5c package.
Due to the fact that clusters formed using the FCM algorithm depend on the fuzzy exponent, this section only shows the four partitions obtained using a fuzzy component of 1.75 and a Euclidian distance.Also, the four classes estimated using this algorithm were not calculated with a homogeneous area.The clustering in the four classes estimated using an FCM algorithm for all the designations assigned by the SNI assessment committees during period 1996-2003 is presented in Table 12.Source: Own data based on results obtained using FuzMe and historical data from the SNI.
It can be seen that the clustering for each SNI designation was very representative, and overall it can be said that the FCM algorithm presented a quite acceptable clustering of almost 97%.The percentage participation of the four classes estimated, in relation to the total applications approved that were grouped using FCM, is presented in Figure 6.Three classes were detected using FCM clustering.However, this clustering was mainly concentrated in two of them: class f1 (accounting for 57.2% of the total applications classified) and class f4 (40%).In other words 97.2% of the applications approved by the SNI were classified into just two classes.This result confirms the result already obtained using other techniques, that the great majority of applications approved by the SNI during period 1996-2003 presented two productive profiles and not four as defined at present.
The distribution of designations classified into the four classes estimated using FCM is shown in Figure 7. Source: Own data based on results obtained using FuzMe and historical data from the SNI.
In Figure 7 it can be seen that Class f1 has a very similar distribution to Class f4 as far as SNI designations are concerned.However, it is striking how well the FCM algorithm separates the two classes identified.
Another important result is that the predominant designation in Classes f1 and f4 was National Researcher Level I.This implies firstly that the productive levels in Classes f1 and f4 revolve around the productive level of an SNI National Researcher Level I, and secondly, that the designation defined as Level I is the main point of intersection between Classes f1 and f4 and could therefore be the distinguishing level between the higher levels of the SNI.These implications strengthen the result that the information requested by this Mexican research system should be different for each level, which in turn could lead to more efficient discrimination between the applications submitted to the group.
The information shown in Figure 7 validates the result that finds that the scientific production notified to the SNI by the approved Mexican researchers does not justify the existence of four levels, but at most two.The FCM algorithm detects two classes or groups of researchers, although the main difference in relation to the results obtained using a c-means algorithm lies in the fact that the fuzzy analysis presents more interaction between the estimated classes for the levels defined by the SNI.However, the designation defined as Level I would involve a majority for these two estimated classes or groups.
In the FCM analysis by areas of knowledge 16 , it is observed now that it was not two classes that were defined predominantly in most of them.In other words the SNI areas of knowledge showed more diversity as far as awarding a designation is concerned.However, in none of these SNI areas were there more than three significant classes.In addition it is clear that in the great majority of these estimated classes the SNI designation with the highest relative participation was Level I.This same diversity could also be seen for the clustering by SNI level, in which two of the system's higher levels (Level I and Level II) predominated.
Therefore, on the basis of the scientific production notified to the SNI, it is no longer so easy to distinguish between these two higher levels of the SNI.

Summary and conclusions
In this article we attempt to evaluate the contribution made to the Mexican science system by the main science policy program: the National System of Researchers.From the results presented, it can be seen that there is room for improvement in Mexico's position in the world science system as regards not only researchersdefined as full-timeper thousand of the economically active population, but also as regards spending on research and experimental development as a proportion of GDP, and even publications and citations per researcher in the ISI.
As a principal measure, the Mexican science and technology system introduced the National System of Researchers program to highlight and help the best Mexican scientists boost their scientific productivity, and, for those who are not yet part of the system, to act as an incentive for improvement.The results show that the proportion of researchers that form part of the SNI remains steady at around 31% of the potential human resources with a real vocation for creating and developing science and technology in Mexico, while the probability of being accepted into the system fell from 79.0% in 1997 to 70.1% in 2008.
The work published by Mexican researchers has had a symbolic contribution in the SNI.However, the participation of Mexican researchers on the international scene has undergone sustained growth according to ISI publications for period 1997-2008.The increase showed a clear correspondence to spending on research and experimental development, i.e. when more investment was made in R&D, an increase in the number of Mexican publications in the ISI was observed.
There have been positive aspects with respect to the SNI, one of which is the differences in assessment criteria between the different areas of knowledge.The research carried out shows that the outputs required are similar between areas, but not the same.This is an example of flexibility and recognizes that not all research can be systematized equally in the same number of research output indicators.Secondly, the SNI program has served to introduce a culture of assessment and accountability as far as funding is concerned.Mexican researchers have internalized the idea that they must submit to assessments in order to receive additional resources and that the periodic nature of these assessments obliges them to work continuously.Thirdly, it appears that quality is prized over quantity, because the higher level SNI researchers are not necessarily notable for their more numerous research outputs, although they do have a greater number of citations and articles per researcher in the ISI.
However, the procedure could still be improved since the results obtained so far cannot be considered completely satisfactory.Firstly, having work published in the ISI is not concomitant with more citations, partly because some of the ISI journals in which work is published (such as the Revista Mexicana de Física) have little international impact.Secondly, and more importantly, the productivity levels of the SNI researchersaccording to the SNI databasewere not as outstanding as would be expected, at least not The algorithms used in this paper show that the SNI, seen as a whole, presents a production profile similar to that of a National Researcher Level I.This result implies that the scientific production notified by the Mexican researchers accepted into this research system does not correspond with the designation assigned by the assessors who made up each of the seven SNI assessment committees.Therefore, the information requested by the SNI should be different for each designation and moreover it should be different in each of the seven areas of knowledge defined by the SNI.This is because the internal criteria to be assessed in each of these areas are different and homogeneous to only a small extent.
The results obtained indicate that there is no justification for the existence of four different levels of researcher (Candidate, Level I, Level II and Level III), but that there should be two levels at most.This together with the low level fit obtained in the regression models leads one to believe that the SNI assessors must use information in addition to that recorded in the SNI database, which as far as possible should be integrated into it.Given that the worst fit comes about in Levels II and III, it is likely that indicators involving the quality of the research carried out (in addition to quantity) had to be included.In this sense it is suggested that variables such as the number of articles and/or citations published in ISI-JCR journals should be incorporated.The two groups that arise from the techniques applied are not clear and in some cases contradict each other.Perhaps the two blocks that appear to be the clearest are Mexican researchers registered in the SNI who are beginning or defining a line of research of their own and Mexican researchers in the SNI who already have experience and a consolidated line of research, at least in national terms.On the basis of the scientific production notified to the SNI in each approved application, the designation defined as National Researcher Level I may well be the designation that separates these two groups of researchers.
It can be understood from the analysis carried out that there is a need to use quality statistical information in order to make the assessment.The science production notified to the SNI should no longer be just one more step in the process for any researcher who wishes to apply for entry into the system.The information gathered by the SNI would be better if it were presented with greater responsibility on the part of the applicant, and as a result the designations issued by each of the SNI assessment committees would be better and more representative.However, the final assessment of an application would still depend on the individual subjective criterion of the assessors who make up these SNI assessment committees.
Finally, this article has presented some techniques to make the SNI selection process more efficient with the aim of, first, gathering quality first-hand information; second, making the information reception process more efficient; and third, making it an instrument to complement SNI peer assessment, as long as it is an improvement on the qualitative information that the assessors appear to use now, which is not included in the current variables.We believe that both the experience shown as regards SNI assessment and the techniques applied could be of use in defining new science policy instruments in other countries with a starting point similar to Mexico as far as research systems are concerned.
For example, during the period of 1996-2003, researchers with an appointment Candidate Researcher showed a distribution to its scientific output, as follows: 19.5% of articles, 11% of citations, 9.8% of academic distinctions and 29.6% of invitations to conferences.These relative weights were preponderant for a Candidate Researcher to obtain a positive assessment of SNI. 7 Although it is normal for the number of citations to increase in the course of a researcher's career simply because there is a longer period in which to be cited.

Figure 1 .
Figure 1.Ratio of R&D spending/GDP in Mexico and Mexican articles as a proportion of total articles in the ISI, 1997-2008.

Figure 3 .
Figure 3. Probabilities for an application to be submitted to the SNI, 1997-2008

Figure 4 .
Figure 4. Items assessed for a successful application, by SNI area for period 1996-2003

(
total observations in each SNI knowledge area, which are not necessarily equal in all areas).(number of parameters estimated in each regression).The purpose of these estimations is to assess to what extent the explanatory variables (research outputs supplied in the application) serve to accurately explain the level (LEVEL variable) assigned to each researcher by the SNI.A good fit would indicate that the decisions taken by the SNI assessors can be explained based only on the indicators available.Absence of a close fit, in contrast, would imply that the assessors' criteria are also based on qualitative aspects not considered in the explanatory variables, and that these variables do not serve equally to accurately classify the researchers in the levels assigned them by the SNI.

Figure 6 .
Figure 6.Participation of the classes estimated using FCM, by SNI level for period 1996-2003

Figure 7 .
Figure 7. Distribution of designations awarded by the SNI in the classes estimated using FCM for period 1996-2003 during period 1997-2002.The researchers registered in the SNI had productivity indicators in the ISI similar to those of all other non-SNI Mexican researchers.In addition, a Mexican researcher registered in the SNI, regardless of designation, accounted for only 0.8 articles during the entiretyof period 1997-2002.This article has also applied different techniques that have made it possible to look more thoroughly into the classification of all the researchers who were assessed positively by the SNI for period 1996-2003.These techniques have enabled us to obtain alternative classifications based on statistical algorithms and to gauge the level of internal consistency of the classifications made by the SNI, based strictly on the quantitative information supplied.

Table 4 . Average number of authorships and co-authorships in the ISI for Mexican researchers currently in the SNI records, 1997-2002 (Insert table 4)
Source: Own data based on the ISIand past production as notified to the SNI.

Table 8 . Summary of correct estimations using LDA for the dependent variable LEVEL, 1996-2003 (Insert table 8)
Source: Own data based on data obtained using SPSS and historical data from the SNI.

Table 11 . Hamming distance matrix for the real averages of the criteria assessed in the SNI, by level for period 1996-2003 (Insert table 11)
Source: Based on historical data from the SNI.