A key objective of the United Nations Development Programme (UNDP) is to assist its member countries in meeting the Millennium Development Goals (MDGs). UNDP pursues this objective in various ways, including through analysis and advice to governments on the progress towards the MDGs (such as support for the preparation and monitoring Poverty Reduction Strategies, or PRSs, in poor countries), assistance for capacity building, and financial and technical support for the preparation and implementation of development programs.

The challenge of achieving the MDGs remains daunting in many countries, including Tajikistan. To do so will require that all development partners, i.e., the government, civil society, private business and donors, make every effort to scale up successful development interventions. Scaling up refers to “expanding, adapting and sustaining successful policies, programs and projects on different places and over time to reach a greater number of people.” Interventions that are successful as pilots but are not scaled up will create localized benefits for a small number of beneficiaries, but they will fail to contribute significantly to close the MDG gap.

This paper aims to assess whether and how well UNDP is supporting scaling up in its development programs in Tajikistan. While the principal purpose of this assessment was to assist the UNDP country program director and his team in Tajikistan in their scaling up efforts, it also contributes to the overall growing body of evidence on the scaling up of development interventions worldwide.

Downloads

• Download Full Paper

Since its 50th anniversary in 1994, the World Bank has been led by four presidents: Lewis Preston until his untimely death in 1995; then James Wolfensohn, who gave the institution new energy, purpose and legitimacy; followed by Paul Wolfowitz, whose fractious management tossed the World Bank into deep crisis; and most recently, Robert Zoellick, who will be remembered for having stabilized the bank and provided effective leadership during its remarkably swift and strong response to the global financial crisis.

Throughout these years of ups and downs in the bank’s leadership, standing and lending, the overall trend of its global role was downhill. While it remains one of the world’s largest multilateral development finance institutions, its position relative to other multilateral financing mechanisms is now much less prominent. Other multilateral institutions have taken over key roles. For example, the European Union agencies and the regional development banks have rapidly expanded their portfolios, and new “vertical funds” such as the Global Fund for AIDS, Tuberculosis and Malaria have become major funding vehicles. At the same time, according to a 2011 OECD Development Assistance Committee report multilateral aid has declined as a share of total aid. Meanwhile, non-governmental aid flows have dramatically increased, including those from major foundations like the Bill and Melinda Gates Foundation, but also from new internet-based channels bundling small individual donations, such as Global Giving. The World Bank— which 20 years ago was still the biggest and most powerful global development agency and hence a ready target for criticism— today is just one of the many institutions that offer for development to the poor and emerging market economies.

Against this backdrop, the World Bank, its members and Dr. Kim face three options in its long-term trajectory over the next 10 to 20 years: 1) the bank can continue on its current path of gradual decline; 2) it might be radically scaled back and eventually eliminated, as other aid channels take over; or 3) it can dramatically reinvent itself as a global finance institution that bundles resources for growing global needs.

There is no doubt in this author’s mind that the World Bank should remain a key part of the global governance architecture, but that requires that the new president forge an ambitious long-term vision for the bank – something that has been lacking for the last 30 years – and then reform the institution and build the authorizing environment that will make it possible to achieve the vision.

Option 1: “Business as Usual” = Continued Gradual Decline

The first option, reflecting the business-as-usual approach that characterized most of the Zoellick years of leadership will mean that the bank will gradually continue to lose in scope, funding and relevance. Its scope will be reduced since the emerging market economies find the institution insufficiently responsive to their needs. They have seen the regional development banks take on increasing importance, as reflected in the substantially greater capital increases in recent years for some of these institutions than for the World Bank in relative terms (and in the case of the Asian Development Bank, even in absolute terms). And emerging market economies have set up their own thriving regional development banks without participation of the industrial countries, such as the Caja Andina de Fomento (CAF) in Latin America and the Eurasian Development Bank in the former Soviet Union. This trend will be reinforced with the creation of a “South Bank” or “BRIC Bank”, an initiative that is currently well underway.

At the same time, the World Bank’s soft loan window, the International Development Association (IDA), will face less support from industrial countries going through deep fiscal crises, heightened competition from other concessional funds, and a perception of reduced need, as many of the large and formerly poor developing countries graduate to middle-income status. It is significant that for the last IDA replenishment much of the increase in resources was due to its growing reliance on advance repayments made by some of its members and commitments against future repayments, thus in effect mortgaging its future financial capacity. The World Bank’s status as a knowledge leader in development will also continue to be challenged with the rise of research from developing countries and growing think tank capacity, as well as a proliferation of private and official agencies doling out advice and technical assistance.

As a result, under this option, over the next 10 to 20 years the World Bank will likely become no more than a shadow of the preeminent global institution it once was. It will linger on but will not be able to contribute substantially to address any of the major global financial, economic or social challenges in the future.
 
Option 2: “The Perfect Storm” = Breaking Up the World Bank

In 1998, the U.S. Congress established a commission to review and advise on the role of the international financial institutions. In 2000, the commission, led by Professor Allan Meltzer, released its recommendations, which included far-reaching changes for the International Monetary Fund and the World Bank, most of them designed to reduce the scope and financial capacities of these institutions in line with the conservative leanings of the majority of the commission’s members. For the World Bank, the “Meltzer Report” called for much of its loan business and financial assets to be devolved to the regional development banks, in effect ending the life of the institution as we know it. The report garnered some attention when it was first issued, but did not have much impact in the way the institution was run in the following 10 years.

In 2010, the U.S. Senate Foreign Relations Committee released a report on the international financial institutions, which called on them to aim toward “succeeding in their development and economic missions and thereby putting them out of business”. However, it did not recommend a drastic restructuring of the multilateral development banks, and instead argued strongly against any dilution of the U.S. veto right, its lock on leadership selection, and its voting share at the IMF and World Bank. While not dramatic in its short-term impact, these recommendations were likely a strong factor in the subsequent decisions made by the Obama administration to oppose a substantial increase in contributions by emerging markets during the latest round of capital increase at the World Bank to push for an American to replace Robert Zoellick as World Bank president. These actions reinforced for emerging market countries that the World Bank would not change sufficiently and quickly enough to serve their interests, and thus helped create the momentum for setting up a new “South Bank.”

While there seems to be no imminent risk of a break-up of the World Bank along the lines recommended by the Meltzer Report, the combination of fiscal austerity and conservative governments in key industrial countries, compounded by a declining interest of the emerging market countries in sustaining the institution’s future, could create the perfect storm for the bank. Specifically, as governments face constrained fiscal resources, confront the increasing fragmentation of the multilateral aid architecture, and take steps to consolidate their own aid agencies, they might conclude that it would be more efficient and fiscally prudent to rationalize the international development system. There is a obvious overlap on the ground in the day-to-day business of the World Bank and that of the regional development banks. This is a reality which is being fostered by the growing decentralization of the World Bank into regional hubs; in fact, a recent evaluation by the World Bank’s Independent Evaluation Group concluded that “[r]ather than functioning as a global institution, the bank is at risk of evolving into six regional banks”. With the growing financial strength, institutional capacity and dynamism, and the apparently greater legitimacy of regional development banks among their regional members, shareholders might eventually decide that consolidation of the World Bank’s operations with those of the regional development banks, in favor of the latter, is the preferred approach.

There are lots of reasons to think that this drastic step would be difficult to take politically, financially and administratively, and therefore the inertia common to the international governance architecture will also prevail in this case. However, the new World Bank president would be well advised to be prepared for the possibility of a “perfect storm” under which the idea of eviscerating the World Bank could gain some traction,. The more the bank is seen to fade away, as postulated under Option 1 above, the greater is the likelihood that Option 2 would be given serious consideration.

Option 3: “A Different World Bank” = Creating a Stronger Global Institution for the Coming Decades

Despite all the criticism and the decline in its relative role as a development finance institution in recent decades, the World Bank is still one of the strongest and most effective development institutions in a world. According to a recent independent ranking of the principal multilateral and bilateral aid institutions by the Brookings Institution and the Center for Global Development “IDA consistently ranks among the best aid agencies in each dimension of quality”.

A third, radically different option from the first two, would build on this strength and ensure that the world has an institution 10 to 20 years from now which helps the global community and individual countries to respond effectively to the many global challenges which the world will undoubtedly face: continued poverty, hunger, conflict and fragility, major infrastructure and energy needs, education and health challenges, and global warming and environmental challenges. On top of this, global financial crises will likely recur and require institutions like the World Bank to help countries provide safety nets and the structural foundations of long-term growth, as the bank has amply demonstrated since 2008. With this as a broad mandate, how could the World Bank respond under new dynamic?

First, it would change its organizational and operating modalities to take a leaf out of the book of the vertical funds, which have been so successful in tackling major development challenges in a focused and scaled-up manner. This means substantially rebalancing the internal matrix between the regional and country departments on the one hand and the technical departments on the other hand. According to the same evaluation cited above, the World Bank has tipped too far toward short-term country priorities and has failed to adequately reflect the need for long-term, dedicated sectoral engagement. The World Bank needs to fortify its reputation as an institution that can muster the strongest technical expertise, fielding team with broad global experience and with first rate regional and country perspective. This does not imply that the World Bank would abandon its engagement at the country level, but it means that it would systematically support the pursuit of long-term sectoral and sub-sectoral strategies at the country level, linked to regional and global initiatives, and involving private-public partnership to assure that development challenges are addressed at scale and in a sustained manner.

Second, recognizing that all countries have unmet needs for which they need long-term finance and best practice in areas such as infrastructure, energy, climate change and environment, the World Bank could become a truly global development institution by opening up its funding windows to all countries, not just an arbitrarily defined subset of developing countries. This would require substantially revising the current graduation rules and possibly the financial instruments. This would mean that the World Bank becomes the global equivalent of the European Investment Bank (EIB) and of the German Kreditanstalt fuer Wiederaufbau (KfW)—development banks that have successfully supported the infrastructure development of the more advanced countries.

Third, the World Bank would focus its own knowledge management activities and support for research and development in developing countries much more on a search for effective and scalable solutions, linked closely to its operational engagement which would be specifically designed to support the scaling up of tested innovations, along the lines pioneered by the Bill and Melinda Gates Foundation.

Fourth, for those countries with strong project management capacities, the World Bank would dramatically simplify its lending processes, following the example of the EIB. This would make it a much more efficient operational institution, making it a more attractive partner to its borrowing member countries, especially the emerging market economies.

Fifth, the membership of the World Bank would fix some fundamental problems with its financial structure and governance. It would invite the emerging market economies to make significantly larger contributions to its capital base in line with their much-enhanced economic and financial capacities. It would revamp the bank’s voting and voice rules to reflect the changed global economic weights and financial contributions of emerging markets. The bank would also explore, based on the experience of the vertical funds, tapping the resources of non-official partners, such as foundations and the private sector as part of its capital and contribution base. Of course, this would bring with it further significant changes in the governance of the World Bank. And the bank would move swiftly to a transparent selection of its leadership on the basis of merit without reference to nationality.

Conclusion: The New World Bank President Needs to Work with the G-20 Leaders to Chart a Course Forward
 
The new president will have to make a choice between these three options. Undoubtedly, the easiest choice is “business-as-usual”, perhaps embellished with some marginal changes that reflect the perspective and new insights that an outsider will bring. There is no doubt that the forces of institutional and political inertia tend to prevent dramatic change. However, it is also possible that Dr. Kim, with his background in a relatively narrow sectoral area may recognize the need for a more vertical approach in the bank’s organizational and operational model. Therefore, he may be more inclined than others to explore Option 3.

If he pursues Option 3, Dr. Kim will need a lot of help. The best place to look for help might be the G-20 leadership. One could hope that at least some of the leaders of the G-20 understand that Options 1 and 2 are not in the interest of their countries and the international community. Hopefully, they would be willing to push their peers to contemplate some radical changes in the multilateral development architecture. This might involve the setting up of a high-level commission as recently recommended by this author, which would review the future of the World Bank as part of a broader approach to rationalize the multilateral system in the interest of greater efficiency and effectiveness. But in setting up such a commission, the G-20 should state a clear objective, namely that the World Bank, perhaps the strongest existing global development institution, should not be gutted or gradually starved out of existence. Instead, it needs to be remade into a focused, effective and truly global institution. If Dr. Kim embraces this vision and develops actionable ideas for the commission and the G-20 leaders to consider and support, then he may bring the right medicine for an ailing giant.

Editor's Note: The "Scaling up in Agriculture, Rural Development and Nutrition" publication is a series of 20 briefs published by the International Food Policy Research Institute. To read the full publication, click here.

Taking successful development interventions to scale is critical if the world is to achieve the Millennium Development Goals and make essential gains in the fight for improved agricultural productivity, rural incomes, and nutrition. How to support scaling up in these three areas, however, is a major challenge. This collection of policy briefs is designed to contribute to a better understanding of the experience to date and the lessons for the future.

Scaling up means expanding, replicating, adapting, and sustaining successful policies, programs, or projects to reach a greater number of people; it is part of a broader process of innovation and learning. A new idea, model, or approach is typically embodied in a pilot project of limited impact; with monitoring and evaluation, the knowledge acquired from the pilot experience can be used to scale up the model to create larger impacts. The process generally occurs in an iterative and interactive cycle, as the experience from scaling up feeds back into new ideas and learning.

The authors of the 20 policy briefs included here explore the experience of scaling up successful interventions in agriculture, rural development, and nutrition under five broad headings: (1) the role of rural community engagement, (2) the importance of value chains, (3) the intricacies of scaling up nutrition interventions, (4) the lessons learned from institutional approaches, and (5) the experience of international aid donors.

There is no blueprint for when and how to take an intervention to scale, but the examples and experiences described in this series of policy briefs offer important insights into how to address the key global issues of agricultural productivity, food insecurity, and rural poverty.

Downloads

• Sample Chapter
• Table of Contents

• {9ABF977A-E4A6-41C8-B030-0FD655E07DBF}, 978-0-8157-2419-3, $29.95 Add to Cart

Editor's Note: Johannes Linn discusses the potential of multilateral development banks in the latest G-20 Research Group briefing book on the St. Petersburg G-20 Summit. Read the full collection here.

The origins of the multilateral development banks (MDBs) lie with the creation of the World Bank at Bretton Woods in 1944. Its initial purpose, as the International Bank for Reconstruction and Development, was the reconstruction of wartorn countries after the Second World War. 

As Europe and Japan recovered in the 1950s, the World Bank turned to providing financial assistance to the developing world. Then came the foundation of the InterAmerican Development Bank (IADB) in 1959, of the African Development Bank (AfDB) in 1964 and of the Asian Development Bank (ADB) in 1966, each to assist the development of countries in their respective regions. The European Bank for Reconstruction and Development (EBRD) was set up in 1991, following the collapse of the Soviet Union, to assist with the transition of countries in the former Soviet sphere. 

The MDBs are thus rooted in two key aspects of the geopolitical reality of the postwar 20th century: the Cold War between capitalist ‘West’ and communist ‘East’, and the division of the world into the industrial ‘North’ and the developing ‘South’. The former aspect was mirrored in the MDBs for many years by the absence of countries from the Eastern Bloc. This was only remedied after the fall of the Bamboo and Iron curtains. The latter aspect remains deeply embedded even today in the mandate, financing pattern and governance structures of the MDBs. 

Changing global financial architecture 

From the 1950s to the 1990s, the international financial architecture consisted of only three pillars: the International Monetary Fund (IMF) and the MDBs represented the multilateral official pillar; the aid agencies of the industrial countries represented bilateral official pillar; and the commercial banks and investors from industrial countries made up the private pillar. 

Today, the picture is dramatically different. Private commercial flows vastly exceed official flows, except during global financial crises. New channels of development assistance have multiplied, as foundations and religious and non-governmental organisations rival the official assistance flows in size. 

The multilateral assistance architecture, previously dominated by the MDBs, is now a maze of multilateral development agencies, with a slew of sub-regional development banks, some exceeding the traditional MDBs in size. For example, the European Investment Bank lends more than the World Bank, and the Caja Andina de Fomento (CAF, the Latin American Development Bank) more than the IADB. There are also a number of large ‘vertical funds’ for specific purposes, such as the International Fund for Agricultural Development and the Global Fund to Fight AIDS, Tuberculosis and Malaria. There are  specialized trust funds, attached to MDBs, but often with their own governance structures.

End of the North-South divide 

Finally, the traditional North-South divide is breaking down, as emerging markets have started to close the development gap, as global poverty has dropped and as many developing countries have large domestic capacities. This means that the new power houses in the South need little financial and technical assistance and are now providing official financial and technical support to their less fortunate neighbors. China’s assistance to Africa outstrips that of the World Bank.

The future for MDBs 

In this changed environment is there a future for MDBs? Three options might be considered: 

1. Do away with the MDBs as a relic of the past. Some more radical market ideologues might argue that, if there ever was a justification for the MDBs, that time is now well past. In 2000, a US congressional commission recommended the less radical solution of shifting the World Bank’s loan business to the regional MDBs. Even if shutting down MDBs were the right option, it is highly unlikely to happen. No multilateral financial institution created after the Second World War has ever been closed. Indeed, recently the Nordic Development Fund was to be shut down, but its owners reversed their decision and it will carry on, albeit with a focus on climate change. 

2. Carry on with business as usual. Currently, MDBs are on a track that, if continued, would mean a weakened mandate, loss of clients, hollowed-out financial strength and diluted technical capacity. Given their tight focus on the fight against poverty, the MDBs will work themselves out of a job as global poverty, according to traditional metrics, is on a dramatic downward trend. 

Many middle-income country borrowers are drifting away from the MDBs, since they find other sources of finance and technical advice more attractive. These include the sub-regional development banks, which are more nimble in disbursing their loans and whose governance is not dominated by the industrial countries. These countries, now facing major long-term budget constraints, will be unable to continue supporting the growth of the MDBs’ capital base. But they are also unwilling to let the emerging market economies provide relatively more funding and acquire a greater voice in these institutions.

Finally, while the MDBs retain professional staff that represents a valuable global asset, their technical strength relative to other sources of advice – and by some measures, even their absolute strength – has been waning. 

If left unattended, this would mean that MDBs 10 years from now, while still limping along, are likely to have lost their ability to provide effective financial and technical services on a scale and with a quality that matter globally or regionally. 

3. Give the MDBs a new mandate, new governance and new financing. If one starts from the proposition that a globalised 21st-century world needs capable global institutions that can provide long-term finance to meet critical physical and social infrastructure needs regionally and globally, and that can serve as critical knowledge hubs in an increasingly interconnected world, then it would be folly to let the currently still considerable institutional and financial strengths of the MDBs wither away.

• The mandate of the MDBs should be adapted to move beyond preoccupation with poverty eradication to focus explicitly on global and regional public goods as a way to help sustain global economic growth and human welfare. Moreover, the MDBs should be able to provide assistance to all their members, not only developing country members. 

• The governance of the MDBs should be changed to give the South a voice commensurate with the greater global role it now plays in economic and political terms. MDB leaders should be selected on merit without consideration of nationality. 

• The financing structure should be matched to give more space to capital contributions from the South and to significantly expand the MDBs’ capital resources in the face of the current severe capital constraints.

In addition, MDB management should be guided by banks’ membership to streamline their operational practices in line with those widely used by sub-regional development banks, and they should be supported in preserving and, where possible, strengthening their professional capacity so that they can serve as international knowledge hubs. 

A new MDB agenda for the G20 

The G20 has taken on a vast development agenda. This is fine, but it risks getting bogged down in the minutiae of development policy design and implementation that go far beyond what global leaders can and should deal with. What is missing is a serious preoccupation of the G20 with that issue on which it is uniquely well equipped to lead: reform of the global financial institutional architecture. 

What better place than to start with than the MDBs? The G20 should review the trends, strengths and weaknesses of MDBs in recent decades and endeavour to create new mandates, governance and financing structures that make them serve as effective pillars of the global institutional system in the 21st century. If done correctly, this would also mean no more need for new institutions, such as the BRICS development bank currently being created by Brazil, Russia, India, China and South Africa. It would be far better to fix the existing institutions than to create new ones that mostly add to the already overwhelming fragmentation of the global institutional system.

Event Information

The year 2015 will be a milestone year, with the adoption of the Sustainable Development Goals (SDGs) and the post-2015 development agenda by world leaders in September; the Addis Ababa Accord on financing for development in July; and the conclusion of climate negotiations at COP21 in Paris in December. The draft Addis Ababa Accord, which focuses on the actions needed to attain the SDGs, highlights the key role envisaged for the multilateral development banks (MDBs) in the post-2015 agenda. Paragraph 65 of the draft accord notes: “We call on the international finance institutions to establish a process to examine the role, scale, and functioning of the multilateral and regional development finance institutions to make them more responsive to the sustainable development agenda.”          

Against this backdrop, on April 18, 2015, the Global Economy and Development program at Brookings held a private roundtable with the leaders of the MDBs and other key stakeholders to discuss the role of the MDBs in supporting the post-2015 development agenda.

The meeting focused on four questions:

1. What does the post-2015 development agenda and the ambitions of the Addis and Paris conferences imply for the MDBs?


Given the ability of the MDBs to leverage shareholder resources, they can be efficient and effective mechanisms for scaling up development cooperation, particularly with respect to the agenda on investing in people and to the financing of sustainable infrastructure. New roles, instruments and partnerships might be needed.

2. How can MDBs best take advantage of the political attention that is being paid to the various conferences in 2015?   


The World Bank and selected regional development banks have launched a series of initiatives to optimize their balance sheets, address other constraints and enhance their catalytic role in crowding in private finance. And new institutions and mechanisms are coming to the fore. But the responses are not coordinated to best take advantage of each MDB’s comparative advantage.

3. What are the key impediments to scaling up the role and engagement of the MDBs?


Views on constraints are likely to differ but discussions should cover policy dialogue, capacity building, capital, leverage, shareholder backing on volume, instruments on leverage and risk mitigation, safeguards, and governance. 

4. How should the MDBs respond to the proposal to establish a process to examine the role, scale and functioning of the multilateral and regional development finance institutions to make them more responsive to the sustainable development agenda?   


A proactive response and engagement on the part of the MDBs would facilitate a better understanding of the contribution that the MDBs can make and greater support among shareholders for a coherent and stepped-up role.

Event Materials

• Participant List
• MDB PostEvent Summary

The replenishment process for concessional resources of the multilateral development banks is broken. We have come to this conclusion after a review of the experience with recent replenishments of multilateral development funds. We also base it on first-hand observation, since one of us was responsible for the World Bank’s International Development Association (IDA) replenishment consultations 20 years ago and recently served as the external chair for the last two replenishment consultations of the International Fund for Agricultural Development (IFAD), which closely follow the common multilateral development bank (MDB) practice. As many of the banks and their donors are preparing for midterm reviews as a first step toward the next round of replenishment consultations, this is a good time to take stock and consider what needs to be done to fix the replenishment process.

So what’s the problem?

Most of all, the replenishment process does not serve its key intended function of setting overall operational strategy for the development funds and holding the institutions accountable for effectively implementing the strategy. Instead, the replenishment consultations have turned into a time-consuming and costly process in which donor representatives from their capitals get bogged down in the minutiae of institutional management that are better left to the boards of directors and the managements of the MDBs. There are other problems, including lack of adequate engagement of recipient countries in donors’ deliberations, the lack of full participation of the donors’ representatives on the boards of the institutions in the process, and inflexible governance structures that serve as a disincentive for non-traditional donors (from emerging countries and from private foundations) to contribute.

But let’s focus on the consultation process. What does it look like? Typically, donor representatives from capitals assemble every three years (or four, in the case of the Asian Development Bank) for a year-long consultation round, consisting of four two-day meetings (including the meeting devoted to the midterm review of the ongoing replenishment and to setting the agenda for the next consultation process). For these meetings, MDB staff prepare, per consultation round, some 20 substantive documents that are intended to delve into operational and institutional performance in great detail. Each consultation round produces a long list of specific commitments (around 40 commitments is not uncommon), which management is required to implement and monitor, and report on in the midterm review. In effect, however, this review covers only half the replenishment cycle, which leads to the reporting, monitoring, and accountability being limited to the delivery of committed outputs (e.g., a specific sector strategy) with little attention paid to implementation, let alone outcomes.

The process is eerily reminiscent of the much maligned “Christmas tree” approach of the World Bank’s structural adjustment loans in the 1980s and 1990s, with their detailed matrixes of conditionality; lack of strategic selectivity and country ownership; focus on inputs rather than outcomes; and lack of consideration of the borrowers’ capacity and costs of implementing the Bank-imposed measures. Ironically, the donors successfully pushed the MDBs to give up on such conditionality (without ownership of the recipient countries) in their loans, but they impose the same kind of conditionality (without full ownership of the recipient countries and institutions) on the MDBs themselves—replenishment after replenishment.

Aside from lack of selectivity, strategic focus, and ownership of the commitments, the consultation process is also burdensome and costly in terms of the MDBs’ senior management and staff time as well as time spent by ministerial staff in donor capitals, with literally thousands of management and staff hours spent on producing and reviewing documentation. And the recent innovation of having donor representatives meet between consultation rounds as working groups dealing with long-term strategic issues, while welcome in principle, has imposed further costs on the MDBs and capitals in terms of preparing documentation and meetings.

It doesn’t have to be that way. Twenty years ago the process was much simpler and less costly. Even today, recent MDB capital increases, which mobilized resources for the non-concessional windows of the MDBs, were achieved with much simpler processes, and the replenishment consultations for special purpose funds, such as the Global Fund for HIV/AIDS, tuberculosis, and malaria and for the GAVI Alliance, are more streamlined than those of the MDBs.

So what’s to be done?

We recommend the following measures to fix the replenishment consultation process:

1. Focus on a few strategic issues and reduce the number of commitments with an explicit consideration of the costs and capacity requirements they imply. Shift the balance of monitoring and accountability from delivery of outputs to implementation and outcomes.
2. Prepare no more than five documents for the consultation process: (i) a midterm review on the implementation of the previous replenishment and key issues for the future; (ii) a corporate strategy or strategy update; (iii) the substantive report on how the replenishment resources will contribute to achieve the strategy; (iv) a financial outlook and strategy document; and (v) the legal document of the replenishment resolution.
3. Reduce the number of meetings for each replenishment round to no more than three and lengthen the replenishment period from three to four years or more.
4. Use the newly established working group meetings between replenishment consultation rounds to focus on one or two long-term, strategic issues, including how to fix the replenishment process.

The initiative for such changes lies with the donor representatives in the capitals, and from our interviews with donor representatives we understand that many of them broadly share our concerns. So this is a good time—indeed it is high time!—for them to act.

Event Information

In 2015, 193 countries adopted the Sustainable Development Goals (SDGs), a new global agenda that is more ambitious than the preceding Millennium Development Goals and aims to make progress on some of the most pressing issues of our time. Goal 4, "To ensure inclusive and quality education for all, with relevant and effective learning outcomes," challenges the international education community to meet universal access plus learning by 2030. We know that access to primary schooling has scaled up rapidly over previous decades, but what can be learned from places where transformational changes in learning have occurred? What can governments, civil society, and the private sector do to more actively scale up quality learning?

On April 18-19, the Center for Universal Education (CUE) at Brookings launched "Millions Learning: Scaling Up Quality Education in Developing Countries," a comprehensive study that examines where learning has improved around the world and what factors have contributed to that process. This two-day event included two sessions. Monday, April 18 focused on the role of global actors in accelerating progress to meeting the SDGs. The second session on Tuesday, April 19 included a presentation of the Millions Learning report followed by panel discussions on the role of financing and technology in scaling education in developing countries.

 Join the conversation on Twitter #MillionsLearning

Video

• Getting millions to learn: What will it take to accelerate progress on meeting the Sustainable Development Goals?
• Scaling quality education: The launch of the Millions Learning report
• Do funders help or hinder scaling in education?
• What role can technology play in scaling education?

Audio

• Getting millions to learn: What will it take to accelerate progress on meeting the Sustainable Development Goals?

Transcript

• Uncorrected Transcript - Day 1 (.pdf)
• Uncorrected Transcript - Day 2 (.pdf)

Event Materials

• 20160418_millions_learning_transcript
• 20160419_millions_learning_transcript

To achieve the desired outcomes of the Sustainable Development Goals as well as the global targets from the Paris COP21 Climate Summit by 2030, governments will have to find ways to meet the top-down objectives with bottom-up approaches. A systematic focus on scaling up successful development interventions could serve to bridge this gap, or what’s been called the “missing middle.” However, the question remains how to actually address the challenge of scaling up.

When Arna Hartmann, adjunct professor of international development, and I first looked at the scaling up agenda in development work in the mid-2000s, we concluded that development agencies were insufficiently focused on supporting the scaling up of successful development interventions. The pervasive focus on one-off projects all too often resulted in what I’ve come to refer to as “pilots to nowhere.” As a first step to fix this, we recommended that each aid organization carry out a review to be sure to focus effectively on scaling up. 

The institutional dimension is critical, given their role in developing and implementing scaling up pathways. Of course, individuals serve as champions, designers, and implementers, but experience illustrates that if individuals lack a strong link to a supportive institution, scaling up is most likely to be short-lived and unsustainable. “Institutions” include many different types of organizations, such as government ministries and departments, private firms and social enterprises, civil society organizations, and both public and private external donors and financiers.

The Brookings book “Getting to Scale: How to Bring Development Solutions to Millions of Poor People” explores the opportunities and challenges that such organizations face, on their own or, better yet, partnering with each other, in scaling up the development impact of their successful interventions.

Over the past decade I have worked with 10 foreign aid institutions—multilateral and bilateral agencies, as well as big global non-governmental organizations—helping them to focus systematically on scaling up operational work and developing approaches to do so. There are common lessons that apply across the board to these agencies, with one salutary example being the International Fund for Agricultural Development (IFAD) which has tackled the scaling up agenda systematically and persistently.

Following are eight takeaway lessons I gleaned from my work with IFAD:

1. Look into the “black box” of institutions. It is not enough to decide that an institution should focus on and support scaling up of successful development interventions. You actually need to look at how institutions function in terms of their mission statement and corporate strategy, their policies and processes, their operational instruments, their budgets, management and staff incentives, and their monitoring and evaluation practices. Check out the Brookings working paper that summarizes the results of a scaling up review of the IFAD.
2. Scaling needs to be pursued institution-wide. Tasking one unit in an organization with innovation and scaling up, or creating special outside entities (like the Global Innovation Fund set up jointly by a number of donor agencies) is a good first step. But ultimately, a comprehensive approach must be mainstreamed so that all operational activities are geared toward scaling up.
3. Scaling up must be championed from the top. The governing boards and leadership of the institutions need to commit to scaling up and persistently stay on message, since, like any fundamental institutional change, effectively scaling up takes time, perhaps a decade or more as with IFAD.
4. The scaling up process must be grown within the institution. External analysis and advice from consultants can play an important role in institutional reviews. But for lasting institutional change, the leadership must come from within and involve broad participation from managers and staff in developing operational policies and processes that are tailored to an institution’s specific culture, tasks, and organizational structure.
5. A well-articulated operational approach for scaling up needs to be put in place. For more on this, take a look at a recent paper by Larry Cooley and I that reviews two helpful operational approaches, which are also covered in Cooley’s blog. For the education sector, the Center for Universal Education at Brookings just published its report “Millions Learning,” which provides a useful scaling up approach specifically tailored to the education sector.
6. Operational staffs need to receive practical guidance and training. It is not enough to tell staff that they have to focus on scaling up and then give them a general framework. They also need practical guidance and training, ideally tailored to the specific business lines they are engaged in. IFAD, for example, developed overall operational guidelines for scaling up, as well as guidance notes for specific area of engagement, including livestock development, agricultural value chains, land tenure security, etc.  This guidance and training ideally should also be extended to consultants working with the agency on project preparation, implementation, and evaluation, as well as to the agency’s local counterpart organizations.
7. New approaches to monitoring and evaluation (M&E) have to be crafted. Typically the M&E for development projects is backward looking and focused on accountability, narrow issues of implementation, and short-term results. Scaling up requires continuous learning, structured experimentation, and innovation based on evidence, including whether the enabling conditions for scaling up are being established. And it is important to monitor and evaluate the institutional mainstreaming process of scaling up to ensure that it is effectively pursued. I’d recommend looking at how the German Agency for International Development (GIZ) carried out a corporate-wide evaluation of its scaling up experience.
8. Scaling up helps aid organizations mobilize financial resources. Scaling up leverages limited institutional resources in two ways: First, an organization can multiply the impact of its own financial capacity by linking up with public and private agencies and building multi-stakeholder coalitions in support of scaling up. Second, when an organization demonstrates that it is pursuing not only one-off results but also scaled up impact, funders or shareholders of the organization tend to be more motivated to support the organization. This certainly was one of the drivers of IFAD’s successful financial replenishment consultation rounds over the last decade.

By adopting these lessons, development organizations can actually begin to scale up to the level necessary to bridge the missing middle. The key will be to assure that a focus on scaling up is not the exception but instead becomes ingrained in the institutional DNA. Simply put, in designing and implementing development programs and projects, the question needs to be answered, “What’s next, if this intervention works?”

I had been mulling a rebuttal to my colleague and friend Jon Rauch’s interesting—but wrong—new Brookings paper praising the role of “hacks, machines, big money, and back room deals” in democracy. I thought the indictments of Chris Christie’s associates last week provided a perfect example of the dangers of all of that, and so of why Jon was incorrect. But in yesterday’s L.A. Times, he beat me to it, himself defending the political morality (if not the efficacy) of their actions, and in the process delivering a knockout blow to his own position.

Bridgegate is a perfect example of why we need fewer "hacks, machines, big money, and back room deals" in our politics, not more. There is no justification whatsoever for government officials abusing their powers, stopping emergency vehicles and risking lives, making kids late for school and parents late for their jobs to retaliate against a mayor who withholds an election endorsement. We vote in our democracy to make government work, not break. We expect that officials will serve the public, not their personal interests. This conduct weakens our democracy, not strengthens it.

It is also incorrect that, as Jon suggests, reformers and transparency advocates are, in part, to blame for the gridlock that sometimes afflicts our American government at every level. As my co-authors and I demonstrated at some length in our recent Brookings paper, “Why Critics of Transparency Are Wrong,” and in our follow-up Op-Ed in the Washington Post, reform and transparency efforts are no more responsible for the current dysfunction in our democracy than they were for the gridlock in Fort Lee. Indeed, in both cases, “hacks, machines, big money, and back room deals” are a major cause of the dysfunction. The vicious cycle of special interests, campaign contributions and secrecy too often freeze our system into stasis, both on a grand scale, when special interests block needed legislation, and on a petty scale, as in Fort Lee. The power of megadonors has, for example, made dysfunction within the House Republican Caucus worse, not better.

Others will undoubtedly address Jon’s new paper at length. But one other point is worth noting now. As in foreign policy discussions, I don’t think Jon’s position merits the mantle of political “realism,” as if those who want democracy to be more democratic and less corrupt are fluffy-headed dreamers. It is the reformers who are the true realists. My co-authors and I in our paper stressed the importance of striking realistic, hard-headed balances, e.g. in discussing our non-absolutist approach to transparency; alas, Jon gives that the back of his hand, acknowledging our approach but discarding the substance to criticize our rhetoric as “radiat[ing] uncompromising moralism.” As Bridgegate shows, the reform movement’s “moralism" correctly recognizes the corrupting nature of power, and accordingly advocates reasonable checks and balances. That is what I call realism. So I will race Jon to the trademark office for who really deserves the title of realist!

I this week had the pleasure of doing a podcast debate with my Brookings colleague Jonathan Rauch on the question of whether we need stronger machines and weaker transparency in American government, or the opposite. Guess which side I took! This has been a long-running water cooler and cafeteria discussion between Jon and myself since I arrived at Brookings almost a year ago. While we find some areas of agreement in the podcast (more than you might think),I remain unconvinced by the so-called “political realist” school that Jonathan is a leader of. As I have previously written and blogged (here, here and here), I think the realists are fantasists, disconnected from the actual reality of politics, including its risks. We need more transparency, not less to deal with, for example, things like corruption risk, particularly in the post-Citizens United era. Indeed, that decision itself embraces the value of a vigorous transparency regime when other safeguards are relaxed. My belief is that Washington works both more efficiently and more ethically under the scrutinizing gaze of the American media, ngo's and public. As former White House ethics czar, I often facilitated administration openness efforts, including as a means of accountability, for example helping put the White House visitor logs online.

Jon and my lively debate covers not only issues of transparency itself but also applies them to other current topics—the Affordable Care Act, Trade Promotion Authority, and much more. The debate was silently moderated by our colleague Ben Wittes as part of his “Chess Clock Debates” series. With only ten minutes on the chess clock each to make our points, it was a concise discussion that hit the fundamentals briskly. Thanks to Ben for inviting us and giving us a public forum to discuss this critical policy issue.

Editor's Note: : We’re hosting a conversation on Markaz on the Iranian nuclear talks, debating the merits of a deal, as well as the broader issues at stake for the United States and the region. This piece originally appeared in The Huffington Post.

As we near what may be the endgame of the current negotiations with Iran, I am reminded of the place where President Obama announced the overarching strategy that helped produce this moment: Prague. After stating his readiness to speak to Iran in a Democratic primary debate in 2007, and following that up postelection in 2009 with a series of initial statements directed to the Iranians, the president chose the Czech capital to lay out his vision of dealing with the dangers of nuclear weapons in April 2009. That included emphasizing that Iran would not be permitted to obtain a nuclear weapon on his watch: "Rules must be binding. Violations must be punished. Words must mean something. The world must stand together to prevent the spread of these weapons."

As a result of that 2009 speech, the president's nuclear strategy became known as the Prague Agenda. I had the privilege to travel with President Obama back to Prague in April 2010 to witness the signing of a major accomplishment in another area under the Prague Agenda, namely the New START treaty. By the following year, April 2011, I was in Prague as U.S. ambassador. That year, and in the each year that followed, we held an annual Prague Agenda conference to assess the steps that had been taken and the challenges that lay ahead.

In the years since, there has been steady progress in each of the four main areas the president laid out on that spring day in Prague in 2009. New START was a step forward on his first objective, to reduce the risks posed by existing nuclear weapons. Another goal, preventing nuclear terror by safeguarding materials and improving safety, has since been the subject of a series of successful Nuclear Security Summits in Washington, Seoul, and The Hague.

I saw first-hand the president's personal commitment to a third objective articulated in Prague: to promote the peaceful use of nuclear energy. I was, for example, present in the Oval Office in October 2011 when the president and the Czech prime minister met. President Obama advocated for the use of civil nuclear power as a part of the Czech energy mix (and also to achieve energy independence from Russia). That approach has been replicated in administration policy supporting civil nuclear energy in the United States and around the world.

Now, with the possible Iran deal, progress under the Prague Agenda's final prong is in reach: holding to account a state which had violated its nuclear obligations under international treaties. I am not of the school that believes the president needs to secure an Iran deal to build his legacy. That was never the case; having known him for almost a quarter of a century, since we were law students together, and having worked for him for six years, first in the White House and then as ambassador, I can attest that those kinds of considerations do not enter into critical decisions like this one. Even the president's strongest critics have to admit that legacy is, as a matter of logic, much less of a consideration after the recent breakthroughs on the Affordable Care Act and on Trade Promotion Authority.

Instead, as the comprehensive nature of the Prague Agenda itself suggests, President Obama is pursuing a deal out of principle. He is acting from his conviction that a good agreement with Iran represents another step toward making the U.S., our allies, and the world safe from nuclear terror. It is that ambition that has driven the president's formulation and consistent pursuit of each of the four elements of the Prague Agenda, the obscure aspects just as much as the headline-making ones.

Of course, as the president himself has repeatedly emphasized, the deal must be a good one. That is why I recently joined a bipartisan group of experts convened by the Washington Institute for Near East Policy in signing a statement laying out criteria for what any deal with Iran must at a minimum contain in five core areas: monitoring and verification; possible military dimensions; advanced centrifuges; sanctions relief; and consequences of violations. We also agreed on the importance of complementing any agreement with a strong deterrence policy and a comprehensive regional strategy. I have been encouraged by the warm reception for our statement from all corners, and by the strong tone struck by the American negotiators in Vienna this week. They recognize that willingness to walk away is the surest path to securing a good deal. If such a deal can be struck that meets the criteria in our bipartisan statement, that will be another stride forward under the Prague Agenda — perhaps the biggest yet.

I had the pleasure earlier this month of welcoming my friend, Czech Republic Foreign Minister Lubomir Zaoralek, here to Brookings for a discussion of critical issues confronting the Europe-U.S. alliance. Foreign Minister Zaoralek was appointed to his current position in January 2014 after serving as a leading figure in the Czech Parliament for many years. He was accompanied by a distinguished delegation that included Dr. Petr Drulak of the Foreign Ministry, and Czech Ambassador Petr Gandalovic. I was fortunate enough to be joined in the discussion by colleagues from Brookings including Fiona Hill, Shadi Hamid, Steve Pifer, and others, as well as representatives of other D.C. think tanks. Our discussion spanned the globe, from how to respond to the Syrian conflict, to addressing Russia’s conduct in Ukraine, to the thaw in U.S.-Cuba relations, to dealing with the refugee crisis in Europe. The conversation was so fascinating that the sixty minutes we had allotted flew by and we ended up talking for two hours—and we still just scratched the surface.

Amb. Eisen and FM Zaoralek, October 2, 2015

Yesterday, we had a visit from Czech State Secretary Tomas Prouza, accompanied by Ambassador Martin Povejsil, the Czech Permanent Envoy to the EU. We also talked about world affairs. In this case, that included perhaps the most important governance matter now confronting the U.S.: the exceptionally entertaining (if not enlightening) presidential primary season. I expressed my opinion that Vice President Biden would not enter the race, only to have him prove me right in his Rose Garden remarks a few hours later. If only all my predictions came true (and as quickly). We at Brookings benefited greatly from the insights of both of these October delegations, and we look forward to welcoming many more from every part of the Czech political spectrum in the months ahead.

Prouza, Eisen, Povejsil, October 21, 2015

Event Information

This program was aired live on CSPAN.org » 

With the demand for its services constantly evolving, the Department of Veterans Affairs (VA) faces complex challenges in providing accessible care to America’s veterans. Amidst a history of long patient wait times, cost overruns, and management concerns, the VA recently conducted a sweeping internal review of its operations.  The result was the new MyVA program.

How will MyVA improve the VA’s care of veterans? What will it do restore public confidence in its efforts? What changes is the VA undergoing to address both internal concerns and modern challenges in veteran care? 

On June 20, Governance Studies at Brookings hosted VA Secretary Robert McDonald. Secretary McDonald described the VA’s transformation strategy and explained how the reforms within MyVA will impact veterans, taxpayers and other stakeholders. He addressed lessons learned not just for the VA but for all government agencies that strive to achieve transformation and improve service delivery.

This event was broadcast live on C-SPAN.

Join the conversation on Twitter at #VASec and @BrookingsGov

Audio

• Can the Department of Veterans Affairs be modernized?

Transcript

• Transcript (.pdf)

Event Materials

• 20160620_veterans_affairs_mcdonald_transcript

How has the Joint Comprehensive Plan of Action (JCPOA)—signed between the P5+1 and Iran one year ago—played out in practice? Several Brookings scholars, many of whom participated prominently in debates last year surrounding official congressional review, offered their views.

Editor's Note: The introduction to the 2015 Brown Center Report on American Education appears below. Use the Table of Contents to navigate through the report online, or download a PDF of the full report.

TABLE OF CONTENTS

Part I: Girls, Boys, and Reading

Part II: Measuring Effects of the Common Core

Part III: Student Engagement

INTRODUCTION

The 2015 Brown Center Report (BCR) represents the 14^th edition of the series since the first issue was published in 2000.  It includes three studies.  Like all previous BCRs, the studies explore independent topics but share two characteristics: they are empirical and based on the best evidence available.  The studies in this edition are on the gender gap in reading, the impact of the Common Core State Standards -- English Language Arts on reading achievement, and student engagement.

Part one examines the gender gap in reading.  Girls outscore boys on practically every reading test given to a large population.  And they have for a long time.  A 1942 Iowa study found girls performing better than boys on tests of reading comprehension, vocabulary, and basic language skills.  Girls have outscored boys on every reading test ever given by the National Assessment of Educational Progress (NAEP)—the first long term trend test was administered in 1971—at ages nine, 13, and 17.  The gap is not confined to the U.S.  Reading tests administered as part of the Progress in International Reading Literacy Study (PIRLS) and the Program for International Student Assessment (PISA) reveal that the gender gap is a worldwide phenomenon.  In more than sixty countries participating in the two assessments, girls are better readers than boys. 

Perhaps the most surprising finding is that Finland, celebrated for its extraordinary performance on PISA for over a decade, can take pride in its high standing on the PISA reading test solely because of the performance of that nation’s young women.  With its 62 point gap, Finland has the largest gender gap of any PISA participant, with girls scoring 556 and boys scoring 494 points (the OECD average is 496, with a standard deviation of 94).   If Finland were only a nation of young men, its PISA ranking would be mediocre.

Part two is about reading achievement, too. More specifically, it’s about reading and the English Language Arts standards of the Common Core (CCSS-ELA).  It’s also about an important decision that policy analysts must make when evaluating public policies—the determination of when a policy begins. How can CCSS be properly evaluated? 

Two different indexes of CCSS-ELA implementation are presented, one based on 2011 data and the other on data collected in 2013.  In both years, state education officials were surveyed about their Common Core implementation efforts.  Because forty-six states originally signed on to the CCSS-ELA—and with at least forty still on track for full implementation by 2016—little variability exists among the states in terms of standards policy.  Of course, the four states that never adopted CCSS-ELA can serve as a small control group.  But variation is also found in how the states are implementing CCSS.  Some states are pursuing an array of activities and aiming for full implementation earlier rather than later.  Others have a narrow, targeted implementation strategy and are proceeding more slowly. 

The analysis investigates whether CCSS-ELA implementation is related to 2009-2013 gains on the fourth grade NAEP reading test.  The analysis cannot verify causal relationships between the two variables, only correlations.  States that have aggressively implemented CCSS-ELA (referred to as “strong” implementers in the study) evidence a one to one and one-half point larger gain on the NAEP scale compared to non-adopters of the standards.  This association is similar in magnitude to an advantage found in a study of eighth grade math achievement in last year’s BCR.  Although positive, these effects are quite small.  When the 2015 NAEP results are released this winter, it will be important for the fate of the Common Core project to see if strong implementers of the CCSS-ELA can maintain their momentum.

Part three is on student engagement.  PISA tests fifteen-year-olds on three subjects—reading, math, and science—every three years.  It also collects a wealth of background information from students, including their attitudes toward school and learning.  When the 2012 PISA results were released, PISA analysts published an accompanying volume, Ready to Learn: Students’ Engagement, Drive, and Self-Beliefs, exploring topics related to student engagement.

Part three provides secondary analysis of several dimensions of engagement found in the PISA report.  Intrinsic motivation, the internal rewards that encourage students to learn, is an important component of student engagement.  National scores on PISA’s index of intrinsic motivation to learn mathematics are compared to national PISA math scores.  Surprisingly, the relationship is negative.  Countries with highly motivated kids tend to score lower on the math test; conversely, higher-scoring nations tend to have less-motivated kids. 

The same is true for responses to the statements, “I do mathematics because I enjoy it,” and “I look forward to my mathematics lessons.”  Countries with students who say that they enjoy math or look forward to their math lessons tend to score lower on the PISA math test compared to countries where students respond negatively to the statements.  These counterintuitive finding may be influenced by how terms such as “enjoy” and “looking forward” are interpreted in different cultures.  Within-country analyses address that problem.  The correlation coefficients for within-country, student-level associations of achievement and other components of engagement run in the anticipated direction—they are positive.  But they are also modest in size, with correlation coefficients of 0.20 or less. 

Policymakers are interested in questions requiring analysis of aggregated data—at the national level, that means between-country data.  When countries increase their students’ intrinsic motivation to learn math, is there a concomitant increase in PISA math scores?  Data from 2003 to 2012 are examined.  Seventeen countries managed to increase student motivation, but their PISA math scores fell an average of 3.7 scale score points.  Fourteen countries showed no change on the index of intrinsic motivation—and their PISA scores also evidenced little change.  Eight countries witnessed a decline in intrinsic motivation.  Inexplicably, their PISA math scores increased by an average of 10.3 scale score points.  Motivation down, achievement up.

Correlation is not causation.  Moreover, the absence of a positive correlation—or in this case, the presence of a negative correlation—is not refutation of a possible positive relationship.  The lesson here is not that policymakers should adopt the most effective way of stamping out student motivation.  The lesson is that the level of analysis matters when analyzing achievement data.  Policy reports must be read warily—especially those freely offering policy recommendations.  Beware of analyses that exclusively rely on within- or between-country test data without making any attempt to reconcile discrepancies at other levels of analysis.  Those analysts could be cherry-picking the data.  Also, consumers of education research should grant more credence to approaches modeling change over time (as in difference in difference models) than to cross-sectional analyses that only explore statistical relationships at a single point in time. 

Downloads

• Download the report

Part III of the 2015 Brown Center Report on American Education

Student engagement refers to the intensity with which students apply themselves to learning in school.  Traits such as motivation, enjoyment, and curiosity—characteristics that have interested researchers for a long time—have been joined recently by new terms such as, “grit,” which now approaches cliché status.  International assessments collect data from students on characteristics related to engagement.  This study looks at data from the Program for International Student Assessment (PISA), an international test given to fifteen-year-olds.  In the U.S., most PISA students are in the fall of their sophomore year.  The high school years are a time when many observers worry that students lose interest in school.

Compared to their peers around the world, how do U.S. students appear on measures of engagement?  Are national indicators of engagement related to achievement?  This analysis concludes that American students are about average in terms of engagement.  Data reveal that several countries noted for their superior ranking on PISA—e.g., Korea, Japan, Finland, Poland, and the Netherlands—score below the U.S. on measures of student engagement.  Thus, the relationship of achievement to student engagement is not clear cut, with some evidence pointing toward a weak positive relationship and other evidence indicating a modest negative relationship. 

Education studies differ in units of analysis.  Some studies report data on individuals, with each student serving as an observation.  Studies of new reading or math programs, for example, usually report an average gain score or effect size representing the impact of the program on the average student.  Others studies report aggregated data, in which test scores or other measurements are averaged to yield a group score. Test scores of schools, districts, states, or countries are constructed like that.  These scores represent the performance of groups, with each group serving as a single observation, but they are really just data from individuals that have been aggregated to the group level.

Aggregated units are particularly useful for policy analysts.  Analysts are interested in how Fairfax County or the state of Virginia or the United States is doing.  Governmental bodies govern those jurisdictions and policymakers craft policy for all of the citizens within the political jurisdiction—not for an individual.  

The analytical unit is especially important when investigating topics like student engagement and their relationships with achievement.  Those relationships are inherently individual, focusing on the interaction of psychological characteristics.  They are also prone to reverse causality, meaning that the direction of cause and effect cannot readily be determined.  Consider self-esteem and academic achievement.  Determining which one is cause and which is effect has been debated for decades.  Students who are good readers enjoy books, feel pretty good about their reading abilities, and spend more time reading than other kids.  The possibility of reverse causality is one reason that beginning statistics students learn an important rule:  correlation is not causation.

Starting with the first international assessments in the 1960s, a curious pattern has emerged. Data on students’ attitudes toward studying school subjects, when examined on a national level, often exhibit the opposite relationship with achievement than one would expect.  The 2006 Brown Center Report (BCR) investigated the phenomenon in a study of “the happiness factor” in learning.^[i]  Test scores of fourth graders in 25 countries and eighth graders in 46 countries were analyzed.  Students in countries with low math scores were more likely to report that they enjoyed math than students in high-scoring countries.  Correlation coefficients for the association of enjoyment and achievement were -0.67 at fourth grade and -0.75 at eighth grade. 

Confidence in math performance was also inversely related to achievement.  Correlation coefficients for national achievement and the percentage of students responding affirmatively to the statement, “I usually do well in mathematics,” were -0.58 among fourth graders and -0.64 among eighth graders.  Nations with the most confident math students tend to perform poorly on math tests; nations with the least confident students do quite well.   

That is odd.  What’s going on?  A comparison of Singapore and the U.S. helps unravel the puzzle.  The data in figure 3-1 are for eighth graders on the 2003 Trends in Mathematics and Science Study (TIMSS).  U.S. students were very confident—84% either agreed a lot or a little (39% + 45%) with the statement that they usually do well in mathematics.  In Singapore, the figure was 64% (46% + 18%).  With a score of 605, however, Singaporean students registered about one full standard deviation (80 points) higher on the TIMSS math test compared to the U.S. score of 504. 

When within-country data are examined, the relationship exists in the expected direction.  In Singapore, highly confident students score 642, approximately 100 points above the least-confident students (551).  In the U.S., the gap between the most- and least-confident students was also about 100 points—but at a much lower level on the TIMSS scale, at 541 and 448.  Note that the least-confident Singaporean eighth grader still outscores the most-confident American, 551 to 541.

The lesson is that the unit of analysis must be considered when examining data on students’ psychological characteristics and their relationship to achievement.  If presented with country-level associations, one should wonder what the within-country associations are.  And vice versa.  Let’s keep that caution in mind as we now turn to data on fifteen-year-olds’ intrinsic motivation and how nations scored on the 2012 PISA.

PISA’s index of intrinsic motivation to learn mathematics comprises responses to four items on the student questionnaire:  1) I enjoy reading about mathematics; 2) I look forward to my mathematics lessons; 3) I do mathematics because I enjoy it; and 4) I am interested in the things I learn in mathematics.  Figure 3-2 shows the percentage of students in OECD countries—thirty of the most economically developed nations in the world—responding that they agree or strongly agree with the statements.  A little less than one-third (30.6%) of students responded favorably to reading about math, 35.5% responded favorably to looking forward to math lessons, 38.2% reported doing math because they enjoy it, and 52.9% said they were interested in the things they learn in math.  A ballpark estimate, then, is that one-third to one-half of students respond affirmatively to the individual components of PISA’s intrinsic motivation index.

Table 3-1 presents national scores on the 2012 index of intrinsic motivation to learn mathematics.  The index is scaled with an average of 0.00 and a standard deviation of 1.00.  Student index scores are averaged to produce a national score.  The scores of 39 nations are reported—29 OECD countries and 10 partner countries.^[ii]  Indonesia appears to have the most intrinsically motivated students in the world (0.80), followed by Thailand (0.77), Mexico (0.67), and Tunisia (0.59).  It is striking that developing countries top the list.  Universal education at the elementary level is only a recent reality in these countries, and they are still struggling to deliver universally accessible high schools, especially in rural areas and especially to girls.  The students who sat for PISA may be an unusually motivated group.  They also may be deeply appreciative of having an opportunity that their parents never had.

The U.S. scores about average (0.08) on the index, statistically about the same as New Zealand, Australia, Ireland, and Canada.  The bottom of the table is extremely interesting.  Among the countries with the least intrinsically motivated kids are some PISA high flyers.  Austria has the least motivated students (-0.35), but that is not statistically significantly different from the score for the Netherlands (-0.33).  What’s surprising is that Korea (-0.20), Finland (-0.22), Japan (-0.23), and Belgium (-0.24) score at the bottom of the intrinsic motivation index even though they historically do quite well on the PISA math test.

Let’s now dig a little deeper into the intrinsic motivation index.  Two components of the index are how students respond to “I do mathematics because I enjoy it” and “I look forward to my mathematics lessons.”  These sentiments are directly related to schooling.  Whether students enjoy math or look forward to math lessons is surely influenced by factors such as teachers and curriculum.  Table 3-2 rank orders PISA countries by the percentage of students who “agree” or “strongly agree” with the questionnaire prompts.  The nations’ 2012 PISA math scores are also tabled.  Indonesia scores at the top of both rankings, with 78.3% enjoying math and 72.3% looking forward to studying the subject.  However, Indonesia’s PISA math score of 375 is more than one full standard deviation below the international mean of 494 (standard deviation of 92).  The tops of the tables are primarily dominated by low-performing countries, but not exclusively so.  Denmark is an average-performing nation that has high rankings on both sentiments.  Liechtenstein, Hong Kong-China, and Switzerland do well on the PISA math test and appear to have contented, positively-oriented students.

Several nations of interest are shaded.  The bar across the middle of the tables, encompassing Australia and Germany, demarcates the median of the two lists, with 19 countries above and 19 below that position.  The United States registers above the median on looking forward to math lessons (45.4%) and a bit below the median on enjoyment (36.6%).  A similar proportion of students in Poland—a country recently celebrated in popular media and in Amanda Ripley’s book, The Smartest Kids in the World,^[iii] for making great strides on PISA tests—enjoy math (36.1%), but only 21.3% of Polish kids look forward to their math lessons, very near the bottom of the list, anchored by Netherlands at 19.8%. 

Korea also appears in Ripley’s book.  It scores poorly on both items.  Only 30.7% of Korean students enjoy math, and less than that, 21.8%, look forward to studying the subject.  Korean education is depicted unflatteringly in Ripley’s book—as an academic pressure cooker lacking joy or purpose—so its standing here is not surprising.  But Finland is another matter.  It is portrayed as laid-back and student-centered, concerned with making students feel relaxed and engaged.  Yet, only 28.8% of Finnish students say that they study mathematics because they enjoy it (among the bottom four countries) and only 24.8% report that they look forward to math lessons (among the bottom seven countries).  Korea, the pressure cooker, and Finland, the laid-back paradise, look about the same on these dimensions.

Another country that is admired for its educational system, Japan, does not fare well on these measures.  Only 30.8% of students in Japan enjoy mathematics, despite the boisterous, enthusiastic classrooms that appear in Elizabeth Green’s recent book, Building a Better Teacher.^[iv]  Japan does better on the percentage of students looking forward to their math lessons (33.7%), but still places far below the U.S.  Green’s book describes classrooms with younger students, but even so, surveys of Japanese fourth and eighth graders’ attitudes toward studying mathematics report results similar to those presented here.  American students say that they enjoy their math classes and studying math more than students in Finland, Japan, and Korea.

It is clear from Table 3-2 that at the national level, enjoying math is not positively related to math achievement.  Nor is looking forward to one’s math lessons.  The correlation coefficients reported in the last row of the table quantify the magnitude of the inverse relationships.  The -0.58 and -0.57 coefficients indicate a moderately negative association, meaning, in plain English, that countries with students who enjoy math or look forward to math lessons tend to score below average on the PISA math test.  And high-scoring nations tend to register below average on these measures of student engagement.  Country-level associations, however, should be augmented with student-level associations that are calculated within each country.

The 2012 PISA volume on student engagement does not present within-country correlation coefficients on intrinsic motivation or its components.  But it does offer within-country correlations of math achievement with three other characteristics relevant to student engagement. Table 3-3 displays statistics for students’ responses to: 1) if they feel like they belong at school; 2) their attitudes toward school, an index composed of four factors;^[v] and 3) whether they had arrived late for school in the two weeks prior to the PISA test. These measures reflect an excellent mix of behaviors and dispositions.

The within-country correlations trend in the direction expected but they are small in magnitude.  Correlation coefficients for math performance and a sense of belonging at school range from -0.02 to 0.18, meaning that the country exhibiting the strongest relationship between achievement and a sense of belonging—Thailand, with a 0.18 correlation coefficient—isn’t registering a strong relationship at all.  The OECD average is 0.08, which is trivial.  The U.S. correlation coefficient, 0.07, is also trivial.  The relationship of achievement with attitudes toward school is slightly stronger (OECD average of 0.11), but is still weak.

Of the three characteristics, arriving late for school shows the strongest correlation, an unsurprising inverse relationship of -0.14 in OECD countries and -0.20 in the U.S.  Students who tend to be tardy also tend to score lower on math tests.  But, again, the magnitude is surprisingly small.  The coefficients are statistically significant because of large sample sizes, but in a real world “would I notice this if it were in my face?” sense, no, the correlation coefficients are suggesting not much of a relationship at all.    

The PISA report presents within-country effect sizes for the intrinsic motivation index, calculating the achievement gains associated with a one unit change in the index.  One of several interesting findings is that intrinsic motivation is more strongly associated with gains at the top of the achievement distribution, among students at the 90^th percentile in math scores, than at the bottom of the distribution, among students at the 10^th percentile. 

The report summarizes the within-country effect sizes with this statement: “On average across OECD countries, a change of one unit in the index of intrinsic motivation to learn mathematics translates into a 19 score-point difference in mathematics performance.”^[vi]  This sentence can be easily misinterpreted.  It means that within each of the participating countries students who differ by one unit on PISA’s 2012 intrinsic motivation index score about 19 points apart on the 2012 math test.  It does not mean that a country that gains one unit on the intrinsic motivation index can expect a 19 point score increase.^[vii]    

Let’s now see what that association looks like at the national level.

PISA first reported national scores on the index of intrinsic motivation to learn mathematics in 2003.  Are gains that countries made on the index associated with gains on PISA’s math test?  Table 3-4 presents a score card on the question, reporting the changes that occurred in thirty-nine nations—in both the index and math scores—from 2003 to 2012.  Seventeen nations made statistically significant gains on the index; fourteen nations had gains that were, in a statistical sense, indistinguishable from zero—labeled “no change” in the table; and eight nations experienced statistically significant declines in index scores. 

The U.S. scored 0.00 in 2003 and 0.08 in 2012, notching a gain of 0.08 on the index (statistically significant).  Its PISA math score declined from 483 to 481, a decline of 2 scale score points (not statistically significant).

Table 3-4 makes it clear that national changes on PISA’s intrinsic motivation index are not associated with changes in math achievement.  The countries registering gains on the index averaged a decline of 3.7 points on PISA’s math assessment.  The countries that remained about the same on the index had math scores that also remain essentially unchanged (-0.09) And the most striking finding: countries that declined on the index (average of -0.15) actually gained an average of 10.3 points on the PISA math scale.  Intrinsic motivation went down; math scores went up.  The correlation coefficient for the relationship over all, not shown in the table, is -0.30.

The analysis above investigated student engagement.  International data from the 2012 PISA were examined on several dimensions of student engagement, focusing on a measure that PISA has employed since 2003, the index of intrinsic motivation to learn mathematics.  The U.S. scored near the middle of the distribution on the 2012 index.  PISA analysts calculated that, on average, a one unit change in the index was associated with a 19 point gain on the PISA math test.  That is the average of within-country calculations, using student-level data that measure the association of intrinsic motivation with PISA score.  It represents an effect size of about 0.20—a positive effect, but one that is generally considered small in magnitude.^[viii]

The unit of analysis matters.  Between-country associations often differ from within-country associations.  The current study used a difference in difference approach that calculated the correlation coefficient for two variables at the national level: the change in intrinsic motivation index from 2003-2012 and change in PISA score for the same time period.  That analysis produced a correlation coefficient of -0.30, a negative relationship that is also generally considered small in magnitude.

Neither approach can justify causal claims nor address the possibility of reverse causality occurring—the possibility that high math achievement boosts intrinsic motivation to learn math, rather than, or even in addition to, high levels of motivation leading to greater learning.  Poor math achievement may cause intrinsic motivation to fall.  Taken together, the analyses lead to the conclusion that PISA provides, at best, weak evidence that raising student motivation is associated with achievement gains.  Boosting motivation may even produce declines in achievement.

Here’s the bottom line for what PISA data recommends to policymakers: Programs designed to boost student engagement—perhaps a worthy pursuit even if unrelated to achievement—should be evaluated for their effects in small scale experiments before being adopted broadly.  The international evidence does not justify wide-scale concern over current levels of student engagement in the U.S. or support the hypothesis that boosting student engagement would raise student performance nationally.

Let’s conclude by considering the advantages that national-level, difference in difference analyses provide that student-level analyses may overlook.

1. They depict policy interventions more accurately.  Policies are actions of a political unit affecting all of its members.  They do not simply affect the relationship of two characteristics within an individual’s psychology. Policymakers who ask the question, “What happens when a country boosts student engagement?” are asking about a country-level phenomenon.

2.  Direction of causality can run differently at the individual and group levels.  For example, we know that enjoying a school subject and achievement on tests of that subject are positively correlated at the individual level.  But they are not always correlated—and can in fact be negatively correlated—at the group level. 

3.  By using multiple years of panel data and calculating change over time, a difference in difference analysis controls for unobserved variable bias by “baking into the cake” those unobserved variables at the baseline.  The unobserved variables are assumed to remain stable over the time period of the analysis.  For the cultural factors that many analysts suspect influence between-nation test score differences, stability may be a safe assumption.  Difference in difference, then, would be superior to cross-sectional analyses in controlling for cultural influences that are omitted from other models.

4.  Testing artifacts from a cultural source can also be dampened.  Characteristics such as enjoyment are culturally defined, and the language employed to describe them is also culturally bounded.  Consider two of the questionnaire items examined above: whether kids “enjoy” math and how much they “look forward” to math lessons.  Cultural differences in responding to these prompts will be reflected in between-country averages at the baseline, and any subsequent changes will reflect fluctuations net of those initial differences.

Downloads

• Download the report