The art in policy Delphi practice: critical decisions for its design and application on foresight exercises
DOI:
https://doi.org/10.26754/ojs_ais/ais.20254712353Keywords:
Foresight, Futures, Project Design, MethodsAbstract
The Delphi method was initially developed in the mid-20th century as a tool for exploring future technological developments, and its variant known as the policy Delphi was developed in 1970 to incorporate the ideas of different stakeholder groups in foresight exercises. A review of more than one hundred exercises that employed this technique up to 2025 allows for a reflection on its practical implementation and on the crucial decisions involved during its design, execution, and analysis phases.
Fourteen crucial decisions have been identified for the design, execution, as well as the analysis and interpretation of foresight exercises, which concern the balance in areas such as theoretical considerations, participation of all stakeholder groups, uncertainty management, breadth of analysis, methodological approaches, the use of collective expertise to examine policy instruments and decisions, case-based experiences, and decision-making informed by dominant patterns of thought.
Downloads
References
Andersen, P. D., & Borup, M. (2009). Foresight and strategy in national research councils and research programmes. Technology Analysis y Strategic Management, 21(8), 917–932. https://doi.org/10.1080/09537320903262280 DOI: https://doi.org/10.1080/09537320903262280
Andersen, P. D., & Rasmussen, B. (2012). Fremsyn: Metoder, praksis og erfaringer. Styrelsen for Forskning og Innovation. Forskning: Analyse og Evaluering No. 1/2012 http://www.fi.dk/publikationer/2012/ny-rapport-om-fremsyn/ny-rapport-om-fremsyn/
Armstrong, J. S. (1985). Long-Range Forecasting (2ª ed.). John Wiley & Sons.
Balthasar, D. (2024). Futuring Fragility: Embracing uncertainty, identifying opportunity, unlocking development. Development Policy Review, 42(S1). https://doi.org/10.1111/dpr.12779 DOI: https://doi.org/10.1111/dpr.12779
Beiderbeck, D., Frevel, N., von der Gracht, H. A., Schmidt, S. L., & Schweitzer, V. M. (2021). Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advances. MethodsX, 8, 101401. https://doi.org/10.1016/j.mex.2021.101.401 DOI: https://doi.org/10.1016/j.mex.2021.101401
Bell, W. (1997). Foundations of futures studies, 2 vol. Transaction Publishers.
Bloor, M., Sampson, H., & Dahlgren, K. (2015). Useful but no Oracle: Reflections on the use of a Delphi Group in a multi-methods policy research study. Qualitative Research, 15(1), 57–70. https://doi.org/10.1177/1468794113504103 DOI: https://doi.org/10.1177/1468794113504103
Collins, J., Hanlon, A., More, S. J., Wall, P. G., & Duggan, V. (2009). Policy Delphi with vignette methodology as a tool to evaluate the perception of equine welfare. Veterinary Journal, 181(1), 63–69. https://doi.org/10.1016/j.tvjl.2009.03.012 DOI: https://doi.org/10.1016/j.tvjl.2009.03.012
Cuhls, K. (2003). From forecasting to foresight processes—new participative foresight activities in Germany. Journal of forecasting, 22(2‐3), 93-111. DOI: https://doi.org/10.1002/for.848
Daheim, C., & Uerz, G. (2008). Corporate foresight in Europe: from trend-based logics to open foresight. Technology Analysis & Strategic Management, 20(3), 321–336. https://doi.org/10.1080/09537320802000047 DOI: https://doi.org/10.1080/09537320802000047
Davis-Stober, C. P., Budescu, D. V., Broomell, S. B., & Dana, J. (2015). The Composition of Optimally Wise Crowds. Decision Analysis, 12(3), 130–143. https://doi.org/10.1287/deca.2015.0315 DOI: https://doi.org/10.1287/deca.2015.0315
de Loë, R. C. (1995). Exploring complex policy questions using the policy Delphi A multi-round, interactive survey method. In Applied Geography (Vol. 15, Issue 1). DOI: https://doi.org/10.1016/0143-6228(95)91062-3
de Loë, R. C., Melnychuk, N., Murray, D., & Plummer, R. (2016). Advancing the State of Policy Delphi Practice: A Systematic Review Evaluating Methodological Evolution, Innovation, and Opportunities. Technological Forecasting and Social Change, 104, 78–88. https://doi.org/10.1016/j.techfore.2015.12.009 DOI: https://doi.org/10.1016/j.techfore.2015.12.009
Edwards, A., Hughes, G., & Lord, N. (2013). Urban security in Europe: Translating a concept in public criminology. European Journal of Criminology, 10(3), 260–283. https://doi.org/10.1177/1477370813483386 DOI: https://doi.org/10.1177/1477370813483386
Fuerth, L. S. (2009). Foresight and anticipatory governance. Foresight, 11(4), 14–32. https://doi.org/10.1108/14636680910982412 DOI: https://doi.org/10.1108/14636680910982412
Godet, M. (2010). Future memories. Technological Forecasting and Social Change, 77(9), 1457–1463. https://doi.org/10.1016/j.techfore.2010.06.008 DOI: https://doi.org/10.1016/j.techfore.2010.06.008
Gordon, T., & Pease, A. (2006). RT Delphi: An efficient,“round-less” almost real time Delphi method. Technological Forecasting and Social Change, 73(4), 321-333. https://doi.org/10.1016/j.techfore.2005.09.005 DOI: https://doi.org/10.1016/j.techfore.2005.09.005
Guimont, D., & Lapointe, D. (2016). Empowering Local Tourism Providers to Innovate through a Living Lab Process: Does Scale Matter? Technology Innovation Management Review, 6(11), 18–25. https://doi.org/10.22215/timreview/1031 DOI: https://doi.org/10.22215/timreview/1031
Havers, S. M., Martin, E., Wilson, A., & Hall, L. (2019). Implementation of government-directed policy in the hospital setting: A modified Delphi study. Health Research Policy and Systems, 17(1). https://doi.org/10.1186/s12961-019-0500-8 DOI: https://doi.org/10.1186/s12961-019-0500-8
Haynes, E., Palermo, C., & Reidlinger, D. P. (2016). Modified Policy-Delphi study for exploring obesity prevention priorities. BMJ Open. https://doi.org/10.1136/bmjopen-2016 DOI: https://doi.org/10.1136/bmjopen-2016-011788
Horton, A. (1999). A simple guide to successful foresight. Foresight, 1(1), 5-9. DOI: https://doi.org/10.1108/14636689910802052
Kadtke, J., & Wells III, L. (2014). Policy Challenges of Accelerating Technological Change: Security Policy and Strategy Implications of Parallel Scientific Revolutions (Número de Septiembre).
Kattirtzi, M., & Winskel, M. (2020). When experts disagree: Using the Policy Delphi method to analyse divergent expert expectations and preferences on UK energy futures. Technological Forecasting and Social Change, 153. https://doi.org/10.1016/j.techfore.2020.119924 DOI: https://doi.org/10.1016/j.techfore.2020.119924
Klenk, N. L., & Hickey, G. M. (2011). A virtual and anonymous, deliberative, and analytic participation process for planning and evaluation: The Concept Mapping Policy Delphi. International Journal of Forecasting, 27(1), 152–165. https://doi.org/10.1016/j.ijforecast.2010.05.002 DOI: https://doi.org/10.1016/j.ijforecast.2010.05.002
Lenz, R. C. (1962). Technological forecasting, Report ASD-TDR 62-414. Nº AD-408,085. Clearing House for Federal Scientific and Technical Literature. DOI: https://doi.org/10.21236/AD0408085
Linstone, H. A., & Turoff, M. (Eds.). (1975). The delphi method (Vol. 1975, pp. 3-12). Reading, MA: Addison-Wesley.
Linstone, H. A., & Turoff, M. (red.) (2002) The Delphi Method - Techniques and Applications, descarga en: https://www.foresight.pl/en/manuals.html?page=8 (20/08/2025)
Loughlin, K. G., & Moore, L. F. (1979). Using Delphi to achieve congruent objectives and activities in a pediatrics department. Academic Medicine, 54(2), 101–106. https://doi.org/10.1097/00001888-197902000-00006 DOI: https://doi.org/10.1097/00001888-197902000-00006
Loveridge, D. (2004). Experts and foresight: review and experience. J. Foresight and Innovation Policy, 1(2), 33–69. DOI: https://doi.org/10.1504/IJFIP.2004.004651
Martínez, R. y Durán, M. (2024) Repensando el papel de las Fuerzas Armadas españolas ante los nuevos desafíos a la seguridad. Centro de Estudios Políticos y Constitucionales.
Maxey, D., & Kezar, A. (2015). Revealing opportunities and obstacles for changing non-tenure-track faculty practices: An examination of stakeholders’ awareness of institutional contradictions. Journal of Higher Education, 86(4), 564–594. https://doi.org/10.1353/jhe.2015.0022 DOI: https://doi.org/10.1080/00221546.2015.11777375
Mehnen, N., Mose, I., & Strijker, D. (2013). The Delphi Method as a Useful Tool to Study Governance and Protected Areas? Landscape Research, 38(5), 607–624. https://doi.org/10.1080/01426397.2012.690862 DOI: https://doi.org/10.1080/01426397.2012.690862
Meskell, P., Murphy, K., Shaw, D. G., & Casey, D. (2014). Insights into the use and complexities of the Policy Delphi technique. DOI: https://doi.org/10.7748/nr2014.01.21.3.32.e342
Miles, I.; Harper, J.C., Georghiou, L.; Keenan, M.; Popper, R. (2008). “The Many Faces of Foresight”. In: The Handbook of Foresight, ed. Luke Georghiou et al., PRIME Series, 3-21. DOI: https://doi.org/10.4337/9781781008768.00010
Moon, N. W., & Baker, P. M. A. (2012). Assessing Stakeholder Perceptions of Workplace Accommodations Barriers: Results from a Policy Research Instrument. Journal of Disability Policy Studies, 23(2), 94–109. https://doi.org/10.1177/1044207311425383 DOI: https://doi.org/10.1177/1044207311425383
Nikolova, B. (2014). The rise and promise of participatory foresight. European Journal of Futures Research, 2(1). https://doi.org/10.1007/s40309-013-0033-2 DOI: https://doi.org/10.1007/s40309-013-0033-2
Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design considerations and applications. Information and Management, 42(1), 15–29. https://doi.org/10.1016/j.im.2003.11.002 DOI: https://doi.org/10.1016/j.im.2003.11.002
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., ... y Alonso-Fernández, S. (2021). Declaración PRISMA 2020: una guía actualizada para la publicación de revisiones sistemáticas. Revista española de cardiología, 74(9), 790-799. DOI: https://doi.org/10.1016/j.rec.2021.07.010
Peter, M. K., & Jarratt, D. G. (2015). The practice of foresight in long-term planning. Technological Forecasting and Social Change, 101, 49–61. https://doi.org/10.1016/j.techfore.2013.12.004 DOI: https://doi.org/10.1016/j.techfore.2013.12.004
Pereira, G. I., da Silva, P. P., & Soule, D. (2018). Policy-adaptation for a smarter and more sustainable EU electricity distribution industry: a foresight analysis. Environment, Development and Sustainability, 20, 231–267. https://doi.org/10.1007/s10668-018-0119-x DOI: https://doi.org/10.1007/s10668-018-0119-x
Popper, R. (2008). How are foresight methods selected? Foresight, 10(6), 62–89. https://doi.org/10.1108/14636680810918586 DOI: https://doi.org/10.1108/14636680810918586
Rikkonen, P., & Tapio, P. (2009). Future prospects of alternative agro-based bioenergy use in Finland-Constructing scenarios with quantitative and qualitative Delphi data. Technological Forecasting and Social Change, 76(7), 978–990. https://doi.org/10.1016/j.techfore.2008.12.001 DOI: https://doi.org/10.1016/j.techfore.2008.12.001
Rourke, F. E. (1984). Bureaucracy, Politics, and Public Policy. In The ANNALS of the American Academy of Political and Social Science (Issue 1). Little Brown. https://doi.org/10.1177/000271626938500151 DOI: https://doi.org/10.1177/000271626938500151
Rowe, G., & Wright, G. (2001). Expert Opinions in Forecasting: The Role of the Delphi Technique. In Principles of forecasting: A handbook for researchers and practitioners (pp. 125–144). https://doi.org/10.1007/978-0-306-47630-3_7 DOI: https://doi.org/10.1007/978-0-306-47630-3_7
Sardar, Z. (2010). The Namesake: Futures; futures studies; futurology; futuristic; foresight-What’s in a name? Futures, 42(3), 177–184. https://doi.org/10.1016/j.futures.2009.11.001 DOI: https://doi.org/10.1016/j.futures.2009.11.001
Shanteau, J., Weiss, D. J., Thomas, R. P., Pounds, J., & Hall, B. (2002). How can you tell if someone is an expert? Empirical assessment of expertise. Emerging Perspectives on Judgment and Decision Research, 620–641. DOI: https://doi.org/10.1017/CBO9780511609978.021
Sharples, M., Graber, R., Harrison, C., & Logan, K. (2009). E-safety and web 2.0 for children aged 11-16. Journal of Computer Assisted Learning, 25(1), 70–84. https://doi.org/10.1111/j.1365-2729.2008.00304.x DOI: https://doi.org/10.1111/j.1365-2729.2008.00304.x
Simonse, L., Simons, D., & Skalska, Z. (2023). Designing foresights by communities: A new groundbreaker role for strategic design. Design Science, 9. https://doi.org/10.1017/dsj.2023.16 DOI: https://doi.org/10.1017/dsj.2023.16
Slocum, M. S., & Lundberg, C. O. (2001). Technology Forecasting: From Emotional to Empirical. Creativity and Innovation Management, 10(2), 140–154. DOI: https://doi.org/10.1111/1467-8691.00215
Spickermann, A., Zimmermann, M., & von der Gracht, H. A. (2014). Surface- and deep-level diversity in panel selection - Exploring diversity effects on response behaviour in foresight. Technological Forecasting and Social Change, 85, 105–120. https://doi.org/10.1016/j.techfore.2013.04.009 DOI: https://doi.org/10.1016/j.techfore.2013.04.009
Surowiecki, J. (2004). The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. Doubleday.
Turoff, M. (1970). The design of a policy Delphi. Technological Forecasting and Social Change, 2(2), 149–171. https://doi.org/10.1016/0040-1625(70)90161-7 DOI: https://doi.org/10.1016/0040-1625(70)90161-7
Uehara, T., Sono, M., Tsuge, T., & Onuma, A. (2021). Can prior informed consent create virtuous cycle between biodiversity conservation and genetic resources utilization? Journal of Environmental Management, 300. https://doi.org/10.1016/j.jenvman.2021.113767 DOI: https://doi.org/10.1016/j.jenvman.2021.113767
Vicente Oliva, S., y Martínez, R. (2024). Policy Delphi como herramienta de prospectiva. En Repensando el papel de las Fuerzas Armadas españolas ante los nuevos desafíos a la seguridad (Cap. 2, pp. 71–100). Centro de Estudios Políticos y Constitucionales.
Voros, J. (2003). A generic foresight process framework. Foresight, 5(3), 10–21. https://doi.org/10.1108/14636680310698379 DOI: https://doi.org/10.1108/14636680310698379
Wallin, J. A. (2005). Bibliometric methods: Pitfalls and possibilities. Basic and Clinical Pharmacology and Toxicology, 97(5), 261–275. https://doi.org/10.1111/j.1742-7843.2005.pto_139.x DOI: https://doi.org/10.1111/j.1742-7843.2005.pto_139.x
Walpole, S. C., Mortimer, F., Inman, A., Braithwaite, I., & Thompson, T. (2015). Exploring emerging learning needs: a UK-wide consultation on environmental sustainability learning objectives for medical education. International Journal of Medical Education, 6, 191–200. https://doi.org/10.5116/ijme.5643.62cd DOI: https://doi.org/10.5116/ijme.5643.62cd
Ward, S., Borden, D. S., Kabo-Bah, A., Fatawu, A. N., & Mwinkom, X. F. (2019). Water resources data, models and decisions: International expert opinion on knowledge management for an uncertain but resilient future. Journal of Hydroinformatics, 21, 32–44. https://doi.org/10.2166/hydro.2018.104 DOI: https://doi.org/10.2166/hydro.2018.104
Williams, P. L., & Webb, C. (1994). The Delphi technique: a methodological discussion. Journal of Advanced Nursing, 19(1), 180–186. DOI: https://doi.org/10.1111/j.1365-2648.1994.tb01066.x
Wu, T., Chen, L., Hui, P., Zhang, C. J., & Li, W. (2015). Hear the whole story. Proceedings of the VLDB Endowment, 8(5), 485–496. https://doi.org/10.14778/2735479.2735482 DOI: https://doi.org/10.14778/2735479.2735482
Zupic, I., & Čater, T. (2015). Bibliometric Methods in Management and Organization. Organizational Research Methods, 18(3), 429–472. https://doi.org/10.1177/1094428114562629 DOI: https://doi.org/10.1177/1094428114562629
Published
Data Availability Statement
The data used in this research are publicly available, and the analysis performed can be accessed upon reasonable request to the author.
Issue
Section
License
Copyright (c) 2025 Silvia Vicente-Oliva
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors publishing in this journal agree to the following terms:
- Authors retain copyright and grant the journal the right to be the first publication of the work as well as licensed under a Creative Commons Attribution-NonCommercial 4.0 license that allows others to share the work with an acknowledgement of authorship and initial publication in this journal.
- Authors may separately establish additional agreements for the non-exclusive distribution of the version of the work published in the journal (e.g., placing it in an institutional repository or publishing it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are allowed and encouraged to disseminate their papers electronically (e.g., in institutional repositories or on their own website) before and during the submission process, as it can lead to productive exchanges, as well as earlier and higher citation of published papers (see The Effect of Open Access).
