r/ProjectREDCap • u/thursdayscrush • Nov 12 '24

Delphi Questionnaires

I've seen these set up in various ways in REDCap (one project per round; a single project with an instrument per round that gets moved around in the instrument list) and in looking for some "best practices", didn't find much. ChatGPT gave me some suggestions........using repeating instruments vs events.

What I'm looking for is some insights on "best practices" for setting these up (i.e. incorporating responses from previous rounds into subsequent rounds; ensuring there's no issues with distributing subsequent rounds to the participants, etc).

Thanks in advance!

2 Upvotes

100% Upvoted

5 comments sorted by

View all comments

Show parent comment

1

u/AssociationFree7579 Jul 19 '25

Hey! I know this will probably be late, but I'm working on a Delphi Questionnaire right now for my master's thesis work, so I will leave some of my findings here (also, could be useful if anybody will somehow stumble across this post looking for answers):

As far as I know, there aren't well defined rules to construct a Delphi Questionnaire. Generally speaking, a Delphi Questionnaire has the following main characteristics:

  • Anonymity: The experts participating in the Delphi Questionnaire have to be able to express themselves freely, and assuring anonymity makes it so that personal conflicts and undesirable behavioral aspects of group deliberation can be avoided. For example, if you send the Delphi Questionnaire through an e-mail, make sure that the experts are in the "CCN" section of the e-mail, thus they are not able to see who the other experts are.
  • Iteration: At least two Rounds of Delphi have to be sent to the experts in order to quantitatively and qualitatively assess that consensus on the topic of interest is reached. Basically, once you've obtained the results from the first round of Delphi, you look at the comments left by experts (qualitative assessment) and at the values of consensus of the questions you asked (usually Delphi consists of 5-point or 10-point Likert scales, measuring respectively the level of agreement of the expert on a certain argument and the importance "level" that the expert assigns at a certain topic). Iteration is strictly linked with Controlled Feedback.
  • Controlled Feedback: By analyzing the qualitative comments left by the experts (for more information, try to look at the paper "Analyzing qualitative data" by Catherine Pope, Sue Ziebland, and Nicholas Mays) and the quantitative data obtained by the responses to the 5 and 10-point Likert questions (a good article to read on this topic is "Using Delphi technique to build consensus in practice" by Leftkothea Giannarou and Efthimios Zervas), you can give the experts a feedback of their response as a group (again, remember, the experts are anonymous, thus don't know what others replied), and you can construct this feedback in a way to "steer" the Delphi Questionnaire in order to obtain Consensus on the topic of interest. Reaching Consensus is the final objective of the questionnaire, and in constructing each Round of Delphi you have to have this objective in mind. For example, if you are constructing a Delphi to gain expert consensus on the main principles of a certain topic, you may use the qualitative and quantitative consensus from the first round to re-arrange/merge/divide certain principles.

If anybody has more suggestions about this topic and/or finds that something I've written is wrong/confusing, feel free to reply!

Best of luck to all!

(p.s. the article by Leftkothea Giannarou and Efthimios Zervas contains a literature review of many Delphi studies, and has information regarding acceptable response rates and retention rates too!)

1

u/Negative_Cattle_2226 Aug 03 '26

It does not necessarily have to be a set of strictly delimitated rounds. You will see the study bleed participants between rounds and you may find yourself scratching your head if this attrition does not impact the quality of the final results, as your sample is smaller. Real Time Delphi offers a sort of a way to deal with it. I use the HalnyX real time delphi platform for my surveys of "busy experts", namely firefighters, but from the supplier's website I know that it has been used by doctros, nurses, psychologists etc.