RDM Tools @ FGB: Qualtrics & Survalyzer

Comparing Qualtrics & Survalyzer

VU Amsterdam has a university-wide contract and processing agreement with Qualtrics. Any VU employee can sign into Qualtrics using their VU single sign-on details. FGB also has a faculty-wide contract and processing agreement with Survalyzer. Access to Survalyzer can be requested by contacting the Faculty Survey Administrator. For Survalyzer, there are a limited number of workspaces available to FGB, so the Survey Admin may have to reject your request for access.

Why are there two questionnaire tools?

Survalyzer was introduced originally because the faculty was able to set up a processing agreement with Survalyzer’s provider. At that time, the VU was unable to get a processing agreement in place with Qualtrics, which limited how Qualtrics could be used in the faculty, particularly for surveys collecting sensitive information. Both options now have processing agreements in place (covering the GDPR requirements), but each option still requires some enhancements to their security measures to allow for the highest risk data to be collected using only one of the options. In other words, if the content of your survey is high-risk (privacy category orange or red), it’s advised to use Survalyzer for collecting the directly identifying information (e.g. names, contact information, digital consent) and Qualtrics for collecting the other research data. This is explained further below in the section on secure use of these tools.

If your data are not high risk (privacy category yellow, green or blue), you can use either Qualtrics or Survalyzer for your entire data collection, whichever better suits your purposes. Survalyzer, however, should only be used when Qualtrics does not meet your needs since Survalyzer has a limited number of workspaces available.

To help you pick an option, you can review the technical specifications of Qualtrics & Survalyzer below:

	Qualtrics	Survalyzer
Questionnaire design options	• User friendly • Wide variety of question options/types • Very flexible • Routing is intuitively programmed • Questions and response codes can be exported as a Word doc • User response data can be exported to Excel, SPSS, CSV, TDF or XML files • Data exported as SPSS are automatically labelled and coded	• User friendly • Many question types but not as varied as Qualtrics • Not as flexible as Qualtrics • Routing is intuitively programmed • Questions and response codes can be exported as a Word doc • User response data can be exported to Excel or SPSS • Data exported into Excel include separate tabs explaining the question variable labels and response codes, as well as the filters and routing used in the questionniare • Data exported as SPSS are automatically labelled and coded
Distribution options	• Several options for survey distribution	• Fewer options for survey distribution
Data storage	• Data are stored on EU servers • Data are backed up and stored for 90 days; any data deletions are only permanent after 90 days have elapsed	• Data are stored on EU servers • Data are backed up for 90 days. When data are deleted they are flagged for 90 days and then deleted, and a back-up of this data is saved for a further 90 days, i.e. data deletions are only permanent after 180 days have elapsed

How do I use Qualtrics and/or Survalyzer securely?

While both Qualtrics and Survalyzer have processing agreements and contracts with our institution, that just covers the legal requirements. There are extra steps you should take to keep the data you collect with these tools secure, especially when collecting higher risk data (privacy category orange or red).

	Qualtrics	Survalyzer
Data Privacy	• Generally avoid using the email distribution system (i.e. where you upload a list of participant e-mails to automatically send out unique survey links)¹. You must avoid this when the data you’ll collect are privacy category orange or red. The instructions below explain an alternative approach. • Under “Survey Options” ensure that: • “Prevent Indexing” is selected • “Secure Participants’ Files” is selected • “Anonymize Response” is selected • In “Survey Flow”, add an “End of Survey” block, then go to “Customize” in this block and select “Do NOT record any personal information and remove panel association”	• Generally avoid using the email distribution system (i.e. where you upload a list of participant e-mails to automatically send out unique survey links)¹. You must avoid this when the data you’ll collect are privacy category orange or red. The instructions below explain an alternative approach.
Data Integrity	• If you are conducting a longitudinal study with multiple follow-up questionnaires, do not use the default question names; give each question a unique name • Test the survey flow before sending the questionnaire to respondents • Before sending out the survey, check the exported data for issues, such as incorrect response coding. If you are randomizing your survey blocks/questions, make sure when exporting your data to select “Export viewing order data for randomized surveys” • Download and save a copy of the questionnaire to serve as documentation of your response codes, question labels and survey routing • It is generally recommended to export the response data in a non-proprietary format such as csv. You can also export an SPSS or Excel copy if that’s easier for you to work with, but a copy of the raw data in csv format is strongly recommended	• If you are conducting a longitudinal study with multiple follow-up questionnaires, do not use the default question names; give each question a unique name • Test the survey flow before sending the questionnaire to respondents • Before sending out the survey, test the exported data for any issues, such as incorrect response coding or issues with how questions are labeled (particularly if your questionnaire has complex routing depending on the respondents’ answers) • Download and save a copy of the questionnaire to serve as documentation of your response codes, question labels and survey routing

How to send out surveys without using the built-in e-mail distrbution system?

There may be some situations (see above) where it is safer for your participants’ privacy to not use Qualtrics’ or Survalyzer’s built-in e-mail distribution system. The following instructions explain how you can bypass these systems for each questionnaire tool.

Qualtrics

Create a spreadsheet with the variable names FirstName and PrimaryEmail on the first row. Under FirstName fill in the ID numbers used to identify your study participants. In e-mail enter a fake e-mail address that follows standard e-mail conventions (e.g. email@email.com). You can fill in the same e-mail address for every row. Save your file as a .csv file (in Excel, when you “Save as” you can use the csv option; don’t worry about the formatting warnings).
In Qualtrics, click on “Contacts” on the top right part of your screen and then click “Create Contact List”.
Give the contact list a name (the name isn’t overly important, especially if you are just sending out your surveys only one time. If you are sending out repeated surveys, then make sure to have a name that makes it clear which sample of participants belongs to which round of the survey, for example by calling your sample “Round1_20200130”)
“Import from a File” is already selected, so click on “Browse” and upload the csv file you made in step 1.
Once the upload is complete you will see that ‘X’ number of “Members” are in your contact list. ‘X’ should be the number of participants you included in the csv file from step 1.
Return to your survey by clicking on “Projects” on the upper right corner of the screen and then select the “Distributions” option.
Of the options on the left-hand side of the screen, select “Personalized Links” and then click the green button on the right side that says “Generate Links”
This will open a window in which you should click on “Select Contacts”>”Use contacts from my library”, and then select the contact list you made in steps 2-5. Click on “Select entire contact list”. (You can also change when the links expire if you choose. The default is after 60 days).
Finally, click “Generate Links” and a csv of your generated survey links will automatically be downloaded.
Now you need to merge this downloaded file with a list of your participant IDs and their respective e-mail addresses, so that you have a list of participant ID numbers, e-mail addresses and the survey link that refers to the participant. Because the generated links won’t create a variable for your participant ID, you had to put the ID numbers under the variable “FirstName”, but you can now change the name of that variable in the downloaded file to the name of your participant ID variable. If you want to send out personalized messages, you can also include participant names in the final merged Excel file. You can use whatever coupling method you wish to merge the survey links to the e-mail addresses by coupling the files together based on the participant ID code; just make sure the final output is an excel file.
Before you start Mail Merge in Microsoft Word, particularly if you are working on a MacOS, make sure your default e-mail program is Microsoft Outlook. That should be the case with Windows users, but for MacOS users, open the MacOS Mail app, select Mail>Preferences and then choose Microsoft Office as your default e-mail reader. If Microsoft Word was running when you changed this default, make sure to quit and restart Microsoft Word before starting Mail Merge.
Additionally, if you are using a functional e-mail account for communication about the study, and you want the e-mail to be sent from that account and not your personal account, go to Microsoft Outlook, click on Outlook>Preferences and then change your default account to the functional e-mail account.
When you are ready to start Mail Merge open a new Word document and click on Mailings>Start Mail Merge and then select “E-mail messages”
Now click on “Select Recipients” and choose “Use an Existing List…”. This will prompt you to upload a file, so now you should upload the merged file you created with the participant ID numbers, e-mail addresses, survey links, and potentially names, if you want to have personalized e-mails. Word will warn you about opening the excel file, but you know it’s a trusted source, so continue onwards. A window will open, asking if you want a specific cell range, but you want the entire workbook, so don’t change anything in this prompt, just click “Ok”.
Ignore the prompt to “Filter Recipients” because you want to e-mail everyone on your list.
Now type the text of your message and when you want to insert information that is specific to the individual participant go to “Insert Merge Field” and select the variable you want to show up in the text. For example, if you want a personal message with the participant’s name in it, type “Dear” and then select “Insert Merge Field” and select the name variable you created. Then you will see the text written as “Dear <>”. Then you could say “Here is the link to the survey:” and insert the survey link, which will end up looking like “Here is the link to the survey: <>”.
Once you are finished with the text, click on “Finish and Merge” and select “Merge to E-mail”. When the pop-up opens, click on “To” and select the e-mail variable. Choose a subject line that you want the participants to see when they receive the e-mail and finally, decide if you want the message to just appear as text within the e-mail (default), html (doesn’t really change much for most e-mails) or as an attachment (this is not recommended because then there will be no text in the e-mail body, just an attachment that includes all the information; participants probably won’t like that).
Select “Mail Merge to Outbox”. You will receive a warning about scripts and viruses, and this will pop-up for each of the e-mails you are going to send. So click on “Don’t notifiy me about this again” and choose to continue with sending the message. Now your e-mails will be sent. Hooray!

Survalyzer

Create an excel sheet with the field names ID, Email, First Name, Last Name, Language, Cell Phone Number, Token and StudyID (or instead of StudyID, if you have a specific name for your participant ID code, such as the name of your study combined with the letters “ID”, you can use that, particularly if you use that specific variable name when referring to your participant identifier in other datasets).
Only fill in the cells beneath StudyID with all of the participant identification numbers that will need to receive an e-mail. All the other columns can remain blank; these column names only need to be included for Survalyzer to be able to read your Excel file.
Go to Survalyzer, select the “Samples” tab and then select “Create New Sample”.
Give the sample a name (the name isn’t overly important, especially if you are just sending out the surveys the one time. If you are sending out repeated surveys, then make sure to have a name that makes it clear which sample of participants belongs to which round of the survey, for example by calling your sample “Round1_20200130”)
Now select “Import from a File” and then upload the excel file you created.
It should report that “X” number of records have been created (X being however many participants to whom you plan to send a survey link).
Once the upload is complete you will end up at a page about the “Members” of your sample. The ID number shown here has been automatically generated by Survalyzer, and you can ignore it. There will be a “!” symbol under e-mail because you didn’t upload any e-mails, but that’s not a problem, since you don’t want to upload any e-mails.
Click on the “<” button at the top of the screen next to the “Members” tab, then return to your survey by selecting the “Surveys” tab.
Open your survey by clicking on it and then select the “Distribute” tab. Below “Distributor Invitations” you will see a list of distribution options if you haven’t already created a distribution method. You can also select an option by scrolling over the green button “Add Distributor” and selecting “Link List”.
You will be directed to a page to select the sample you created, e.g. based on the name we created above, you would see “Round1_20200130”; click on that name to select that sample.
Once the Link List is generated you can click on “Download Link List” and an Excel file with unique Survey links will be generated.
Now you need to merge this downloaded Excel file with a list of your participant IDs and their respective e-mail addresses, so that you have a list of participant ID numbers, e-mail addresses and the survey link that refers to the participant. If you want to send out personalized messages, you can also include participant names in the final merged Excel file. You can use whatever coupling method you wish to merge the survey links to the e-mail addresses by coupling the files together based on the participant ID code; just make sure the final output is an excel file.
Before you start Mail Merge in Microsoft Word, particularly if you are working on a MacOS, make sure your default e-mail program is Microsoft Outlook. That should be the case with Windows users, but for MacOS users, open the MacOS Mail app, select Mail>Preferences and then choose Microsoft Office as your default e-mail reader. If Microsoft Word was running when you changed this default, make sure to quit and restart Microsoft Word before starting Mail Merge.
Additionally, if you are using a functional e-mail account for communication about the study, and you want the e-mail to be sent from that account and not your personal account, go to Microsoft Outlook, click on Outlook>Preferences and then change your default account to the functional e-mail account.
When you are ready to start Mail Merge open a new Word document and click on Mailings>Start Mail Merge and then select “E-mail messages”
Now click on “Select Recipients” and choose “Use an Existing List…”. This will prompt you to upload a file, so now you should upload the merged file you created with the participant ID numbers, e-mail addresses, survey links, and potentially names, if you want to have personalized e-mails. Word will warn you about opening the excel file, but you know it’s a trusted source, so continue onwards. A window will open, asking if you want a specific cell range, but you want the entire workbook, so don’t change anything in this prompt, just click “Ok”.
Ignore the prompt to “Filter Recipients” because you want to e-mail everyone on your list.
Now type the text of your message and when you want to insert information that is specific to the individual participant go to “Insert Merge Field” and select the variable you want to show up in the text. For example, if you want a personal message with the participant’s name in it, type “Dear” and then select “Insert Merge Field” and select the name variable you created. Then you will see the text written as “Dear <>”. Then you could say “Here is the link to the survey:” and insert the survey link, which will end up looking like “Here is the link to the survey: <>”.
Once you are finished with the text, click on “Finish and Merge” and select “Merge to E-mail”. When the pop-up opens, click on “To” and select the e-mail variable. Choose a subject line that you want the participants to see when they receive the e-mail and finally, decide if you want the message to just appear as text within the e-mail (default), html (doesn’t really change much for most e-mails) or as an attachment (this is not recommended because then there will be no text in the e-mail body, just an attachment that includes all the information; participants probably won’t like that).
Select “Mail Merge to Outbox”. You will receive a warning about scripts and viruses, and this will pop-up for each of the e-mails you are going to send. So click on “Don’t notifiy me about this again” and choose to continue with sending the message. Now your e-mails will be sent. Hooray!

What do I do when I’m done my research or when I’m done collecting data with Qualtrics/Survalyzer?

If your data collection is finished, you should export all of the data you need from your survey tool and store the data on an appropriate storage location (or locations).

Neither Qualtrics nor Survalyzer should be used for data storage during your research! And they definitely shouldn’t be used as an archive for long-term storage of your data after your research is complete!!!

Once you’ve safely exported all of your data and stored it in a secure location, you must delete the data from your workspace. Lastly, if you are using Survalyzer and you no longer need your workspace, contact the Faculty Survey Administrator to let them know that your workspace can be deleted (to make space for someone else).

If your data are lower risk and you opt to collect directly identifying data (such as names, contact information and consent) simultaneously with the other research data, you should export the directly identifying data as soon as possible and delete them from Qualtrics/Survalyzer↩︎