This article discusses how digital tools can be incorporated throughout the evaluation process and provides useful information on selecting digital tools to suit your evaluation needs. Parallel to the increase in online interactions, is the explosion of mobile technology use such as smart phones, mobile applications, and other digital devices. In this new era of the digital age, researchers can no longer afford to overlook digital tools (Gilbert, Jackson, & di Gregorio, 2014). As society evolves, it is important for researchers and evaluators to evolve in the ways in which we conceptualize, collect, and analyze data. More specifically, you will learn about the following:
- The use of digital tools to support the entire process of conducting an evaluation – including data collection, data analysis, and writing
- Factors to consider when choosing to adopt different digital tools for research
The list of tools presented in this article is not intended to be exhaustive, as new digital tools are created regularly, but provide examples of how digital tools can be used throughout an evaluation.
Supporting evaluations through digital tools
Data collection
Evaluations may involve collecting data through surveys. Popular online platforms used to collect survey data include Survey Monkey and Google Docs. Google Docs affords flexibility for exporting survey responses, which can help programs streamline data collection, data entry, and data analysis. For example, I worked with a non-profit organization that previously relied on paper and pencil surveys for their pre and post assessments. By creating the participant surveys in Google Docs, we were able to link survey entries directly into an auto-populated Excel Sheet, also saved on Google Docs. The Excel Sheet was then programmed to aggregate the survey responses as new entries were received. This process saved the organization time and organized the data in a way that was easy to access and allowed the organization to review data in real-time.
You may also have program components that include social networking sites, blogs, or microblogs. Digital tools, like NVivo and DiscoverText, make it easy for organizations to extract data directly from their social media sites or other Internet pages to analyze the conversations that are taking place online as well. DiscoverText, for example, is a text analytics software that combines machine learning and user coding to extract and analyze data from online sources. The tool allows users to extract large amounts of data from the web in real-time, e.g. from Twitter, Facebook, Google, based on user defined rules. In addition to importing data into DiscoverText, users can create data sets and code data directly within the platform as well.
Figure 1 Example of Twitter data imported into DiscoverText
Figure 2 Creating a Dataset using DiscoverText
Data Analysis
The use of digital tools for data analysis is typically the first thought when mentioning the use of digital tools to conduct a research or evaluation study. Evaluations can certainly incorporate the use of digital tools to support data analyses. For example, statistical software packages, like SPSS, are commonly used to analyze survey data. In addition, qualitative data from your evaluation, like focus group and interview transcripts, can be analyzed using computer-assisted qualitative data analysis software (CAQDAS), like NVivo. Data analysis software allows users to organize large amounts of data, examine relationships between variables through statistical analysis and testing (quantitative data), develop deductive and inductive codes and themes to describe and/or interpret data (qualitative data), and create visual representations of your data, like charts, figures, concept maps, and even word clouds.
Figure 3 Word Cloud created in NVivo Plus to visualize word frequencies from interview transcript
Furthermore, if your evaluation includes collecting interview, focus groups, or other forms of audio data, you will likely need to transcribe the data before further analysis. For interviews and focus groups, several tools exist to help researchers transcribe recordings, like Trint, Express Scribe, and oTranscribe. Dragon Dictation is voice recognition app that can also assist with transcribing.
Writing
After data is collected and analyzed, digital tools can be used for writing the findings for your evaluation report as well. Scrivener, for an example, is a platform that can be used to organize and structure long documents such as reports and manuscripts. Similar to using sticky notes, Scrivener allows users to organize ideas for a document, in addition to, typing the document text directly within Scrivener. In Scrivener, users can outline and work on parts of the documents separately in smaller sections before compiling each section together into one document.
Selecting Tools
A suggestion when selecting tools for data analysis is to first consider the research and analytical tasks that you will be engaging in and the type of software needed to engage in those tasks (Gilbert et al., 2014). For example, tools for data analysis or transcription have varying capabilities and features. Cost and budget are also important factors to consider when selecting digital tools for your evaluation. There are some free digital tools, like oTranscribe, and also tools available with free trial periods. Finally, consider the time and training necessary to learn the tool and explore the extent to which tutorial services are available through the company.
As with developing other skills, application and practice are key to building competency in digital tools. Given dedicated time to learn, digital tools can be used to enhance organization and efficiency of the evaluation process.
Author: Ayanna Robinson
Ayanna Robinson has worked in the field of evaluation for seven years, evaluating programs on a range of topics including obesity, HIV, injury prevention, and nutrition. She has a Ph.D. in Health Promotion and Behavior and a Master of Public Health. She provides evaluation support for Creative Research Solutions as an evaluation subcontractor.
- Gilbert, L. S., Jackson, K., & di Gregorio, S. (2014). Tools for Analyzing Qualitative Data: The History and Relevance of Qualitative Data Analysis Software. Handbook of Research on Educational Communications & Technology, 221. Retrieved from http://proxy-+remote.galib.uga.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=edb&AN=93782184&site=eds-live