Swapping Human Labor for AI

Key Considerations for Using Generative AI

Generative artificial intelligence offers a wide array of capabilities: compiling meeting agendas, drafting emails, transcribing notes, and even generating code. However, a crucial question often arises: should these tasks be performed by AI? Before Swapping Human Labor for AI, read on.

John Horton, an MIT Sloan associate professor and leader of the IDE research group, who specializes in AI labor and online marketplaces, notes that effectively working with AI requires more than just knowing its functions. “It’s not a trivial task, learning how to work well with a machine,” Horton said. “There’s still the challenge of figuring out how to ask good questions or make effective requests.”

Since the release of ChatGPT last fall, a powerful AI tool for answering questions, engaging in conversation, and generating text, both businesses and consumers have been exploring its potential. The critical question for employers considering replacing human labor with AI is not whether AI can perform a task but whether integrating AI with human capabilities is worthwhile.

For a human-AI interaction to be effective, several factors need to align. Humans must pose the right questions and evaluate the AI’s responses promptly. Horton emphasizes, “Is that going to be more efficient than just having the person do the task directly?”

During the 2023 IDE Annual Conference, Horton suggested four key questions to consider when determining the suitability of AI for various tasks:

How Much Time Does the Task Require Without AI Assistance? AI is often praised for its ability to complete tasks much faster than humans. “If you have a task that takes a lot of time normally, that’s a prime candidate for AI substitution,” Horton explained. Tasks AI excels at, such as drafting meeting agendas or coding, are prevalent across many jobs. The impact of generative AI will depend on how widely applicable and critical these tasks are within labor markets.
How Highly Paid Are the People Performing This Task? AI can significantly reduce costs by automating tasks typically performed by highly paid employees. According to a McKinsey report, generative AI could add $2.6 trillion to $4.4 trillion annually to the global economy. “Replacing high-cost tasks with AI is more attractive,” Horton noted, highlighting that automation can democratize tasks currently performed by highly skilled workers.
How Capable Is the AI of Completing the Task Correctly? The effectiveness of AI is a major factor. Horton observed that while AI can now handle many complex tasks previously thought to be challenging, users still need to craft precise prompts to achieve desired outcomes. He shared an example of using ChatGPT to code in Perl, where modifying the prompt led to successful results. “You can achieve the right outcome even without knowing how to perform the task yourself,” Horton said.
How Easy Is It for Humans to Determine the Accuracy of AI Output? Quick and accurate evaluation of AI-generated results is essential. For some tasks, such as creating meeting agendas, this is straightforward. However, for others, verifying the AI’s accuracy can be challenging. Horton pointed out that tasks requiring substantial validation or testing, like coding, might not justify the use of AI if the evaluation cost is too high. “If the evaluation step is too cumbersome, then AI might not be worth the effort.”

Looking ahead, Horton envisions improvements in AI technology, not only in the models themselves but also in user interfaces that simplify prompting and result evaluation. “It’s easy to imagine these tasks becoming more integrated, making the evaluation process simpler and more automated,” Horton said. In the future, AI could potentially enhance both prompting and evaluation tasks.

For further insights, the original article was published on MIT Sloan on August 28, 2023.