Summarizing Long-Form Audio with Speech-to-Text and Claude: Automating Meeting Reports

Table of Contents

Background: Turning Long Audio into "Readable Information"

Audio content — economic events, internal meetings, interviews — carries a high time cost to listen through in full on the spot, and there is a constant need to grasp just the key points quickly. The subject here is Fed Chair Powell's speech at the Jackson Hole symposium on August 23, 2024. The goal was to capture, in a structured form and in as little time as possible, remarks that could shape the direction of monetary policy.

Manually transcribing and then summarizing tens of minutes of audio is not realistic. So I evaluated a setup that automates the whole flow by transcribing the full text with speech recognition and then having an LLM summarize the result.

Approach: Combining Speech-to-Text and Claude with a Clear Division of Roles

The pipeline is built by clearly dividing roles between two services with different strengths.

• Google Cloud Speech-to-Text: handles speech recognition (transcription), turning audio into text
• Claude: handles translation and summarization of the transcript

The key thing to note is that Speech-to-Text is strictly a recognition engine that turns speech into text as-is; it does not translate. English audio yields English text. Processing that involves meaning — Japanese summarization or translation — was designed to be left to Claude in the later stage.

The processing flow is as follows:

1. Feed the meeting audio into Google Cloud Speech-to-Text and obtain the full English text.
2. Pass the resulting text to Claude and instruct it to translate into Japanese and organize the key points.
3. Claude outputs a summary structured by topic.

By separating speech recognition from semantic processing, you can leverage each service's strengths while keeping the intermediate result (the raw transcript) available for verification.

Results: Extracting the Speech's Key Points in Structured Form

Claude's summary organized the speech's key points by topic, with neither gaps nor excess. The summary actually obtained was as follows.

1. Economic conditions

   • Inflation has fallen substantially and is now around 2.5%.
   • The labor market has cooled and the unemployment rate is 4.3%, still low by historical standards.
   • The economy continues to grow solidly.

2. Monetary policy outlook

   • Upside risks to inflation have diminished, while downside risks to employment have risen.
   • The time to adjust policy has come. The direction toward rate cuts is clear, but the timing and pace depend on incoming data.

3. Analysis of inflation and employment

   • Pandemic-related distortions in supply and demand, along with shocks to energy and commodity markets, were the main drivers of high inflation.
   • The unwinding of these factors, together with the Fed's restrictive monetary policy, brought inflation down without a sharp rise in unemployment.

4. Key lessons

   • Anchored inflation expectations and forceful central-bank action made disinflation possible without economic slack.
   • This experience is consistent with the predictions of standard economic models.

5. Looking ahead

   • Learning from the pandemic economy is ongoing.
   • Through its five-year policy review, the Fed intends to incorporate new ideas while preserving the strengths of its framework.

In closing, Chair Powell emphasized the importance of humbly taking in the lessons from an extraordinary period and applying them flexibly to current challenges. A summary at sufficient granularity to follow the original speech's line of argument was obtained in a short time.

Accuracy Considerations and Cautions in Production

To incorporate this pipeline into real work, it is important to understand the accuracy characteristics of speech recognition.

A representative point confirmed in this evaluation is that recognition of proper nouns has its quirks. For example, there were places where "Powell" was recognized as "Powel." Domain-specific terms such as personal names, company names, and technical jargon are prone to misrecognition, so for accuracy-critical uses it is effective to pre-register frequent terms using Speech-to-Text's vocabulary-adaptation (phrase hints) feature, or to correct spelling variants in the later LLM stage.

Also, because the speech-recognition output becomes the summarization input directly, errors at the recognition stage can propagate into the summary. For important figures and proper nouns, it is safer to keep a process of cross-checking against the raw transcript. The role-division setup is well-suited to operation precisely because it makes such intermediate verification possible.

Summary

With a two-stage design — transcribing the full text via speech recognition, then translating and summarizing with an LLM — long meeting audio was converted into structured information in a short time. By separating recognition from semantic processing, accuracy issues can be addressed incrementally. For consultation on adopting AI/LLM use or audio processing, use the contact form.