- Token consumption: High AI token use drives costs; organisations like Uber overspend annual budgets quickly due to rapid consumption.
- Document formats increase tokens: PDFs significantly raise token count; converting to Markdown is recommended to streamline usage.
- Strategies to minimise token use: Focus on relevant text, start fresh chats, and limit response lengths to save tokens.
- Cautious AI reliance: Users value AI productivity but actively fact-check outputs to avoid inaccuracies in generated content.
- Tool recommendations: Use Microsoft Markdown converter for document prep and AI tools to assist with prompt crafting.
Notes
AI Token Usage and Cost Management
The team focused on understanding and reducing excessive AI token consumption to control escalating costs.
- Token consumption explained with LEGO analogy clarifying how AI breaks down inputs into tokens, driving cost based on token use (07:41)
- Malcolm described tokens as small building blocks that AI assembles and analyses to generate outputs.
- Text tokens correspond roughly to every four characters, while images, audio, and video are broken down into smaller segments for analysis.
- Free AI models offer token limits per session, such as 128,000 tokens for ChatGPT, 200,000 for Claude, and 1 million for Gemini.
- Token usage directly impacts cost, with companies like Uber exhausting annual AI budgets in a few months due to high token consumption.
- PDF and document formats significantly increase token usage due to extra metadata and formatting (12:16)
- PDFs include spacing, fonts, tables, images, and metadata that multiply token count compared to plain text.
- Tables and images in documents add complexity, greatly increasing tokens needed for processing.
- Malcolm recommended converting documents to Markdown (MD) format, which strips unnecessary formatting and is AI-friendly.
- Tools like Microsoft’s MarkItdown converter and batch file converters can automate this conversion to reduce token use.
- Practical strategies to minimize token consumption during AI interactions were emphasized (18:55)
- Copy-pasting only relevant text segments from documents rather than entire files reduces tokens used.
- Starting a new chat session after completing a task prevents the AI from reprocessing long chat histories and saves tokens.
- Summarizing previous chats and feeding concise summaries into new chats preserves context while limiting token use.
- Using simpler, older AI models when advanced features aren’t needed cuts costs.
- Setting output length limits (e.g., 100 words) avoids unnecessarily long AI responses that consume tokens.
- Disabling unused advanced features like web search and deep thinking in AI tools can reduce token consumption (24:01)
- Malcolm advised turning off features not needed for a task to prevent extra token usage from external data searches.
- This approach helps keep AI usage efficient and cost-effective.
AI Tool Usage and Workflow Integration
The group shared current AI tool adoption and challenges integrating AI into legal workflows.
- Murphy remains the primary AI tool for legal workflows, with ongoing efforts to optimize its integration (25:53)
- A explained Murphy is being tested with litigation workflows but faces delays due to developer turnover and backend capacity upgrades.
- The team is working closely with Tim’s group as guinea pigs to refine these workflows.
- Despite challenges, Murphy is the mainstay AI platform for document drafting and legal tasks.
- ChatGPT is used for drafting and prompt refinement, complementing prompts with Murphy(27:17)
- A uses ChatGPT’s voice-to-text to quickly create initial drafts and refine prompts before entering them into Murphy.
- This two-step approach helps overcome Murphy’s less conversational interface and improves output quality.
- She also uses ChatGPT to rewrite Murphy-generated drafts into her own style, increasing personalization.
- AI prompt crafting and iteration is an evolving skill emphasized by the team (29:00)
- A highlighted the importance of improving prompt quality by cross-checking outputs across different AI tools to avoid generic or inaccurate results.
- Malcolm shared that prompt wording can significantly impact AI results and that copying and modifying prompts between chats helps manage token use and accuracy.
- This iterative process is necessary due to AI’s occasional errors and token limits.
- The team values AI as a productivity aid but remains cautious about output accuracy (31:10)
- A noted AI outputs can seem impressive initially but often recycle the same content, requiring human review and fact-checking.
- Using multiple AI tools to review and improve outputs is a practical approach to mitigate AI “slop.”
- This cautious stance balances AI benefits with risks of overreliance.
Process Improvements for AI Efficiency
The meeting identified specific process changes to improve AI usage and reduce waste.
- Avoiding long continuous chat sessions to save tokens (19:53)
- Malcolm explained that long chats cause AI to reprocess entire conversation history with each new input, increasing token use.
- The recommendation is to start a fresh chat after completing a topic to reset the token count.
- Summarizing long chats before continuing preserves context but limits token consumption.
- Selective input submission reduces unnecessary token processing (21:46)
- Copying only the relevant text from documents or chat history into AI prompts avoids feeding irrelevant bulk data into the model.
- This focused approach improves efficiency and reduces cost.
- Limiting output length for AI responses prevents token overuse (23:07)
- Setting a maximum word count (e.g., 100 words) ensures concise answers that meet user needs without token bloat.
- This also improves user experience by delivering direct answers quickly.
- Reusing and modifying original prompts instead of new chats when correcting AI errors (23:57)
- Malcolm advised editing the original prompt text in the same chat to refine AI responses rather than starting new chats that consume more tokens.
- This strategy saves tokens and speeds up problem resolution.
Strategic Insights on AI Cost Control
The discussion underscored broader strategic concerns about AI cost growth and usage awareness.
- Awareness of AI token costs is rising due to rapid consumption and budget overruns (06:31)
- Malcolm cited industry examples like Microsoft and Uber cutting AI token use after budgets blew out early in the year.
- This has raised internal urgency around efficient AI use.
- Understanding AI token mechanics is critical to managing costs (07:19)
- The team sees education on token structure as foundational to smarter AI consumption.
- Malcolm’s detailed explanation aimed to empower users to make informed decisions.
- Choosing appropriate AI models based on task complexity balances cost and capability (22:30)
- Using simpler, less expensive AI models for routine tasks helps control spending without sacrificing performance where advanced models aren’t needed.
- This strategic use of AI tiers reflects a cost-conscious deployment philosophy.
Team AI Adoption and User Experience
Personal experiences and attitudes toward AI tools revealed practical considerations for adoption.
- Users rely heavily on voice-to-text and drafting support from AI (27:17)
- A’s workflow of dictating text, refining prompts in ChatGPT, then using Murphy illustrates a hybrid human-AI approach that suits legal work style.
- This method reduces typing effort and speeds up drafting.
- AI tone and style adaptation enhances user acceptance (27:22)
- AI’s ability to learn and mimic individual writing style makes outputs feel personalized and easier to edit.
- This personalization increases reliance on AI for routine communication.
- Users recognize AI limitations and engage in active fact-checking (31:10)
- The team does not accept AI output at face value but uses multiple tools to cross-verify.
- This approach mitigates risks of generic or incorrect AI-generated content.
- The evolving nature of AI tools requires ongoing learning and experimentation (28:55)
- A noted that prompt crafting and tool selection constantly change, requiring users to find their own effective methods over time.
- The team values sharing tips and updates to improve collective AI proficiency.
Resource and Tool Recommendations
Practical tool suggestions and resource sharing aimed to improve AI workflows and efficiency.
- Microsoft MarkItdown converter and batch file converters recommended for document preparation (15:02)
- Malcolm shared links to these tools in chat for converting PDFs and Word docs into AI-friendly Markdown format.
- These tools strip excess formatting and reduce token usage.
- Web-based and Mac-specific converters offered as alternatives (16:43)
- Options include web converters for quick tests and a Mac-specific file-to-text Markdown converter for Apple users.
- This broadens accessibility across platforms.
- Best practices for chat management shared to optimize token use (19:53)
- Starting new chat sessions for new topics and summarizing ongoing chats saves tokens.
- Slash commands like “/clear” may work on some platforms to reset chats, though Claude did not recognise it.
- Encouragement to use AI tools that assist with prompt crafting (28:44)
- Malcolm noted tools exist that help create better prompts, complementing manual crafting and experimentation.
- This can improve AI output quality and reduce token waste.
Action items
Malcolm Pearson
- Share Microsoft Store Markdown converter and batch file converter links in chat for team to review (15:59)
- Link shared in the chat…