This usability study evaluated the new AI chatbot interface with 24 participants across 48 sessions, totaling 18.5 hours of recorded interaction data. The overall System Usability Scale (SUS) score of 78.4 places the interface in the “Good” category, with particularly strong performance in ease-of-use (4.5/5) and perceived speed (4.6/5).
However, two critical usability issues emerged: the export conversation flow achieved only 62% task completion (unacceptable for a core feature), and document upload caused confusion for 17% of participants. Both issues stem from discoverability problems rather than functional bugs—the features work correctly once found.
The interface succeeds at its core value proposition—fast, clean AI interaction—but fails on secondary features that power users rely on. Addressing the export and upload flows before launch is critical; other feedback (dark mode, code highlighting) can be prioritized for post-launch iterations.
Why This Study Matters
AI chatbot interfaces are becoming commoditized—users have experience with ChatGPT, Claude, Gemini, and numerous competitors. This raises the bar for new entrants: users arrive with strong mental models and expectations formed by existing products.
Our interface takes a deliberately minimalist approach, prioritizing focus and speed over feature density. This study tests whether this tradeoff resonates with users or creates frustration when expected features are harder to find.
Importantly, we recruited participants from our target user base (technical professionals) rather than general consumers, as their needs and expectations differ significantly from casual users.
Research Questions
- 1
Can users complete core tasks (send message, upload document, export) without assistance?
- 2
Does the minimalist design feel “clean” or “incomplete” to technical users?
- 3
How do developer expectations (keyboard shortcuts, code handling) differ from other user segments?
- 4
What feature gaps would prevent adoption as a daily-driver AI assistant?
Study Design
Mixed-methods approach combining moderated task-based testing (Session 1) with unmoderated exploratory sessions (Session 2). Each participant completed both sessions 2-5 days apart.
Participant Recruitment
24 participants recruited via professional networks, screened for: weekly AI tool usage, technical role, and no prior exposure to our interface. Compensation: $75 gift card.
Data Collection
Full session recordings (video + audio + screen), click heatmaps via Hotjar, System Usability Scale questionnaires, and semi-structured exit interviews.
Task Scenarios
Participants completed 5 standardized tasks designed to test core functionality and edge cases:
Participant Demographics
Task Completion Metrics
| Task | Success Rate | Avg Time | Errors |
|---|---|---|---|
| Send first message | 100% | 8s | 0 |
| Upload a document | 83% | 24s | 4 |
| Modify AI response | 71% | 45s | 7 |
| Export conversation | 62% | 38s | 9 |
| Change AI persona | 88% | 15s | 3 |
Understanding the Failures
Task 4: Export Conversation (62% Success)
The export function is located in a hamburger menu at the top-right of the conversation view. However, participants overwhelmingly looked for it in two other locations: (1) at the bottom of the conversation near the input field, and (2) via right-click context menu on messages.
- • 71% of failures tried scrolling to find export
- • Average time-to-abandonment: 38 seconds
- • 4 participants verbalized “Where is export?”
- • Add export icon to conversation header bar
- • Implement Cmd/Ctrl+E keyboard shortcut
- • Add right-click context menu on messages
Task 2: Upload Document (83% Success)
While most participants eventually found the upload button (paperclip icon), confusion arose around what file types are supported and what happens after upload. The interface provides no feedback on supported formats until an error occurs.
- • 3 participants tried drag-and-drop (failed)
- • 2 attempted to paste file from clipboard
- • Confusion about PDF vs image handling
- • Enable drag-and-drop with visual drop zone
- • Show tooltip listing supported formats
- • Add upload progress indicator
Task 3: Modify AI Response (71% Success)
The “edit” functionality allows users to request changes to AI responses. However, the trigger (double-clicking response text) was not discoverable. Users expected either an edit button or the ability to directly type changes.
This is a deliberate design tradeoff: we chose double-click to keep the interface clean, avoiding per-message edit buttons. However, 29% failure rate suggests we need either progressive disclosure (show button on hover) or better onboarding to teach this interaction pattern.
Developers (n=10)
Highest task completion rates across all segments (87% avg). Strongly preferred keyboard navigation—67% attempted keyboard shortcuts before using mouse. Unanimous request for dark mode.
Syntax highlighting, Vim keybindings, API access for automation
Product Managers (n=6)
Most frustrated by export limitations—want PDF and Word format options in addition to Markdown. Valued conversation history and search features more than other segments.
Folder organization, conversation templates, team sharing features
Designers (n=5)
Most positive about the minimal aesthetic (4.8/5 design satisfaction). However, criticized lack of image generation/editing features compared to competing products.
Image generation, brand color customization, Figma plugin integration
Click Heatmap Analysis
Heatmap Insights
Click density analysis reveals three distinct interaction zones. The primary action zone (CTA button, search bar) received 58% of all clicks. The navigation zone (top menu) received 31%. The discovery zone (profile, footer links) received only 11% despite containing important features.
This confirms that users focus heavily on the central conversation area and rarely explore peripheral UI elements—supporting the case for surfacing critical actions (like export) in the primary zone.
Scroll Depth Analysis
In longer conversations (10+ messages), only 34% of participants scrolled back to review earlier context. This suggests either (a) users trust AI to maintain context, or (b) scrolling UX is friction-heavy.
Consider adding a “conversation summary” or “jump to topic” feature for long conversations rather than relying on manual scroll navigation.
“The response speed is impressive, but I wish I could edit my prompts after sending them.”
P07 - Developer“Finally, an AI interface that does not feel cluttered. Very clean design.”
P12 - Designer“I could not figure out how to export my conversation. The button was hidden.”
P19 - PMAffinity Map: Key Themes
1Critical: Fix Export Flow
Only 62% task completion rate. Move export button to conversation header and add keyboard shortcut (Cmd+E).
2High: Improve File Upload
Add drag-and-drop support and clearer file type indicators. Current upload flow causes confusion.
3Medium: Add Dark Mode
Requested by 67% of developer participants. Add system preference detection with manual override.
4Low: Code Highlighting
Implement syntax highlighting for code blocks. Consider Prism.js for language detection.
Implementation Priority Matrix
| Issue | Impact | Effort | Timeline |
|---|---|---|---|
| Export button relocation | High | Low | Pre-launch |
| Drag-and-drop upload | Medium | Medium | Pre-launch |
| Edit response discoverability | Medium | Low | Pre-launch |
| Dark mode | Medium | Medium | V1.1 |
| Syntax highlighting | Low | Low | V1.1 |
Study Limitations
- Sample composition: Heavy representation of developers (42%) may skew findings toward technical user needs. Consumer user testing needed for broader product-market fit assessment.
- Lab environment: Moderated sessions may introduce observer effects. Participants might behave differently than in natural work contexts.
- Short-term focus: Study captured first-use experience. Longitudinal research needed to understand habit formation and feature discovery over time.
- Desktop only: All testing conducted on desktop browsers. Mobile experience requires separate evaluation.
Planned Follow-Up Studies
- A/B Test Export Redesign: Validate that header placement improves task completion. Target: 90%+ success rate.
- Diary Study: 2-week longitudinal study with 15 participants to understand feature discovery and daily usage patterns.
- Competitive Benchmarking: Identical task scenarios tested against ChatGPT and Claude interfaces for baseline comparison.
- Accessibility Audit: WCAG 2.1 AA compliance testing with assistive technology users.