2023 conversational ai application

AI Developer Assistant.

Trimble Developer Assistant is an AI-powered tool that helps developers find and execute API calls using Trimble's developer catalog.

ROLE

Senior UX Designer

timeline

1 year (2023)

team

Design, Engineering, Data, AI

Problem

API instructions and documentation were scattered across the site by business unit, causing delays in application integrations and decreased adoption.

Solution

Built an AI assistant for API calls and extended it to a multi-agent model for complex development queries.

Outcome

50% of users found the AI behavior predictive.

Overview

Trimble Developer Assistant was an initiative resulting from a strategic decision to leverage current technologies to improve customer experience when integrating with Trimble API's. Target users are developers, integrators, and engineering leads who research and integrate Trimble APIs.

The AI is primarily inspired and modeled from Gorilla LLM with Elasticsearch as the engine and Azure AI services for resources and management.

Tip💡: Besides information architecture and user goals, I find it helpful to learn about the underlying system architecture and the individual resources that make up the system. A product could be comprised of multiple services, resources, and processes. For more information on service design, check out this article by NN/G.

My goal in this effort is to calibrate user trust with the overall user experience and obtain user feedback on the predictability and accuracy of the AI model.

To achieve this, I've outlined the following criteria:

  • Competence: Does it improve the experience or satisfactorily address the user's needs?

  • Reliability: How well does the product deliver on its expected capabilities?

  • Predictability: Does the product fit the users' mental modes and habits?

  • Benevolence: Is the product transparent about the value it provides for the users? Are those values intended for the user good?

Create an optimized API workflow experience with AI and increase Trimble API adoption.

Process

We were given a one-month deadline to design and develop an alpha version. Our goal was to showcase it during Trimble's Dimensions 2023 conference in November.

To meet this challenge, I collaborated with the product and development teams and produced a project plan. I divided the effort between pre and post-conference, outlining the activities for each phase.

Given the time constraint, I planned for the following artifacts:

  • Competitive Analysis: to compare standard UI components with those of existing products in the market and establish a baseline.

  • User Interviews: to validate the product concept and learn the end user's mental model.

  • User testing plan: a study to evaluate the alpha version and receive user feedback.

Problem

After conducting a competitive analysis, we established a UI baseline for the alpha version and identified potential gaps. One such gap is the user's inability to group their AI interactions into topics.

Check out the 📖 Dev Assistant UX Comp Analysis 2023 for a more detailed view.

During the user interview, we discovered that our customers visit multiple touchpoints to obtain API documentation and instructions.

While an AI assistant can simplify the process of obtaining API calls, it's crucial to calibrate user expectations.

Tip💡: I utilize Dovetail AI to summarize user transcripts immediately after an interview while the conversation details remain fresh. Afterward, I review the check for accuracy, then identify high-level themes converted to word tags and apply them to the transcript when I review it later. For more information on using Dovetail AI, check out this resource.

Solutions

Give users control over their chat history.

New feature added to platform: Users can create lists and pin chat interactions based on them. This helps group AI interactions by time and topic, making it easier to search for API documentation.

Calibrate user trust.

To ensure user trust, I developed a standard AI response script for situations where an API call is unavailable. I started by brainstorming initial ideas with ChatGPT to help kickstart my scriptwriting process, and then I tailored the content to fit our product. My plan is to test the scripts during the alpha release and make any necessary revisions in the following year.

Users can now provide explicit feedback through reactions, chat suggestions, and support channels.

Extend functionality through a multi-agent model.

Given strong early interest, we explored extending the product to a multi-agent model using the AutoGen framework.

For more information on the AutoGen framework, check out the doc in Github.

Outcomes

0

% of testers who found the AI behavior to be almost predictive.

0

% of users found the assistant usable.

Reflections

user data synthesis

All data collected during the alpha test should be analyzed to identify new features or areas for improvement between the two products. If there is not enough participation, an additional user test may be necessary with an open group of testers.

Create Feature Backlog

After synthesis, a backlog for any gaps in the AI experience is needed, along with a roadmap for delivery. Alignment with product and development is important for roadmap refinement and program increment planning.

Refine api personas

To improve the assistant's functionality, we need to identify the different types of users interacting with it. Conducting a Jobs-To-Be-Done (JTBD) exercise will help us uncover the APIs users search for, the challenges they face with AI models, and potential complementary features.

INTEGRATION

We need to establish how users will discover the assistant and how it integrates with the broader Trimble digital experience. At what point in the Trimble digital experience will users come across the assistant? Which touchpoints will be used to introduce the assistant? What functionalities of the assistant will be highlighted for each touchpoint?

Home

Avatar

Panel