Despite their similarities, an MCP Server is not an API Gateway, Microsoft engineers advise. It can do so much more, so let it.
Slapping a protocol wrapper over an internal API may be a tempting shortcut to quickly build a MCP server, but the end product may only flummox an agent by offering too many choices, which would lead to context confusion, and subpar results.
Programming for the agents requires a different workflow, a different way of engineering, and even a different way of thinking, advised a recent post from a program team at Microsoft. The post was authored by Microsoft technical program manager Pieter de Bruin, senior program manager Eric Imasogie, and principal software engineering manager Tianqi Zhang.
The post shared the lessons they learned from building an MCP server for Visual Studio to help developers work through new tasks.
“We are sharing not just what was built, but why the decisions we made mattered and what surprised us on this journey,” the authors write.
Indeed, content-rich organizations could use Microsoft’s teachings as a blueprint for building their own MCP servers. Spoiler: It’s a completely different approach than building an API server, however similar they may first seem initially.
What Microsoft’s MCP Server Offers Visual Studio Developers
The main idea for the Microsoft Learn MCP server was to reduce the amount of context switching the developer makes flipping between documents and the IDE. This feature was launched in June as part of Visual Studio and made generally available in November.
Typically, the code jockey embarks on a new task, such as adding a database connectivity to a Node.js application, by searching through the documentation, perhaps saving a browser tab for each bit of relevant info. Then they flip back and forth between these pages and the IDE as they step through the task.
The MCP server tries to automate this task as much as possible. Once configured into Visual Studio, the service is accessible through GitHub Copilot chat, in agent mode.
Ask it a question (“How do you connect Node.js to Postgres?”) and the server rifles through the Microsoft documentation and builds a step-by-step instruction set and then can even apply those changes to the project itself, so explains a Microsoft video tutorial on the feature.
Pretty sweet.
The Microsoft MCP Stack
Essentially, this MCP server sits in front of Microsoft’s Learn knowledge service, Ask Learn, an API-fronted RAG-based knowledge service for Microsoft documentation that debuted in 2023. The MCP just requires a short list of tools that are available. It can negotiate the capabilities at runtime. Clients connect over Streamable HTTP Transport. Azure App Service handles the transport and session handling, which was implemented using the C# SDK for MCP.
To keep the documentation fresh, the MCP Server taps into the same content vector store used by Ask Learn. It uses Azure OpenAI, an enterprise-friendly platform for the OpenAI models. Enterprise security is handled by the Azure API Management.
Let MCP Do the Driving
The chief lesson the Microsoft folks offered here was “Your API is not an MCP tool.”
A developer should “design tools for the agent workflow, not to mirror internal APIs,” the trio of authors wrote. Internal APIs are usually just a collection of knobs designed to be twiddled for specific needs. With MCP, your system design should be cognizant that the agents will make decisions for themselves.
Microsoft’s approach supporting these agents is surprisingly simple. The MCP server essentially is given two main tools: search and fetch (a third one is also offered for finding code snippets in specific languages).
Contrast this with the Learn API itself, which has a veritable treasure trove of parameters from which developers can choose (topK, index selection, thresholds, OData filters, etc). Lots of choices there, but also lots of complexity.
The Microsoft engineers devised a simpler abstraction, one that mimics how human users, not machines, look up information: Search then click. “The client can first find the best match and then ground the answer with the full-page Markdown,” they wrote.
This MCP Has an Informed Opinion
To help guide the agent along, the MCP server includes built-in automated evaluation that is deployed in a feedback loop, with the output fed back into the model as input context. This tool evaluates how tools work based on the agents’ success metrics, routing the refinements back to the agent when the session is refreshed.
In other words, the MCP is very opinionated. And with good reason. Developers working in Visual Studio tend to have the same sorts of questions: “Most requests are related to coding tasks, explaining, and troubleshooting,” the authors write. So the search and fetch tools were optimized for these intentions.
Thus the agent gets smarter the more it uses the tools. As more agents use the search and fetch functions, the accuracy and citation quality of the results improve, Microsoft asserts.
For a certain type of question, a description may advise the client to go directly to the “troubleshooting guide” of the docs, rather than just scanning from the beginning. This can save Microsoft computational costs and save the user time.
Letting Go of the Hard Code
Still, programmers are a stubborn lot, and some will try to hard-code help for the client. In fact, this has been an issue for Microsoft. When it renamed the “question” parameter to “query” on the server, about 2-5% of the requests that subsequently came in broke, because they were hardcoded to “question,” even though it was completely unnecessary.
For Microsoft, such specificity amounts to “tool space interference,” and its researchers penned a missive explaining why it’s such a bad idea to overspecify. It notes the performance of an LLM drops in direct proportion to the number of tools it has been given – a conclusion OpenAI also agreed with, setting a limit of 128 tools per session.
Keep in mind that Microsoft went with an opinionated approach for building an MCP server because it best fits a tightly-controlled enterprise development environment with highly-curated data. There are more unopinionated approaches, such as Clawbot, if you just want to build a bridge into the wild west of AI usage. For organizations with rich troves of data and services, however, an opinionated but simple architecture may be the way to go, if they want their users to make full use of its resources.
