Old Tricks for the New Dog: Why Traditional Technology Sourcing Best Practice Is Relevant for Cutting-Edge AI

Posted
By

Since the release of OpenAI’s ChatGPT, the intense hype around large language models (LLMs) and complex AI systems has exploded. Organizations have rushed to both try and buy these new tools. Along with it, a flood of commentary continues to flow regarding how to use the tools productively and responsibly, along with the legal issues that might arise through such use. Those topics are certainly novel—but when it comes to procuring AI tools, what if the key to successfully purchasing the products is not?

As businesses rush to engage players—big or small—in the AI space, it might be tempting to think that sourcing AI tools require a completely new set of skills. It is even more tempting to hurry to market to avoid falling behind the zeitgeist that seems to have developed over implementing the new technology. However, the best practices that lead to a “good deal” are much the same as traditional technology procurement, and careful strategy has always been in style.

The ubiquity of AI transactions underscores the need for diligent and strategic approaches to procurement. Moreover, organizations need a strategy that ensures its purchase of AI appropriately meets business needs, protects the organization from undue risk, and provides the best commercial value. The tools needed for success may be ones you have already put into place in prior sourcing engagements. These practices are (1) engaging vendors in a competitive, robust RFP process; (2) participating in pilots, or low-dollar/low-risk trials of new technology; and (3) negotiating your agreement for the product to avoid undue risk for your organization.

  1. Competitive RFPs Result in Better Commercial Value
    When it comes to any product, competition breeds excellence. Being the only show in town often leads to complacency and can result in too much leverage for the provider, which leaves the customer without recourse for bad outcomes. Neither are elements of a successful long-term service relationship. To fix the problem, businesses should consider engaging in a robust, competitive bidding or RFP process.

As part of the RFP process, it is best to compare each participant based on a number of elements, depending on and weighted according to business priority. This includes the following:

    • Price: Perhaps the most obvious benefit to performing an RFP is to compare price and encourage providers to bid against others as a mechanism to reduce an organization’s overall cost of procurement. While many AI providers are offering a one-size-fits-all pricing structure for the new services, pitting the dollars against one another may encourage movement across the market. Some new vocabulary may be necessary for AI services, where the price model is often based on “tokens” and “characters.” But the cost per token, or cost per user (if subscription- or license-based), is not the only, or even most effective, way to compare price. Instead, organizations should more comprehensively compare the total cost of ownership, or TCO). Holistically taking into consideration additional hosting, implementation, training or governance costs, opportunity cost related to leveraging one solution over another, as well as credits or financial incentives available, enables a more complete and clearer financial picture. According to the World Economic Forum’s guide to AI procurement, there is one dissimilarity in comparing the TCO of AI products as compared to other technology services. Typically, the TCO for non-AI solutions spans three to seven years, but AI deployments often extend over a more protracted duration, necessitating an adjustment in cost considerations to encompass this extended timeframe.
    • Product Performance and Service Levels: There seems to be an influx of tools on the AI market that might accomplish the same thing. But successful procurement requires nuanced consideration of a product’s capability to meet its users needs, and comparing across the available products may be the best way to get the complete picture. Qualitatively, require RFP participants to explain the scope of their capability (i.e., the “what”), and back it up with specifications and documentation detailing their solution (i.e., the “how”). Quantitatively, contrast each provider’s commitment to meeting a defined set of precise and reproducible service levels. There are opportunities for AI-specific creativity in defining the service levels, such as requiring a percentage of responses to be free from hallucinations or to meet subjective comprehensibility ratings. But classic service levels, like uptime, maintenance response times and defect rates are also important elements to compare between providers.
    • Disaster Recovery and Business Continuity: An RFP is also an opportunity to perform vendor due diligence before getting in too deep with a provider. Particularly in light of the nascency of the AI space, using the RFP to query a provider’s efficacy is sound practice to avoid going into business with unreliable vendors, or entities that are unsophisticated and therefore unable to consistently perform. As part of the due diligence process during the bidding phase, require each provider to clearly articulate their ability to continue operations and provide the particular service in the face of disaster or financial difficulty. Doing so at this stage not only prevents developing a reliance on a risky product down the road, but it also enables an organization to choose a product based on values that align with its own risk profile.

In choosing participants of the RFP, businesses should consider a mix of insourcing, incumbent providers, and new providers. While new providers are cropping up seemingly daily in the AI space, there is something to be said for also looking internally, and at the existing supplier base, which could round out a more successful RFP.

The choice to insource may not be available for developing complex AI systems, which require significant technological brainpower, massive amounts of computing power and data, and, of course, time. However, insourcing may be achieved by degrees. Rather than relying on the third-party provider for the data sets necessary to train an AI model, consider products that leverage only data domestic to your organization. Alternatively, consider products that can be run on-premise rather than on a remote cloud. Governance can also be insourced instead of left up to the cloud provider, such as performing in-house content filtering or watermarking.

Many RFPs include not only new providers, but (hopefully) friendly faces with whom a customer already has an ongoing relationship. Many players in the AI space, such as the large hyperscalers, likely already provide other services to many customers. Including incumbents in the RFP provides an opportunity to leverage the efficiency of an already negotiated master agreement that may be leveraged for new services. There may also be strong long-standing relationships that will be an asset to the provider’s performance. However, be careful not to give incumbents more credit than they deserve. First, the existing agreement with such a provider may not be suited to AI services. Second, an existing provider may lull one into accepting poor or non-innovative performance.

Fostering a competitive environment through robust RFP processes is, and always has been, fundamental to cultivating excellence and accountability in service relationships. By evaluating participants based on diverse criteria such as price, product performance and disaster recovery capabilities, organizations can make informed decisions that align with their strategic objectives. Moreover, the inclusion of a mix of insourcing, incumbent providers, and new entrants ensures a comprehensive evaluation that optimizes the potential for successful outcomes. Ultimately, leveraging these practices not only mitigates risk but also promotes innovation and efficiency in the procurement of AI services.

  1. Piloting the Product Ensures Business Needs Are Met
    The second non-new concept in sourcing that can be leveraged in procurement of AI services is to try before you buy. Many service providers offer the ability to pilot the AI service for a limited period of time, or engage in a trial or beta release of the product free of charge before committing to a costly long-term engagement. The advantages of piloting an AI service is, most obviously, being able to see if the product actually works, as well as enabling safer initial uses, encouraging AI literacy that can prevent misuse later, and more cautiously determining business needs.

AI services are touted as being able to essentially do it all. Whether AI can measure up remains to be seen, though. If a provider offers the opportunity to test the product before significant financial outlay, it may enable an organization to determine if performance meets expectations. But an added benefit to the limited use is the ability to also limit the risk. Often the pilots can only be deployed with a limited user base, or using ring-fenced data. These sandbox environments provide an organization a chance to issue spot for what risks need to be better cared for with broader deployment.

Trials also enable everyone from the organization’s front-line users to the company’s high-level executives, to develop a baseline literacy in how AI functions. Oftentimes, there is misunderstandings of the products functionality, and little comprehension as to what new and different monitoring and governance is necessary for AI use. Creating a plan in advance around audits, transparency, use policies, and other elements, and then pressure testing them during a trial, can increase the general literacy and prepare the organization for wider implementation.

Finally, the benefit of a trial is that it lets the business consider if it is “sure” AI is what it needs to meet its objectives. Since AI is such a hot topic, organizations are clamoring to engage vendors, leaving little room for sound circumspection. Implementing a pilot may give the business time to recalibrate, as well as avoid over-purchasing an unnecessary product or number of subscriptions.

In spite of the benefits of piloting AI products, there are certain pitfalls. Consider the following:

    • Avoid entering into legal terms that are binding for an extended period, or that reduce the negotiating leverage for purchase of the live product.
    • Beware of contracts that unduly shield the provider from all risk associated with AI products. Certain legal terms such as intellectual property, data protection, limitations of liability, and indemnities should still be critically reviewed by legal counsel.
    • Implement internal policies requiring that the trial use cases are appropriately limited and monitored carefully.
  1. Negotiating Protective Contracts Avoids Undue Risk for the Organization
    As with traditional sourcing engagements, there is prophylactic value in negotiating a strong contract to memorialize the engagement. Doing so protects each party’s best interests, while fairly allocating risks that may come to fruition down the road. While careful negotiation is not a new concept, admittedly the legal issues related to AI require special considerations, some of which we have summarized in a prior article detailing the legal risks of AI systems in technology services. For example, a well-negotiated contract for AI services should address:
    • Allocation of intellectual property rights in the AI model’s input, training and output
    • Security and privacy protections applicable to the AI model
    • Representations and warranties related to AI performance
    • Limitations of liability and indemnification provisions that specifically account for the functioning of AI.

It is likely that a provider’s off-the-shelf or standard agreement is insufficient to care for these unique needs, so customers should be sure to review these types of agreements carefully. Customers could, instead, consider using custom, AI-specific services agreements that are more customer friendly.

Effective negotiation of these contracts of course (says the lawyer) benefits from sage legal counsel—but the lawyers alone may not be as successful without support from the organization’s business and management that the various issues reflected in the contract are imperative in order to move forward with the engagement. As such, avoid leaving the negotiation up to the lawyers, and ensure all stakeholders remain involved and supportive of drafting and signing a contract that mutually benefits both parties.

It may seem that the past year has been a long, incessant lesson in artificial intelligence. According to Bloomberg Law, at least 41%, of the S&P 500 companies mentioned AI in their most recent 10-K report—so nearly everyone who is anyone is buying this technology or selling it. But, while the technology itself may be novel, the principles governing sound procurement remain steadfast. By adhering to established practices, such as engaging vendors in a competitive RFP process, participating in pilots, and skillfully negotiating agreements, organizations can navigate the complexities of AI acquisitions with confidence and prudence.

For questions on the procurement of AI services, and negotiating services agreements with such service providers, Pillsbury’s global sourcing and technology transactions professionals can assist.

RELATED ARTICLES

Earning Your Trust: The Need for “Explainability” in AI Systems

Artificial Intelligence Systems and Risks in Commercial Contracting