Morris Matsa

API Gateway Innovator, interested in compilers, languages, performance.

8 minute read

Which public endpoints and open source libraries are using GraphQL Cost Analysis in 2021, and which methods are they using?

This article is Part 4 in a 7-Part Series. (7 posts to date, more planned)

Now that you know that GraphQL cost analysis is important, and the three high-level methods of calculating GraphQL cost, we investigate what people are actually doing today.

There are a plethora of uses of GraphQL in production around the industry today. The goal here is not to survey them all, but rather to look at a few of each type to gather a sense of the various options for cost analysis and the tradeoffs being made.

In this post, let’s look at a few concrete examples in each of:

Then in the next post, I plan to discuss examples of:

  • Professional products
  • Research
  • Human offload

These assessments are based on public documentation. Where documentation is vague, I’ve tried to accurately represent that in my assessment. For example, where a server calculates an actual cost, is it doing Dynamic Query Analysis or Query Response Analysis? I’ve tried to indicate whichever one the text of the documentation hinted at, but if someone wants to correct me please let me know and I can update this page.

Terminology

Before we start, let’s settle on a little bit of nomenclature.

  • For the methods of GraphQL cost analysis, I’m using Static Query Analysis, Dynamic Query Analysis, and Query Response Analysis as defined in an earlier post.
  • Field Cost is a GraphQL cost metric that estimates how expensive it is for the GraphQL server to execute a query.
  • Type Cost is a GraphQL cost metric that estimates how much data will be returned over the wire as a response to a query.
  • Threat Protection is stopping a query before its executed because there is too high a cost for that single query.
  • Rate Limiting is limiting the total cost incurred from running queries over a time interval.
  • Monetization can involve charging an API consumer based on the cost of the queries they send.
  • Slicing Arguments are arguments on a GraphQL field which express the maximum size of a returned list.

There are many other names used for Slicing Arguments, including Pagination Arguments, Limit Arguments, List Arguments, and Size Arguments. We have chosen to use Slicing Arguments because the GraphQL Foundation learn pages define this process as slicing, as does the Relay Connections Spec.

Public Endpoints

The big public GraphQL endpoints are all open about their use of cost analysis. Here are a few examples:

Github

GitHub seems to use:

  • Static Query Analysis with slicing arguments to calculate an overall Type Cost. For this analysis, where all weights for nodes in the connection pattern are 1 and others are 0, and the calculated cost is used for threat protection.
  • Dynamic Query Analysis to calculate an overall Field Cost. This analysis uses custom weights depending on what each resolver function does, and the calculated cost is used for rate limiting.

Here’s GitHub documentation of their threat protection. It includes cost analysis that is enforced as an extended layer of schema validation: Github's Custom Weights Table

And here’s an example of how GitHub documentation explains their cost analysis computation: Github's Custom Weights Table

Shopify

Shopify seems to use:

  • Static Query Analysis with slicing arguments to compute a Field Cost, with some special field weights. It uses this calculated cost for threat protection.
  • Query Response Analysis on the response, and it uses this calculated cost for rate limiting.

Here is an example of Shopify advertising two alternative API Plans that cost different amounts of money and provide different rate limits: Shopify's Custom Weights Table

Here is an example of Shopify’s documentation of custom weights for GraphQL cost analysis: Shopify's Custom Weights Table

Yelp

Yelp seems to use:

  • Query Response Analysis, with special field weights and slicing arguments, to calculate a Type Cost. It uses this calculated cost for rate limiting.

Here is an example of Yelp advertising their custom weights: Yelp's Custom Weights Table

And here is the message they return when someone exceeds their daily rate limit:

You have reached the GraphQL daily points limit for this API key. The limit will reset at midnight GMT.

Standalone Open Source Libraries

Taking a step back from endpoints with custom cost analysis, there are a number of open source projects doing cost analysis of GraphQL queries:

pa-bru/graphql-cost-analysis

pa-bru/graphql-cost-analysis is a JavaScript package that does Static Query Analysis to calculate Field Cost, for use by both threat protection and rate limiting. The GraphQL schema can be augmented by either internal directives or an external JSON map, from type names and field names. Either way, the extra information includes both custom weights and named slicing arguments.

Here is a small part of an example from their documentation of what the directives look like:

  first(limit: Int): First
    @cost(multipliers: ["limit"], useMultipliers: true, complexity: 2)

Again from their documentation, this is what their config map looks like:

const myCostMap = {
  Query: {
    first: {
      multipliers: ['limit'],
      useMultipliers: true,
      complexity: 3,
    },
  },
}

This project hasn’t been updated recently, but it has inspired later projects.

stems/graphql-depth-limit

stems/graphql-depth-limit is specific to calculating the depth of the GraphQL query.

They explicitly point to other libraries for more thorough cost analysis and make the case that the simpler approach can be superior:

Deciding exactly when a GraphQL query is “too complex” is a nuanced and subtle art. It feels a bit like deciding how many grains of sand are needed to compose a “pile of sand”. Some other libraries have the developer assign costs to parts of the schema, and adds the cumulative costs for each query.

I think that this is an important point. For many customers I’ve seen, depth-checking alone was not enough to protect their backend resources and therefore not a possibility, but if it is an option for you then it can be much simpler. It would be reasonable to build it in as base support for graphql-js and other common implementations.

This repo hasn’t been updated recently — of course that’s part of the benefit of using a simple algorithm.

slicknode/graphql-query-complexity

slicknode/graphql-query-complexity provides a much more flexible approach to Field Cost, allowing users to customize every field’s cost calculation with custom JavaScript code. They provide useful basic “custom” functions out-of-the-box to allow setting the weights in the GraphQL Schema, but the framework is extensible to any algorithm you need.

Here is an example of how their built-in option works for adding directives to the SDL:

  listScalar(limit: Int): String @complexity(value: 2, multipliers: ["limit"])

And here’s an example from their documentation of programatically building up the GraphQL schema types with a complexity extension which can then be read by their provided cost-calculation JavaScript function:

const Post = new GraphQLObjectType({
  name: 'Post',
  fields: () => ({
    title: { type: GraphQLString },
    text: {
      type: GraphQLString,
      extensions: {
        complexity: 5
      },
    },
  }),
});

This is the most advanced of the standalone libraries that I’ve seen, even just in its handling of the “built-in” methods, besides that it provides the “escape hatch” of allowing general JavaScript cost functions. While the general function capability is modeled after support in Sangria, I think that it is new in this library to also provide the simplicity of a list of slicing arguments and a cost weight.

I like the general functions as both an internal mechanism for the implementation and an extended ability for advanced users. Of course, many big companies don’t even have the resources in-house to configure weights in a configuration map or in directives, so I’m not sure how many of them would be using that extensibility.

Big advantage: As of February 2021, this library is the most actively maintained of the standalone libraries.

4Catalyzer/graphql-validation-complexity

4Catalyzer/graphql-validation-complexity does Static Query Analysis to calculate a Field Cost which is used for threat protection, and it makes this calculated cost available for rate limiting. It uses custom weights which are specified on the GraphQL schema types.

Here is an example of how it adds weights in the SDL:

type CustomCostItem {
  expensiveField: ExpensiveItem @cost(value: 50)
  expensiveList: [MyItem] @costFactor(value: 100)
}

It also allows specifying the custom weights directly in the constructed GraphQL schema:

const expensiveField = {
  type: ExpensiveItem,
  extensions: {
    getCost: () => 50,
  },
};

const expensiveList = {
  type: new GraphQLList(MyItem),
  extensions: {
    getCostFactor: () => 100,
  },
};

Version 0.4.2 was released in August 2020.

Summary

For the standalone open source libraries:

  • One library only checks query depth.
  • Three include cost analysis with custom cost weights for calculating Field Cost, while none calculate Type Cost.
  • Two include an ability to use slicing arguments to guide list size estimates.

For the major public endpoints discussed:

  • All three use cost analysis based on custom cost weights and slicing arguments.
  • All three use cost analysis for rate limiting, and two of them also use it for threat protection.
  • Two of them calculate Type Cost. Two of them calculate Field Cost.
  • Two of them use Static Query Analysis, two of them use Query Response Analysis, and one uses Dynamic Query Analysis.

In the next post I plan to extend this survey to include some products and academic research.

Please send me more examples of public endpoints or open source libraries using GraphQL cost analysis, or any corrections to the analysis here.

This article is Part 4 in a 7-Part Series. (7 posts to date, more planned)

You May Also Enjoy

Getting started with Rust

less than 1 minute read

If this short video series inspired you to try out Rust for yourself, then how can you get started learning the language?

Trivial example of safety in Rust

less than 1 minute read

To understand how Rust’s memory safety works and what it gives you, we look at a trivial example of the Rust compiler enforcing memory safety.

Is Rust difficult to learn?

less than 1 minute read

Rust is more complicated and difficult to learn than most programming languages. How long does it take to learn?

Is Rust fast?

less than 1 minute read

Speed is not the reason why most people should use Rust, but Rust programs run fast. Is Rust fast enough?