Using JSON schema with statically typed object-oriented languages — Illustrated in Kotlin | by Martin Devillers | Aug, 2022

Photo by Rodion Kutsaev on Unsplash

JSON Schema is an impractical standard to describe JSON APIs. Here’s how to make the best of it anyways, so that client developers can leverage schemas published by server developers.

But first, a bit of history.

Around the year 2010, JSON overcame XML as the most popular data exchange format, following the advent of REST APIs which came as a replacement for SOAP, which itself relied on XML. JSON has many advantages that make it a better format than XML for exchanging data, most notably its simplicity. Compare the JSON specification with the XML specification and you immediately get a pretty good idea of the difference. Seriously, try to print the XML specification, it’s 31 pages long, and it mentions dynamic processing instructions within XML documents. On the other hand, JSON can be summarized using a few simple diagrams, making it much easier to support it in many different environments. In addition to this simplicity, JSON is much less verbose, it has native support for numeric types and booleans, not to mention that it’s directly interoperable with Javascript. All these assets lead to widespread adoption of JSON, which in turn led to an ecosystem being developed around it, with various libraries and tooling.

With this advent of JSON came a need to develop tools to describe the structure of JSON documents, much like XML had the concept of XML schema. A first draft of JSON schema was published in 2010, but it was in 2013 that Draft 4 of of JSON schema was finalized, which was to become the first widely adopted iteration (unfortunately, as of today there still hasn’t been a version of JSON schema published which isn’t considered a “draft”). This release already contained most of the core concepts used by JSON schema today.

JSON Schema is a vocabulary that allows you to annotate and validate JSON documents.

Similarly, around that time, the OpenAPI specification started being developed to provide a language to fully describe a REST API. The first version was published in 2011, and version 3 (widely adopted today) was released in 2017.

The OpenAPI Specification (OAS) defines a standard, programming language-agnostic interface description for HTTP APIs, which allows both humans and computers to discover and understand the capabilities of a service without requiring access to source code, additional documentation, or inspection of network traffic. When properly defined via OpenAPI, a consumer can understand and interact with the remote service with a minimal amount of implementation logic. Similar to what interface descriptions have done for lower-level programming, the OpenAPI Specification removes guesswork in calling a service.

JSON being the de-facto standard for REST API content, and JSON schema the most widely adopted standard for describing JSON documents, OpenAPI itself chose to use a derived version of JSON schema to support the formal description of JSON documents.

OpenAPI 3.0 uses an extended subset of JSON Schema Specification Wright Draft 00 (aka Draft 5) to describe the data formats. “Extended subset” means that some keywords are supported and some are not, some keywords have slightly different usage than in JSON Schema, and additional keywords are introduced.

OpenApi eventually grew to become arguably the most widely-used standard for formally describing REST APIs, which themselves use JSON as a data format, hence making JSON Schema the most widely adopted tool for describing JSON API requests and responses. Following the popularity of these new standards, a wide set of tooling progressively developed around OpenAPI and JSON Schema, notably for documentation and automated checks, but also for automatically generating the code handling API calls.

In languages like Kotlin or Swift (for mobile applications), it’s convenient to declare classes whose structure structure matches that of the JSON exchanged in the API, using as a source of truth the schema describing the APIs. Ideally, there would therefore be a direct relationship between the concepts in JSON schema and in the type system of the languages using the APIs. This is an important requirement to build code generators for the API calls using the specification, which not only is convenient and saves time but perhaps even more importantly provides guarantees of correctness and safety both for the client and server developers.

As it turns out though, linking JSON schema constructs to those of the type system of typical object-oriented language like Kotlin isn’t so simple.

The following simple example of JSON, exchanged in an API, contains the name of a person, and indicates whether this person is active.

{
"name": "John Doe",
"active": true
}

To describe this data structure, a JSON schema like the following would be used. It specifies an object containing only a name property, which is of string type, and an active property, which is of boolean type.

{
"type": "object",
"properties: {
"name": {
"type": "string"
},
"active": {
"type": "boolean"
}
}
}

The application code would then typically declare a class like the following to represent these JSON objects received or sent through the API.

class Person(
val name: String,
val active: Boolean
)

Using tool like Moshi, Gson, Jackson, or Kotlin Serialization, serializing an instance of this class is guaranteed to produce JSON which is valid to send in a request to the API. Likewise, if receiving a response valid according to this schema, it should always be successfully parsed to produce an instance of this class, and the resulting instance would capture all the data from the response.

Using classes like this one to represent JSON objects exchanged in APIs guarantees the correctness of the JSON data handled by the application (if the class itself is correct), and provides more convenience for integration and maintenance.

Naming

Unfortunately, JSON schema doesn’t have a way to provide a definition of names for types, and doesn’t enforce any format format for naming properties.

Having a common name for classes would be beneficial to make sure all implementations use the same terminology, and would facilitate the implementation of code generators which consume the API specification. The JSON schema annotations specification does include a title and a description though, so as a convention the title can be used to contain the name of the type. Alternatively, the last part of the $id URI could be used as a name. In any case, it’s useful to have an established convention to define the name of types.

{
"title": "User",
"type": "object",
"properties: {
"name": {
"type": "string"
}
}
}

For properties themselves, the key of the property (or some derivation of it) is typically used to name it. Establishing conventions on JSON property names makes it easier to map them to the names of class properties. For example, although properties in JSON are technically allowed to contain spaces, this should be disallowed in practice. A common practice is to only use snake case names for properties, like in the following User type.

{
"title": "User",
"type": "object",
"properties: {
"first_name": {
"type": "string"
},
"last_name": {
"type": "string
}
}
}

These property names could easily be transformed to camel case to match the conventions of the target languages, provided that the JSON serialization tool being used is able to handle this mapping as well, possibly through annotations. The native code uses its native conventions.

class User(
val firstName: String,
val lastName: String
)

Likewise, enumerated values should be constrained. They should generally only allow strings with a certain format, to be easily mapped to naming conventions for enums in different languages. The following JSON describes a status value which must be a among those defined in a predefined set.

{ 
"title": "status"
"enum": ["started", "in_progress", "done"]
}

Which leads to the following enum declaration, in uppercase to match platform naming conventions.

enum class Status {
STARTED,
IN_PROGRESS,
DONE
}

For all names used in JSON schemas, strict conventions should be enforced, to guarantee that code generators are able to easily map the structures to valid declarations in object-oriented languages.

Required vs optional properties

JSON properties may or be optional or not, which in most object-oriented languages amounts to determining whether their value may be null.

However, in JSON objects, null values may be represented in one of two different ways: the property might be entirely missing from an object, or its value might be null. JavaScript is essentially the only language which also has this distinction in its type system (unsurprisingly, since JSON is a derivative of Javascript objects), through the undefinedvalue, which is distinct from null. Other languages typically have difficulty representing these two different concepts, conflating them to only null. The sensible thing to do for the schema to be compatible across different languages is therefore that it doesn’t give a different semantic meaning to a property having a null value versus being omitted, and establishes a convention on which one is used.

Correspondingly, JSON schema provides two different concepts to represent these two different forms of nullability, required properties in objects and the null type for properties.

Using the required keyword for objects, properties in corresponding classes may be missing if and only if they are not explicitly marked as required.

{
"type": "object",
"properties": {
"email": {
"type": "string"
},
"phone": {
"type": "string"
}
},
"required": {
"email"
}
}

In the above schema, the email property is marked as required, therefore it’s non-nullable, while the phone property is nullable because it’s not explicitly marked as required. This leads to the following class to represent this object.

class ContactInfo(
val email: String,
val phone: String?
)

However, if an instance of this class with a null phone value is serialized to JSON, the resulting object will only be valid according to the JSON schema if the serialization omitted the phone property entirely. Omitting null values is an option that many serializers support, but nevertheless it can be prone to error. Alternatively, in order to allow for null to be passed as a value, the property’s type must be declared to allow null.

{
"type": "object",
"properties": {
"email": {
"type": "string"
},
"phone": {
"type": [ "string", "null" ]
}
},
"required": {
"email"
}
}

This way, JSON objects without the phone value are valid.

{
"email" "[email protected]"
}

As are JSON object with null as the value for phone, since it’s still not required.

{
"email" "[email protected]",
"phone": null
}

As explained above, these two objects should be considered semantically identical.

Required properties should be explicitly declared as such. For optional properties, either establish conventions on using null vs omitting the property, or support both indiscriminately.

Additional Properties

By default, JSON schema ignores unrecognized properties within objects, allowing them to be present in JSON objects while only performing validation on the properties which are declared. Consider the following schema.

{
"type": "object",
"properties: {
"name": {
"type": "string"
}
}
}

The following JSON, containing an unknown age property, is considered valid according to this schema.

{
"name: "John Smith"
"age": 27
}

Whether or not to allow this behavior is actually something which depends on the context. On one hand, disallowing unknown properties makes the contract more strict. On the other, it makes it less flexible.

  • Servers should disallow extra properties in the body of client requests which they receive, in order to perform strict validation of JSON to avoid accidental errors (for example typos). However this means that to remain backwards-compatible with out-of-date clients schemas must never fully remove properties in future iterations, they may only deprecate them.
  • Clients should allow extra properties in the body of server responses which they receive, in order to be forwards-compatible with future server updates. This means that servers will be able to add new properties to their responses without breaking any client-side validation.

To enforce this behavior, the JSON schema additionalProperties keyword is available, which can be set to false to disallow properties which aren’t explicitly declared.

In general, however, schemas should never rely to perform their semantics on unknown properties which could be allowed. All useful properties should be explicitly declared.

JSON has 2 distinct numeric types, and so correspondingly does JSON schema: integer and number (for floating-point values). Both of these types represent arbitrarily large numbers, and in the case of number with arbitrary precision. On the other hand, most statically typed languages use a fixed size memory space for numeric values, meaning that both the range and the precision are bounded.

Use conventions for numeric types, and avoid floating-point values when possible.

Integer

For integer values, if there is no minimum and maximum enforced. In order to be rigorous with the schema, clients should declare properties to be of BigInteger type.

{
"type": "object",
"properties: {
"count": {
"type": "integer"
}
}
}

The above schema would therefore yield the following class to capture the data.

class Records(
val count: BigInteger
)

This is correct, but it’s not the most practical solution. BigInteger isn’t as efficient as a primitive Int or Long, and it’ll have to be propagated to the entire code for this captured value. On the other hand, by declaring in the schema a range of allowed values, it becomes explicit that the value can be captured simply using an Int.

{
"type": "object",
"properties: {
"count": {
"type": "integer",
"min": -2147483648,
"max": 2147483647
}
}
}

This schema allows the safe use of an (32-byte) Int value for the count property, since the min and max constraints match exactly with Int.MIN_VALUE and Int.MAX_VALUE.

class Records(
val count: Int
)

Likewise, declaring a min and max value corresponding exactly to Long.MIN_VALUE and Long.MAX_VALUE would be an explicit indication that the property could be represented using a (64-byte) Long.

Using these declaration is however verbose. Some tools generate the schema automatically, which mitigates this problem, otherwise it can actually be easier to simply agree on a convention for the bounds of all integer values.

Float

Most statically typed languages implement the IEEE 754 norm to represent floating-point values in memory. Unfortunately, this norm doesn’t match well with the JSON representation of number, which is just a string representing the value in base 10. This means that it’s entirely possible to run into a variant of the 0.1+0.2≠0.3 problem if capturing JSON number values using a Float, or even a Double, as well as running into a host of other issues with precision and bounds. This is not something which can be fixed using constraints on the JSON schema.

{
"type": "object",
"properties": {
"length: {
"type": "number"
}
}
}

Technically, in order to be certain to fully capture the data from a JSON number value like amount declared in the schema above, it needs to be declared as a BigDecimal .

class Size(
val length: BigDecimal
)

The best alternative to using BigDecimal is to agree on conventions on the bounds of values and their expected precision. If all languages using the API rely on the same memory representation for floating-point values, then this becomes a non-issue. A sound principle is to avoid floating-point values in JSON whenever possible (as a reminder, never use Float and Double for monetary calculations).

Data types in object-oriented languages typically use collections types like List , Set, and Map. JSON schema provides constructs which can be used to describe these data structures.

List and Set

List values are directly mapped to JSON arrays, and are described in JSON schema using the array type. To allow the generic type of the list to be defined, the items allowed in the JSON array must always be defined in the schema.

{
"type": "object",
"properties": {
"members: {
"type": "array",
"items": {
"type": "string"
}
}
}
}

The above schema corresponds to the following class declaration.

class Group(
val members: List<String>
)

Declaring a Set is very similar, only requiring the uniqueItems JSON schema keyword to be added.

{
"type": "object",
"properties": {
"members: {
"type": "array",
"items": {
"type": "string"
},
"uniqueItems": true
}
}
}

In this case, the above schema corresponds to the following class declaration.

class Group(
val members: Set<String>
)

The type of items in JSON arrays must always be defined, to allow generic collections to be generated.

This is a constraint that OpenAPI itself also imposes on schemas for arrays.

Maps

In some cases, it’s useful to have properties in a JSON which correspond to a map structure, with dynamic keys which vary from one request to another. A class like the following holds data mapping from revision ids to author names.

class Revisions(
val authors: Map<String, String>
)

This data structure would yield a JSON like the following. The keys in the JSON object could however change completely over time, which is why it’s important that the content is captured as a Map.

{
"authors": {
"d76fb677": "Simone Paul",
"da6dc75a": "Carla Eaton"
}
}

This type of JSON is described in JSON schema using the additionalProperties keyword, providing a schema describing the type of values in the map. Since the data structure is a JSON object, the keys in the Map are always strings.

{
"type": "object",
"properties": {
"authors: {
"type": "object",
"additionalProperties: {
"type": "string"
}
}
}
}

It’s important to notice that if the schema was to provides both definitions of some fixed properties and some additionalProperties, it would become difficult to build a classes to represent it. Object definitions which declare additional properties should therefore never also declare regular properties, so that they can clearly by identified as Map values in the type system of the target language.

JSON schema definitions which declare additionalProperties instead of properties correspond to a Map from strings to values of the specified target type.

Many types actually support direct serialization and deserialization to and from a string, making them easy to transport as JSON values. The format keyword provides definitions for these types.

JSON schema actually has several built-in formats already supported in the specification. For example, to support a date, use the date format.

{
"birth_date": "1998-09-09"
}

The above JSON object example would be described using the following JSON schema.

{
"type": "object",
"properties: {
"birth_date {
"type": "string"
"format": "date"
}
}
}

Which leads to the following class declaration to capture the data from this object.

class Person(
val birthday: LocalDate
)

Many built-in formats can be directly mapped to standard classes. The definition of the formats might not perfectly match the behavior of the classes, but usually they’re close enough to be usable in practice.

To support additional class definitions in JSON schema, use custom formats, as allowed by the JSON schema specification.

[…] custom formats may also be used, as long as the parties exchanging the JSON documents also exchange information about the custom format types

For example, for a field containing a ISO time zone identifier, the following schema uses a custom time-zone format.

{
"type": "object",
"properties: {
"zone {
"type": "string"
"format": "time-zone"
}
}
}

Which would lead to the following class being declared.

class Event(
val zone: ZoneId
)

Consumers of the schema have to all agree that the semantics of the time-zone format correspond to the behavior of the ZoneId class.

Use the format keyword to describe custom types which are serialized to strings in JSON.

Many modern languages contain some notion of restricted class hierarchies (formally called algebraic data types in type theory), whether it’s through sealed classes in Kotlin or union and intersection types in Typescript. These types represent in the type system the concept that a value may be one of two data structures.

This corresponds to the oneOf statement in JSON schema. However, JSON schema itself doesn’t provide any restriction on the usage of oneOf. It’s technically correct to define a value as being either numeric or boolean, even though for many languages this is tricky to represent in the type system (or it requires wrapping the value in an object, which is cumbersome). The usage of oneOf should therefore be restricted to objects.

The following schema describes an identifier which could be either a database id (a 64-byte integer) or a UUID string.

{
"oneOf": [
{
"type": "object",
"properties": {
"id": {
"type": "integer"
}
}
},
{
"type": "object",
"properties": {
"uuid": {
"type": "string"
}
}
}
]
}

The following JSON object would then be a valid identifier for a database id.

{
"id": 8123
}

While the following JSON would be a valid identifier using a UUID.

{
"uuid": "d408e29a-415d-11ec-ba1c-0242ac130002"
}

This identifier value would typically be represented using a sealed class in Kotlin.

class Identifier { 
class Database(
val id: Long
) : Transaction()
class Universal(
val uuid: UUID
) : Transaction()
}

However, while this data structure is convenient to for serialization to JSON, it often doesn’t doesn’t play well with JSON parsers. There’s no clear and obvious indication in the schema of what is the criteria to distinguish between a database id and a UUID. It can be helpful to add an explicit criteria, for instance by establishing a convention that each case in a union types must have a distinct const value identifying it, which by convention would always be called type.

{
"oneOf":[
{
"type":"object",
"properties":{
"type": {
"const": "database"
},
"id":{
"type":"string"
}
}
},
{
"type":"object",
"properties":{
"type": {
"const": "universal"
},
"uuid":{
"type":"string"
}
}
}
]
}

The class generated from this schema remains identical, because the type property is erased. It’s only meant be used during parsing to determine which subclass of Transaction to instantiate, and it would be included in the resulting object during serialization. The JSON for a universal id would therefore be like the following.

{ 
"type": "universal",
"id": "d408e29a-415d-11ec-ba1c-0242ac130002"
}

This can be useful to make some tooling work better, but given that JSON by default doesn’t provide any guarantees on the order of fields, it’s still far from a perfect solution.

If the language doesn’t support union types, another possibility is to merge the oneOf statement into a single type. This is only possible if there’s no conflict between different cases on fields with the same name.

class Transaction(
val id: Long?,
val uuid: UUID?
)

This is a useful fallback, but this type loses information on the semantics of different combinations of properties.

Defining union types in the JSON schema provides a clearer contract and allows more type safety in some languages. However, this is a complex feature to combine with the type systems of different languages and their respective JSON tooling, and may require additional conventions to be compatible with them.

On a side note, this is actually a feature for which XML is much better suited, because nodes are inherently typed.

This catalog of conventions to establish relationships between JSON schema concepts and object-oriented language concepts should give a good idea of the difficulty when trying to build API code based on a JSON schema specification. At the heart of the problem is the fact that JSON schema is designed as a JSON validation tool, not as a descriptive tool for API contents. This makes it difficult to use to use it to describe classes which represent JSON objects.

Enforcing the schema

Given that JSON schema is a validation tool, the question which arises is whether to use it to perform strict validation of the JSON exchanged in an API. Tools like Ajv are convenient for this, providing a highly configurable validation library to ensure that JSON passes checks based on the schema, and returning clear errors if not.

However, it’s generally not useful for clients to perform strict validation on the JSON received from the server. This adds extra overhead even though the server is the source of truth for the JSON contract. For this to be viable though, the server implementation must ensure that the JSON it returns in its responses is valid. If not, inconsistencies can lead to nasty issues for clients. For example, in Kotlin the nullability of properties is erased at runtime, which can lead some JSON parsers succeeding in deserializing an object even when some of its required properties are missing, causing unexpected errors later on at runtime when accessing the property. The server should be sure in the content it provides that required properties are never missing, whether by performing automated validation on its own responses or by guaranteeing it through its own data types derived from JSON schema.

On the other hand, it can be useful for servers to use JSON schema to automatically validate the JSON sent by clients in their requests. This can facilitate the development of the API with clearer errors for invalid JSON in request body.

The server is responsible for enforcing JSON schema, ensuring that API requests and responses conform to the contract.The client merely consumes the schema for its code, and the structure of the JSON classes themselves serves as a guarantee for correctness.

JSON schema for tooling

This isn’t to say that there isn’t a place for the full JSON schema specification in programming. It remains a great tool in many cases, for example when describing manually maintained JSON configuration files. In these case many of the restrictions prevented above aren’t relevant. Many IDEs have out-of-the-box support for JSON schema, for example IntelliJ IDEA allows using using custom JSON schemas and Visual Studio Code supports JSON Schema integration. Providing a schema triggers the validation of JSON files with highlighting and auto-completion. JSON schema has its critics, but the availability of tooling for it continues to make it a great option for many use cases.

If you’re manually maintaining the JSON schema describing an API, it’s actually a good idea to fork the meta-schema of JSON schema from the specification. This way, by applying this meta-schema to the schemas maintained locally, developers are sure to not accidentally introduce into them some constructs which would be difficult for consumers of the schema to interpret. Use the full JSON schema specification in your meta-schema, which restricts the use of JSON schema in the specifications describing the API, which are used by clients.

{
"type": "object",
"properties": {
"type": {
"const": "object",
},
"properties": {
"type": "object",
"patternProperties": {
"^[a-z]+(_[a-z]+)*$": {
"oneOf": [
{
"const": {
"type": "string"
}
},
{
"const": {
"type": "integer",
"min": -2147483648,
"max": 2147483647
}
}
]
}
},
"additionalProperties": false
}
},
"additionalProperties": false
}

This meta-schema, which would validate the schemas which describe the API, only allows an object which may only contain properties with a lowercase snake case name and which must be of type string or 32-bit integer (for which the range must always be explicitly specified). This meta-schema would be used to enforce this restriction on developers, most likely as part of CI checks on the schemas maintained manually. It’s a very simple and restrictive example, real meta-schemas would most likely allow for more complex structures, but it shows the basic idea of using meta-schemas, which themselves use the full extent of JSON-schema specification, since they’re only fed into JSON schema tooling, for validation of API schemas to make sure these match established conventions and restrictions.

Use JSON schema for validation of manually maintained JSON files, to assist with writing and validating them. Use meta-schemas to validate schemas and restrict them.

Alternatives

Are there better options to build an API and describe its structure for clients? The main one which comes to mind is protocol buffers, developed by Google. This standard was designed first and foremost with code generators in mind, integrating the primitive data types common to most statically typed languages. However, while it’s a fairly popular format, which supports Kotlin, it’s still not as ubiquitous as JSON, which can harm its portability.

You could also create your own descriptor format for JSON APIs, but you would lose the benefit of much of the tooling which comes from JSON schema itself, which is very useful for instance for automated documentation generation and automated JSON validation.

Making the most of JSON Schema

Making the most out of JSON schema for APIs therefore requires careful conventions, which must be strictly enforced. Popular libraries to generate code from schemas do themselves also have to introduce conventions and restrictions, which generally resemble the ones presented in this article. The scope of JSON schema is far too broad. Many keywords of JSON schema deliberately haven’t been presented in this article because they’re inconvenient to represent in object-oriented languages (regular expressions for strings, custom integer range, …). If you decide to allow them in the JSON schema for your APIs, be sure to think about the impact for developers who will write the code for the API.

Restrict the JSON Schema used to describe your API to use only predefined conventions, so that it presents patterns which are simple to translate to object-oriented types.

Martin Devillers

Source link