RFC: Updating data types in a backwards compatible way

[kjetilkjeka]

Note: This RFC is still highly incomplete. As long as the [WIP] tag is in place I will edit this issue rather freely

Requirements

• This RFC requires RFC: Separate versioning of data types #35 to land.
• This RFC is contradicting to Future: Freezing DSDL signature #4, it requires this suggestion to be abandoned.

Motivation

Let's assume #35 lands. DSDL is now pretty great, we even have a way to talk about versions which make it trivial to see what kind of versions that are compatible. The problem is that the implementations don't care. They will see two data types with different signature even when they only differ a minor version and should be regarded as compatible.

This RFC will attempt to solve this issue by using the same data type signature for all compatible frames.

Explanation

When the dsdl is compiled for a data type, it should always use the data type signature of the oldest compatible message. This is the version on the form @version <MAJOR>.0.0 where <MAJOR> matches the data type under compilation.

The exception is not yet stabilized messages (major version 0). For those the dsdl compiler should use the data signature of the message itself.

This will require that all versions with the current major version is kept in the dsdl file. Several data type versions in same file is already accepted if we accept the use <DataType> version x.y.z from #35.

Examples

Let's use the same frame as from #35
This makes the file initialy looking like

@version 1.0.0
uint2 state
void2 reserved1
void4 reserved2

But after updating the version a couple of times, it quickly looks like

@version 1.0.0
uint2 state
void2 reserved1
void4 reserved2

@version 1.0.1
uint2 state # i like comments
void2 reserved1
void4 reserved2

@version 1.0.2
uint2 STATE_IDLE = 0
uint2 STATE_TRANSMITTING = 1
uint2 STATE_RECEIVING = 2
uint2 STATE_ERROR = 3
uint2 state # i like comments

void2 reserved1
void4 reserved2

When the major version is incremented, old versions may be removed (unless they're depended upon by other frames)

@version 2.0.0
uint2 STATE_OK = 0
uint2 STATE_STARTING = 1
uint2 STATE_ERROR = 3
uint2 state # i like comments

void2 reserved1
void4 reserved2

When updating to next major version, the old on is scrapped as well

@version 3.0.0
uint2 STATUS_OK = 0
uint2 STATUS_STARTING = 1
uint2 STATUS_ERROR = 3
uint2 status # i like comments

void2 reserved1
void4 reserved2

With the last additions, this ends up looking like

@version 3.0.0
uint2 STATUS_OK = 0
uint2 STATUS_STARTING = 1
uint2 STATUS_ERROR = 3
uint2 status # i like comments

void2 reserved1
void4 reserved2

@version 3.0.1
uint2 STATUS_OK = 0
uint2 STATUS_STARTING = 1
uint2 STATUS_ERROR = 3
uint2 status # This is the status

void2 reserved1
void4 reserved2

@version 3.1.0
uint2 STATUS_OK = 0
uint2 STATUS_STARTING = 1
uint2 STATUS_ERROR = 3
uint2 status # This is the status

uint3 STATE_IDLE = 0
uint3 STATE_TRANSMITTING = 1
uint3 STATE_RECEIVING = 2
uint3 STATE_ERROR = 7
void3 state

void3 reserved

Assuming the use <DataType> version x.y.z syntax from #35 is accepted. If a message depend on
2.0.0 after upgrading to 3.0.0 the dsdl file will look as following.

@version 2.0.0
uint2 STATE_OK = 0
uint2 STATE_STARTING = 1
uint2 STATE_ERROR = 3
uint2 state # i like comments

void2 reserved1
void4 reserved2

@version 3.0.0
uint2 STATUS_OK = 0
uint2 STATUS_STARTING = 1
uint2 STATUS_ERROR = 3
uint2 status # i like comments

void2 reserved1
void4 reserved2

@version 3.0.1
uint2 STATUS_OK = 0
uint2 STATUS_STARTING = 1
uint2 STATUS_ERROR = 3
uint2 status # This is the status

void2 reserved1
void4 reserved2

@version 3.1.0
uint2 STATUS_OK = 0
uint2 STATUS_STARTING = 1
uint2 STATUS_ERROR = 3
uint2 status # This is the status

uint3 STATE_IDLE = 0
uint3 STATE_TRANSMITTING = 1
uint3 STATE_RECEIVING = 2
uint3 STATE_ERROR = 7
void3 state

void3 reserved

Implementation

This can be implemented in a backward compatible way. If #35 is accepted, no data types will list its version and thus be in version 0.0.0. If this feature is accepted before any data type increments its version to 1.1.0 (is stabilized and improved upon). This feature will be fully backward compatible.

This should be pretty equivalent in implementation overhead as #4 while solving the same problem in (what I believe is) a cleaner way.

Alternatives

The alternative proposed in #4 could be used instead. But if the addition of seperate versioning lands, i feel like this proposal is a more hollistic approach.

Drawbacks

• Compiling dsdl becomes a more complex task
• For a message to be compatible it will no longer require it be identical. However, it must contain the same "base version" in the dsdl it was compiled from. (To the same extent as Future: Freezing DSDL signature #4).

[kjetilkjeka]

@pavel-kirienko I'm close to a finished suggestion here as well.

This ended up beeing much closer to what you propsed in #4 than what I originally suggested. Do you see any lacking bits I should add before removing [WIP] tag?

[pavel-kirienko]

@kjetilkjeka Sorry about my unresponsiveness, I've been travelling and will continue doing so until tomorrow. I should take a closer look at this and provide a meaningful response around the weekend. Thanks!

(edit weeks later: for the benefit of posterity I must indicate that my response has been published in a different thread at #35)

[kjetilkjeka]

Closing this as a superseeding proposal is soon to come