RFC: Separate versioning of data types

[kjetilkjeka]

Summary

Instead of talking about DSDL versions and stabilization of DSDL as a whole, this RFC suggests the alternative of versioning data types individually. This will ease the task of stabilization, allowing it to happen sooner, giving a more flexible way to describe compatibility between DSDL, and enabling a way to accept bit compatible data types without freezing data type signatures [#4]. Some robustness is also gained by having ways to check if single frame data types have conflicting definitions.

Motivation

There is an obvious need to stabilize DSDL. Every time a message is changed every software depending on it breaks. There is also a need to not prematurely stabilize. Being stuck with badly functioning data types will be the consequence, and it won't be fixable until next major version.

With the current system (waiting for dsdl 1.0) no stabilization is realistically within reach. For every battle proven data type, there is a data type that is not yet ready as well. By keeping on adding data types when it is convenient a dsdl stabilization is further pushed in the future as well.

By accepting separate versioning of data types, these problems should be much easier to handle. The proven parts of dsdl can be stabilized while allowing rapid development of the non-stabilized parts. This also allows software that only relies on stable messages to be stabilized, this is a huge advantage for uavcan users.

The data type signatures are really great! They let you know when you're using the same data types, and even enforce this programmatically for multi frames. The problem is that when you do have a miss match, it's not trivial to find out which unit is erroneous. Separate versioning of data types give us tools to talk about the version of the messages. For instance, if a unit (outside our control) states that it sends 0.2 frames and we know that we currently accept 0.3 frames, we will be much closer to fix the problem. It will also make it possible to (non-automatically) detect different single frame transfers, this would not otherwise be noticed as no CRC is used.

There are a lot of things that can go wrong when stabilizing DSDL. By stabilizing one data type at a time we make eventual "screw ups" less possible and more handleable. By combining this with using gained knowledge from previous stabilizations to stabilize new data types, stabilization should have a much lower chance of causing headaches.

Explanation

The approach to versioning is based upon the proven idea of semantic versioning but adapted to make sense for dsdl. The version information is encoded directly in the name of the file where the definition is contained such as <ID>.<Name>.<Major version>.<Minor version>.uavcan. For example 341.NodeStatus.1.0.uavcan

• A change that respects bit compatibility, and doesn't break code compatibility is allowed to be done by incrementing minor version. This holds even if the data type signature is changed.
• All other changes must increment the major version.
• The exception to this is if the more important version numbers are zeros. E.g. 0.2 is allowed to break systems dependent on 0.1.
• Version 0.0 is special since no data types tagged with this version number should be considered the same or compatible in any way (everything goes).

Examples

Let's use the following frame as an example

#version 1.0
uint2 state
void2 reserved1
void4 reserved2

If we were to add a comment we would have to increment the minor version (this doesn't change data type signature nor bit compatibility)

#version 1.1
uint2 state # i like comments
void2 reserved1
void4 reserved2

In the same way, adding constants would only require an increment of the minor version

#version 1.2
uint2 STATE_IDLE = 0
uint2 STATE_TRANSMITTING = 1
uint2 STATE_RECEIVING = 2
uint2 STATE_ERROR = 3
uint2 state # i like comments

void2 reserved1
void4 reserved2

But removing/changing a constant would require a major version (since it breaks backward compatibility)

#version 2.0
uint2 STATE_OK = 0
uint2 STATE_STARTING = 1
uint2 STATE_ERROR = 3
uint2 state # i like comments

void2 reserved1
void4 reserved2

Changing the name of fields or constants would not retain code compatibility and will also require a major version increment.

#version 3.0
uint2 STATUS_OK = 0
uint2 STATUS_STARTING = 1
uint2 STATUS_ERROR = 3
uint2 status # i like comments

void2 reserved1
void4 reserved2

Removing/editing a comment will both keep data type signature and code compatibility, thus only requiring minor version increment

#version 3.1
uint2 STATUS_OK = 0
uint2 STATUS_STARTING = 1
uint2 STATUS_ERROR = 3
uint2 status # This is the status

void2 reserved1
void4 reserved2

Using new fields in a bit compatible way or changing the name of void fields only require a minor version update.

#version 3.2
uint2 STATUS_OK = 0
uint2 STATUS_STARTING = 1
uint2 STATUS_ERROR = 3
uint2 status # This is the status

uint3 STATE_IDLE = 0
uint3 STATE_TRANSMITTING = 1
uint3 STATE_RECEIVING = 2
uint3 STATE_ERROR = 7
void3 state

void3 reserved

Implementation

This feature should be regarded as a low impact feature. The feature itself will not change the generated code. Meaning that by treating files without version as 0.0 (and forbidding explicit version 0.0) this change will be fully backward compatible. This means that for example 341.NotStatus.uavcan "really" means 341.NodeStatus.0.0.uavcan, while explicitly writing 341.NodeStatus.0.0.uavcan will result in an error.

Open questions

How do we handle versioning of data types that uses other data types?

In some way or another we will have to specify the exact version the data type is dependent on. One way to do this would be as following

@use Command version 0.2
Command[<=15] commands

Another alternative would be

Command.0.2[<=15] commands

The following rules must be adopted

• You must not have a stable data type depending on an unstable data type
• When updating to a dependency with higher major/minor version, you must increment the major/minor version
• The exception beeing that in version 0.0 you're not required explicitely list the dependency. This is required to ensure backward compatibility..

Further work

• A way to accept bit compatible frames (RFC: Updating data types in a backwards compatible way #36)
• Creating CI tests to see if the rules of versioning are upheld.

[kjetilkjeka]

@pavel-kirienko I think I'm close to suggesting a complete system for versioning. This will be the foundation for accepting bit-compatible frames. Take a quick look before I remove the WIP tag (one of the upcoming days) and say if you see contradictions or unspecified behaviour.

[pavel-kirienko]

Currently, the specification states that the name of a namespace cannot begin with a digit. As such, abc.xyz.Def is a valid name, whereas abc.xyz.Def.1.2 is not. The unexpected side effect of this is that we can encode the version information of a data type definition directly in the name of the file where the definition is contained. For example: 341.NodeStatus.1.0.uavcan.

If the above approach were adopted, it would provide us with an uber convenient way to specify the version of dependencies. Instead of this:

@version 0.0.3
@use Command version 0.0.2
Command[<=15] commands

...we could write this:

Command.0.0.2[<=15] commands

Having the version number encoded in the name has the following advantages over the current proposition:

• Having all versions defined in the same file would make definitions large and hard to grasp for us humans. Service type definitions would be hard to parse (for humans, not machines) because of multiple service response markers occurring in the same text file.
• Having the version number defined in the file name would allow people to gain more information about data type sets simply by viewing directory listings, without having to open each file separately.
• It is coherent with the proposed approach of specifying dependency versions.
• It matches well the approach we're already using: uavcan.equipment.gnss.Fix & uavcan.equipment.gnss.Fix2.

---

The response structure of the service uavcan.protocol.GetDataTypeInfo would have to be extended with a list of supported versions of the requested data type. The maximum number of reported version numbers would have to be limited. Shall the number of supported data types exceed the limit, the oldest versions should be omitted from the response. Does that make sense?

The limit could be set at 10 versions.

---

I understand the value of semantic versioning in the context of software versioning, but it is not evident to me why is it required to have the explicit patch version in DSDL. The information conveyed by the patch version is equally accessible via version control systems. I propose to remove the patch version from the RFC and keep only those that affect the semantics and/or the binary layout of data types: major and minor.

---

I think it is perfectly fine to make the version numbers mandatory from the start. That would promote good data type design practices and make the default data type set more consistent. In order to switch to the forced versioning, we would have to go over the existing definitions and manually assign each one a version number, which would be trivial to do.

---

Have you had a chance to work out how exactly different versions should be handled when receiving and emitting transfers?

My take on this is that when receiving a transfer, the receiver would compute the data type signature for each known version of the data type until a match is found. Having found a match, the receiver should unmarshal the transfer and do either:

1. attempt to automatically convert the unmarshaled data structure to the newest known version of its data type, or
2. simply report to the application that a data structure of version such and such has been received, and let the application handle that the way it sees fit.

The first approach is the most convenient from the standpoint of the application designer; the second approach is the easiest to implement.

When emitting a transfer, the library should select the newest version by default, but the application designer should have a way to specify the version manually.

[kjetilkjeka]

Currently, the specification states that the name of a namespace cannot begin with a digit. As such, abc.xyz.Def is a valid name, whereas abc.xyz.Def.1.2 is not. The unexpected side effect of this is that we can encode the version information of a data type definition directly in the name of the file where the definition is contained. For example: 341.NodeStatus.1.0.uavcan

Good suggestion, I agree with the advantages from your list. I support this.

If the above approach were adopted, it would provide us with an uber convenient way to specify the version of dependencies. Instead of this:

@version 0.0.3
@use Command version 0.0.2
Command[<=15] commands

...we could write this:

 Command.0.0.2[<=15] 

I'm not sure if I like it. My problems with it are:

• All fields using the same message should probably be forced to be the same version. This is enforced by separating version from type.
• When updating version, it would be "nicer" to only have to write the new version one place.

The response structure of the service uavcan.protocol.GetDataTypeInfo would have to be extended with a list of supported versions of the requested data type. The maximum number of reported version numbers would have to be limited. Shall the number of supported data types exceed the limit, the oldest versions should be omitted from the response. Does that make sense?

The limit could be set at 10 versions.

I ended up writing this up a bit different from the original suggestion we talked about.

First of all, I split up versioning and backward compatibility into two seperate RFC

• RFC: Separate versioning of data types #35 regards versioning
• RFC: Updating data types in a backwards compatible way #36 is about how messages can be backward compatible

Second of all I ended up suggesting something closer to your original proposal. This only requires using 1 signature. Check out #36 for full description.

I understand the value of semantic versioning in the context of software versioning, but it is not evident to me why is it required to have the explicit patch version in DSDL. The information conveyed by the patch version is equally accessible via version control systems. I propose to remove the patch version from the RFC and keep only those that affect the semantics and/or the binary layout of data types: major and minor.

After thinking about this I fully agree with you. Let's ditch the patch version.

I think it is perfectly fine to make the version numbers mandatory from the start. That would promote good data type design practices and make the default data type set more consistent. In order to switch to the forced versioning, we would have to go over the existing definitions and manually assign each one a version number, which would be trivial to do.

A 0.0 version would make everything fully backward compatible. But to go over definitions and initially assign 0.1 version to everything is fine as well. I think either one will be ok.

Have you had a chance to work out how exactly different versions should be handled when receiving and emitting transfers?

Again, my proposal is a bit changed compared to what we were originally discussion. Check out #36

[pavel-kirienko]

• All fields using the same message should probably be forced to be the same version.
• When updating version, it would be "nicer" to only have to write the new version one place.

Why though. What do we gain by introducing this restriction? By adding it, we condemn ourselves to a far uglier, in my opinion, way of specifying the versions of dependencies. My point is that the restriction has costs.

The response structure of the service uavcan.protocol.GetDataTypeInfo would have to be extended with a list of supported versions of the requested data type. The maximum number of reported version numbers would have to be limited. Shall the number of supported data types exceed the limit, the oldest versions should be omitted from the response. Does that make sense?
The limit could be set at 10 versions.

I ended up writing this up a bit different from the original suggestion we talked about.

I just went and re-read the proposition you referred to, and it doesn't seem to address the case when a new version of the data type is not backward compatible with the older one, so that multiple signatures per data type are needed. My proposition still holds.

[kjetilkjeka]

All fields using the same message should probably be forced to be the same version.
When updating version, it would be "nicer" to only have to write the new version one place.

Why though. What do we gain by introducing this restriction? By adding it, we condemn ourselves to a far uglier, in my opinion, way of specifying the versions of dependencies. My point is that the restriction has costs.

You say it's far uglier, I think it's nicer. It's (mostly) just a semantic thing, and both will probably work fine. If the concept of versioning is accepted we can discuss the semantics after?

The response structure of the service uavcan.protocol.GetDataTypeInfo would have to be extended with a list of supported versions of the requested data type. The maximum number of reported version numbers would have to be limited. Shall the number of supported data types exceed the limit, the oldest versions should be omitted from the response. Does that make sense?
The limit could be set at 10 versions.

I ended up writing this up a bit different from the original suggestion we talked about.

I just went and re-read the proposition you referred to, and it doesn't seem to address the case when a new version of the data type is not backward compatible with the older one, so that multiple signatures per data type are needed. My proposition still holds.

#36 states "When the dsdl is compiled for a data type, it should always use the data type signature of the oldest compatible message. This is the version on the form @version <MAJOR>.0.0 where <MAJOR> matches the data type under compilation."

We know that when a new incompatible data type is added the major version will be incremented (or minor if unstable). This means that the new data type is now the "oldest compatible" and its signature will be compiled from itself.

The proposal is somewhere between my original proposal and your original proposal (from #4). To be more precise it's something like automating your original proposal by using versioning.

Should I edit the issues to show the following things?

• patch versioning is out
• filenames should contain version at the end
• semantics for "imported" data types is a open question

[pavel-kirienko]

You say it's far uglier, I think it's nicer. It's (mostly) just a semantic thing, and both will probably work fine. If the concept of versioning is accepted we can discuss the semantics after?

I fully support the proposed concept of versioning, so consider it accepted. Except for the thing below which still needs sorting out.

We know that when a new incompatible data type is added the major version will be incremented (or minor if unstable). This means that the new data type is now the "oldest compatible" and its signature will be compiled from itself.

Complete removal of older definitions when adding a new major version partially defeats the purpose of the whole concept of versioning. Say, we made a new message v2.0, which new nodes can use. There will be older nodes which may continue using v1.x for an indefinite period of time. In order to guarantee compatibility with them, designers of newer applications will want to support both v1.x and v2.0. The current versioning system does not account for that.

Therefore it should be possible to define more than one major version of any message type. That means that we're going to need more than one signature per data type.

Does that make sense?

Should I edit the issues to show the following things?

Absolutely.

[kjetilkjeka]

So I'm finally back at this. But it's been a long time. I'll sum up a few things so I know we are still on the same page. @pavel-kirienko shout out if there are unsettled questions or disagreements.

• We are certain that we want the .<major_version>.<minor_version>.uavcan format for DSDL message versioning.
• We want to specify version dependency explicitly like Command.0.0.2[<=15]
• We must keep the old major version of a message but we're allowed to overwrite minor versions (unless major version is 0) e.g we must have 1.7 and 2.1 (and 0.1, 0.2, 0.3), but not both 1.6 and 1.7.
• At least for now, we want to treat file names without version as 0.0 to prevent breakage.

[pavel-kirienko]

Seems sensible.

Have you thought about the best way to carry the version information over the wire? How does a node know which version of DSDL to use to decode a given transfer? Two things come to mind, not sure if they make sense:

• Try the newest version; if decoding fails (e.g. not enough data or too much data), try an older version;
• Mix version ID in the transfer CRC together with the Data Type Signature.

[kjetilkjeka]

I've not yet been able to work out all the details around how we should ensure compatibility. I feel certain that we should have a notion of version and that this suggestion not will put us in a worse situation at the uavcan protocol level.

What I'm currently thinking is that:

• The CRC should "always" fail for incompatible frames.
• When a message with a field not supported for all compatible messages is sent, it should be ignored in nodes that do not support the field.
• When designing new versions of messages we need to be aware that a new message with a 0 value is indistinguishable from an old message with a void field.

(compatible) frame sends a message to a node supporting an old (compatible) frame the extra fields

I will start doing the groundwork with dsdl versioning, and we can discuss message compatibility together with the other protocol changes ("version bit, TAO and CRC changes)

[kjetilkjeka]

This comment is a bit hasty, I'm going to write up a more thought through proposal later. Consider this as a heads up for something we need to think about for the new version of the uavcan protocol.

The reason for initializing the transfer CRC with the data type signature was to ensure that the received message was the exact same message as the one transmitted. As we're aiming to have a notion of compatiblity between messages this is perhaps not what we want anymore.

By introducing versioning of messages, a message (with its version) will (should) be immutable once published. This makes it easier to reason about compatibility from the definitions, and should be easy enough to write a checker that enforces some of the properties. This may also make it more acceptable to not enforce compatbility as strictly as it has been for the new version of the protocol.

What I'm thinking is instead of initializing the CRC with the normalized message (hashed) we instead initialize it with a structure containing:

• namespace + Name
• Major version (and minor version if major version == 0)
• A dsdl-ish definition that shows layout of fields with dynamic sized fields (dynamic arrays, unions, structures with dynamic fields etc).
• (perhaps some other things that cannot change and needs to be enforced to keep compatibility)

@pavel-kirienko can you see anything that in principle won't work with this approach?

cc: @thirtytwobits since you're currently working on supporting the new protocol version.

[kjetilkjeka]

OpenCyphal-Garage/uavcan.github.io#40 and OpenCyphal-Garage/uavcan.github.io#41 proposes how backwards compatibility can be ensured for the new protocol version. This proposal is a bit more "from the ground up" and doesn't feel as much as an afterthought as what we've discussed before.

[pavel-kirienko]

Here I want to admit that when facing difficult problems, my design workflow may involve a bit of mindless wandering around the campus. Can be quite enlightening.

Thanks, Kjetil. It took me some time to rebuild the big picture in my head since I was engaging with completely different matters lately. I believe that part of the reasoning in this and related proposals are based on a false assumption (#35 (comment)):

A change that respects bit compatibility, and doesn't break code compatibility is allowed to be done by incrementing minor version. This holds even if the data type signature is changed.

Our experience indicates that this is an exceptionally rare circumstance. All of the changes that we implemented to the existing data type definitions so far ended up being backward-incompatible at the bit level.

Another important idea to keep in mind is that even a bit-compatible change to a data type may still render the resulting definition semantically incompatible with the old definition. An example of that is provided in the current specification where two fields are swapped names (although this is not the only error mode where a semantic incompatibility can be introduced). This is why I don't think the idea to abandon the stricter DSDL signature we have now in favor of a semantic-invariant serialization signature you proposed is a viable one.

An automatic way of ensuring semantic compatibility without the unnecessary restrictions of the existing DSDL signature logic is probably possible, but it would require some complex in-depth analysis of data types. I don't think the benefits of that worth the effort, at least not for v1.0. If there ever is a need for a human to squelch a false-positive DSDL incompatibility error, we could (as an easier alternative) consider resurrecting this old proposal: #4.

I'm sorry that I failed to recognize this problem with your proposal earlier; as I said, I just got back to UAVCAN-related issues recently, so it was difficult to spot from outside.

---

I suggest keeping only the major version number. As a side effect, this will also eliminate this ugly special case you mentioned earlier (#44 (comment)):

Append Ver0 to the typename for every minor version that has 0 as its major version.

Then, as far as serialization/deserialization logic is concerned, different versions of a data type are to be treated as completely different data types, at least for now. Eventually, it would be quite interesting to explore the possibility of automatic data type promotion from older versions to the newer versions upon reception, but that would be a local node-specific change that doesn't need to reflect in the protocol specification whatsoever. Consider it a bit of syntax sugar for users of high-level libraries like libuavcan or uavcan.rs.

A remaining issue, as I see it, is how do we convey the version information over the wire? In the case of CAN FD, I suggest to add a single byte per transfer, containing the version number of the encoded data type. The byte should be probably allocated either before the payload or after the padding bytes. It might be sensible to use only 6 of the 8 bits for the version number (assuming that 64 versions are enough for everybody?) and keep the two major bits reserved for future use. That will eliminate the need for the trial-and-error approach to decoding, and also alleviate the problem with single-frame transfers which do not carry any DSDL signature information (because they don't have CRC).

The case of CAN 2.0 is trickier because the extremely limited amount of data per frame does not easily permit us just to throw away a whole byte. I expect CAN FD to completely displace CAN 2.0 eventually, so it might be sensible to view CAN 2.0 as a semi-deprecated transport that eventually is going to fall out of use. In that spirit, it might be acceptable to resort to the trial-and-error approach to decoding, starting with the newest available version and going to the older versions if the deserializer runs out of data, encounters too much data, or the CRC ends up being incorrect.

Do you think I am thinking in the right direction?

[kjetilkjeka]

I've taken some time to think through all of this once more. I came to a different conclusion than what you did. Let's discuss....

First of all, we're really discussing two separate things here. One of them is whether minor version is a useful concept, and the other is whether populating a void field should be possible to do while being compatible with the former data type. I think the minor version have value even if we do not allow compatible population of void fields. I've split my reasoning up in two parts.

Why we should have a minor version

Stability guarantees

Since consts are not being included in the data type signature a change of a const value or name can break compatibility without changing the data type signature. This means that the data type signature is not enough on it self to argue for compatibility. This versioning proposal gives the complete toolbox required to reason about compatibility, if we follow some rules.

One of these rules needs to be that a released version of a data type can never change without changing its version number. If this is not true, it would be possible to have incompatible definitions for the same version and that means that versioning doesn't make sense at all. This means that editing comments or adding consts can not be done without releasing a new version. If no minor version existed there would suddenly exist two incompatible versions that really should have been compatible (only differed by a const definition).

Furthermore, if a const were to change in a way that was incompatible we could bump up the major version. This would express that semantic compatibility had been broken in a way that would not be possible if there existed no versioning, or every change bumped the major version.

Experimentation before stabilization

Another important part of this proposal is the ability to experiment with data types before stabilizing them. This requires a way to distinguish between stable versions and unstable versions. Even though versions with major version 0, like 0.1, also will be guaranteed immutable they will also convey information that we will not be reluctant to push new experimental versions.

Once a 1.0 version is published the datatype should be tested and most faults should be fixed. This matches the observation that "compatible changes" are more rare than breaking changes. This do not imply that the few small changes are of little value, I see a lot of value in being able to reformulate comments, add consts and populate explicitly reserved void fields. This allow us to fix small details in hindsight. Only in the few cases where big mistakes were made, or the future makes the original data type obsolete a 2.0 version should be published.

Some numbers

Our experience indicates that this is an exceptionally rare circumstance. All of the changes that we implemented to the existing data type definitions so far ended up being backward-incompatible at the bit level.

To gather some data on this, I went through all the commits in the DSDL repo and checked those named "Merge pull request...". I counted the number of changes to a datatype requiring a major or minor version based on bit layout or semantic differences. I found:

• 6 cases of changing data types that would require a major version change
• 4 cases of changing data types that would require a minor version change (2/4 is comment only changes)
• 12 cases of adding new data types.

I think that these numbers alone makes it attractive to have a way to change things without breaking anything. But more importantly i believe looking at the current number of changes may be misleading for two reasons. The first on is simple, there will always be more big changes in the start. The second one is that when these changes were made there was no way of making any significant changes without breaking stuff (due to the restrictive nature of data type signature). Perhaps there was changes that would have been desirable to do but not important enough to justify breakage.

Why we should initialize CRC with something that allow us to populate void fields

The current data type signature does not guarantee compatibility.

The most important argument for not relaxing the signature used for initializing CRC is that it cannot be used a a proof for compatibility. However, the assumption that equal data type signatures guarantees compatibility is not correct as consts are not included in the data type signature. It would be easy to break semantic compatibility (while retaining the same data type signature) by changing the value of two error code consts. I believe this example is quite close to the example (from the previous comment) for field names.

What we should do instead is to manually enforce some simple rules. One of those we currently have to follow is that const names are not to be swapped. It would be quite simple to follow the equivalent rule for field names when merging PRs. And with the dsdl linter/checker (mentioned below) this would be automated.

The current compatibility checks only work for some types.

A lot of the data types are being sent without any compatibility checking, either static or dynamic. And what data types are being checked corresponds to the arbitrary condition if it's a multi frame transfer. Single frame transfers are currently always being sent without any explicit checking. When using CAN-FD most of the transfers will fit in a single frame transfer and almost none of the transfers will have the data type signature verified. This means that by keeping a strict data type signature in the CRC we sacrifice a lot of flexibility to verify (allegedly in an overly strict way) compatibility for only a few of the data types.

This also mean that we cannot rely on the current mechanisms for compatibility checks. This means in turn that we either have to add dynamic checking to single frame transfers or do more verification statically.

The current compatibility checks hinder simple future proofing

If we want to be able to stabilize definitions without shooting ourself in the foot we are going to want as much flexibility in future proofing as possible. Reserving a field as a void and populating it at a later point is one of the most important ways to do this (together with adding consts at a later point).

I do not think we gain enough safety by keeping the current strictness to justify this loss in future proofing capabilities.

Writing the dsdl checker tool is not hard.

The current rust uavcan parser works fine for all happy paths. But if you try to parse dsdl that are not adhering to specs, all bets are off. This was super to get things running, but not acceptable in the long run. Improving error handling in the parser is the only difficult thing about writing such dsdl linter. The good thing is that I know how to do it, and actually already did half the work last week. This was something that had to done anyway.

Once the parser is up to speed, I consider writing the following checks trivial. This would also leave us in a better condition to detect compatibility between dsdl than with the current data type signature since it also works for consts and single frame transfers.

• [Error] Published definition changed (The checked dsdl contains changes to a published defintion).
• [Warning] Rename const (const with same value but different name with same major version).
• [Warning] Change const (const with same name but different value with same major version).
• [Error] Incomatible bit layout (message type with same major version but different bit layout, people friendly error message will be given).
• [Warning] Renamed field (a field with the same bit placement and type info but different name).
• [Error] Incompatible field (a field with the same name but different type int2 -> uint2 ).
• [Error] Removed field (a field with was replaced with a void field for a newer data type int2 -> void2 ).

The capabilities of good error reporting is also useful when compiling custom dsdl for use with uavcan.rs. The work done with dsdl linting will with a reasonable amount or work benefit uavcan.rs and the uavcan ecosystem as a whole.

If we decide that static checking is the way to go, I can have a couple of these lints up and running (on the old parser) in just a day or two. And have all of them, including the new parser up and running in about 2 weeks, this also leaves the required time to update uavcan.rs to the new protocol (a lot of this work is done).

Other

Convey dsdl version number of the wire

A remaining issue, as I see it, is how do we convey the version information over the wire?

I do not think we should include message definition version numbers over the wire every time the message is being sent. Remember that once the message definitions are stabilized they're only supposed to be broken in rare cases. And i do not think adding a byte to handle this is worth the overhead.

In the rare cases they're broken we have the following mechanisms to handle this:

• We have explicit versioning allowing us to convey information about breakage very clearly. X supports version 1 of this message while Y supports version 2. Being able to clearly express the problem helps a lot. Or, (in a changelog) the new FW uses version 2 of this message (breaking change).
• And as an answer to your question, we add a service that can ask a device at run time which message version it supports. We make it recommended to support this service. Uavcan gui tool (or a similar tool) can ask all connected devices at run time which message version they support to detect breakage. Advanced nodes can also perform these checks when new nodes enter the network. We add this service to a core part of the protocol that cannot be updated to version 2 (together with other core definitions like NodeStatus).

CAN 2.0 deprecation

it might be sensible to view CAN 2.0 as a semi-deprecated transport that eventually is going to fall out of use

I think this is overly optimistic. I think realistically CAN 2.0 is going to be around for a long time, and we should support it as best we can. I think we should to as little extent as possible differentiate between the transfer protocol. I think we also should be conservative with the CAN-FD data usage, even though its much cheaper than CAN 2.0 we should stay on the "as low overhead as possible" side.

[thirtytwobits]

I'm both coming into this conversation very, very late and also trying to learn enough about UAVCAN to be a useful participant so please be gentle.

CAN 2.0 deprecation

2.0 isn't going anywhere anytime soon as FD is only just now (i.e. 2018) becoming available for large-scale production and with the massive existing deployments of this protocol it may not go away for decades if ever. While the UAV space is much less encumbered with legacy it borrows heavily from automotive (in civilian sectors) and is currently too small to support its own hardware ecosystem. Because of that we will inherent automotive's legacy and will continue to see 2.0 (and LIN) peripherals available in microprocessor lines that we otherwise find useful.

Convey dsdl version number of the wire

I'm definitely against a per-message versioning scheme. There just isn't enough bandwidth to handle this in 2.0 and it's even onerous in FD.

I'd recommend an approach where the node allocation operation was expanded to also include a protocol level negotiation. For some safety-critical systems the node id server would want to decline to enable nodes that were not compatible with it. This could be used to ensure a system remained inactive unless/until all safety-critical nodes were found to be running a compatible version of the protocol. This doesn't help for systems that statically allocate node identifiers so this proposal requires more thought. I'll float it here and develop it more if the group thinks there's merit to a service-based approach.

Writing the dsdl checker tool is not hard.

Perhaps not but even it if were hard it is extremely valuable. I'm all for developing a tool suite that could analyze DSDL at compile-time and also be evolved into a static CAN network analyzer going forward.

The current compatibility checks hinder simple future proofing

So @kjetilkjeka , my understanding is that the following is still your recommendation? :

What I'm thinking is instead of initializing the CRC with the normalized message (hashed) we instead initialize it with a structure containing:

• namespace + Name
  Major version (and minor version if major version == 0)
• A dsdl-ish definition that shows layout of fields with dynamic sized fields (dynamic arrays, unions, structures with dynamic fields etc).
• (perhaps some other things that cannot change and needs to be enforced to keep compatibility)