[Flex] Add callbacks to indicate start/end of stage processing

[mmd-osm]

Today, a Lua script has a global section, which gets executed right at the beginning, and a few callback functions for nodes, ways, and relations. However, there's no way for the script to figure out when the Flex backend enters or exits a given stage.

This could be useful to do some initialization steps upon entry of a stage, or trigger the creation of additional DB indexes when exiting stage 3 (see #1212).

As an alternative, providing a global "setup" and "teardown" function callback, similar to the way unit tests work, would be a good starting point.

[joto]

Do you have an actual, current use case for this?

We should not just add features that seem nice or useful, because we have to support them until the end of times...

[mmd-osm]

The main use cases I had in mind were the inclusion of external script logic into the Lua file to faciliate the overall processing, and have the creation/update process in one place. As an example, osm-fr has a set of additional scripts to create indices where needed. I think those could be easily integrated into an "End of osm2pgsql processing reached" function in the Lua script.

https://github.com/osm-fr/osm2pgsql-import-tools/blob/master/requetes-sql-indexes-et-autre/indexes-admin_level.sql
https://github.com/osm-fr/osm2pgsql-import-tools/blob/master/requetes-sql-indexes-et-autre/indexes-ref.sql
https://github.com/osm-fr/osm2pgsql-import-tools/blob/master/requetes-sql-indexes-et-autre/index-ref-sandre.sql

Another use case could be the calculation of simplified geometries at the end of processing according to custom rules. The following example uses database trigger for this purpose. This could be also implemented as SQL statement which is executed towards the end of osm2pgsql processing.

https://github.com/osm-fr/osm2pgsql-import-tools/blob/master/requetes-sql-indexes-et-autre/colonne-geometries-simplifi%C3%A9es-polygon-administratifs.sql

Another use case according to "https://blog.jochentopf.com/2020-05-10-new-flex-output-in-osm2pgsql.html" -> Use cases is maybe in line with "much more interesting for non-rendering use cases. Whether you want to create statistics of certain OSM tags": Assuming we keep the stats in memory for some time, thereby avoiding frequent database updates, we need some way to find out, that the last node/way/relation has been processed now, and we still need to store our Lua buffer on the database before osm2pgsql finishes. This is only possible if the script is notified about the overall end of processing.

I know this concept is very generic, and I tried to cover different use cases that should be of general interest. All of them worked with an additional "end of processing" function, which would be the minimum scope for this issue.

[joto]

So far I don't see any reason why this can't be done outside osm2pgsql. In fact the scripts you mentioned presumable already do this. So at the moment I don't see any reason to make osm2pgsql more complicated.

[mmd-osm]

How would you handle the last use case I mentioned outside of osm2pgsql?

[joto]

@mmd-osm This is not about hypothetical use cases you invent that I should find hypothetical solutions for. I have other things to do. Issues are for pointing out actual current problems with the software.

[lonvia]

The whole handling of post-processing is a topic we need to think about on a grander scale. It does not only concern the flex output. It's really not a good idea to add some interim solution that may or may not solve a problem but has the real danger of restricting us later because we need to support it forever.

Closing this for now. We need to get the basic functionality of the flex backend sorted out first. Once this runs stable, we can start thinking about what else to do.

[mmd-osm]

@joto: I quoted a use case from your blog post, which I couldn't figure out how to map on the current Flex backend implementation. Maybe I'm missing something here, but I really don't know how you would address this "create statistics of certain OSM tags" use case.

The whole handling of post-processing is a topic we need to think about on a grander scale. It does not only concern the flex output

@lonvia : thanks, this makes sense. It's really hard to find out from the outside where you're heading with this project, and what your plans are. That's why I keep opening issues you're dismissing now and then.