We have decided to drop, completely, using ruby-bbcode-to-md in the migration process and to perform all of the legacy bbcode transformation in the preprocessing routine in the migration script.
In fact, during my initial test runs, I did not use ruby-bbcode-to-md but there was a lot of forum BBCODE which was not properly transformed; so after days of problems and searching the discourse meta site and other Internet spaces, the only recommended "solution" from Discourse and their users was ruby-bbcode-to-md.
We now know, that this "solution" is seriously flawed. ruby-bbcode-to-md is a mangler.
During my initial test runs, it was obvious that there was another alternative, and that alternative was to greatly modify and enhance the preprocessing routine to do the required bbcode-to-markdown transformations in there.
Basically, the migration script is a kind of kluge of various routines and plugins, because migration is not well supported by Discourse, especially forums which use legacy bbcode.
This is totally understandable, as no one likes migrating legacy data from one system to another. It is perfectly normal for the Discourse team to cringe at this, and they do, believe me. However, in this case I am very disappointed in Discourse team member Neil Lalonde, which I will discuss later. He owes us a big apology but I am sure we will never get one.
In an nutshell, the migration script preprocesses the original post pagetext in two routines. One (the first one) is a Ruby preprocessing routine and the second is a Discourse plugin based on a Ruby gem, ruby-bbcode-to-md , "maintained" by Mr. Lalonde, which we now have confirmed is a serious code mangler.
On that note, I recommend Mr. Lalonde put a big scarlet banner across his GitHub ruby-bbcode-to-md repo; WARNING THIS PLUGIN IS A CODE MANGLER.
Around a week or so ago, I considered, around test run 2, to do all the bbcode to md transformations in the migration in the preprocessing routing in the migration script, but research indicated that the ruby-bbode-to-md plugin worked. I did a lot of research on this and here was no indication this plugin was unsupported, obsolete, or problematic.
So, I decided to go with the Neil Lalonde's ruby-bbode-to-md plugin, which looked like it was well supported. That decision was a mistake. However, now we know. It's time to move on.
Yesterday, I posted this issues over at meta Discourse and Neil Lalonde washed his hands of his plugin, closed and deleted my bug report, even thought the bugs are confirmed and serious. This was quite shocking to see a Discourse team member resort to such heavy-handed tactics when their code repo is proven to be seriously flawed. So forget ruby-bbode-to-md. The developer has shown his true colors as a developer. He owes us a serious apology but since he is "Leader" status as meta, I'm sure nothing good will come of this. He just shuts me down and deletes my bug reports, I assume to protect his reputation. Shocking and unexpected, really.
Normally, I never call out any software developer by name, but his heavy-handed and unprofessional behavior yesterday, regarding serious flaws in his own Discourse plugin repo on GitHub, is not acceptable. Other migration teams need to be warned how much of a mangler ruby-bbcode-to-md is.
On a very positive note:
@Scrutinizer has offered to add the required bbcode to md transistions in the migration script preprocessing routine. This is a cleaner solution (and was one I considered earlier when I was doing all this alone). So, I am very pleased that Jeroen has volunteered to help. He deserves a special badge for that. It's very helpful to have someone else writing code, testing and helping out! Thank you!
Moving forward, we are washing our hands, completely, of ruby-bbode-to-md and I will write some posts to warn other migration teams to avoid ````ruby-bbode-to-md``` like the black plague.
If someone want's to rewrite ruby-bbode-to-md the code mangler, please feel free, but we have what we consider to be a better solution for the migration and that is to do the required transitions in the preprocessing routing.
I have no problem if anyone wants to rewrite the ruby-bbcode-to-md gem; maybe that plugin will be useful in the future for those who might want to post legacy bbcode; but for me, I prefer we migration to markdown 100% and avoid all legacy bbcode.
That's all for now.