Just posted with a Duckduckgo browser.
I still don't have a clue how to properly post code ...edit: now I got it!
It looks like the import from Unix.com is missing some "[" characters within formatted code.
@Scrutinizer is leading the effort to modify the Ruby import script to refine the preprocessing method so please start a new topic or message him with links and details.
Thanks
@MadeInGermany, thanks for reporting. Do you have an example for me, so I can try to fix it in preprocessing?
Example:
The opening [[ are missing (in the last two posts).
Preliminary bug hunting shows this bug does not happen in the preprocessing method in the migration script; and instead looks like a bug in the ruby-bbcode-to-md plugin.
I'm waiting on @Scrutinizer to review my findings before we decide if we are going to post a bug report over at meta or not.
FYI, you can look at the plugin code here, if interested:
Thanks so much for finding and reporting this.
Update:
Bug report submitted:
https://meta.discourse.org/t/ruby-bbcode-to-md-removes/145662
Confirmed bug in ruby-bbcode-to-md
cat test.rb
require 'ruby-bbcode-to-md'
hello = "[[ $string2 =~ ROS[0-9]+$ ]] && echo OK || echo NOK".bbcode_to_md
puts hello
ruby test.rb
$string2 =~ ROS[0-9]+$ ]] && echo OK || echo NOK
LOL... that is the first Ruby gem I have ever debugged :). It was easier than I imagined.
Just did this on my Mac:
ruby install ruby-bbcode-to-md
and then wrote some simple code and tested to confirm the bug.... guess I'z a b learn'in ruby nowza....
Update: Sent message (email) to developers notifying them of this confirmed bug:
Posted link to bug report on meta:
Do you want to take a shot at debugging and rebuilding this Ruby gem?
You can probably debug and fix this in a short time, even if you never used Ruby before 
Otherwise, I am going to have to stop all the migration work I'm doing and do this myself, because I don't think we are going to see a bug fix from the developers or anyone at meta.
1 Like
Honestly, I don’t understand what is going on. I’m like from another planet. It’s better to keep such a fighter like me under lock and key so that he doesn’t touch anything. I'm sorry
1 Like
It is not only [[
The following thread is missing many single [ characters in my post (bash-4 script):
1 Like
Yea, it looks like it is just ripping all the [ out of the text.
Just updated the bug report on meta:
As our team as pointed out, it's more buggy than [[
$ cat test.rb
require 'ruby-bbcode-to-md'
hello = 'for key in "${!aa[@]}"'.bbcode_to_md
puts hello
$ ruby test.rb
for key in "${!aa@]}"
ruby-bbcode-to-md is a black hole for left square brackets in a code fragment.
Updated the bug report over there:
https://meta.discourse.org/t/ruby-bbcode-to-md-removes/145662/8
1 Like
Looks like we are not going to get any help at meta.
They are washing their hands of this plugin:
Of course, that is not what their documentation said before we migrated.
Now, it is like.... "oh, you mean that old code..... " LOL
Looks like we are totally on our own on this one.
Anyone want to rewrite it?
I have stopped the visit our new community banner on the legacy site and turned off Google indexing (again).
The code which has been migrated is too mangled due to the bug in the ruby-bbcode-to-md plugin.
The guy responsible for ruby-bbcode-to-md washed his hands of this code and closed my bug report because I told him the truth.
ruby-bbcode-to-md is total garbage and ruined our entire migration.
I am really in bad spirits over this mangled code problem.
Not because of the bad plugin, but because of having my bug report shut down and deleted over at meta by the maintainer of the plugin, even though it is confirmed, a serious bug.
Let us be clear to all migrators:
ruby-bbcode-to-md is total garbage; it is a mangler gem, and polluted our entire migration.
However, we are moving on from this negative experience, tired and weary warriors of the migration.
As I understand the problem:
- All Code within Code-Tags of www.unix.com has been migrated to community.unix.com.
- That means that the literals [code]...[/code] and [icode]...[/code] has successfully moved to ```...``` literals
- The problem is with what is inside the new Code-Tags(= ```). This data is defective - supposedly beyond any repair - meaning it must be retransferred.
The problem seems not to be so serious to me and solvable. What can be serious is, if there are further transformation issues, that are not known at this time and which may be detected later, when a lot of new posts had been written into the new platform, so the loss of a switch back would be a lot greater than now.
Besides that basic thought, here is some idea about the current problem:
What about this solution (in Pseudo-Code) for database transformation?
foreach (@all_article_ids as $id) {
if article_time($id) < $before_migration then
$old_post = get_old_site_post($id)
$code_elements = get_code_elements_old($old_post)
$transformed_code_elements = transform_code_to_new_code_tags($code_elements)
replace_code_elements($id, get_code_elements_new(), $transformed_code_elements)
end-if
end-foreach
Personally I would expect the transformation from bb-code tags to markdown not that complicated - but as often experienced - I would expect things to get more complicated when starting to get the work done 
- Basically the code transformations to transform from [code](...)[/code] to ```$1``` (=regex capture/replace).
- There maybe some additional issues with other bbcode tags within the code tags(I do not remember if that was possible).
- There maybe character encoding issues, which can be very complex.(I always avoided character encoding when possible, because I didn't fully understand it).
In fact most if not all bbcode tags are allowed within [code] and [icode]. You could even have [code] within [code].
But maybe it was not used at all.
Personally I only used color within [code].
Of course this ruby gem is fixable...
We have decided to drop, completely, using ruby-bbcode-to-md in the migration process and to perform all of the legacy bbcode transformation in the preprocessing routine in the migration script.
In fact, during my initial test runs, I did not use ruby-bbcode-to-md but there was a lot of forum BBCODE which was not properly transformed; so after days of problems and searching the discourse meta site and other Internet spaces, the only recommended "solution" from Discourse and their users was ruby-bbcode-to-md.
We now know, that this "solution" is seriously flawed. ruby-bbcode-to-md is a mangler.
During my initial test runs, it was obvious that there was another alternative, and that alternative was to greatly modify and enhance the preprocessing routine to do the required bbcode-to-markdown transformations in there.
Basically, the migration script is a kind of kluge of various routines and plugins, because migration is not well supported by Discourse, especially forums which use legacy bbcode.
This is totally understandable, as no one likes migrating legacy data from one system to another. It is perfectly normal for the Discourse team to cringe at this, and they do, believe me. However, in this case I am very disappointed in Discourse team member Neil Lalonde, which I will discuss later. He owes us a big apology but I am sure we will never get one.
In an nutshell, the migration script preprocesses the original post pagetext in two routines. One (the first one) is a Ruby preprocessing routine and the second is a Discourse plugin based on a Ruby gem, ruby-bbcode-to-md , "maintained" by Mr. Lalonde, which we now have confirmed is a serious code mangler.
On that note, I recommend Mr. Lalonde put a big scarlet banner across his GitHub ruby-bbcode-to-md repo; WARNING THIS PLUGIN IS A CODE MANGLER.
Around a week or so ago, I considered, around test run 2, to do all the bbcode to md transformations in the migration in the preprocessing routing in the migration script, but research indicated that the ruby-bbode-to-md plugin worked. I did a lot of research on this and here was no indication this plugin was unsupported, obsolete, or problematic.
So, I decided to go with the Neil Lalonde's ruby-bbode-to-md plugin, which looked like it was well supported. That decision was a mistake. However, now we know. It's time to move on.
Yesterday, I posted this issues over at meta Discourse and Neil Lalonde washed his hands of his plugin, closed and deleted my bug report, even thought the bugs are confirmed and serious. This was quite shocking to see a Discourse team member resort to such heavy-handed tactics when their code repo is proven to be seriously flawed. So forget ruby-bbode-to-md. The developer has shown his true colors as a developer. He owes us a serious apology but since he is "Leader" status as meta, I'm sure nothing good will come of this. He just shuts me down and deletes my bug reports, I assume to protect his reputation. Shocking and unexpected, really.
Normally, I never call out any software developer by name, but his heavy-handed and unprofessional behavior yesterday, regarding serious flaws in his own Discourse plugin repo on GitHub, is not acceptable. Other migration teams need to be warned how much of a mangler ruby-bbcode-to-md is.
On a very positive note:
@Scrutinizer has offered to add the required bbcode to md transistions in the migration script preprocessing routine. This is a cleaner solution (and was one I considered earlier when I was doing all this alone). So, I am very pleased that Jeroen has volunteered to help. He deserves a special badge for that. It's very helpful to have someone else writing code, testing and helping out! Thank you!
Moving forward, we are washing our hands, completely, of ruby-bbode-to-md and I will write some posts to warn other migration teams to avoid ````ruby-bbode-to-md``` like the black plague.
If someone want's to rewrite ruby-bbode-to-md the code mangler, please feel free, but we have what we consider to be a better solution for the migration and that is to do the required transitions in the preprocessing routing.
I have no problem if anyone wants to rewrite the ruby-bbcode-to-md gem; maybe that plugin will be useful in the future for those who might want to post legacy bbcode; but for me, I prefer we migration to markdown 100% and avoid all legacy bbcode.
That's all for now.
1 Like